<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hello,<div class=""><br class=""></div><div class="">Do you know more about?</div><div class=""><br class=""></div><div class="">In addition, do you know how to « activate » RDMA for my volume with Intel/QLogic QDR? Currently, i mount my volumes with RDMA transport-type option (both in server and client side) but I notice all streams are using TCP stack -and my bandwith never exceed 2.0-2.5Gbs (250-300MB/s).</div><div class=""><br class=""></div><div class="">Thanks in advance,</div><div class="">Geoffrey<br class=""><div apple-content-edited="true" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">------------------------------------------------------<br class="">Geoffrey Letessier<br class="">Responsable informatique & ingénieur système<br class="">UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br class="">Institut de Biologie Physico-Chimique<br class="">13, rue Pierre et Marie Curie - 75005 Paris<br class="">Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr" class="">geoffrey.letessier@ibpc.fr</a></div>
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">Le 2 juin 2015 à 23:45, Geoffrey Letessier <<a href="mailto:geoffrey.letessier@cnrs.fr" class="">geoffrey.letessier@cnrs.fr</a>> a écrit :</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=windows-1252" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Ben,<div class=""><br class=""></div><div class="">I just check my messages log files, both on client and server, and I dont find any hung task you notice on yours.. </div><div class=""><br class=""></div><div class="">As you can read below, i dont note the performance issue in a simple DD but I think my issue is concerning a set of small files (tens of thousands nay more)…</div><div class=""><br class=""></div><div class=""><div class=""><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">[root@nisus test]# ddt -t 10g /mnt/test/</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">Writing to /mnt/test/ddt.8362 ... syncing ... done.</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">sleeping 10 seconds ... done.</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">Reading from /mnt/test/ddt.8362 ... done.</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">10240MiB KiB/s CPU%</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">Write 114770 4</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">Read 40675 4</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class=""><br class=""></div></div><div class="">for info: /mnt/test concerns the single v2 GlFS volume</div><div class=""><br class=""></div><div apple-content-edited="true" class=""><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">[root@nisus test]# ddt -t 10g /mnt/fhgfs/</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">Writing to /mnt/fhgfs/ddt.8380 ... syncing ... done.</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">sleeping 10 seconds ... done.</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">Reading from /mnt/fhgfs/ddt.8380 ... done.</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">10240MiB KiB/s CPU%</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">Write 102591 1</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">Read 98079 2</div><div class=""><br class=""></div></div></div><div class="">Do you have a idea how to tune/optimize performance settings? and/or TCP settings (MTU, etc.)?</div><div class=""><br class=""></div><div class=""><div class=""><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">---------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">| | UNTAR | DU | FIND | TAR | RM |</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">---------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">| single | ~3m45s | ~43s | ~47s | ~3m10s | ~3m15s |</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">---------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">| replicated | ~5m10s | ~59s | ~1m6s | ~1m19s | ~1m49s |</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">---------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">| distributed | ~4m18s | ~41s | ~57s | ~2m24s | ~1m38s |</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">---------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">| dist-repl | ~8m18s | ~1m4s | ~1m11s | ~1m24s | ~2m40s |</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">---------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">| native FS | ~11s | ~4s | ~2s | ~56s | ~10s |</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">---------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">| BeeGFS | ~3m43s | ~15s | ~3s | ~1m33s | ~46s |</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">---------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">| single (v2) | ~3m6s | ~14s | ~32s | ~1m2s | ~44s |</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">---------------------------------------------------------------</div></div><div class="">for info: </div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>-BeeGFS is a distributed FS (4 bricks, 2 bricks per server and 2 servers)</div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>- single (v2): simple gluster volume with default settings</div></div><div class=""><br class=""></div><div class="">I also note I obtain the same tar/untar performance issue with FhGFS/BeeGFS but the rest (DU, FIND, RM) looks like to be OK.</div><div class=""><br class=""></div><div class="">Thank you very much for your reply and help.</div><div class="">Geoffrey<br class=""><div apple-content-edited="true" class="">
<div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">-----------------------------------------------<br class="">Geoffrey Letessier<br class=""><br class="">Responsable informatique & ingénieur système<br class="">CNRS - UPR 9080 - Laboratoire de Biochimie Théorique<br class="">Institut de Biologie Physico-Chimique<br class="">13, rue Pierre et Marie Curie - 75005 Paris<br class="">Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@cnrs.fr" class="">geoffrey.letessier@cnrs.fr</a></div>
</div>
<br class=""><div class=""><div class="">Le 2 juin 2015 à 21:53, Ben Turner <<a href="mailto:bturner@redhat.com" class="">bturner@redhat.com</a>> a écrit :</div><br class="Apple-interchange-newline"><blockquote type="cite" class=""><div style="font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">I am seeing problems on 3.7 as well. Can you check /var/log/messages on both the clients and servers for hung tasks like:<br class=""><br class="">Jun 2 15:23:14 gqac006 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br class="">Jun 2 15:23:14 gqac006 kernel: iozone D 0000000000000001 0 21999 1 0x00000080<br class="">Jun 2 15:23:14 gqac006 kernel: ffff880611321cc8 0000000000000082 ffff880611321c18 ffffffffa027236e<br class="">Jun 2 15:23:14 gqac006 kernel: ffff880611321c48 ffffffffa0272c10 ffff88052bd1e040 ffff880611321c78<br class="">Jun 2 15:23:14 gqac006 kernel: ffff88052bd1e0f0 ffff88062080c7a0 ffff880625addaf8 ffff880611321fd8<br class="">Jun 2 15:23:14 gqac006 kernel: Call Trace:<br class="">Jun 2 15:23:14 gqac006 kernel: [<ffffffffa027236e>] ? rpc_make_runnable+0x7e/0x80 [sunrpc]<br class="">Jun 2 15:23:14 gqac006 kernel: [<ffffffffa0272c10>] ? rpc_execute+0x50/0xa0 [sunrpc]<br class="">Jun 2 15:23:14 gqac006 kernel: [<ffffffff810aaa21>] ? ktime_get_ts+0xb1/0xf0<br class="">Jun 2 15:23:14 gqac006 kernel: [<ffffffff811242d0>] ? sync_page+0x0/0x50<br class="">Jun 2 15:23:14 gqac006 kernel: [<ffffffff8152a1b3>] io_schedule+0x73/0xc0<br class="">Jun 2 15:23:14 gqac006 kernel: [<ffffffff8112430d>] sync_page+0x3d/0x50<br class="">Jun 2 15:23:14 gqac006 kernel: [<ffffffff8152ac7f>] __wait_on_bit+0x5f/0x90<br class="">Jun 2 15:23:14 gqac006 kernel: [<ffffffff81124543>] wait_on_page_bit+0x73/0x80<br class="">Jun 2 15:23:14 gqac006 kernel: [<ffffffff8109eb80>] ? wake_bit_function+0x0/0x50<br class="">Jun 2 15:23:14 gqac006 kernel: [<ffffffff8113a525>] ? pagevec_lookup_tag+0x25/0x40<br class="">Jun 2 15:23:14 gqac006 kernel: [<ffffffff8112496b>] wait_on_page_writeback_range+0xfb/0x190<br class="">Jun 2 15:23:14 gqac006 kernel: [<ffffffff81124b38>] filemap_write_and_wait_range+0x78/0x90<br class="">Jun 2 15:23:14 gqac006 kernel: [<ffffffff811c07ce>] vfs_fsync_range+0x7e/0x100<br class="">Jun 2 15:23:14 gqac006 kernel: [<ffffffff811c08bd>] vfs_fsync+0x1d/0x20<br class="">Jun 2 15:23:14 gqac006 kernel: [<ffffffff811c08fe>] do_fsync+0x3e/0x60<br class="">Jun 2 15:23:14 gqac006 kernel: [<ffffffff811c0950>] sys_fsync+0x10/0x20<br class="">Jun 2 15:23:14 gqac006 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b<br class=""><br class="">Do you see a perf problem with just a simple DD or do you need a more complex workload to hit the issue? I think I saw an issue with metadata performance that I am trying to run down, let me know if you can see the problem with simple DD reads / writes or if we need to do some sort of dir / metadata access as well.<br class=""><br class="">-b<br class=""><br class="">----- Original Message -----<br class=""><blockquote type="cite" class="">From: "Geoffrey Letessier" <<a href="mailto:geoffrey.letessier@cnrs.fr" class="">geoffrey.letessier@cnrs.fr</a>><br class="">To: "Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com" class="">pkarampu@redhat.com</a>><br class="">Cc:<span class="Apple-converted-space"> </span><a href="mailto:gluster-users@gluster.org" class="">gluster-users@gluster.org</a><br class="">Sent: Tuesday, June 2, 2015 8:09:04 AM<br class="">Subject: Re: [Gluster-users] GlusterFS 3.7 - slow/poor performances<br class=""><br class="">Hi Pranith,<br class=""><br class="">I’m sorry but I cannot bring you any comparison because comparison will be<br class="">distorted by the fact in my HPC cluster in production the network technology<br class="">is InfiniBand QDR and my volumes are quite different (brick in RAID6<br class="">(12x2TB), 2 bricks per server and 4 servers into my pool)<br class=""><br class="">Concerning your demand, in attachments you can find all expected results<br class="">hoping it can help you to solve this serious performance issue (maybe I need<br class="">play with glusterfs parameters?).<br class=""><br class="">Thank you very much by advance,<br class="">Geoffrey<br class="">------------------------------------------------------<br class="">Geoffrey Letessier<br class="">Responsable informatique & ingénieur système<br class="">UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br class="">Institut de Biologie Physico-Chimique<br class="">13, rue Pierre et Marie Curie - 75005 Paris<br class="">Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr" class="">geoffrey.letessier@ibpc.fr</a><br class=""><br class=""><br class=""><br class=""><br class="">Le 2 juin 2015 à 10:09, Pranith Kumar Karampuri < <a href="mailto:pkarampu@redhat.com" class="">pkarampu@redhat.com</a> > a<br class="">écrit :<br class=""><br class="">hi Geoffrey,<br class="">Since you are saying it happens on all types of volumes, lets do the<br class="">following:<br class="">1) Create a dist-repl volume<br class="">2) Set the options etc you need.<br class="">3) enable gluster volume profile using "gluster volume profile <volname><br class="">start"<br class="">4) run the work load<br class="">5) give output of "gluster volume profile <volname> info"<br class=""><br class="">Repeat the steps above on new and old version you are comparing this with.<br class="">That should give us insight into what could be causing the slowness.<br class=""><br class="">Pranith<br class="">On 06/02/2015 03:22 AM, Geoffrey Letessier wrote:<br class=""><br class=""><br class="">Dear all,<br class=""><br class="">I have a crash test cluster where i’ve tested the new version of GlusterFS<br class="">(v3.7) before upgrading my HPC cluster in production.<br class="">But… all my tests show me very very low performances.<br class=""><br class="">For my benches, as you can read below, I do some actions (untar, du, find,<br class="">tar, rm) with linux kernel sources, dropping cache, each on distributed,<br class="">replicated, distributed-replicated, single (single brick) volumes and the<br class="">native FS of one brick.<br class=""><br class=""># time (echo 3 > /proc/sys/vm/drop_caches; tar xJf ~/linux-4.1-rc5.tar.xz;<br class="">sync; echo 3 > /proc/sys/vm/drop_caches)<br class=""># time (echo 3 > /proc/sys/vm/drop_caches; du -sh linux-4.1-rc5/; echo 3 ><br class="">/proc/sys/vm/drop_caches)<br class=""># time (echo 3 > /proc/sys/vm/drop_caches; find linux-4.1-rc5/|wc -l; echo 3<br class=""><blockquote type="cite" class="">/proc/sys/vm/drop_caches)<br class=""></blockquote># time (echo 3 > /proc/sys/vm/drop_caches; tar czf linux-4.1-rc5.tgz<br class="">linux-4.1-rc5/; echo 3 > /proc/sys/vm/drop_caches)<br class=""># time (echo 3 > /proc/sys/vm/drop_caches; rm -rf linux-4.1-rc5.tgz<br class="">linux-4.1-rc5/; echo 3 > /proc/sys/vm/drop_caches)<br class=""><br class="">And here are the process times:<br class=""><br class="">---------------------------------------------------------------<br class="">| | UNTAR | DU | FIND | TAR | RM |<br class="">---------------------------------------------------------------<br class="">| single | ~3m45s | ~43s | ~47s | ~3m10s | ~3m15s |<br class="">---------------------------------------------------------------<br class="">| replicated | ~5m10s | ~59s | ~1m6s | ~1m19s | ~1m49s |<br class="">---------------------------------------------------------------<br class="">| distributed | ~4m18s | ~41s | ~57s | ~2m24s | ~1m38s |<br class="">---------------------------------------------------------------<br class="">| dist-repl | ~8m18s | ~1m4s | ~1m11s | ~1m24s | ~2m40s |<br class="">---------------------------------------------------------------<br class="">| native FS | ~11s | ~4s | ~2s | ~56s | ~10s |<br class="">---------------------------------------------------------------<br class=""><br class="">I get the same results, whether with default configurations with custom<br class="">configurations.<br class=""><br class="">if I look at the side of the ifstat command, I can note my IO write processes<br class="">never exceed 3MBs...<br class=""><br class="">EXT4 native FS seems to be faster (roughly 15-20% but no more) than XFS one<br class=""><br class="">My [test] storage cluster config is composed by 2 identical servers (biCPU<br class="">Intel Xeon X5355, 8GB of RAM, 2x2TB HDD (no-RAID) and Gb ethernet)<br class=""><br class="">My volume settings:<br class="">single: 1server 1 brick<br class="">replicated: 2 servers 1 brick each<br class="">distributed: 2 servers 2 bricks each<br class="">dist-repl: 2 bricks in the same server and replica 2<br class=""><br class="">All seems to be OK in gluster status command line.<br class=""><br class="">Do you have an idea why I obtain so bad results?<br class="">Thanks in advance.<br class="">Geoffrey<br class="">-----------------------------------------------<br class="">Geoffrey Letessier<br class=""><br class="">Responsable informatique & ingénieur système<br class="">CNRS - UPR 9080 - Laboratoire de Biochimie Théorique<br class="">Institut de Biologie Physico-Chimique<br class="">13, rue Pierre et Marie Curie - 75005 Paris<br class="">Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@cnrs.fr" class="">geoffrey.letessier@cnrs.fr</a><br class=""><br class=""><br class=""><br class="">_______________________________________________<br class="">Gluster-users mailing list <a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class=""><a href="http://www.gluster.org/mailman/listinfo/gluster-users" class="">http://www.gluster.org/mailman/listinfo/gluster-users</a><br class=""><br class=""><br class=""><br class=""><br class="">_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class=""><a href="http://www.gluster.org/mailman/listinfo/gluster-users" class="">http://www.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></body></html>