<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><span style="font-size: 11px;">Hi Ben</span><div><span style="font-size: 11px;"><br></span></div><div><span style="font-size: 11px;">Here the expected output:</span></div><div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">[root@node048 ~]# iperf3 -c 10.0.4.1</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">Connecting to host 10.0.4.1, port 5201</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">[ 4] local 10.0.5.48 port 44151 connected to 10.0.4.1 port 5201</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">[ ID] Interval Transfer Bandwidth Retr Cwnd</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">[ 4] 0.00-1.00 sec 1.86 GBytes 15.9 Gbits/sec 0 8.24 MBytes </span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">[ 4] 1.00-2.00 sec 1.94 GBytes 16.7 Gbits/sec 0 8.24 MBytes </span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">[ 4] 2.00-3.00 sec 1.95 GBytes 16.8 Gbits/sec 0 8.24 MBytes </span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">[ 4] 3.00-4.00 sec 1.86 GBytes 16.0 Gbits/sec 0 8.24 MBytes </span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">[ 4] 4.00-5.00 sec 1.85 GBytes 15.8 Gbits/sec 0 8.24 MBytes </span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">[ 4] 5.00-6.00 sec 1.89 GBytes 16.2 Gbits/sec 0 8.24 MBytes </span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">[ 4] 6.00-7.00 sec 1.90 GBytes 16.3 Gbits/sec 0 8.24 MBytes </span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">[ 4] 7.00-8.00 sec 1.88 GBytes 16.1 Gbits/sec 0 8.24 MBytes </span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">[ 4] 8.00-9.00 sec 1.88 GBytes 16.2 Gbits/sec 0 8.24 MBytes </span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">[ 4] 9.00-10.00 sec 1.87 GBytes 16.1 Gbits/sec 0 8.24 MBytes </span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">- - - - - - - - - - - - - - - - - - - - - - - - -</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">[ ID] Interval Transfer Bandwidth Retr</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">[ 4] 0.00-10.00 sec 18.9 GBytes 16.2 Gbits/sec 0 sender</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">[ 4] 0.00-10.00 sec 18.9 GBytes 16.2 Gbits/sec receiver</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 13px;"><span style="font-size: 11px;"><br></span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">iperf Done.</span></div><div><span style="font-size: 11px;"><br></span></div><div><span style="font-size: 11px;">Here are all shell commands i used for volume creation with RDMA transport-type:</span></div><div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); position: static; z-index: auto;"><span style="font-size: 11px;">gluster volume create vol_home replica 2 transport rdma,tcp ib-storage1:/export/brick_home/brick1/ ib-storage2:/export/brick_home/brick1/ ib-storage3:/export/brick_home/brick1/ ib-storage4:/export/brick_home/brick1/ ib-storage1:/export/brick_home/brick2/ ib-storage2:/export/brick_home/brick2/ ib-storage3:/export/brick_home/brick2/ ib-storage4:/export/brick_home/brick2/ force</span></div></div><div><span style="font-size: 11px;"><br class="webkit-block-placeholder"></span></div><div><span style="font-size: 11px;">and below the current volume information:</span></div><div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">[root@lucifer ~]# gluster volume info vol_home</span></div><p style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 13px;"><span style="font-size: 11px;"> <br class="webkit-block-placeholder"></span></p><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">Volume Name: vol_home</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">Type: Distributed-Replicate</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">Volume ID: f6ebcfc1-b735-4a0e-b1d7-47ed2d2e7af6</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">Status: Started</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">Number of Bricks: 4 x 2 = 8</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">Transport-type: tcp,rdma</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">Bricks:</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">Brick1: ib-storage1:/export/brick_home/brick1</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">Brick2: ib-storage2:/export/brick_home/brick1</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">Brick3: ib-storage3:/export/brick_home/brick1</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">Brick4: ib-storage4:/export/brick_home/brick1</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">Brick5: ib-storage1:/export/brick_home/brick2</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">Brick6: ib-storage2:/export/brick_home/brick2</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">Brick7: ib-storage3:/export/brick_home/brick2</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">Brick8: ib-storage4:/export/brick_home/brick2</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">Options Reconfigured:</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">performance.stat-prefetch: on</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">performance.flush-behind: on</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">features.default-soft-limit: 90%</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">features.quota: on</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">diagnostics.brick-log-level: CRITICAL</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">auth.allow: localhost,127.0.0.1,10.*</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">nfs.disable: on</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">performance.cache-size: 64MB</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">performance.write-behind-window-size: 1MB</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">performance.quick-read: on</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">performance.io-cache: on</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">performance.io-thread-count: 64</span></div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">nfs.enable-ino32: on</span></div></div><div><span style="font-size: 11px;"><br></span></div><div><span style="font-size: 11px;">and below my mount command:</span></div><div><div style="margin: 0px; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 11px;">mount -t glusterfs -o transport=rdma,direct-io-mode=disable,enable-ino32 ib-storage1:vol_home /home</span></div></div><div><br></div><div>I dont obtain any error with RDMA option but transport type silently fall back to TCP.</div><div><span style="font-size: 11px;"><br></span></div><div><span style="font-size: 11px;">Did i make any mistake in my settings?</span></div><div><span style="font-size: 11px;"><br></span></div><div><span style="font-size: 11px;">Can you tell me more about block size and other tunings i should do on my rdma volumes?</span></div><div><span style="font-size: 11px;"><br></span></div><div><span style="font-size: 11px;">Thanks in advance,</span></div><div><span style="font-size: 11px;">Geoffrey</span></div><div apple-content-edited="true"><span style="font-size: 11px;">
------------------------------------------------------<br>Geoffrey Letessier<br>Responsable informatique & ingénieur système<br>UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr">geoffrey.letessier@ibpc.fr</a>
</span></div>
<span style="font-size: 11px;"><br></span><div><div><span style="font-size: 11px;">Le 8 juin 2015 à 18:22, Ben Turner <<a href="mailto:bturner@redhat.com">bturner@redhat.com</a>> a écrit :</span></div><span style="font-size: 11px;"><br class="Apple-interchange-newline"></span><blockquote type="cite"><div style="font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-size: 11px;">----- Original Message -----<br></span><blockquote type="cite"><span style="font-size: 11px;">From: "Geoffrey Letessier" <<a href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a>><br>To: "Ben Turner" <<a href="mailto:bturner@redhat.com">bturner@redhat.com</a>><br>Cc: "Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>>,<span class="Apple-converted-space"> </span><a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>Sent: Monday, June 8, 2015 8:37:08 AM<br>Subject: Re: [Gluster-users] GlusterFS 3.7 - slow/poor performances<br><br>Hello,<br><br>Do you know more about?<br><br>In addition, do you know how to « activate » RDMA for my volume with<br>Intel/QLogic QDR? Currently, i mount my volumes with RDMA transport-type<br>option (both in server and client side) but I notice all streams are using<br>TCP stack -and my bandwith never exceed 2.0-2.5Gbs (250-300MB/s).<br></span></blockquote><span style="font-size: 11px;"><br>That is a little slow for the HW you described. Can you check what you get with iperf just between the clients and servers?<span class="Apple-converted-space"> </span><a href="https://iperf.fr/">https://iperf.fr/</a><span class="Apple-converted-space"> </span> With replica 2 and 10G NW you should see ~400 MB / sec sequential writes and ~600 MB / sec reads. Can you send me the output from gluster v info? You specify RDMA volumes at create time by running gluster v create blah transport rdma, did you specify RDMA when you created the volume? What block size are you using in your tests? 1024 KB writes perform best with glusterfs, and the block size gets smaller perf will drop a little bit. I wouldn't write in anything under 4k blocks, the sweet spot is between 64k and 1024k.<br><br>-b<br><br></span><blockquote type="cite"><span style="font-size: 11px;"><br>Thanks in advance,<br>Geoffrey<br>------------------------------------------------------<br>Geoffrey Letessier<br>Responsable informatique & ingénieur système<br>UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr">geoffrey.letessier@ibpc.fr</a><br><br></span><blockquote type="cite"><span style="font-size: 11px;">Le 2 juin 2015 à 23:45, Geoffrey Letessier <<a href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a>> a<br>écrit :<br><br>Hi Ben,<br><br>I just check my messages log files, both on client and server, and I dont<br>find any hung task you notice on yours..<br><br>As you can read below, i dont note the performance issue in a simple DD but<br>I think my issue is concerning a set of small files (tens of thousands nay<br>more)…<br><br>[root@nisus test]# ddt -t 10g /mnt/test/<br>Writing to /mnt/test/ddt.8362 ... syncing ... done.<br>sleeping 10 seconds ... done.<br>Reading from /mnt/test/ddt.8362 ... done.<br>10240MiB KiB/s CPU%<br>Write 114770 4<br>Read 40675 4<br><br>for info: /mnt/test concerns the single v2 GlFS volume<br><br>[root@nisus test]# ddt -t 10g /mnt/fhgfs/<br>Writing to /mnt/fhgfs/ddt.8380 ... syncing ... done.<br>sleeping 10 seconds ... done.<br>Reading from /mnt/fhgfs/ddt.8380 ... done.<br>10240MiB KiB/s CPU%<br>Write 102591 1<br>Read 98079 2<br><br>Do you have a idea how to tune/optimize performance settings? and/or TCP<br>settings (MTU, etc.)?<br><br>---------------------------------------------------------------<br>| | UNTAR | DU | FIND | TAR | RM |<br>---------------------------------------------------------------<br>| single | ~3m45s | ~43s | ~47s | ~3m10s | ~3m15s |<br>---------------------------------------------------------------<br>| replicated | ~5m10s | ~59s | ~1m6s | ~1m19s | ~1m49s |<br>---------------------------------------------------------------<br>| distributed | ~4m18s | ~41s | ~57s | ~2m24s | ~1m38s |<br>---------------------------------------------------------------<br>| dist-repl | ~8m18s | ~1m4s | ~1m11s | ~1m24s | ~2m40s |<br>---------------------------------------------------------------<br>| native FS | ~11s | ~4s | ~2s | ~56s | ~10s |<br>---------------------------------------------------------------<br>| BeeGFS | ~3m43s | ~15s | ~3s | ~1m33s | ~46s |<br>---------------------------------------------------------------<br>| single (v2) | ~3m6s | ~14s | ~32s | ~1m2s | ~44s |<br>---------------------------------------------------------------<br>for info:<br><span class="Apple-tab-span" style="white-space: pre;"> </span>-BeeGFS is a distributed FS (4 bricks, 2 bricks per server and 2 servers)<br><span class="Apple-tab-span" style="white-space: pre;"> </span>- single (v2): simple gluster volume with default settings<br><br>I also note I obtain the same tar/untar performance issue with FhGFS/BeeGFS<br>but the rest (DU, FIND, RM) looks like to be OK.<br><br>Thank you very much for your reply and help.<br>Geoffrey<br>-----------------------------------------------<br>Geoffrey Letessier<br><br>Responsable informatique & ingénieur système<br>CNRS - UPR 9080 - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a><br><<a href="mailto:geoffrey.letessier@cnrs.fr">mailto:geoffrey.letessier@cnrs.fr</a>><br>Le 2 juin 2015 à 21:53, Ben Turner <<a href="mailto:bturner@redhat.com">bturner@redhat.com</a><br><<a href="mailto:bturner@redhat.com">mailto:bturner@redhat.com</a>>> a écrit :<br><br></span><blockquote type="cite"><span style="font-size: 11px;">I am seeing problems on 3.7 as well. Can you check /var/log/messages on<br>both the clients and servers for hung tasks like:<br><br>Jun 2 15:23:14 gqac006 kernel: "echo 0 ><br>/proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>Jun 2 15:23:14 gqac006 kernel: iozone D 0000000000000001 0<br>21999 1 0x00000080<br>Jun 2 15:23:14 gqac006 kernel: ffff880611321cc8 0000000000000082<br>ffff880611321c18 ffffffffa027236e<br>Jun 2 15:23:14 gqac006 kernel: ffff880611321c48 ffffffffa0272c10<br>ffff88052bd1e040 ffff880611321c78<br>Jun 2 15:23:14 gqac006 kernel: ffff88052bd1e0f0 ffff88062080c7a0<br>ffff880625addaf8 ffff880611321fd8<br>Jun 2 15:23:14 gqac006 kernel: Call Trace:<br>Jun 2 15:23:14 gqac006 kernel: [<ffffffffa027236e>] ?<br>rpc_make_runnable+0x7e/0x80 [sunrpc]<br>Jun 2 15:23:14 gqac006 kernel: [<ffffffffa0272c10>] ?<br>rpc_execute+0x50/0xa0 [sunrpc]<br>Jun 2 15:23:14 gqac006 kernel: [<ffffffff810aaa21>] ?<br>ktime_get_ts+0xb1/0xf0<br>Jun 2 15:23:14 gqac006 kernel: [<ffffffff811242d0>] ? sync_page+0x0/0x50<br>Jun 2 15:23:14 gqac006 kernel: [<ffffffff8152a1b3>] io_schedule+0x73/0xc0<br>Jun 2 15:23:14 gqac006 kernel: [<ffffffff8112430d>] sync_page+0x3d/0x50<br>Jun 2 15:23:14 gqac006 kernel: [<ffffffff8152ac7f>]<br>__wait_on_bit+0x5f/0x90<br>Jun 2 15:23:14 gqac006 kernel: [<ffffffff81124543>]<br>wait_on_page_bit+0x73/0x80<br>Jun 2 15:23:14 gqac006 kernel: [<ffffffff8109eb80>] ?<br>wake_bit_function+0x0/0x50<br>Jun 2 15:23:14 gqac006 kernel: [<ffffffff8113a525>] ?<br>pagevec_lookup_tag+0x25/0x40<br>Jun 2 15:23:14 gqac006 kernel: [<ffffffff8112496b>]<br>wait_on_page_writeback_range+0xfb/0x190<br>Jun 2 15:23:14 gqac006 kernel: [<ffffffff81124b38>]<br>filemap_write_and_wait_range+0x78/0x90<br>Jun 2 15:23:14 gqac006 kernel: [<ffffffff811c07ce>]<br>vfs_fsync_range+0x7e/0x100<br>Jun 2 15:23:14 gqac006 kernel: [<ffffffff811c08bd>] vfs_fsync+0x1d/0x20<br>Jun 2 15:23:14 gqac006 kernel: [<ffffffff811c08fe>] do_fsync+0x3e/0x60<br>Jun 2 15:23:14 gqac006 kernel: [<ffffffff811c0950>] sys_fsync+0x10/0x20<br>Jun 2 15:23:14 gqac006 kernel: [<ffffffff8100b072>]<br>system_call_fastpath+0x16/0x1b<br><br>Do you see a perf problem with just a simple DD or do you need a more<br>complex workload to hit the issue? I think I saw an issue with metadata<br>performance that I am trying to run down, let me know if you can see the<br>problem with simple DD reads / writes or if we need to do some sort of<br>dir / metadata access as well.<br><br>-b<br><br>----- Original Message -----<br></span><blockquote type="cite"><span style="font-size: 11px;">From: "Geoffrey Letessier" <<a href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a><br><<a href="mailto:geoffrey.letessier@cnrs.fr">mailto:geoffrey.letessier@cnrs.fr</a>>><br>To: "Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a><br><<a href="mailto:pkarampu@redhat.com">mailto:pkarampu@redhat.com</a>>><br>Cc:<span class="Apple-converted-space"> </span><a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><span class="Apple-converted-space"> </span><<a href="mailto:gluster-users@gluster.org">mailto:gluster-users@gluster.org</a>><br>Sent: Tuesday, June 2, 2015 8:09:04 AM<br>Subject: Re: [Gluster-users] GlusterFS 3.7 - slow/poor performances<br><br>Hi Pranith,<br><br>I’m sorry but I cannot bring you any comparison because comparison will<br>be<br>distorted by the fact in my HPC cluster in production the network<br>technology<br>is InfiniBand QDR and my volumes are quite different (brick in RAID6<br>(12x2TB), 2 bricks per server and 4 servers into my pool)<br><br>Concerning your demand, in attachments you can find all expected results<br>hoping it can help you to solve this serious performance issue (maybe I<br>need<br>play with glusterfs parameters?).<br><br>Thank you very much by advance,<br>Geoffrey<br>------------------------------------------------------<br>Geoffrey Letessier<br>Responsable informatique & ingénieur système<br>UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: 01 58 41 50 93 - eMail:<span class="Apple-converted-space"> </span><a href="mailto:geoffrey.letessier@ibpc.fr">geoffrey.letessier@ibpc.fr</a><br><<a href="mailto:geoffrey.letessier@ibpc.fr">mailto:geoffrey.letessier@ibpc.fr</a>><br><br><br><br><br>Le 2 juin 2015 à 10:09, Pranith Kumar Karampuri <<span class="Apple-converted-space"> </span><a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a><br><<a href="mailto:pkarampu@redhat.com">mailto:pkarampu@redhat.com</a>> > a<br>écrit :<br><br>hi Geoffrey,<br>Since you are saying it happens on all types of volumes, lets do the<br>following:<br>1) Create a dist-repl volume<br>2) Set the options etc you need.<br>3) enable gluster volume profile using "gluster volume profile <volname><br>start"<br>4) run the work load<br>5) give output of "gluster volume profile <volname> info"<br><br>Repeat the steps above on new and old version you are comparing this<br>with.<br>That should give us insight into what could be causing the slowness.<br><br>Pranith<br>On 06/02/2015 03:22 AM, Geoffrey Letessier wrote:<br><br><br>Dear all,<br><br>I have a crash test cluster where i’ve tested the new version of<br>GlusterFS<br>(v3.7) before upgrading my HPC cluster in production.<br>But… all my tests show me very very low performances.<br><br>For my benches, as you can read below, I do some actions (untar, du,<br>find,<br>tar, rm) with linux kernel sources, dropping cache, each on distributed,<br>replicated, distributed-replicated, single (single brick) volumes and the<br>native FS of one brick.<br><br># time (echo 3 > /proc/sys/vm/drop_caches; tar xJf<br>~/linux-4.1-rc5.tar.xz;<br>sync; echo 3 > /proc/sys/vm/drop_caches)<br># time (echo 3 > /proc/sys/vm/drop_caches; du -sh linux-4.1-rc5/; echo 3<br></span><blockquote type="cite"><span style="font-size: 11px;"><br></span></blockquote><span style="font-size: 11px;">/proc/sys/vm/drop_caches)<br># time (echo 3 > /proc/sys/vm/drop_caches; find linux-4.1-rc5/|wc -l;<br>echo 3<br></span><blockquote type="cite"><span style="font-size: 11px;">/proc/sys/vm/drop_caches)<br></span></blockquote><span style="font-size: 11px;"># time (echo 3 > /proc/sys/vm/drop_caches; tar czf linux-4.1-rc5.tgz<br>linux-4.1-rc5/; echo 3 > /proc/sys/vm/drop_caches)<br># time (echo 3 > /proc/sys/vm/drop_caches; rm -rf linux-4.1-rc5.tgz<br>linux-4.1-rc5/; echo 3 > /proc/sys/vm/drop_caches)<br><br>And here are the process times:<br><br>---------------------------------------------------------------<br>| | UNTAR | DU | FIND | TAR | RM |<br>---------------------------------------------------------------<br>| single | ~3m45s | ~43s | ~47s | ~3m10s | ~3m15s |<br>---------------------------------------------------------------<br>| replicated | ~5m10s | ~59s | ~1m6s | ~1m19s | ~1m49s |<br>---------------------------------------------------------------<br>| distributed | ~4m18s | ~41s | ~57s | ~2m24s | ~1m38s |<br>---------------------------------------------------------------<br>| dist-repl | ~8m18s | ~1m4s | ~1m11s | ~1m24s | ~2m40s |<br>---------------------------------------------------------------<br>| native FS | ~11s | ~4s | ~2s | ~56s | ~10s |<br>---------------------------------------------------------------<br><br>I get the same results, whether with default configurations with custom<br>configurations.<br><br>if I look at the side of the ifstat command, I can note my IO write<br>processes<br>never exceed 3MBs...<br><br>EXT4 native FS seems to be faster (roughly 15-20% but no more) than XFS<br>one<br><br>My [test] storage cluster config is composed by 2 identical servers<br>(biCPU<br>Intel Xeon X5355, 8GB of RAM, 2x2TB HDD (no-RAID) and Gb ethernet)<br><br>My volume settings:<br>single: 1server 1 brick<br>replicated: 2 servers 1 brick each<br>distributed: 2 servers 2 bricks each<br>dist-repl: 2 bricks in the same server and replica 2<br><br>All seems to be OK in gluster status command line.<br><br>Do you have an idea why I obtain so bad results?<br>Thanks in advance.<br>Geoffrey<br>-----------------------------------------------<br>Geoffrey Letessier<br><br>Responsable informatique & ingénieur système<br>CNRS - UPR 9080 - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: 01 58 41 50 93 - eMail:<span class="Apple-converted-space"> </span><a href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a><br><<a href="mailto:geoffrey.letessier@cnrs.fr">mailto:geoffrey.letessier@cnrs.fr</a>><br><br><br><br>_______________________________________________<br>Gluster-users mailing list<span class="Apple-converted-space"> </span><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br><<a href="mailto:Gluster-users@gluster.org">mailto:Gluster-users@gluster.org</a>><br><a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><br><<a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a>><br><br><br><br><br>_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><span class="Apple-converted-space"> </span><<a href="mailto:Gluster-users@gluster.org">mailto:Gluster-users@gluster.org</a>><br><a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><br><<a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a>></span></blockquote></blockquote></blockquote></blockquote></div></blockquote></div><br></div></body></html>