<div dir="ltr">I am afraid I am not experienced enough to be much more useful.<div><br></div><div>My guess is that, since client is writing synchronously to all node (to keep data coherent), it's going as fast as the slowest brick.</div><div><br></div><div>Then small files are often slow because TCP windows doesn't have time to grow up.</div><div>That's why I gave you some kernel tuning to help TCP Windows to get bigger faster.</div><div><br></div><div>Do you use latest version (3.7.1) ?</div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature">Cordialement,<br>Mathieu CHATEAU<br><a href="http://www.lotp.fr" target="_blank">http://www.lotp.fr</a></div></div>
<br><div class="gmail_quote">2015-06-20 11:01 GMT+02:00 Geoffrey Letessier <span dir="ltr"><<a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank">geoffrey.letessier@cnrs.fr</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Hello Mathieu,<div><br></div><div>Thanks for replying.</div><div><br></div><div>Previously, i’ve never notice such throughput performances (around 1GBs for 1 big file) but.... The situation with a « big » set of small files wasn’t amazing but not such bad than today.</div><div><br></div><div>The problem seems to concern exclusively the size of each file.</div><div>"proof": </div><div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><div style="margin:0px">[root@node056 tmp]# dd if=/dev/zero of=masterfile bs=1M count=1000</div><div style="margin:0px">1000+0 enregistrements lus</div><div style="margin:0px">1000+0 enregistrements écrits</div><div style="margin:0px">1048576000 octets (1,0 GB) copiés, 2,09139 s, 501 MB/s</div><div style="margin:0px"><div style="margin:0px">[root@node056 tmp]# time split -b 1000000 -a 12 masterfile # 1MB per file</div><div style="margin:0px;min-height:11px"><br></div><div style="margin:0px">real<span style="white-space:pre-wrap"> </span>0m42.841s</div><div style="margin:0px">user<span style="white-space:pre-wrap"> </span>0m0.004s</div><div style="margin:0px">sys<span style="white-space:pre-wrap"> </span>0m1.416s</div><div><div style="margin:0px">[root@node056 tmp]# rm -f xaaaaaaaaa* && sync</div></div><div></div><div style="margin:0px">[root@node056 tmp]# time split -b 5000000 -a 12 masterfile # 5 MB per file</div><div style="margin:0px;min-height:11px"><br></div><div style="margin:0px">real<span style="white-space:pre-wrap"> </span>0m17.801s</div><div style="margin:0px">user<span style="white-space:pre-wrap"> </span>0m0.008s</div><div style="margin:0px">sys<span style="white-space:pre-wrap"> </span>0m1.396s</div><div><div style="margin:0px">[root@node056 tmp]# rm -f xaaaaaaaaa* && sync</div><div style="margin:0px">[root@node056 tmp]# time split -b 10000000 -a 12 masterfile # 10MB per file</div><div style="margin:0px;min-height:11px"><br></div><div style="margin:0px">real<span style="white-space:pre-wrap"> </span>0m9.686s</div><div style="margin:0px">user<span style="white-space:pre-wrap"> </span>0m0.008s</div><div style="margin:0px">sys<span style="white-space:pre-wrap"> </span>0m1.451s</div><div style="margin:0px">[root@node056 tmp]# rm -f xaaaaaaaaa* && sync</div><div style="margin:0px">[root@node056 tmp]# time split -b 20000000 -a 12 masterfile # 20MB per file</div><div style="margin:0px;min-height:11px"><br></div><div style="margin:0px">real<span style="white-space:pre-wrap"> </span>0m9.717s</div><div style="margin:0px">user<span style="white-space:pre-wrap"> </span>0m0.003s</div><div style="margin:0px">sys<span style="white-space:pre-wrap"> </span>0m1.399s</div><div style="margin:0px">[root@node056 tmp]# rm -f xaaaaaaaaa* && sync</div></div><div><div style="margin:0px">[root@node056 tmp]# time split -b 1000000 -a 12 masterfile # 10MB per file</div><div style="margin:0px;min-height:11px"><br></div><div style="margin:0px">real<span style="white-space:pre-wrap"> </span>0m40.283s</div><div style="margin:0px">user<span style="white-space:pre-wrap"> </span>0m0.007s</div><div style="margin:0px">sys<span style="white-space:pre-wrap"> </span>0m1.390s</div></div><div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px"><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255)">[root@node056 tmp]# rm -f xaaaaaaaaa* && sync</div></div></div></div></div><div><br></div><div>Higher is the generated file size, best is the performance (IO throughput and running time)… ifstat output is coherent from both client/node and server side..</div><div><br></div><div>a new test:</div><div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">[root@node056 tmp]# dd if=/dev/zero of=masterfile bs=1M count=10000</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">10000+0 enregistrements lus</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">10000+0 enregistrements écrits</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">10485760000 octets (10 GB) copiés, 23,0044 s, 456 MB/s</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">[root@node056 tmp]# rm -f xaaaaaaaaa* && sync</div></div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">[root@node056 tmp]# time split -b 10000000 -a 12 masterfile # 10MB per file</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><div style="margin:0px;min-height:11px"><br></div><div style="margin:0px">real<span style="white-space:pre-wrap"> </span>1m43.216s</div><div style="margin:0px">user<span style="white-space:pre-wrap"> </span>0m0.038s</div><div style="margin:0px">sys<span style="white-space:pre-wrap"> </span>0m13.407s</div><div><br></div></div><div><br></div><div>So the performance per file is the same (despite of 10x more files)</div><div><br></div><div>So, i dont understand why, to get the best performance, i need to create file with a size of 10MB or more.</div><div><br></div><div>Here are my volume reconfigured options:</div><div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">performance.cache-max-file-size: 64MB</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">performance.read-ahead: on</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">performance.write-behind: on</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">features.quota-deem-statfs: on</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">performance.stat-prefetch: on</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">performance.flush-behind: on</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">features.default-soft-limit: 90%</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">features.quota: on</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">diagnostics.brick-log-level: CRITICAL</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">auth.allow: localhost,127.0.0.1,10.*</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">nfs.disable: on</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">performance.cache-size: 1GB</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">performance.write-behind-window-size: 4MB</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">performance.quick-read: on</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">performance.io-cache: on</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">performance.io-thread-count: 64</div><div style="margin:0px;font-size:10px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">nfs.enable-ino32: off</div></div><div><br></div><div>It’s not a local cache trouble because:</div><div><span style="white-space:pre-wrap"> </span>1- it’s disabled in my mount command <span style="background-color:rgb(0,0,0);color:rgb(255,255,255);font-family:Menlo;font-size:10px">mount -t glusterfs -o transport=rdma,</span><span style="background-color:rgb(0,0,0);font-family:Menlo;font-size:10px"><font color="#f5ec00">direct-io-mode=disable</font></span><span style="background-color:rgb(0,0,0);color:rgb(255,255,255);font-family:Menlo;font-size:10px">,enable-ino32 ib-storage1:vol_home /home</span></div><div><span style="white-space:pre-wrap"> </span>2- i made my test also playing with <span style="background-color:rgb(0,0,0);color:rgb(255,255,255);font-family:Menlo;font-size:10px">/proc/sys/vm/drop_caches</span></div><div><span style="white-space:pre-wrap"> </span>3- I note the same ifstat output from both client and server side which is coherent with the computing of bandwidth (file sizes / time (considering the replication).</div><div><br></div><div>I think it’s not an infiniband network trouble but here are my [not default] settings:</div><div>connected mode with MTU set to 65520 </div><div><br></div><div>Do you confirm my feelings? If yes, do you have any other idea?</div><div><br></div><div>Thanks again and thanks by advance,</div><span class=""><div>Geoffrey</div><div>
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">-----------------------------------------------<br>Geoffrey Letessier<br><br>Responsable informatique & ingénieur système<br>CNRS - UPR 9080 - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: <a href="tel:01%2058%2041%2050%2093" value="+33158415093" target="_blank">01 58 41 50 93</a> - eMail: <a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank">geoffrey.letessier@cnrs.fr</a></div>
</div>
<br></span><div><div class="h5"><div><div>Le 20 juin 2015 à 09:12, Mathieu Chateau <<a href="mailto:mathieu.chateau@lotp.fr" target="_blank">mathieu.chateau@lotp.fr</a>> a écrit :</div><br><blockquote type="cite"><div dir="ltr">Hello,<div><br></div><div>for the replicated one, is it a new issue or you just didn't notice before ? Same baseline as before?</div><div><br></div><div>I also have slowness with small files/many files.</div><div><br></div><div>For now I could only tune up things with:</div><div><br></div><div>On gluster level:</div><div><div>gluster volume set myvolume performance.io-thread-count 16</div><div>gluster volume set myvolume performance.cache-size 1GB</div><div>gluster volume set myvolume nfs.disable on</div><div>gluster volume set myvolume readdir-ahead enable</div><div>gluster volume set myvolume read-ahead disable</div></div><div><br></div><div>On network level (client and server) (I don't have infiniband):</div><div><div style="font-family:Tahoma;text-align:-webkit-auto;font-size:inherit">sysctl -w vm.swappiness=0</div><div style="font-family:Tahoma;text-align:-webkit-auto;font-size:inherit">sysctl -w net.core.rmem_max=67108864</div><div style="font-family:Tahoma;text-align:-webkit-auto;font-size:inherit">sysctl -w net.core.wmem_max=67108864<br>
# increase Linux autotuning TCP buffer limit to 32MB<br>
sysctl -w net.ipv4.tcp_rmem="4096 87380 33554432"<br>
sysctl -w net.ipv4.tcp_wmem="4096 65536 33554432"<br>
# increase the length of the processor input queue<br>
sysctl -w net.core.netdev_max_backlog=30000<br>
# recommended default congestion control is htcp<br>
sysctl -w net.ipv4.tcp_congestion_control=htcp</div></div><div style="font-family:Tahoma;text-align:-webkit-auto;font-size:inherit"><br></div><div style="font-family:Tahoma;text-align:-webkit-auto;font-size:inherit"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small">But it's still really slow, even if better</span><br></div></div><div class="gmail_extra"><br clear="all"><div><div>Cordialement,<br>Mathieu CHATEAU<br><a href="http://www.lotp.fr" target="_blank">http://www.lotp.fr</a></div></div>
<br><div class="gmail_quote">2015-06-20 2:34 GMT+02:00 Geoffrey Letessier <span dir="ltr"><<a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank">geoffrey.letessier@cnrs.fr</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Re,<div><br></div><div>For comparison, here is the output of the same script run on a distributed only volume (2 servers of the 4 previously described, 2 bricks each):<div><span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################ UNTAR time consumed ################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div></span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>1m44.698s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m8.891s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m8.353s</span></div><span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# DU time consumed ##################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div></span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">554M<span style="white-space:pre-wrap"> </span>linux-4.1-rc6</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>0m21.062s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m0.100s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m1.040s</span></div><span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# FIND time consumed ################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">52663</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div></span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>0m21.325s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m0.104s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m1.054s</span></div><span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# GREP time consumed ################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">7952</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div></span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>0m43.618s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m0.922s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m3.626s</span></div><span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# TAR time consumed #################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div></span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>0m50.577s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m29.745s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m4.086s</span></div><span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# RM time consumed ##################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div></span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>0m41.133s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m0.171s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m2.522s</span></div><div><br></div><div>The performances are amazing different!</div><span><div><br></div><div>Geoffrey</div><div>
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">-----------------------------------------------<br>Geoffrey Letessier<br><br>Responsable informatique & ingénieur système<br>CNRS - UPR 9080 - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: <a href="tel:01%2058%2041%2050%2093" value="+33158415093" target="_blank">01 58 41 50 93</a> - eMail: <a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank">geoffrey.letessier@cnrs.fr</a></div>
</div>
<br></span><div><div><div><div>Le 20 juin 2015 à 02:12, Geoffrey Letessier <<a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank">geoffrey.letessier@cnrs.fr</a>> a écrit :</div><br></div></div><blockquote type="cite"><div><div><div style="word-wrap:break-word">Dear all,<div><br></div><div>I just noticed on my main volume of my HPC cluster my IO operations become impressively poor.. </div><div><br></div><div>Doing some file operations above a linux kernel sources compressed file, the untar operation can take more than 1/2 hours for this file (roughly 80MB and 52 000 files inside) as you read below:</div><div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################ UNTAR time consumed ################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>32m42.967s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m11.783s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m15.050s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# DU time consumed ##################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">557M<span style="white-space:pre-wrap"> </span>linux-4.1-rc6</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>0m25.060s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m0.068s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m0.344s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# FIND time consumed ################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">52663</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>0m25.687s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m0.084s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m0.387s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# GREP time consumed ################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">7952</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>2m15.890s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m0.887s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m2.777s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# TAR time consumed #################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>1m5.551s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m26.536s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m2.609s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# RM time consumed ##################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>2m51.485s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m0.167s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m1.663s</span></div></div><div><br></div><div>For information, this volume is a distributed replicated one and is composed by 4 servers with 2 bricks each. Each bricks is a 12-drives RAID6 vdisk with nice native performances (around 1.2GBs).</div><div><br></div><div>In comparison, when I use DD to generate a 100GB file on the same volume, my write throughput is around 1GB (client side) and 500MBs (server side) because of replication:</div><div>Client side:</div><div><div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[root@node056 ~]# ifstat -i ib0</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> ib0 </span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> KB/s in KB/s out</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> 3251.45 1.09e+06</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> 3139.80 1.05e+06</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> 3185.29 1.06e+06</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> 3293.84 1.09e+06</span></div></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">...</span></div><div><br></div><div>Server side:</div><div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[root@lucifer ~]# ifstat -i ib0</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> ib0 </span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> KB/s in KB/s out</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">561818.1 1746.42</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">560020.3 1737.92</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">526337.1 1648.20</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">513972.7 1613.69</span></div></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">...</span></div><div><br></div><div>DD command:</div><div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[root@node056 ~]# dd if=/dev/zero of=/home/root/test.dd bs=1M count=100000</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">100000+0 enregistrements lus</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">100000+0 enregistrements écrits</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">104857600000 octets (105 GB) copiés, 202,99 s, 517 MB/s</span></div></div><div><br></div><div>So this issue doesn’t seem coming from the network (which is Infiniband technology in this case)</div><div><br></div><div>You can find in attachments a set of files:</div><div><span style="white-space:pre-wrap"> </span>- mybench.sh: the bench script</div><div><span style="white-space:pre-wrap"> </span>- benches.txt: output of my "bench"</div><div><span style="white-space:pre-wrap"> </span>- profile.txt: gluster volume profile during the "bench"</div><div><span style="white-space:pre-wrap"> </span>- vol_status.txt: gluster volume status</div><div><span style="white-space:pre-wrap"> </span>- vol_info.txt: gluster volume info</div><div><br></div><div>Can someone help me to fix it (it’s very critical because this volume is on a HPC cluster in production).</div><div><br></div><div>Thanks by advance,</div><div>Geoffrey</div><div>
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">-----------------------------------------------<br>Geoffrey Letessier<br><br>Responsable informatique & ingénieur système<br>CNRS - UPR 9080 - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: <a href="tel:01%2058%2041%2050%2093" value="+33158415093" target="_blank">01 58 41 50 93</a> - eMail: <a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank">geoffrey.letessier@cnrs.fr</a></div>
</div>
</div></div></div></div><span><benches.txt></span><div style="word-wrap:break-word"><div></div></div><span><mybench.sh></span><div style="word-wrap:break-word"><div></div></div><span><profile.txt></span><div style="word-wrap:break-word"><div></div></div><span><vol_info.txt></span><div style="word-wrap:break-word"><div></div></div><span><vol_status.txt></span><div style="word-wrap:break-word"><div></div></div></blockquote></div><br></div></div></div><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div><br></div></div></div></div></blockquote></div><br></div>