<div dir="ltr">Hello,<div><br></div><div>for the replicated one, is it a new issue or you just didn't notice before ? Same baseline as before?</div><div><br></div><div>I also have slowness with small files/many files.</div><div><br></div><div>For now I could only tune up things with:</div><div><br></div><div>On gluster level:</div><div><div>gluster volume set myvolume performance.io-thread-count 16</div><div>gluster volume set myvolume performance.cache-size 1GB</div><div>gluster volume set myvolume nfs.disable on</div><div>gluster volume set myvolume readdir-ahead enable</div><div>gluster volume set myvolume read-ahead disable</div></div><div><br></div><div>On network level (client and server) (I don't have infiniband):</div><div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">sysctl -w vm.swappiness=0</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">sysctl -w net.core.rmem_max=67108864</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">sysctl -w net.core.wmem_max=67108864<br>
# increase Linux autotuning TCP buffer limit to 32MB<br>
sysctl -w net.ipv4.tcp_rmem="4096 87380 33554432"<br>
sysctl -w net.ipv4.tcp_wmem="4096 65536 33554432"<br>
# increase the length of the processor input queue<br>
sysctl -w net.core.netdev_max_backlog=30000<br>
# recommended default congestion control is htcp<br>
sysctl -w net.ipv4.tcp_congestion_control=htcp</div></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small">But it's still really slow, even if better</span><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature">Cordialement,<br>Mathieu CHATEAU<br><a href="http://www.lotp.fr" target="_blank">http://www.lotp.fr</a></div></div>
<br><div class="gmail_quote">2015-06-20 2:34 GMT+02:00 Geoffrey Letessier <span dir="ltr"><<a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank">geoffrey.letessier@cnrs.fr</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Re,<div><br></div><div>For comparison, here is the output of the same script run on a distributed only volume (2 servers of the 4 previously described, 2 bricks each):<div><span class=""><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################ UNTAR time consumed ################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div></span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>1m44.698s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m8.891s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m8.353s</span></div><span class=""><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# DU time consumed ##################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div></span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">554M<span style="white-space:pre-wrap"> </span>linux-4.1-rc6</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>0m21.062s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m0.100s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m1.040s</span></div><span class=""><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# FIND time consumed ################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">52663</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div></span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>0m21.325s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m0.104s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m1.054s</span></div><span class=""><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# GREP time consumed ################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">7952</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div></span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>0m43.618s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m0.922s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m3.626s</span></div><span class=""><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# TAR time consumed #################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div></span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>0m50.577s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m29.745s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m4.086s</span></div><span class=""><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# RM time consumed ##################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div></span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>0m41.133s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m0.171s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m2.522s</span></div><div><br></div><div>The performances are amazing different!</div><span class=""><div><br></div><div>Geoffrey</div><div>
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">-----------------------------------------------<br>Geoffrey Letessier<br><br>Responsable informatique & ingénieur système<br>CNRS - UPR 9080 - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: <a href="tel:01%2058%2041%2050%2093" value="+33158415093" target="_blank">01 58 41 50 93</a> - eMail: <a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank">geoffrey.letessier@cnrs.fr</a></div>
</div>
<br></span><div><div><div class="h5"><div>Le 20 juin 2015 à 02:12, Geoffrey Letessier <<a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank">geoffrey.letessier@cnrs.fr</a>> a écrit :</div><br></div></div><blockquote type="cite"><div><div class="h5"><div style="word-wrap:break-word">Dear all,<div><br></div><div>I just noticed on my main volume of my HPC cluster my IO operations become impressively poor.. </div><div><br></div><div>Doing some file operations above a linux kernel sources compressed file, the untar operation can take more than 1/2 hours for this file (roughly 80MB and 52 000 files inside) as you read below:</div><div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################ UNTAR time consumed ################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>32m42.967s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m11.783s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m15.050s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# DU time consumed ##################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">557M<span style="white-space:pre-wrap"> </span>linux-4.1-rc6</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>0m25.060s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m0.068s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m0.344s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# FIND time consumed ################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">52663</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>0m25.687s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m0.084s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m0.387s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# GREP time consumed ################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">7952</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>2m15.890s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m0.887s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m2.777s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# TAR time consumed #################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>1m5.551s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m26.536s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m2.609s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">################# RM time consumed ##################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#######################################################</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0);min-height:11px"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">real<span style="white-space:pre-wrap"> </span>2m51.485s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">user<span style="white-space:pre-wrap"> </span>0m0.167s</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sys<span style="white-space:pre-wrap"> </span>0m1.663s</span></div></div><div><br></div><div>For information, this volume is a distributed replicated one and is composed by 4 servers with 2 bricks each. Each bricks is a 12-drives RAID6 vdisk with nice native performances (around 1.2GBs).</div><div><br></div><div>In comparison, when I use DD to generate a 100GB file on the same volume, my write throughput is around 1GB (client side) and 500MBs (server side) because of replication:</div><div>Client side:</div><div><div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[root@node056 ~]# ifstat -i ib0</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> ib0 </span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> KB/s in KB/s out</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> 3251.45 1.09e+06</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> 3139.80 1.05e+06</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> 3185.29 1.06e+06</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> 3293.84 1.09e+06</span></div></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">...</span></div><div><br></div><div>Server side:</div><div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[root@lucifer ~]# ifstat -i ib0</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> ib0 </span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> KB/s in KB/s out</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">561818.1 1746.42</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">560020.3 1737.92</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">526337.1 1648.20</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">513972.7 1613.69</span></div></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">...</span></div><div><br></div><div>DD command:</div><div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[root@node056 ~]# dd if=/dev/zero of=/home/root/test.dd bs=1M count=100000</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">100000+0 enregistrements lus</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">100000+0 enregistrements écrits</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">104857600000 octets (105 GB) copiés, 202,99 s, 517 MB/s</span></div></div><div><br></div><div>So this issue doesn’t seem coming from the network (which is Infiniband technology in this case)</div><div><br></div><div>You can find in attachments a set of files:</div><div><span style="white-space:pre-wrap"> </span>- mybench.sh: the bench script</div><div><span style="white-space:pre-wrap"> </span>- benches.txt: output of my "bench"</div><div><span style="white-space:pre-wrap"> </span>- profile.txt: gluster volume profile during the "bench"</div><div><span style="white-space:pre-wrap"> </span>- vol_status.txt: gluster volume status</div><div><span style="white-space:pre-wrap"> </span>- vol_info.txt: gluster volume info</div><div><br></div><div>Can someone help me to fix it (it’s very critical because this volume is on a HPC cluster in production).</div><div><br></div><div>Thanks by advance,</div><div>Geoffrey</div><div>
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">-----------------------------------------------<br>Geoffrey Letessier<br><br>Responsable informatique & ingénieur système<br>CNRS - UPR 9080 - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: <a href="tel:01%2058%2041%2050%2093" value="+33158415093" target="_blank">01 58 41 50 93</a> - eMail: <a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank">geoffrey.letessier@cnrs.fr</a></div>
</div>
</div></div></div></div><span><benches.txt></span><div style="word-wrap:break-word"><div></div></div><span><mybench.sh></span><div style="word-wrap:break-word"><div></div></div><span><profile.txt></span><div style="word-wrap:break-word"><div></div></div><span><vol_info.txt></span><div style="word-wrap:break-word"><div></div></div><span><vol_status.txt></span><div style="word-wrap:break-word"><div></div></div></blockquote></div><br></div></div></div><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>