<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Hello,<div><br></div><div>@Krutika: Thanks for transferring my issue.</div><div><br></div><div>Everything is becoming completely crazy; other quotas are exploding. Indeed, after having remove my previous quota in failure, some other quotas have grown up as you can read below:</div><div><br></div><div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">[root@lucifer ~]# gluster volume quota vol_home list</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"> Path Hard-limit Soft-limit Used Available</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">--------------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">/baaden_team 20.0TB 90% 15.1TB 4.9TB</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">/sterpone_team 14.0TB 90% 25.5TB 0Bytes</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">/simlab_team 5.0TB 90% 1.3TB 3.7TB</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">/sacquin_team 10.0TB 90% 8.3TB 1.7TB</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">/admin_team 1.0TB 90% 17.0GB 1007.0GB</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">/amyloid_team 7.0TB 90% 6.4TB 577.5GB</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">/amyloid_team/nguyen 4.0TB 90% 3.7TB 312.7GB</div><div><br></div><div><br></div><div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">[root@lucifer ~]# pdsh -w cl-storage[1,3] du -sh /export/brick_home/brick*/sterpone_team</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">cl-storage1: 3,1T<span class="Apple-tab-span" style="white-space:pre"> </span>/export/brick_home/brick1/sterpone_team</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">cl-storage1: 2,3T<span class="Apple-tab-span" style="white-space:pre"> </span>/export/brick_home/brick2/sterpone_team</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">cl-storage3: 2,7T<span class="Apple-tab-span" style="white-space:pre"> </span>/export/brick_home/brick1/sterpone_team</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">cl-storage3: 2,9T<span class="Apple-tab-span" style="white-space:pre"> </span>/export/brick_home/brick2/sterpone_team</div></div><div>=> ~11TB (not 25.5TB!!!)</div><div><br></div><div><br></div><div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">[root@lucifer ~]# pdsh -w cl-storage[1,3] du -sh /export/brick_home/brick*/baaden_team</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">cl-storage1: 4,2T<span class="Apple-tab-span" style="white-space:pre"> </span>/export/brick_home/brick1/baaden_team</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">cl-storage3: 3,7T<span class="Apple-tab-span" style="white-space:pre"> </span>/export/brick_home/brick1/baaden_team</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">cl-storage1: 3,6T<span class="Apple-tab-span" style="white-space:pre"> </span>/export/brick_home/brick2/baaden_team</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">cl-storage3: 3,5T<span class="Apple-tab-span" style="white-space:pre"> </span>/export/brick_home/brick2/baaden_team</div></div><div>=> ~15TB (not 14TB).</div><div><br></div><div>Etc.</div><div><br></div><div>Do you please help me to urgently solve this issue because this situation is blocking and I must stop the production until.</div><div><br></div><div>Do you think upgrading storage cluster into 3.7.1 (the latest) version of GlusterFS could fix the problem?</div><div><br></div><div>Thanks by advance,</div><div>Geoffrey</div><div><div apple-content-edited="true">
------------------------------------------------------<br>Geoffrey Letessier<br>Responsable informatique & ingénieur système<br>UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr">geoffrey.letessier@ibpc.fr</a>
</div>
<br><div><div>Le 27 juin 2015 à 08:13, Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>> a écrit :</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="font-family: garamond, 'new york', times, serif; font-size: 12pt;"><div>Copying Vijai and Raghavendra for help...</div><div><br></div><div>-Krutika</div><hr id="zwchr"><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Geoffrey Letessier" <<a href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a>><br><b>To: </b>"Krutika Dhananjay" <<a href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>><br><b>Sent: </b>Saturday, June 27, 2015 2:13:52 AM<br><b>Subject: </b>Re: [Gluster-users] GlusterFS 3.5.3 - untar: very poor performance<br><div><br></div>Hi Krutika,<div><br></div><div>Since I have re-enabled the quota feature on my volume vol_home, one defined quota is become like crazy… And it’s a very very very big problem for us.</div><div><br></div><div>During all the day, after having re-enabled it, i noted the used space detected growing up (without any user IO on)..</div><div><br></div><div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" data-mce-style="margin: 0px; font-size: 11px; font-family: Menlo; color: #ffffff; background-color: #000000;">[root@lucifer ~]# gluster volume quota vol_home list|grep derreumaux_team</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" data-mce-style="margin: 0px; font-size: 11px; font-family: Menlo; color: #ffffff; background-color: #000000;">/derreumaux_team 14.0TB 80% 13.7TB 357.2GB</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" data-mce-style="margin: 0px; font-size: 11px; font-family: Menlo; color: #ffffff; background-color: #000000;">[root@lucifer ~]# gluster volume quota vol_home list /derreumaux_team</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" data-mce-style="margin: 0px; font-size: 11px; font-family: Menlo; color: #ffffff; background-color: #000000;"> Path Hard-limit Soft-limit Used Available</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" data-mce-style="margin: 0px; font-size: 11px; font-family: Menlo; color: #ffffff; background-color: #000000;">--------------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" data-mce-style="margin: 0px; font-size: 11px; font-family: Menlo; color: #ffffff; background-color: #000000;">/derreumaux_team 14.0TB 80% 13.1TB 874.1GB</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" data-mce-style="margin: 0px; font-size: 11px; font-family: Menlo; color: #ffffff; background-color: #000000;">[root@lucifer ~]# pdsh -w cl-storage[1,3] du -sh /export/brick_home/brick*/derreumaux_team</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" data-mce-style="margin: 0px; font-size: 11px; font-family: Menlo; color: #ffffff; background-color: #000000;">cl-storage3: 590G<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>/export/brick_home/brick1/derreumaux_team</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" data-mce-style="margin: 0px; font-size: 11px; font-family: Menlo; color: #ffffff; background-color: #000000;">cl-storage3: 611G<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>/export/brick_home/brick2/derreumaux_team</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" data-mce-style="margin: 0px; font-size: 11px; font-family: Menlo; color: #ffffff; background-color: #000000;">cl-storage1: 567G<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>/export/brick_home/brick1/derreumaux_team</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" data-mce-style="margin: 0px; font-size: 11px; font-family: Menlo; color: #ffffff; background-color: #000000;">cl-storage1: 564G<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>/export/brick_home/brick2/derreumaux_team</div><div><br class="webkit-block-placeholder"></div><div>As you can see in these 3 command lines, i obtain 3 different results but, the worse, it’s quota system est very very far from the real disk used space (13.7TB <> 13.1TB <<>> 2.3TB).</div><div><br></div><div>Can you please help to fix it very quickly because all this group is completely block by exceeded quota.</div><div><br></div><div>Thank you so much by advance,</div><div>Have a nice week-end,</div><div>Geoffrey</div><div>------------------------------------------------------<br>Geoffrey Letessier<br>Responsable informatique & ingénieur système<br>UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr" target="_blank" data-mce-href="mailto:geoffrey.letessier@ibpc.fr">geoffrey.letessier@ibpc.fr</a></div><br><div><div>Le 26 juin 2015 à 10:29, Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank" data-mce-href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>> a écrit :</div><br class="Apple-interchange-newline"><blockquote><div style="font-family: garamond, 'new york', times, serif; font-size: 12pt;" data-mce-style="font-family: garamond, 'new york', times, serif; font-size: 12pt;"><div>No but if you are saying it is 3.5.3 rpm version, then that bug does not exist there.<br></div><div>And still it is weird how you are seeing such bad performance. :-/<br></div><div>Anything suspicious in the logs?<br></div><div><br></div><div>-Krutika<br></div><hr id="zwchr"><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" data-mce-style="border-left-width: 2px; border-left-style: solid; border-left-color: #1010ff; margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;"><b>From: </b>"Geoffrey Letessier" <<a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank" data-mce-href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a>><br><b>To: </b>"Krutika Dhananjay" <<a href="mailto:kdhananj@redhat.com" target="_blank" data-mce-href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>><br><b>Sent: </b>Friday, June 26, 2015 1:27:16 PM<br><b>Subject: </b>Re: [Gluster-users] GlusterFS 3.5.3 - untar: very poor performance<br><div><br></div>No , it’s the 3.5.3 RPMS version if found on your reposity (published on novembre 2014).<div class="">So, you suggest me to simply upgrade all servers and clients with the new 3.5.4 version? Wouldn't it be better to upgrade all the system (servers and clients) to the 3.7.1 version?</div><div class=""><br class=""></div><div class="">Geoffrey<br class=""><div class=""><div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="" data-mce-style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">------------------------------------------------------<br class="">Geoffrey Letessier<br class="">Responsable informatique & ingénieur système<br class="">UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br class="">Institut de Biologie Physico-Chimique<br class="">13, rue Pierre et Marie Curie - 75005 Paris<br class="">Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr" class="" target="_blank" data-mce-href="mailto:geoffrey.letessier@ibpc.fr">geoffrey.letessier@ibpc.fr</a><br></div></div><br class=""><div><blockquote class=""><div class="">Le 26 juin 2015 à 09:03, Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" class="" target="_blank" data-mce-href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>> a écrit :</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family: garamond, 'new york', times, serif; font-size: 12pt;" class="" data-mce-style="font-family: garamond, 'new york', times, serif; font-size: 12pt;"><div class="">Also, so are you running 3.5.3 rpms on the clients? Or is it a patched version with more fixes on top of 3.5.3?<br class=""></div><div class="">The reason I ask this is because there was one performance issue introduced after 3.5.3 and fixed by 3.5.4 in replication module. I'm wondering if that could be causing the issue you experience.<br class=""></div><div class=""><br class=""></div><div class="">-Krutika<br class=""></div><hr id="zwchr" class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class="" data-mce-style="border-left-width: 2px; border-left-style: solid; border-left-color: #1010ff; margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;"><b class="">From: </b>"Geoffrey Letessier" <<a href="mailto:geoffrey.letessier@cnrs.fr" class="" target="_blank" data-mce-href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a>><br class=""><b class="">To: </b>"Krutika Dhananjay" <<a href="mailto:kdhananj@redhat.com" class="" target="_blank" data-mce-href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>><br class=""><b class="">Sent: </b>Friday, June 26, 2015 10:05:26 AM<br class=""><b class="">Subject: </b>Re: [Gluster-users] GlusterFS 3.5.3 - untar: very poor performance<br class=""><div class=""><br class=""></div>Hi Krutika,<div class=""><br class=""></div><div class="">Oops, I disable quota manager without saving configuration. Could you tell me how to retrieve quota list information?</div><div class=""><br class=""></div><div class="">I’m gonna test the untar in the meantime.</div><div class=""><br class=""></div><div class="">Geoffrey<br class=""><div class="">------------------------------------------------------<br class="">Geoffrey Letessier<br class="">Responsable informatique & ingénieur système<br class="">UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br class="">Institut de Biologie Physico-Chimique<br class="">13, rue Pierre et Marie Curie - 75005 Paris<br class="">Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr" target="_blank" class="" data-mce-href="mailto:geoffrey.letessier@ibpc.fr">geoffrey.letessier@ibpc.fr</a><br class=""></div><br class=""><div class=""><div class="">Le 26 juin 2015 à 04:56, Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank" class="" data-mce-href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>> a écrit :</div><br class="Apple-interchange-newline"><blockquote class=""><div style="font-family: garamond, 'new york', times, serif; font-size: 12pt;" class="" data-mce-style="font-family: garamond, 'new york', times, serif; font-size: 12pt;"><div class="">Hi,<br class=""></div><div class=""><br class=""></div><div class="">So i tried out kernel src tree untar locally on a plain replicate (1x2) volume and it took me 7m30sec on an average. This was on vms and there was no rdma and there was no quota enabled.</div><div class="">Could you try the same thing on a volume without quota to see if it makes a difference to the perf?</div><div class=""><br class=""></div><div class="">-Krutika</div><div class=""><br class=""></div><hr id="zwchr" class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class="" data-mce-style="border-left-width: 2px; border-left-style: solid; border-left-color: #1010ff; margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;"><b class="">From: </b>"Geoffrey Letessier" <<a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank" class="" data-mce-href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a>><br class=""><b class="">To: </b>"Krutika Dhananjay" <<a href="mailto:kdhananj@redhat.com" target="_blank" class="" data-mce-href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>><br class=""><b class="">Sent: </b>Wednesday, June 24, 2015 10:21:13 AM<br class=""><b class="">Subject: </b>Re: [Gluster-users] GlusterFS 3.5.3 - untar: very poor performance<br class=""><div class=""><br class=""></div>Hi Krutika,<div class=""><br class=""></div><div class="">OK, thank you very much by advance.</div><div class="">Concerning quota system, are you in touch with Vijaykumar? Because I’m still waiting for a answer since a couple of days, nay more.</div><div class=""><br class=""></div><div class="">One more time, thank you.</div><div class="">Have a nice day (in France it’s 6:50 AM).</div><div class="">Geoffrey</div><div class=""><div class=""><div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="" data-mce-style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">-----------------------------------------------<br class="">Geoffrey Letessier<br class=""><div class=""><br class=""></div>Responsable informatique & ingénieur système<br class="">CNRS - UPR 9080 - Laboratoire de Biochimie Théorique<br class="">Institut de Biologie Physico-Chimique<br class="">13, rue Pierre et Marie Curie - 75005 Paris<br class="">Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank" class="" data-mce-href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a><br class=""></div></div><br class=""><div class=""><div class="">Le 24 juin 2015 à 05:55, Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank" class="" data-mce-href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>> a écrit :</div><br class="Apple-interchange-newline"><blockquote class=""><div style="font-family: garamond, 'new york', times, serif; font-size: 12pt;" class="" data-mce-style="font-family: garamond, 'new york', times, serif; font-size: 12pt;"><div class="">Ok so for anything related to replication, I could help you out.</div><div class="">But for quota, it would be better to ask Vijaikumar Mallikarjuna or Raghavendra G on the mailing list.<br class=""></div><div class="">I used to work on quota, long time back. But now I am not in touch with the component anymore and do not know of the latest changes to it.<br class=""></div><div class="">For the performance issue, I will try linux kernel src untar on my machines and let you know what I find.<br class=""></div><div class=""><br class=""></div><div class="">-Krutika<br class=""></div><div class=""><br class=""></div><hr id="zwchr" class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class="" data-mce-style="border-left-width: 2px; border-left-style: solid; border-left-color: #1010ff; margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;"><b class="">From: </b>"Geoffrey Letessier" <<a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank" class="" data-mce-href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a>><br class=""><b class="">To: </b>"Krutika Dhananjay" <<a href="mailto:kdhananj@redhat.com" target="_blank" class="" data-mce-href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>><br class=""><b class="">Sent: </b>Monday, June 22, 2015 9:00:52 PM<br class=""><b class="">Subject: </b>Re: [Gluster-users] GlusterFS 3.5.3 - untar: very poor performance<br class=""><div class=""><br class=""></div>Hi Krutika,<div class=""><br class=""></div><div class="">Sorry for the delay but i was in meeting all the day. </div><div class=""><br class=""></div><div class=""><blockquote class=""><div class=""><div class="" style="font-family: garamond, 'new york', times, serif; font-size: 12pt;" data-mce-style="font-family: garamond, 'new york', times, serif; font-size: 12pt;"><div class="">Good to hear from you as well. :)<br class=""></div></div></div></blockquote><div class=""><div class=""><div class="" style="font-family: garamond, 'new york', times, serif; font-size: 12pt;" data-mce-style="font-family: garamond, 'new york', times, serif; font-size: 12pt;"><div class="">;-)</div><div class=""><blockquote style="font-family: Helvetica; font-size: 12px;" class="" data-mce-style="font-family: Helvetica; font-size: 12px;"><div class=""><div class="" style="font-family: garamond, 'new york', times, serif; font-size: 12pt;" data-mce-style="font-family: garamond, 'new york', times, serif; font-size: 12pt;"><div class="">So you are seeing this bad performance only in 3.5.3? Any other releases you tried this test on, where the results were much better with replication?<br class=""></div></div></div></blockquote><div class=""><div class=""><div class="" style="font-family: garamond, 'new york', times, serif; font-size: 12pt;" data-mce-style="font-family: garamond, 'new york', times, serif; font-size: 12pt;"><div class="">Yes but I’m not sure my issue is only concerning this specific release. A few days ago, the untar process (with the same version of GlusterFS) took around 8 minutes, now around 32 minutes. 8 was too much but what about 32 minutes? :)</div><div class=""><br class=""></div><div class="">That said, my matter is only concerning small files because if i play with dd (or other) with only 1 big file all is OK (client write throughput: ~1GBs => ~500MBs in each replica)</div><div class=""><br class=""></div><div class="">If i run my bench on my only distributed volume i get a good performance (untar: ~1m44s, etc.)..</div><div class=""><br class=""></div><div class="">In addition, i dunno if it can be important, I have some troubles with GlusterFS group quota: there are a lot of conflicts between quota size and actual file size which dont match and a lot of "quota xattrs not found" messages with quota-verify glusterfs app. -you can find in attachment an extract of quota-verify outputs. </div></div></div></div></div><div class=""><br class=""></div><div class=""><blockquote style="font-family: Helvetica; font-size: 12px;" class="" data-mce-style="font-family: Helvetica; font-size: 12px;"><div class=""><div class="" style="font-family: garamond, 'new york', times, serif; font-size: 12pt;" data-mce-style="font-family: garamond, 'new york', times, serif; font-size: 12pt;"><div class="">If so, could you please let me know? Meanwhile let me try the untar myself on my vms to see what could be causing the perf issue.<br class=""></div></div></div></blockquote><div class=""><div class=""><div class="" style="font-family: garamond, 'new york', times, serif; font-size: 12pt;" data-mce-style="font-family: garamond, 'new york', times, serif; font-size: 12pt;"><div class="">OK, thanks. </div><div class=""><br class=""></div><div class="">See you,</div><div class="">Geoffrey</div></div></div></div></div></div></div></div><div class=""><div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="" data-mce-style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">------------------------------------------------------<br class="">Geoffrey Letessier<br class="">Responsable informatique & ingénieur système<br class="">UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br class="">Institut de Biologie Physico-Chimique<br class="">13, rue Pierre et Marie Curie - 75005 Paris<br class="">Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr" class="" target="_blank" data-mce-href="mailto:geoffrey.letessier@ibpc.fr">geoffrey.letessier@ibpc.fr</a><br class=""></div></div><br class=""><div class=""><blockquote class=""><div class="">Le 22 juin 2015 à 11:35, Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" class="" target="_blank" data-mce-href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>> a écrit :</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family: garamond, 'new york', times, serif; font-size: 12pt;" class="" data-mce-style="font-family: garamond, 'new york', times, serif; font-size: 12pt;"><div class="">Hi Geoffrey,<br class=""></div><div class=""><br class=""></div><div class="">Good to hear from you as well. :)<br class=""></div><div class="">Ok so you say disabling write-behind does not help. Makes me wonder what the problem could be.<br class=""></div><div class="">So you are seeing this bad performance only in 3.5.3? Any other releases you tried this test on, where the results were much better with replication?<br class=""></div><div class="">If so, could you please let me know? Meanwhile let me try the untar myself on my vms to see what could be causing the perf issue.<br class=""></div><div class=""><br class=""></div><div class="">-Krutika<br class=""></div><div class=""><br class=""></div><hr id="zwchr" class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class="" data-mce-style="border-left-width: 2px; border-left-style: solid; border-left-color: #1010ff; margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;"><b class="">From: </b>"Geoffrey Letessier" <<a href="mailto:geoffrey.letessier@cnrs.fr" class="" target="_blank" data-mce-href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a>><br class=""><b class="">To: </b>"Krutika Dhananjay" <<a href="mailto:kdhananj@redhat.com" class="" target="_blank" data-mce-href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>><br class=""><b class="">Sent: </b>Monday, June 22, 2015 10:14:26 AM<br class=""><b class="">Subject: </b>Re: [Gluster-users] GlusterFS 3.5.3 - untar: very poor performance<br class=""><div class=""><br class=""></div>Hi Krutika,<div class=""><br class=""></div><div class="">It’s good to read you again :)</div><div class=""><br class=""></div><div class="">Here are my answers:</div><div class="">1- could you remind me how to know if self-heal is currently in progress? I dont note any special neither mount-point (except /var/run/gluster/vol_home one) nor dedicated process; but maybe i look in the wrong place..</div><div class="">2- OK, I just disabled write-behind parameter and rerun the bench. I’ll let you know more about when I will arrive at my office (I’m still at home at this time).</div><div class=""><br class=""></div><div class="">See you and thanks you for helping. </div><div class="">Geoffrey</div><div class=""><div class=""><div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="" data-mce-style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">-----------------------------------------------<br class="">Geoffrey Letessier<br class=""><div class=""><br class=""></div>Responsable informatique & ingénieur système<br class="">CNRS - UPR 9080 - Laboratoire de Biochimie Théorique<br class="">Institut de Biologie Physico-Chimique<br class="">13, rue Pierre et Marie Curie - 75005 Paris<br class="">Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank" class="" data-mce-href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a><br class=""></div></div><br class=""><div class=""><div class="">Le 22 juin 2015 à 04:35, Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank" class="" data-mce-href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>> a écrit :</div><br class="Apple-interchange-newline"><blockquote class=""><div style="font-family: garamond, 'new york', times, serif; font-size: 12pt;" class="" data-mce-style="font-family: garamond, 'new york', times, serif; font-size: 12pt;"><div class="">Hi Geoffrey,<br class=""></div><div class=""><br class=""></div><div class="">1. Was self-heal also in progress while I/O was happening on the volume?<br class=""></div><div class="">2. Also, there seem to be quite a few fsyncs which could possibly have slowed things down a bit. Could you disable write-behind and try<br class=""></div><div class=""> getting the time stats one more time to eliminate the possibility of write-behind's presence causing out-of-order writes to increase the number of fsyncs<br class=""></div><div class=""> by the replication module.<br class=""></div><div class=""><br class=""></div><div class="">-Krutika<br class=""></div><hr id="zwchr" class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class="" data-mce-style="border-left-width: 2px; border-left-style: solid; border-left-color: #1010ff; margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;"><b class="">From: </b>"Geoffrey Letessier" <<a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank" class="" data-mce-href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a>><br class=""><b class="">To: </b><a href="mailto:gluster-users@gluster.org" target="_blank" class="" data-mce-href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br class=""><b class="">Sent: </b>Saturday, June 20, 2015 6:04:40 AM<br class=""><b class="">Subject: </b>Re: [Gluster-users] GlusterFS 3.5.3 - untar: very poor performance<br class=""><div class=""><br class=""></div>Re,<div class=""><br class=""></div><div class="">For comparison, here is the output of the same script run on a distributed only volume (2 servers of the 4 previously described, 2 bricks each):<div class=""><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">################ UNTAR time consumed ################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>1m44.698s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m8.891s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m8.353s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">################# DU time consumed ##################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">554M<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>linux-4.1-rc6</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m21.062s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m0.100s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m1.040s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">################# FIND time consumed ################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">52663</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m21.325s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m0.104s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m1.054s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">################# GREP time consumed ################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">7952</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m43.618s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m0.922s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m3.626s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">################# TAR time consumed #################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m50.577s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m29.745s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m4.086s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">################# RM time consumed ##################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m41.133s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m0.171s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m2.522s</span></div><div class=""><br class=""></div><div class="">The performances are amazing different!</div><div class=""><br class=""></div><div class="">Geoffrey</div><div class=""><div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="" data-mce-style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">-----------------------------------------------<br class="">Geoffrey Letessier<br class=""><div class=""><br class=""></div>Responsable informatique & ingénieur système<br class="">CNRS - UPR 9080 - Laboratoire de Biochimie Théorique<br class="">Institut de Biologie Physico-Chimique<br class="">13, rue Pierre et Marie Curie - 75005 Paris<br class="">Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank" class="" data-mce-href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a><br class=""></div></div><br class=""><div class=""><div class="">Le 20 juin 2015 à 02:12, Geoffrey Letessier <<a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank" class="" data-mce-href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a>> a écrit :</div><br class="Apple-interchange-newline"><blockquote class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="" data-mce-style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Dear all,<div class=""><br class=""></div><div class="">I just noticed on my main volume of my HPC cluster my IO operations become impressively poor.. </div><div class=""><br class=""></div><div class="">Doing some file operations above a linux kernel sources compressed file, the untar operation can take more than 1/2 hours for this file (roughly 80MB and 52 000 files inside) as you read below:</div><div class=""><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">################ UNTAR time consumed ################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>32m42.967s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m11.783s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m15.050s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">################# DU time consumed ##################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">557M<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>linux-4.1-rc6</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m25.060s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m0.068s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m0.344s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">################# FIND time consumed ################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">52663</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m25.687s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m0.084s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m0.387s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">################# GREP time consumed ################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">7952</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>2m15.890s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m0.887s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m2.777s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">################# TAR time consumed #################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>1m5.551s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m26.536s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m2.609s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">################# RM time consumed ##################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000; min-height: 11px;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"><br class=""></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>2m51.485s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m0.167s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>0m1.663s</span></div></div><div class=""><br class=""></div><div class="">For information, this volume is a distributed replicated one and is composed by 4 servers with 2 bricks each. Each bricks is a 12-drives RAID6 vdisk with nice native performances (around 1.2GBs).</div><div class=""><br class=""></div><div class="">In comparison, when I use DD to generate a 100GB file on the same volume, my write throughput is around 1GB (client side) and 500MBs (server side) because of replication:</div><div class="">Client side:</div><div class=""><div class=""><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">[root@node056 ~]# ifstat -i ib0</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"> ib0 </span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"> KB/s in KB/s out</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"> 3251.45 1.09e+06</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"> 3139.80 1.05e+06</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"> 3185.29 1.06e+06</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"> 3293.84 1.09e+06</span></div></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">...</span></div><div class=""><br class=""></div><div class="">Server side:</div><div class=""><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">[root@lucifer ~]# ifstat -i ib0</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"> ib0 </span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;"> KB/s in KB/s out</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">561818.1 1746.42</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">560020.3 1737.92</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">526337.1 1648.20</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">513972.7 1613.69</span></div></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">...</span></div><div class=""><br class="webkit-block-placeholder"></div><div class="">DD command:</div><div class=""><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">[root@node056 ~]# dd if=/dev/zero of=/home/root/test.dd bs=1M count=100000</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">100000+0 enregistrements lus</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">100000+0 enregistrements écrits</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; font-family: Menlo; color: #ffffff; background-color: #000000;"><span style="font-size: 9px;" class="" data-mce-style="font-size: 9px;">104857600000 octets (105 GB) copiés, 202,99 s, 517 MB/s</span></div></div><div class=""><br class=""></div><div class="">So this issue doesn’t seem coming from the network (which is Infiniband technology in this case)</div><div class=""><br class=""></div><div class="">You can find in attachments a set of files:</div><div class=""><span class="Apple-tab-span" style="white-space: pre;" data-mce-style="white-space: pre;"> </span>- mybench.sh: the bench script</div><div class=""><span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>- benches.txt: output of my "bench"</div><div class=""><span class="Apple-tab-span" style="white-space: pre;" data-mce-style="white-space: pre;"> </span>- profile.txt: gluster volume profile during the "bench"</div><div class=""><span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>- vol_status.txt: gluster volume status</div><div class=""><span class="Apple-tab-span" style="white-space:pre" data-mce-style="white-space: pre;"> </span>- vol_info.txt: gluster volume info</div><div class=""><br class=""></div><div class="">Can someone help me to fix it (it’s very critical because this volume is on a HPC cluster in production).</div><div class=""><br class=""></div><div class="">Thanks by advance,</div><div class="">Geoffrey</div><div class=""><div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="" data-mce-style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">-----------------------------------------------<br class="">Geoffrey Letessier<br class=""><div class=""><br class=""></div>Responsable informatique & ingénieur système<br class="">CNRS - UPR 9080 - Laboratoire de Biochimie Théorique<br class="">Institut de Biologie Physico-Chimique<br class="">13, rue Pierre et Marie Curie - 75005 Paris<br class="">Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank" class="" data-mce-href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a><br class=""></div></div></div></div><span class=""><benches.txt></span><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="" data-mce-style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class=""><br class=""></div></div><span class=""><mybench.sh></span><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="" data-mce-style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class=""><br class=""></div></div><span class=""><profile.txt></span><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="" data-mce-style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class=""><br class=""></div></div><span class=""><vol_info.txt></span><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="" data-mce-style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class=""><br class=""></div></div><span class=""><vol_status.txt></span><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="" data-mce-style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class=""><br class=""></div></div></blockquote></div><br class=""></div></div><br class="">_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="" target="_blank" data-mce-href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br class=""><a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank" class="" data-mce-href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><br class=""></blockquote><div class=""><br class=""></div></div></blockquote></div><br class=""></div></blockquote><div class=""><br class=""></div></div></div></div></blockquote></div></div><br class=""><div class=""><br class=""></div></blockquote><div class=""><br class=""></div></div></blockquote></div><br class=""></div></blockquote><div class=""><br class=""></div></div></blockquote></div><br class=""></div></blockquote><div class=""><br class=""></div></div></div></div></blockquote></div><br class=""></div></blockquote><div><br></div></div></blockquote></div><br></div></blockquote><div><br></div></div></blockquote></div><br></div></div></body></html>