<html><body><div style="font-family: garamond,new york,times,serif; font-size: 12pt; color: #000000"><div>Hi Geoffrey,<br></div><div><br></div><div>1. Was self-heal also in progress while I/O was happening on the volume?<br></div><div>2. Also, there seem to be quite a few fsyncs which could possibly have slowed things down a bit. Could you disable write-behind and try<br></div><div> getting the time stats one more time to eliminate the possibility of write-behind's presence causing out-of-order writes to increase the number of fsyncs<br></div><div> by the replication module.<br></div><div><br></div><div>-Krutika<br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Geoffrey Letessier" <geoffrey.letessier@cnrs.fr><br><b>To: </b>gluster-users@gluster.org<br><b>Sent: </b>Saturday, June 20, 2015 6:04:40 AM<br><b>Subject: </b>Re: [Gluster-users] GlusterFS 3.5.3 - untar: very poor performance<br><div><br></div>Re,<div><br></div><div>For comparison, here is the output of the same script run on a distributed only volume (2 servers of the 4 previously described, 2 bricks each):<div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">################ UNTAR time consumed ################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre"> </span>1m44.698s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre"> </span>0m8.891s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m8.353s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">################# DU time consumed ##################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">554M<span class="Apple-tab-span" style="white-space:pre"> </span>linux-4.1-rc6</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre"> </span>0m21.062s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.100s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m1.040s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">################# FIND time consumed ################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">52663</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre"> </span>0m21.325s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.104s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m1.054s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">################# GREP time consumed ################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">7952</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre"> </span>0m43.618s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.922s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m3.626s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">################# TAR time consumed #################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre"> </span>0m50.577s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre"> </span>0m29.745s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m4.086s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">################# RM time consumed ##################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre"> </span>0m41.133s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.171s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m2.522s</span></div><div><br></div><div>The performances are amazing different!</div><div><br></div><div>Geoffrey</div><div>
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">-----------------------------------------------<br>Geoffrey Letessier<br><div><br></div>Responsable informatique & ingénieur système<br>CNRS - UPR 9080 - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank">geoffrey.letessier@cnrs.fr</a><br data-mce-bogus="1"></div>
</div>
<br><div><div>Le 20 juin 2015 à 02:12, Geoffrey Letessier <<a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank">geoffrey.letessier@cnrs.fr</a>> a écrit :</div><br class="Apple-interchange-newline"><blockquote><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Dear all,<div><br></div><div>I just noticed on my main volume of my HPC cluster my IO operations become impressively poor.. </div><div><br></div><div>Doing some file operations above a linux kernel sources compressed file, the untar operation can take more than 1/2 hours for this file (roughly 80MB and 52 000 files inside) as you read below:</div><div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">################ UNTAR time consumed ################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre"> </span>32m42.967s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre"> </span>0m11.783s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m15.050s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">################# DU time consumed ##################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">557M<span class="Apple-tab-span" style="white-space:pre"> </span>linux-4.1-rc6</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre"> </span>0m25.060s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.068s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.344s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">################# FIND time consumed ################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">52663</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre"> </span>0m25.687s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.084s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.387s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">################# GREP time consumed ################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">7952</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre"> </span>2m15.890s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.887s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m2.777s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">################# TAR time consumed #################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre"> </span>1m5.551s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre"> </span>0m26.536s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m2.609s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">################# RM time consumed ##################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">#######################################################</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); min-height: 11px;"><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">real<span class="Apple-tab-span" style="white-space:pre"> </span>2m51.485s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.167s</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m1.663s</span></div></div><div><br></div><div>For information, this volume is a distributed replicated one and is composed by 4 servers with 2 bricks each. Each bricks is a 12-drives RAID6 vdisk with nice native performances (around 1.2GBs).</div><div><br></div><div>In comparison, when I use DD to generate a 100GB file on the same volume, my write throughput is around 1GB (client side) and 500MBs (server side) because of replication:</div><div>Client side:</div><div><div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">[root@node056 ~]# ifstat -i ib0</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> ib0 </span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> KB/s in KB/s out</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 3251.45 1.09e+06</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 3139.80 1.05e+06</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 3185.29 1.06e+06</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 3293.84 1.09e+06</span></div></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">...</span></div><div><br></div><div>Server side:</div><div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">[root@lucifer ~]# ifstat -i ib0</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> ib0 </span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> KB/s in KB/s out</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">561818.1 1746.42</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">560020.3 1737.92</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">526337.1 1648.20</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">513972.7 1613.69</span></div></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">...</span></div><div><br class="webkit-block-placeholder"></div><div>DD command:</div><div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">[root@node056 ~]# dd if=/dev/zero of=/home/root/test.dd bs=1M count=100000</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">100000+0 enregistrements lus</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">100000+0 enregistrements écrits</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">104857600000 octets (105 GB) copiés, 202,99 s, 517 MB/s</span></div></div><div><br></div><div>So this issue doesn’t seem coming from the network (which is Infiniband technology in this case)</div><div><br></div><div>You can find in attachments a set of files:</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>- mybench.sh: the bench script</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- benches.txt: output of my "bench"</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>- profile.txt: gluster volume profile during the "bench"</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- vol_status.txt: gluster volume status</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- vol_info.txt: gluster volume info</div><div><br></div><div>Can someone help me to fix it (it’s very critical because this volume is on a HPC cluster in production).</div><div><br></div><div>Thanks by advance,</div><div>Geoffrey</div><div>
<div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">-----------------------------------------------<br>Geoffrey Letessier<br><div><br></div>Responsable informatique & ingénieur système<br>CNRS - UPR 9080 - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank">geoffrey.letessier@cnrs.fr</a><br data-mce-bogus="1"></div>
</div>
</div></div><span><benches.txt></span><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div></div></div><span><mybench.sh></span><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div></div></div><span><profile.txt></span><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div></div></div><span><vol_info.txt></span><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div></div></div><span><vol_status.txt></span><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div></div></div></blockquote></div><br></div></div><br>_______________________________________________<br>Gluster-users mailing list<br>Gluster-users@gluster.org<br>http://www.gluster.org/mailman/listinfo/gluster-users</blockquote><div><br></div></div></body></html>