<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Dear all,<div><br></div><div>I have a crash test cluster where i’ve tested the new version of GlusterFS (v3.7) before upgrading my HPC cluster in production. </div><div>But… all my tests show me very very low performances.</div><div><br></div><div>For my benches, as you can read below, I do some actions (untar, du, find, tar, rm) with linux kernel sources, dropping cache, each on distributed, replicated, distributed-replicated, single (single brick) volumes and the native FS of one brick.</div><div><br></div><div><div># time (echo 3 > /proc/sys/vm/drop_caches; tar xJf ~/linux-4.1-rc5.tar.xz; sync; echo 3 > /proc/sys/vm/drop_caches)</div><div># time (echo 3 > /proc/sys/vm/drop_caches; du -sh linux-4.1-rc5/; echo 3 > /proc/sys/vm/drop_caches)</div><div># time (echo 3 > /proc/sys/vm/drop_caches; find linux-4.1-rc5/|wc -l; echo 3 > /proc/sys/vm/drop_caches)</div><div># time (echo 3 > /proc/sys/vm/drop_caches; tar czf linux-4.1-rc5.tgz linux-4.1-rc5/; echo 3 > /proc/sys/vm/drop_caches)</div><div># time (echo 3 > /proc/sys/vm/drop_caches; rm -rf linux-4.1-rc5.tgz linux-4.1-rc5/; echo 3 > /proc/sys/vm/drop_caches)</div><div><br></div><div>And here are the process times:</div><div><br></div><div><div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">---------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| | UNTAR | DU | FIND | TAR | RM |</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">---------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| single | ~3m45s | ~43s | ~47s | ~3m10s | ~3m15s |</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">---------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| replicated | ~5m10s | ~59s | ~1m6s | ~1m19s | ~1m49s |</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">---------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| distributed | ~4m18s | ~41s | ~57s | ~2m24s | ~1m38s |</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">---------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| dist-repl | ~8m18s | ~1m4s | ~1m11s | ~1m24s | ~2m40s |</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">---------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| native FS | ~11s | ~4s | ~2s | ~56s | ~10s |</div><div style="margin: 0px; font-size: 10px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">---------------------------------------------------------------</div></div></div><div><br></div><div>I get the same results, whether with default configurations with custom configurations.</div><div><br></div><div>if I look at the side of the ifstat command, I can note my IO write processes never exceed 3MBs...</div><div><br></div><div>EXT4 native FS seems to be faster (roughly 15-20% but no more) than XFS one</div><div><br></div><div>My [test] storage cluster config is composed by 2 identical servers (biCPU Intel Xeon X5355, 8GB of RAM, 2x2TB HDD (no-RAID) and Gb ethernet)</div><div><br></div><div>My volume settings:</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>single: 1server 1 brick</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>replicated: 2 servers 1 brick each</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>distributed: 2 servers 2 bricks each</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>dist-repl: 2 bricks in the same server and replica 2</div><div><br></div><div>All seems to be OK in gluster status command line.</div><div><br></div><div>Do you have an idea why I obtain so bad results?</div><div>Thanks in advance.</div><div>Geoffrey</div><div>-----------------------------------------------</div><div apple-content-edited="true"><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Geoffrey Letessier<br><br>Responsable informatique & ingénieur système<br>CNRS - UPR 9080 - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a></div>
</div>
<br></div></body></html>