<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div>Hi,</div><div><br></div><div>Since i've upgraded GlusterFS from 3.5.3 to 3.7.x, trying to solve my quota miscalculation and poor performances (as advised by the user support team), we are still out-of-production for roughly 7 weeks because of a lot of v3.7.x issues we meet:</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- T-files apparition. I notice a lot of T files (with permissions --- --- --- T) located in my brick paths. Vijay has explained me T-files appear when a re-name is performed or when an add/remove brick is performed; but the problem is, since I've completely re-created (with RAID initialization, etc.) and import my data into the new volume, i've renamed nothing and never add nor delete any brick. </div><div>So, why these T-files are present in my new volume??? For example, for my /derreumaux_team directory, I have 13891 real files and 704 T-files totalized in the brick paths…</div><div>How to clean it, avoiding side effets?</div><div><br></div><div>The first time I noticed this kind of files, it was after having set a quota under the real path size which has resulted in some quota explosions (quota daemon failure) and T-files apparitions...</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- 7006 files in split-brain status after having back transfert data (30TB, 6.2M files) from a backup server into my just created volume. Thanks to Mathieu Chateau who help me putting me on road (GFID vs real file path), this problem has been manually fixed.</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- log issue. After having created only one file (35GB), I can notice more than 186000 new lines in brick log files. I can stop them setting brick-log-level to CRITICAL but I guess this issue gravely impact the IO performances and throughput. Vijay told me having fixed this problem in the code but I apparently need to wait the new release to take advantage of… Very nice for the production!</div><div><br></div><div>Actually, if I dont set brick-log-level to CRITICAL, i can fill my /var partition (10GB) in less than 1 day making some tests/benchs in the volume… </div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- volume healing issue: slightly less than 14000 files was in a bad situation (# gluster volume heal <vol_home> info) and a new forced heal in my volume make no change. Thanks to Krutika and Pranith, this is problem is currently fixed.</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- du/df/stat/etc. hangs cause of RDMA protocol. This problem seems to not occur anymore since I’ve upgraded my GlusterFS v3.7.2 to v3.7.3. This was probably due to the brick crashes (after a few minutes or a few days after having [re]start the volume) with RDMA transport-type we had. I noticed it only with v3.7.2 version.</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- quota problem: after having forced (with success) the quota re-calculation (with a simple DU for each defined quotas), after a couple of days with good values, the quota daemon failed again (some quota explosions, etc.)</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- a lot of warnings in TAR operations on replicated volumes: </div><div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">tar: linux-4.1-rc6/sound/soc/codecs/wm8962.c : fichier modifié pendant sa lecture</div></div><div><br></div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- low I/O performances and throughput:</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>1- if I enable to quota feature, my IO throughput is divided by 2. So, for the moment, i disabled this feature… (only since I’ve upgraded GlusterFS into 3.7.x version)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>2- since I’ve upgraded GlusterFS from 3.5.3 to 3.7.3, my I/O performance and throughput is lower than before, as you can read below. (keeping in mind i’ve disable quota feature)</div><div><br></div><div><div>IO operation tests with a Linux kernel archive (80MB tar ball file, ~53000 files, 550MB uncompressed):</div><div></div></div><div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| PRODUCTION HARDWARE |</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| | UNTAR | DU | FIND | GREP | TAR | RM |</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; background-color: rgb(0, 0, 0);"><font color="#f5ec00" class="">| native FS | ~16s | ~0.1s | ~0.1s | ~0.1s | ~24s | ~3s |</font></div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| GlusterFS version 3.5.3<span class="Apple-tab-span" style="white-space:pre"> </span> <span class="Apple-tab-span" style="white-space:pre"> </span> |</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| distributed | ~2m57s | ~23s | ~22s | ~49s | ~50s | ~54s |</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| dist-repl | ~29m56s | ~1m5s | ~1m04s | ~1m32s | ~1m31s | ~2m40s |</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div></div><div><div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| GlusterFS version 3.7.3<span class="Apple-tab-span" style="white-space: pre;"> </span> <span class="Apple-tab-span" style="white-space: pre;"> </span> |</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| distributed | ~2m49s | ~20s | ~29s | ~58s | ~60s | ~41s |</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| dist-repl | ~28m24s | ~51s | ~37s | ~1m16s | ~1m14s | ~1m17s |</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div></div></div><div>*:</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- distributed: 4 bricks (2 bricks on 2 servers)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- dist-repl: 4 bricks (2 bricks on 2 servers) for each replicas, 2 replicas.</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- native FS: each brick path (XFS)</div><div><br></div><div>And the craziest thing is I did the same test on a crashtest storage cluster (2 old Dell servers, all brick are single 2TB hard drive 7.2k, 2 bricks for each server) and the performance exceeds the production hardware performance (4 recent servers, 2 bricks each, each brick is 24TB RAID6 with good LSI RAID controllers (1 controller for 1 brick):</div><div><span style="background-color: rgb(0, 0, 0); color: rgb(255, 255, 255); font-family: Menlo; font-size: 11px;">------------------------------------------------------------------------</span></div><div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| CRASHTEST HARDWARE |</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| | UNTAR | DU | FIND | GREP | TAR | RM |</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; background-color: rgb(0, 0, 0);"><font color="#f5ec00" class="">| native FS | ~19s | ~0.2s | ~0.1s | ~1.2s | ~29s | ~2s |</font></div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| single | ~3m45s | ~43s | ~47s | | ~3m10s | ~3m15s |</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| single v2* | ~3m24s | ~13s | ~33s | ~1m10s | ~46s | ~48s |</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| single NFS | ~23m51s | ~3s | ~1s | ~27s | ~36s | ~13s |</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><div class="" style="margin: 0px;">------------------------------------------------------------------------</div></div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| replicated | ~5m10s | ~59s | ~1m6s | | ~1m19s | ~1m49s |</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| distributed | ~4m18s | ~41s | ~57s | | ~2m24s | ~1m38s |</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">| dist-repl | ~7m1s | ~19s | ~31s | ~1m34s | ~1m26s | ~2m11s |</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; background-color: rgb(0, 0, 0);"><span style="color: rgb(255, 255, 255);">| FhGFS(dist) | ~3m33s | ~15s | ~2s | ~1m31s | ~1m31s | ~52s |</span></div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">------------------------------------------------------------------------</div><div class="" style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">*: with default parameters</div></div><div><br></div><div><br></div>Concerning the throughput (for both writes and reads operations), in the production hardware, it was around 600MBs (dist-repl volume) and 1.1GBs (distributed volume) with GlusterFS version 3.5.3 with TCP network transport-type (RDMA never worked in my storage cluster before 3.7.x version of GlusterFS).<div>Now, it is around 500-600MBs with RDMA and 150-300MBs with TCP (for dist-repl volume), and around 600-700MBs with RDMA and 500-600 with TCP for distributed volume.</div><div><br></div><div>Could you help to back into production our HPC center, solving above-mentioned issues? Or do you advise me to downgrade into v3.5.3 (the more stable version I’d known since I use GlusterFS in production)? or move on ?;-)</div><div><br></div><div>Thanks in advance.</div><div>Geoffrey<br><div apple-content-edited="true">
------------------------------------------------------<br>Geoffrey Letessier<br>Responsable informatique & ingénieur système<br>UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr">geoffrey.letessier@ibpc.fr</a>
</div>
<br></div></body></html>