<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Hi,<div><br></div><div>No idea to help me fix this issue? (big logs, small write performance (/4), etc.)</div><div><br></div><div>For comparison, here to volumes: </div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- home: distributed on 4 bricks / 2 nodes (and replicated on 4 other bricks / 2 other nodes):</div><div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"># ddt -t 35g /home</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Writing to /home/ddt.24172 ... syncing ... done.</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sleeping 10 seconds ... done.</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Reading from /home/ddt.24172 ... done.</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">33792MiB KiB/s CPU%</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Write 103659 1</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Read 391955 3</span></div></div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- workdir: distributed on 4 bricks / 2 nodes (one the same RAID volumes and servers than home):</div><div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"># ddt -t 35g /workdir</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Writing to /workdir/ddt.24717 ... syncing ... done.</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sleeping 10 seconds ... done.</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Reading from /workdir/ddt.24717 ... done.</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">35840MiB KiB/s CPU%</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Write 738314 4</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Read 536497 4</span></div></div><div><br></div><div>For information, previously on 3.5.3-2 version, I obtained roughly 1.1GBs for workdir volume and ~550-600MBs for home.</div><div><br></div><div>All my tests (CP, RSYNC, etc.) provides me the same result (write throughput between 100MBs and 150MBs)</div><div><br></div><div>Thanks.</div><div>Geoffrey<br><div apple-content-edited="true">
------------------------------------------------------<br>Geoffrey Letessier<br>Responsable informatique & ingénieur système<br>UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr">geoffrey.letessier@ibpc.fr</a>
</div>
<br><div><div>Le 5 août 2015 à 10:40, Geoffrey Letessier <<a href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a>> a écrit :</div><br class="Apple-interchange-newline"><blockquote type="cite"><meta http-equiv="Content-Type" content="text/html charset=windows-1252"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Hello,<div><br></div><div>In addition, knowing I have reactivated the log (brick-log-level = INFO not CRITICAL) only for the file creation duration (i.e. a few minutes), do you have noticed the log sizes and the number of lines inside:</div><div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"># ls -lh storage*</span></div><div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">-rw------- 1 letessier staff 18M 5 aoû 00:54 storage1__export-brick_home-brick1-data.log</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">-rw------- 1 letessier staff 2,1K 5 aoû 00:54 storage1__export-brick_home-brick2-data.log</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">-rw------- 1 letessier staff 15M 5 aoû 00:56 storage2__export-brick_home-brick1-data.log</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">-rw------- 1 letessier staff 2,1K 5 aoû 00:54 storage2__export-brick_home-brick2-data.log</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">-rw------- 1 letessier staff 47M 5 aoû 00:55 storage3__export-brick_home-brick1-data.log</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">-rw------- 1 letessier staff 2,1K 5 aoû 00:54 storage3__export-brick_home-brick2-data.log</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">-rw------- 1 letessier staff 47M 5 aoû 00:55 storage4__export-brick_home-brick1-data.log</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">-rw------- 1 letessier staff 2,1K 5 aoû 00:55 storage4__export-brick_home-brick2-data.log</span></div></div><div><span style="font-size: 9px;"><br></span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"># wc -l storage*</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 55381 storage1__export-brick_home-brick1-data.log</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 17 storage1__export-brick_home-brick2-data.log</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 41636 storage2__export-brick_home-brick1-data.log</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 17 storage2__export-brick_home-brick2-data.log</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 270360 storage3__export-brick_home-brick1-data.log</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 17 storage3__export-brick_home-brick2-data.log</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 270358 storage4__export-brick_home-brick1-data.log</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 17 storage4__export-brick_home-brick2-data.log</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 637803 total</span></div><div><br></div><div>If the let brick-log-level to INFO, the brick log files in each server will consume all my /var partition capacity within only a few hours/days…</div><div><br></div><div>Thanks in advance,</div><div>Geoffrey</div><div apple-content-edited="true">
------------------------------------------------------<br>Geoffrey Letessier<br>Responsable informatique & ingénieur système<br>UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr">geoffrey.letessier@ibpc.fr</a>
</div>
<br><div><div>Le 5 août 2015 à 01:12, Geoffrey Letessier <<a href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a>> a écrit :</div><br class="Apple-interchange-newline"><blockquote type="cite"><meta http-equiv="Content-Type" content="text/html charset=windows-1252"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Hello,<div><br></div><div>Since the problem motioned previously (all errors noticed in brick log files), i notice a very very bad performance: i can note my write performance divided by 4 than previously -knowing it was not so good before.</div><div>Now, a write of a 33GB file, my write throughput is around 150MBs (with Infiniband), before it was around 550-600MBs; and this, both with RDMA and TCP protocol.</div><div><br></div><div>During this test, more than 40 000 error lines (as the following) were added to the brick log files.</div><div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); position: static; z-index: auto;">[2015-08-04 22:34:27.337622] E [dict.c:1418:dict_copy_with_ref] (-->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(server_resolve_inode+0x60) [0x7f021c6f7410] -->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(resolve_gfid+0x88) [0x7f021c6f7188] -->/usr/lib64/libglusterfs.so.0(dict_copy_with_ref+0xa4) [0x7f0229cba674] ) 0-dict: invalid argument: dict [Argument invalide]</div></div><div><br></div><div><br></div><div>All brick log files are in attachments.</div><div><br></div><div>Thanks in advance for all your help and fix,</div><div>Best,</div><div>Geoffrey</div><div><br></div><div>PS: question: is it possible to easily downgrade GlusterFS to a previous version from 3.7 (for example: v3.5)?</div><div><br></div><div><div apple-content-edited="true">
------------------------------------------------------<br>Geoffrey Letessier<br>Responsable informatique & ingénieur système<br>UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr">geoffrey.letessier@ibpc.fr</a>
</div>
</div></div><span><bricks-logs.tgz></span><meta http-equiv="Content-Type" content="text/html charset=windows-1252"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div></div></div></blockquote></div><br></div></div></blockquote></div><br></div></body></html>