<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div>Hi Vijay, </div><div><br></div><div>My brick logs issue and big performance problem have begun when I upgraded Gluster into 3.7.3 version; before write throughput was good enough (~500MBs) -but not as good as with GlusterFS 3.5.3 (especially with distributed volumes)- and didn’t notice these problème with brick-logs.</div><div><br></div><div>OK… in live:</div><div><br></div><div>i just disable to quota for my home volume and now my performance appears to be relatively better (around 300MBs) but i still see the logs (from storage1 and its replicate storage2) growing up with only this kind of lines:</div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); position: static; z-index: auto;"><span style="font-size: 9px;">[2015-08-07 11:16:51.746142] E [dict.c:1418:dict_copy_with_ref] (-->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(server_resolve_inode+0x60) [0x7f85e9a6a410] -->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(resolve_gfid+0x88) [0x7f85e9a6a188] -->/usr/lib64/libglusterfs.so.0(dict_copy_with_ref+0xa4) [0x3e99c20674] ) 0-dict: invalid argument: dict [Argument invalide]</span></div><div><br></div><div>After a few minutes: my write throughput seems to be now correct (~550MBs) but the log are still growing up (to not say exploding). So one part of the problem looks like taking its origin in the quota system management.</div><div>… after a few minutes (and still only 1 client connected), now it is the read operation which is very very slow… -I’m gonna become crazy! :/-</div><div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"># ddt -t 50g /home/</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Writing to /home/ddt.11293 ... syncing ... done.</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sleeping 10 seconds ... done.</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Reading from /home/ddt.11293 ... done.</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">35840MiB KiB/s CPU%</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Write 568201 5</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Read 567008 4</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"># ddt -t 50g /home/</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Writing to /home/ddt.11397 ... syncing ... done.</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sleeping 10 seconds ... done.</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Reading from /home/ddt.11397 ... done.</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">51200MiB KiB/s CPU%</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Write 573631 5</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Read 164716 1</span></div></div><div><br></div><div>and my log are still exploding…</div><div><br></div><div>After having re-enabled the quota on my volume: </div><div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"># ddt -t 50g /home/</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Writing to /home/ddt.11817 ... syncing ... done.</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sleeping 10 seconds ... done.</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Reading from /home/ddt.11817 ... done.</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">51200MiB KiB/s CPU%</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Write 269608 3</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Read 160219 1</span></div></div><div><br></div><div>Thanks </div><div>Geoffrey </div><div apple-content-edited="true">
------------------------------------------------------<br>Geoffrey Letessier<br>Responsable informatique & ingénieur système<br>UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr">geoffrey.letessier@ibpc.fr</a>
</div>
<br><div><div>Le 7 août 2015 à 06:28, Vijaikumar M <<a href="mailto:vmallika@redhat.com">vmallika@redhat.com</a>> a écrit :</div><br class="Apple-interchange-newline"><blockquote type="cite">
<meta content="text/html; charset=windows-1252" http-equiv="Content-Type">
<div bgcolor="#FFFFFF" text="#000000">
<tt>Hi Geoffrey,</tt><tt><br>
</tt><tt><br>
</tt><tt>Some performance improvements has been done in quota in
glusterfs-3.7.3.</tt><tt><br>
</tt><tt>Could you upgrade to glusterfs-3.7.3 and see if this helps</tt><tt><br>
</tt><tt><br>
</tt><tt>Thanks,</tt><tt><br>
</tt><tt>Vijay</tt><br>
<br>
<br>
<div class="moz-cite-prefix">On Friday 07 August 2015 05:02 AM,
Geoffrey Letessier wrote:<br>
</div>
<blockquote cite="mid:6BC6BC5D-39E2-4B3C-B076-E36428499679@cnrs.fr" type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
Hi,
<div><br>
</div>
<div>No idea to help me fix this issue? (big logs, small write
performance (/4), etc.)</div>
<div><br>
</div>
<div>For comparison, here to volumes: </div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>-
home: distributed on 4 bricks / 2 nodes (and replicated on 4
other bricks / 2 other nodes):</div>
<div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"># ddt -t 35g /home</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Writing to /home/ddt.24172 ...
syncing ... done.</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sleeping 10 seconds ... done.</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Reading from /home/ddt.24172 ...
done.</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">33792MiB KiB/s CPU%</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Write 103659 1</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Read 391955 3</span></div>
</div>
<div><br>
</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>-
workdir: distributed on 4 bricks / 2 nodes (one the same RAID
volumes and servers than home):</div>
<div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"># ddt -t 35g /workdir</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Writing to /workdir/ddt.24717 ...
syncing ... done.</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">sleeping 10 seconds ... done.</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Reading from /workdir/ddt.24717 ...
done.</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">35840MiB KiB/s CPU%</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Write 738314 4</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Read 536497 4</span></div>
</div>
<div><br>
</div>
<div>For information, previously on 3.5.3-2 version, I obtained
roughly 1.1GBs for workdir volume and ~550-600MBs for home.</div>
<div><br>
</div>
<div>All my tests (CP, RSYNC, etc.) provides me the same result
(write throughput between 100MBs and 150MBs)</div>
<div><br>
</div>
<div>Thanks.</div>
<div>Geoffrey<br>
<div apple-content-edited="true">
------------------------------------------------------<br>
Geoffrey Letessier<br>
Responsable informatique & ingénieur système<br>
UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br>
Institut de Biologie Physico-Chimique<br>
13, rue Pierre et Marie Curie - 75005 Paris<br>
Tel: 01 58 41 50 93 - eMail: <a moz-do-not-send="true" href="mailto:geoffrey.letessier@ibpc.fr">geoffrey.letessier@ibpc.fr</a>
</div>
<br>
<div>
<div>Le 5 août 2015 à 10:40, Geoffrey Letessier <<a moz-do-not-send="true" href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a>>
a écrit :</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;">Hello,
<div><br>
</div>
<div>In addition, knowing I have reactivated the log
(brick-log-level = INFO not CRITICAL) only for the file
creation duration (i.e. a few minutes), do you have
noticed the log sizes and the number of lines inside:</div>
<div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"># ls -lh storage*</span></div>
<div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">-rw------- 1 letessier
staff 18M 5 aoû 00:54
storage1__export-brick_home-brick1-data.log</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">-rw------- 1 letessier
staff 2,1K 5 aoû 00:54
storage1__export-brick_home-brick2-data.log</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">-rw------- 1 letessier
staff 15M 5 aoû 00:56
storage2__export-brick_home-brick1-data.log</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">-rw------- 1 letessier
staff 2,1K 5 aoû 00:54
storage2__export-brick_home-brick2-data.log</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">-rw------- 1 letessier
staff 47M 5 aoû 00:55
storage3__export-brick_home-brick1-data.log</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">-rw------- 1 letessier
staff 2,1K 5 aoû 00:54
storage3__export-brick_home-brick2-data.log</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">-rw------- 1 letessier
staff 47M 5 aoû 00:55
storage4__export-brick_home-brick1-data.log</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">-rw------- 1 letessier
staff 2,1K 5 aoû 00:55
storage4__export-brick_home-brick2-data.log</span></div>
</div>
<div><span style="font-size: 9px;"><br>
</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"># wc -l storage*</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 55381
storage1__export-brick_home-brick1-data.log</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 17
storage1__export-brick_home-brick2-data.log</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 41636
storage2__export-brick_home-brick1-data.log</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 17
storage2__export-brick_home-brick2-data.log</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 270360
storage3__export-brick_home-brick1-data.log</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 17
storage3__export-brick_home-brick2-data.log</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 270358
storage4__export-brick_home-brick1-data.log</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 17
storage4__export-brick_home-brick2-data.log</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;"> 637803 total</span></div>
<div><br>
</div>
<div>If the let brick-log-level to INFO, the brick log
files in each server will consume all my /var
partition capacity within only a few hours/days…</div>
<div><br>
</div>
<div>Thanks in advance,</div>
<div>Geoffrey</div>
<div apple-content-edited="true">
------------------------------------------------------<br>
Geoffrey Letessier<br>
Responsable informatique & ingénieur système<br>
UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br>
Institut de Biologie Physico-Chimique<br>
13, rue Pierre et Marie Curie - 75005 Paris<br>
Tel: 01 58 41 50 93 - eMail: <a moz-do-not-send="true" href="mailto:geoffrey.letessier@ibpc.fr">geoffrey.letessier@ibpc.fr</a>
</div>
<br>
<div>
<div>Le 5 août 2015 à 01:12, Geoffrey Letessier <<a moz-do-not-send="true" href="mailto:geoffrey.letessier@cnrs.fr">geoffrey.letessier@cnrs.fr</a>>
a écrit :</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space; -webkit-line-break:
after-white-space;">Hello,
<div><br>
</div>
<div>Since the problem motioned previously (all
errors noticed in brick log files), i notice a
very very bad performance: i can note my write
performance divided by 4 than previously
-knowing it was not so good before.</div>
<div>Now, a write of a 33GB file, my write
throughput is around 150MBs (with Infiniband),
before it was around 550-600MBs; and this, both
with RDMA and TCP protocol.</div>
<div><br>
</div>
<div>During this test, more than 40 000 error
lines (as the following) were added to the brick
log files.</div>
<div>
<div style="margin: 0px; font-size: 11px;
font-family: Menlo; color: rgb(255, 255, 255);
background-color: rgb(0, 0, 0); position:
static; z-index: auto;">[2015-08-04
22:34:27.337622] E
[dict.c:1418:dict_copy_with_ref]
(-->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(server_resolve_inode+0x60)
[0x7f021c6f7410]
-->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(resolve_gfid+0x88)
[0x7f021c6f7188]
-->/usr/lib64/libglusterfs.so.0(dict_copy_with_ref+0xa4)
[0x7f0229cba674] ) 0-dict: invalid argument:
dict [Argument invalide]</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>All brick log files are in attachments.</div>
<div><br>
</div>
<div>Thanks in advance for all your help and fix,</div>
<div>Best,</div>
<div>Geoffrey</div>
<div><br>
</div>
<div>PS: question: is it possible to easily
downgrade GlusterFS to a previous version from
3.7 (for example: v3.5)?</div>
<div><br>
</div>
<div>
<div apple-content-edited="true">
------------------------------------------------------<br>
Geoffrey Letessier<br>
Responsable informatique &
ingénieur système<br>
UPR 9080 - CNRS - Laboratoire de Biochimie
Théorique<br>
Institut de Biologie Physico-Chimique<br>
13, rue Pierre et Marie Curie - 75005 Paris<br>
Tel: 01 58 41 50 93 - eMail: <a moz-do-not-send="true" href="mailto:geoffrey.letessier@ibpc.fr">geoffrey.letessier@ibpc.fr</a>
</div>
</div>
</div>
<span><bricks-logs.tgz></span>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
</blockquote></div><br></body></html>