<div dir="ltr">Will look into the issue from RDMA side and let you know if we find something.<div><br></div><div>A few more details from your setup are required.</div><div><br></div><div>1. ls -l on all the brick mounts.</div><div>2. Did you change transport of the volume with the volume on?</div><div><br></div><div>Raghavendra Talur</div><div><br></div><div>On Sun, Aug 9, 2015 at 2:23 PM, Geoffrey Letessier <span dir="ltr"><<a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank">geoffrey.letessier@cnrs.fr</a>></span> wrote:<br></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Hi Mathieu,<div><br></div><div>First of all, thanks for replying.</div><div><br></div><div>I’ve done your proposal but there’s no change: my brick logs are still growing up in the server where the file is written, with this kind of lines:</div><div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[...]</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[2015-08-09 08:16:57.856987] W [marker-quota.c:3379:_mq_initiate_quota_txn] 0-vol_home-marker: parent is NULL for <gfid:64c302ab-2171-4656-8e5f-47e474de80b6>, aborting updation txn</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[2015-08-09 08:16:57.857691] W [marker-quota.c:3379:_mq_initiate_quota_txn] 0-vol_home-marker: parent is NULL for <gfid:64c302ab-2171-4656-8e5f-47e474de80b6>, aborting updation txn</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[2015-08-09 08:16:57.858403] W [marker-quota.c:3379:_mq_initiate_quota_txn] 0-vol_home-marker: parent is NULL for <gfid:64c302ab-2171-4656-8e5f-47e474de80b6>, aborting updation txn</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[2015-08-09 08:16:57.859226] W [marker-quota.c:3379:_mq_initiate_quota_txn] 0-vol_home-marker: parent is NULL for <gfid:64c302ab-2171-4656-8e5f-47e474de80b6>, aborting updation txn</span></div><div style="margin:0px;background-color:rgb(0,0,0)"><font color="#ffffff" face="Menlo"><span style="font-size:9px">[2015-08-09 08:16:57.859982] W [marker-quota.c:3379:_mq_initiate_quota_txn] 0-vol_home-marker: parent is NULL for <gfid:64c302ab-2171-4656-8e5f-47e474de80b6>, aborting updating txn</span></font><span style="color:rgb(255,255,255);font-family:Menlo;font-size:9px;white-space:pre-wrap"> </span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">The message "W [MSGID: 113001] [posix.c:3700:posix_get_ancestry_non_directory] 0-vol_home-posix: listxattr failed on/export/brick_home/brick2/data/.glusterfs/64/c3/64c302ab-2171-4656-8e5f-47e474de80b6 [Aucun fichier ou dossier de ce type]" repeated 149711 times between [2015-08-09 08:15:17.811919] and [2015-08-09 08:16:57.859754]</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[2015-08-09 08:16:59.629692] W [MSGID: 113001] [posix.c:3700:posix_get_ancestry_non_directory] 0-vol_home-posix: listxattr failed </span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">on/export/brick_home/brick2/data/.glusterfs/64/c3/64c302ab-2171-4656-8e5f-47e474de80b6 [Aucun fichier ou dossier de ce type]</span></div></div><div style="margin:0px;background-color:rgb(0,0,0)"><font color="#ffffff" face="Menlo"><span style="font-size:9px">[...]</span></font></div><div><br></div><div>and here the ddt output:</div><div><span class=""><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"># ddt -t 35g /home/</span></div></span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Writing to /home/ddt.12247 ... syncing ... done.</span></div><span class=""><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sleeping 10 seconds ... done.</span></div></span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Reading from /home/ddt.12247 ... done.</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">35840MiB KiB/s CPU%</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Write 184737 2</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Read 484209 3</span></div></div><div><br></div><div>For just a write of only one 35GB file (with a blank log files before) :</div><div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"># grep "parent is NULL" /var/log/glusterfs/bricks/export-brick_home-brick2-data.log|wc -l</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">286720</span></div></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"><br></span></div><div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"># grep "xattr" /var/log/glusterfs/bricks/export-brick_home-brick2-data.log|wc -l</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">5</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"><br></span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"># wc -l /var/log/glusterfs/bricks/export-brick_home-brick2-data.log</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">286733 /var/log/glusterfs/bricks/export-brick_home-brick2-data.log</span></div></div><div><br></div><div>and the other kind of lines in the brick log file:</div><div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"># grep -vE "(xattr|parent is NULL)" /var/log/glusterfs/bricks/export-brick_home-brick2-data.log</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[2015-08-09 08:13:16.368705] I [MSGID: 115034] [server.c:397:_check_for_auth_option] 0-/export/brick_home/brick2/data: skip format check for non-addr auth option auth.login./export/brick_home/brick2/data.allow</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[2015-08-09 08:13:16.368858] I [MSGID: 115034] [server.c:397:_check_for_auth_option] 0-/export/brick_home/brick2/data: skip format check for non-addr auth option auth.login.dffafb7e-3ff2-4e91-b30b-eb87c6cfe621.password</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[2015-08-09 08:13:16.368953] E [MSGID: 115041] [server.c:833:reconfigure] 0-vol_home-server: Reconfigure not found for transport</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[2015-08-09 08:13:16.377119] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[2015-08-09 08:13:16.393164] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[2015-08-09 08:13:16.402136] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[2015-08-09 08:13:16.410998] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[2015-08-09 08:22:16.000685] E [MSGID: 113104] [posix-handle.c:154:posix_make_ancestryfromgfid] 0-vol_home-posix: could not read the link from the gfid handle /export/brick_home/brick2/data/.glusterfs/b3/7a/b37a7750-f250-4ab4-8b29-bba519b6dc69 [Aucun fichier ou dossier de ce type]</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[2015-08-09 08:32:17.000668] E [MSGID: 113104] [posix-handle.c:154:posix_make_ancestryfromgfid] 0-vol_home-posix: could not read the link from the gfid handle /export/brick_home/brick2/data/.glusterfs/b3/7a/b37a7750-f250-4ab4-8b29-bba519b6dc69 [Aucun fichier ou dossier de ce type]</span></div></div><div><br></div><div>No change in logs if i run the command with a simple user but a slightly better performance for write but a slightly lower performance for read:</div><div><span class=""><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">$ ddt -t 35g /home/admin_team/letessier/</span></div></span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Writing to /home/admin_team/letessier/ddt.12489 ... syncing ... done.</span></div><span class=""><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sleeping 10 seconds ... done.</span></div></span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Reading from /home/admin_team/letessier/ddt.12489 ... done.</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">35840MiB KiB/s CPU%</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Write 280981 3</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Read 313502 2</span></div></div><div><div><br></div><div>Any other idea?</div></div><div><br></div><div>Frankly, I'm very frustrated for having stopped our scientific computing production more than six weeks ago and, due to cascading issues in GlusterFS, to not be able to restart it for the moment and…</div><div><br></div><div>Thanks again,</div><div>Geoffrey</div><div><span class=""><div>
------------------------------------------------------<br>Geoffrey Letessier<br>Responsable informatique & ingénieur système<br>UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr" target="_blank">geoffrey.letessier@ibpc.fr</a>
</div>
<br></span><div><div class="h5"><div><div>Le 8 août 2015 à 10:02, Mathieu Chateau <<a href="mailto:mathieu.chateau@lotp.fr" target="_blank">mathieu.chateau@lotp.fr</a>> a écrit :</div><br><blockquote type="cite"><div dir="ltr">Maybe related to the insecure port issue reported ?<div><br></div><div>try with :<br><div><br><div>gluster volume set xxx server.allow-insecure on<br></div></div></div></div><div class="gmail_extra"><br clear="all"><div><div>Cordialement,<br>Mathieu CHATEAU<br><a href="http://www.lotp.fr/" target="_blank">http://www.lotp.fr</a></div></div>
<br><div class="gmail_quote">2015-08-07 23:47 GMT+02:00 Geoffrey Letessier <span dir="ltr"><<a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank">geoffrey.letessier@cnrs.fr</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>I’m not really sure to well understand your answer.</div><div><br></div><div>I try to set inode-lru-limit to 1, I can not notice any good effect. </div><div><br></div><div>When i re-run ddt application, I can note 2 kinds of messages:</div><div><span style="background-color:rgb(0,0,0);color:rgb(255,255,255);font-family:Menlo;font-size:9px">[2015-08-07 21:29:21.792156] W [marker-quota.c:3379:_mq_initiate_quota_txn] 0-vol_home-marker: parent is NULL for <gfid:5a32328a-7fd9-474e-9bc6-cafde9c41af7>, aborting updation txn</span></div><div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[2015-08-07 21:29:21.792176] W [marker-quota.c:3379:_mq_initiate_quota_txn] 0-vol_home-marker: parent is NULL for <gfid:5a32328a-7fd9-474e-9bc6-cafde9c41af7>, aborting updation txn</span></div></div><div><br></div><div>and/or:</div><div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[2015-08-07 21:44:19.279971] E [marker-quota.c:2990:mq_start_quota_txn_v2] 0-vol_home-marker: contribution node list is empty (31d7bf88-b63a-4731-a737-a3dce73b8cd1)</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[2015-08-07 21:41:26.177095] E [dict.c:1418:dict_copy_with_ref] (-->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(server_resolve_inode+0x60) [0x7f85e9a6a410] -->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(resolve_gfid+0x88) [0x7f85e9a6a188] -->/usr/lib64/libglusterfs.so.0(dict_copy_with_ref+0xa4) [0x3e99c20674] ) 0-dict: invalid argument: dict [Argument invalide]</span></div></div><div><br></div><div>And concerning the bad IO performance?</div><div><br></div><div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[letessier@node031 ~]$ ddt -t 35g /home/admin_team/letessier/</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Writing to /home/admin_team/letessier/ddt.25259 ... syncing ... done.</span></div><span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sleeping 10 seconds ... done.</span></div></span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Reading from /home/admin_team/letessier/ddt.25259 ... done.</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">35840MiB KiB/s CPU%</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Write 277451 3</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Read 188682 1</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[letessier@node031 ~]$ logout</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[root@node031 ~]# ddt -t 35g /home/</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Writing to /home/ddt.25559 ... syncing ... done.</span></div><span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sleeping 10 seconds ... done.</span></div></span><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Reading from /home/ddt.25559 ... done.</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">35840MiB KiB/s CPU%</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Write 196539 2</span></div><div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Read 438944 3</span></div></div><div>Notice the read/write throughput differences when i’m root and when i’m a simple user.</div><div><br></div><div>Thanks.</div><div><span>Geoffrey<br><div>
------------------------------------------------------<br>Geoffrey Letessier<br>Responsable informatique & ingénieur système<br>UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br>Institut de Biologie Physico-Chimique<br>13, rue Pierre et Marie Curie - 75005 Paris<br>Tel: <a href="tel:01%2058%2041%2050%2093" value="+33158415093" target="_blank">01 58 41 50 93</a> - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr" target="_blank">geoffrey.letessier@ibpc.fr</a>
</div>
<br></span><div><div><div><div>Le 7 août 2015 à 14:57, Vijaikumar M <<a href="mailto:vmallika@redhat.com" target="_blank">vmallika@redhat.com</a>> a écrit :</div><br><blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000">
<br>
<br>
<div>On Friday 07 August 2015 05:34 PM,
Geoffrey Letessier wrote:<br>
</div>
<blockquote type="cite">
<div>Hi Vijay, </div>
<div><br>
</div>
<div>My brick logs issue and big performance problem have begun
when I upgraded Gluster into 3.7.3 version; before write
throughput was good enough (~500MBs) -but not as good as with
GlusterFS 3.5.3 (especially with distributed volumes)- and
didn’t notice these problème with brick-logs.</div>
<div><br>
</div>
<div>OK… in live:</div>
<div><br>
</div>
<div>i just disable to quota for my home volume and now my
performance appears to be relatively better (around 300MBs) but
i still see the logs (from storage1 and its replicate storage2)
growing up with only this kind of lines:</div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">[2015-08-07
11:16:51.746142] E [dict.c:1418:dict_copy_with_ref]
(-->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(server_resolve_inode+0x60)
[0x7f85e9a6a410]
-->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(resolve_gfid+0x88)
[0x7f85e9a6a188]
-->/usr/lib64/libglusterfs.so.0(dict_copy_with_ref+0xa4)
[0x3e99c20674] ) 0-dict: invalid argument: dict [Argument
invalide]</span></div>
<div><br>
</div>
</blockquote>
<tt>We have root caused log issue, bug# 1244613 tracks this issue</tt><tt><br>
</tt><br>
<br>
<blockquote type="cite">
<div>After a few minutes: my write throughput seems to be now
correct (~550MBs) but the log are still growing up (to not say
exploding). So one part of the problem looks like taking its
origin in the quota system management.</div>
<div>… after a few minutes (and still only 1 client connected),
now it is the read operation which is very very slow… -I’m gonna
become crazy! :/-</div>
<div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"># ddt -t 50g /home/</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Writing to /home/ddt.11293 ...
syncing ... done.</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sleeping 10 seconds ... done.</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Reading from /home/ddt.11293 ...
done.</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">35840MiB KiB/s CPU%</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Write 568201 5</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Read 567008 4</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"># ddt -t 50g /home/</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Writing to /home/ddt.11397 ...
syncing ... done.</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sleeping 10 seconds ... done.</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Reading from /home/ddt.11397 ...
done.</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">51200MiB KiB/s CPU%</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Write 573631 5</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Read 164716 1</span></div>
</div>
<div><br>
</div>
<div>and my log are still exploding…</div>
<div><br>
</div>
<div>After having re-enabled the quota on my volume: </div>
<div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"># ddt -t 50g /home/</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Writing to /home/ddt.11817 ...
syncing ... done.</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sleeping 10 seconds ... done.</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Reading from /home/ddt.11817 ...
done.</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">51200MiB KiB/s CPU%</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Write 269608 3</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Read 160219 1</span></div>
</div>
<div><br>
</div>
<div>Thanks </div>
<div>Geoffrey </div>
<div>
------------------------------------------------------<br>
Geoffrey Letessier<br>
Responsable informatique & ingénieur système<br>
UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br>
Institut de Biologie Physico-Chimique<br>
13, rue Pierre et Marie Curie - 75005 Paris<br>
Tel: <a href="tel:01%2058%2041%2050%2093" value="+33158415093" target="_blank">01 58 41 50 93</a> - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr" target="_blank">geoffrey.letessier@ibpc.fr</a>
</div>
<br>
<div>
<div>Le 7 août 2015 à 06:28, Vijaikumar M <<a href="mailto:vmallika@redhat.com" target="_blank">vmallika@redhat.com</a>>
a écrit :</div>
<br>
<blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000"> <tt>Hi Geoffrey,</tt><tt><br>
</tt><tt><br>
</tt><tt>Some performance improvements has been done in
quota in glusterfs-3.7.3.</tt><tt><br>
</tt><tt>Could you upgrade to glusterfs-3.7.3 and see if
this helps</tt><tt><br>
</tt><tt><br>
</tt><tt>Thanks,</tt><tt><br>
</tt><tt>Vijay</tt><br>
<br>
<br>
<div>On Friday 07 August 2015 05:02
AM, Geoffrey Letessier wrote:<br>
</div>
<blockquote type="cite">
Hi,
<div><br>
</div>
<div>No idea to help me fix this issue? (big logs, small
write performance (/4), etc.)</div>
<div><br>
</div>
<div>For comparison, here to volumes: </div>
<div><span style="white-space:pre-wrap">
</span>- home: distributed on 4 bricks / 2 nodes (and
replicated on 4 other bricks / 2 other nodes):</div>
<div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"># ddt -t 35g /home</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Writing to /home/ddt.24172
... syncing ... done.</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sleeping 10 seconds ...
done.</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Reading from /home/ddt.24172
... done.</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">33792MiB KiB/s CPU%</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Write 103659 1</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Read 391955 3</span></div>
</div>
<div><br>
</div>
<div><span style="white-space:pre-wrap">
</span>- workdir: distributed on 4 bricks / 2 nodes (one
the same RAID volumes and servers than home):</div>
<div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"># ddt -t 35g /workdir</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Writing to
/workdir/ddt.24717 ... syncing ... done.</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">sleeping 10 seconds ...
done.</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Reading from
/workdir/ddt.24717 ... done.</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">35840MiB KiB/s CPU%</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Write 738314 4</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">Read 536497 4</span></div>
</div>
<div><br>
</div>
<div>For information, previously on 3.5.3-2 version, I
obtained roughly 1.1GBs for workdir volume and
~550-600MBs for home.</div>
<div><br>
</div>
<div>All my tests (CP, RSYNC, etc.) provides me the same
result (write throughput between 100MBs and 150MBs)</div>
<div><br>
</div>
<div>Thanks.</div>
<div>Geoffrey<br>
<div>
------------------------------------------------------<br>
Geoffrey Letessier<br>
Responsable informatique & ingénieur système<br>
UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br>
Institut de Biologie Physico-Chimique<br>
13, rue Pierre et Marie Curie - 75005 Paris<br>
Tel: <a href="tel:01%2058%2041%2050%2093" value="+33158415093" target="_blank">01 58 41 50 93</a> - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr" target="_blank">geoffrey.letessier@ibpc.fr</a>
</div>
<br>
<div>
<div>Le 5 août 2015 à 10:40, Geoffrey Letessier <<a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank">geoffrey.letessier@cnrs.fr</a>>
a écrit :</div>
<br>
<blockquote type="cite">
<div style="word-wrap:break-word">Hello,
<div><br>
</div>
<div>In addition, knowing I have reactivated the
log (brick-log-level = INFO not CRITICAL) only
for the file creation duration (i.e. a few
minutes), do you have noticed the log sizes and
the number of lines inside:</div>
<div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#
ls -lh storage*</span></div>
<div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">-rw-------
1 letessier staff 18M 5 aoû 00:54
storage1__export-brick_home-brick1-data.log</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">-rw-------
1 letessier staff 2,1K 5 aoû 00:54
storage1__export-brick_home-brick2-data.log</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">-rw-------
1 letessier staff 15M 5 aoû 00:56
storage2__export-brick_home-brick1-data.log</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">-rw-------
1 letessier staff 2,1K 5 aoû 00:54
storage2__export-brick_home-brick2-data.log</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">-rw-------
1 letessier staff 47M 5 aoû 00:55
storage3__export-brick_home-brick1-data.log</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">-rw-------
1 letessier staff 2,1K 5 aoû 00:54
storage3__export-brick_home-brick2-data.log</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">-rw-------
1 letessier staff 47M 5 aoû 00:55
storage4__export-brick_home-brick1-data.log</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">-rw-------
1 letessier staff 2,1K 5 aoû 00:55
storage4__export-brick_home-brick2-data.log</span></div>
</div>
<div><span style="font-size:9px"><br>
</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">#
wc -l storage*</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> 55381
storage1__export-brick_home-brick1-data.log</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">
17
storage1__export-brick_home-brick2-data.log</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> 41636
storage2__export-brick_home-brick1-data.log</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">
17
storage2__export-brick_home-brick2-data.log</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> 270360
storage3__export-brick_home-brick1-data.log</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">
17
storage3__export-brick_home-brick2-data.log</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> 270358
storage4__export-brick_home-brick1-data.log</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px">
17
storage4__export-brick_home-brick2-data.log</span></div>
<div style="margin:0px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)"><span style="font-size:9px"> 637803
total</span></div>
<div><br>
</div>
<div>If the let brick-log-level to INFO, the
brick log files in each server will consume
all my /var partition capacity within only a
few hours/days…</div>
<div><br>
</div>
<div>Thanks in advance,</div>
<div>Geoffrey</div>
<div>
------------------------------------------------------<br>
Geoffrey Letessier<br>
Responsable informatique &
ingénieur système<br>
UPR 9080 - CNRS - Laboratoire de Biochimie
Théorique<br>
Institut de Biologie Physico-Chimique<br>
13, rue Pierre et Marie Curie - 75005 Paris<br>
Tel: <a href="tel:01%2058%2041%2050%2093" value="+33158415093" target="_blank">01 58 41 50 93</a> - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr" target="_blank">geoffrey.letessier@ibpc.fr</a>
</div>
<br>
<div>
<div>Le 5 août 2015 à 01:12, Geoffrey
Letessier <<a href="mailto:geoffrey.letessier@cnrs.fr" target="_blank">geoffrey.letessier@cnrs.fr</a>>
a écrit :</div>
<br>
<blockquote type="cite">
<div style="word-wrap:break-word">Hello,
<div><br>
</div>
<div>Since the problem motioned previously
(all errors noticed in brick log files),
i notice a very very bad performance: i
can note my write performance divided by
4 than previously -knowing it was not so
good before.</div>
<div>Now, a write of a 33GB file, my write
throughput is around 150MBs (with
Infiniband), before it was around
550-600MBs; and this, both with RDMA and
TCP protocol.</div>
<div><br>
</div>
<div>During this test, more than 40 000
error lines (as the following) were
added to the brick log files.</div>
<div>
<div style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(255,255,255);background-color:rgb(0,0,0)">[2015-08-04
22:34:27.337622] E
[dict.c:1418:dict_copy_with_ref]
(-->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(server_resolve_inode+0x60)
[0x7f021c6f7410]
-->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(resolve_gfid+0x88)
[0x7f021c6f7188]
-->/usr/lib64/libglusterfs.so.0(dict_copy_with_ref+0xa4)
[0x7f0229cba674] ) 0-dict: invalid
argument: dict [Argument invalide]</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>All brick log files are in
attachments.</div>
<div><br>
</div>
<div>Thanks in advance for all your help
and fix,</div>
<div>Best,</div>
<div>Geoffrey</div>
<div><br>
</div>
<div>PS: question: is it possible to
easily downgrade GlusterFS to a previous
version from 3.7 (for example: v3.5)?</div>
<div><br>
</div>
<div>
<div>
------------------------------------------------------<br>
Geoffrey Letessier<br>
Responsable informatique &
ingénieur système<br>
UPR 9080 - CNRS - Laboratoire
de Biochimie Théorique<br>
Institut de Biologie Physico-Chimique<br>
13, rue Pierre et Marie Curie -
75005 Paris<br>
Tel: <a href="tel:01%2058%2041%2050%2093" value="+33158415093" target="_blank">01 58 41 50 93</a> - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr" target="_blank">geoffrey.letessier@ibpc.fr</a>
</div>
</div>
</div>
<span><bricks-logs.tgz></span>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</blockquote>
<br>
</div>
</blockquote></div><br></div></div></div></div><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div><br></div></div></div></div><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div></div>