<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<br>
<div class="moz-cite-prefix">On 04/13/2016 03:29 AM, Ashish Pandey
wrote:<br>
</div>
<blockquote
cite="mid:1494436596.44109296.1460543353059.JavaMail.zimbra@redhat.com"
type="cite">
<div style="font-family: times new roman, new york, times, serif;
font-size: 12pt; color: #000000">
<div>Hi Chen,<br>
</div>
<div><br>
</div>
<div>What do you mean by "instantly get inode locked and teared
down <br>
the whole cluster" ? Do you mean that whole disperse volume
became unresponsive?<br>
</div>
<div><br>
</div>
<div>I don't have much idea about features.lock-heal so can't
comment how can it help you.<br>
</div>
</div>
</blockquote>
<br>
So who should get added to this email that would have an idea? Let's
get that person looped in.<br>
<br>
<blockquote
cite="mid:1494436596.44109296.1460543353059.JavaMail.zimbra@redhat.com"
type="cite">
<div style="font-family: times new roman, new york, times, serif;
font-size: 12pt; color: #000000">
<div><br>
</div>
<div>Could you please explain second part of your mail? What
exactly are you trying to do and what is the setup?</div>
<div> Also volume info, logs statedumps might help.<br>
</div>
<div><br>
</div>
<div>-----<br>
</div>
<div>Ashish <br>
</div>
<div><br>
</div>
<div><br>
</div>
<hr id="zwchr">
<div
style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"
data-mce-style="color: #000; font-weight: normal; font-style:
normal; text-decoration: none; font-family:
Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Chen
Chen" <a class="moz-txt-link-rfc2396E" href="mailto:chenchen@smartquerier.com"><chenchen@smartquerier.com></a><br>
<b>To: </b>"Ashish Pandey" <a class="moz-txt-link-rfc2396E" href="mailto:aspandey@redhat.com"><aspandey@redhat.com></a><br>
<b>Cc: </b><a class="moz-txt-link-abbreviated" href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
<b>Sent: </b>Wednesday, April 13, 2016 3:26:53 PM<br>
<b>Subject: </b>Re: [Gluster-users] Need some help on
Mismatching xdata / Failed combine iatt / Too many fd<br>
<div><br>
</div>
Hi Ashish and other Gluster Users,<br>
<div><br>
</div>
When I put some heavy IO load onto my cluster (a rsync
operation, <br>
~600MB/s), one of the node instantly get inode locked and
teared down <br>
the whole cluster. I've already turned on "features.lock-heal"
but it <br>
didn't help.<br>
<div><br>
</div>
My clients is using a round-robin tactic to mount servers,
hoping to <br>
average the pressure. Could it be caused by a race between NFS
servers <br>
on different nodes? Should I instead create a dedicated NFS
Server with <br>
huge memory, no brick, and multiple Ethernet cables?<br>
<div><br>
</div>
I really appreciate any help from you guys.<br>
<div><br>
</div>
Best wishes,<br>
Chen<br>
<div><br>
</div>
PS. Don't know why the native fuse client is 5 times inferior
than the <br>
old good NFSv3.<br>
<div><br>
</div>
On 4/4/2016 6:11 PM, Ashish Pandey wrote:<br>
> Hi Chen,<br>
><br>
> As I suspected, there are many blocked call for inodelk
in sm11/mnt-disk1-mainvol.31115.dump.1459760675.<br>
><br>
> =============================================<br>
> [xlator.features.locks.mainvol-locks.inode]<br>
> path=/home/analyzer/softs/bin/GenomeAnalysisTK.jar<br>
> mandatory=0<br>
> inodelk-count=4<br>
> lock-dump.domain.domain=mainvol-disperse-0:self-heal<br>
> lock-dump.domain.domain=mainvol-disperse-0<br>
> inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0,
len=0, pid = 1, owner=dc2d3dfcc57f0000, client=0x7ff03435d5f0,
connection-id=sm12-8063-2016/04/01-07:51:46:892384-mainvol-client-0-0-0,
blocked at 2016-04-01 16:52:58, granted at 2016-04-01 16:52:58<br>
> inodelk.inodelk[1](BLOCKED)=type=WRITE, whence=0,
start=0, len=0, pid = 1, owner=1414371e1a7f0000,
client=0x7ff034204490,
connection-id=hw10-17315-2016/04/01-07:51:44:421807-mainvol-client-0-0-0,
blocked at 2016-04-01 16:58:51<br>
> inodelk.inodelk[2](BLOCKED)=type=WRITE, whence=0,
start=0, len=0, pid = 1, owner=a8eb14cd9b7f0000,
client=0x7ff01400dbd0,
connection-id=sm14-879-2016/04/01-07:51:56:133106-mainvol-client-0-0-0,
blocked at 2016-04-01 17:03:41<br>
> inodelk.inodelk[3](BLOCKED)=type=WRITE, whence=0,
start=0, len=0, pid = 1, owner=b41a0482867f0000,
client=0x7ff01800e670,
connection-id=sm15-30906-2016/04/01-07:51:45:711474-mainvol-client-0-0-0,
blocked at 2016-04-01 17:05:09<br>
> =============================================<br>
><br>
> This could be the cause of hang.<br>
> Possible Workaround -<br>
> If there is no IO going on for this volume, we can
restart the volume using - gluster v start <volume-name>
force. This will restart the nfs process too which will
release the locks and<br>
> we could come out of this issue.<br>
><br>
> Ashish<br>
<div><br>
</div>
-- <br>
Chen Chen<br>
Shanghai SmartQuerier Biotechnology Co., Ltd.<br>
Add: Add: 3F, 1278 Keyuan Road, Shanghai 201203, P. R. China<br>
Mob: +86 15221885893<br>
Email: <a class="moz-txt-link-abbreviated" href="mailto:chenchen@smartquerier.com">chenchen@smartquerier.com</a><br>
Web: <a class="moz-txt-link-abbreviated" href="http://www.smartquerier.com">www.smartquerier.com</a><br>
<div><br>
</div>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></div>
<div><br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>