<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div>Hi Chen,<br></div><div><br></div><div>What do you mean by "instantly get inode locked and teared down <br>the whole cluster" ? Do you mean that whole disperse volume became unresponsive?<br></div><div><br></div><div>I don't have much idea about features.lock-heal so can't comment how can it help you.<br></div><div><br></div><div>Could you please explain second part of your mail? What exactly are you trying to do and what is the setup?</div><div> Also volume info, logs statedumps might help.<br></div><div><br></div><div>-----<br></div><div>Ashish <br></div><div><br></div><div><br></div><hr id="zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Chen Chen" <chenchen@smartquerier.com><br><b>To: </b>"Ashish Pandey" <aspandey@redhat.com><br><b>Cc: </b>gluster-users@gluster.org<br><b>Sent: </b>Wednesday, April 13, 2016 3:26:53 PM<br><b>Subject: </b>Re: [Gluster-users] Need some help on Mismatching xdata / Failed combine iatt / Too many fd<br><div><br></div>Hi Ashish and other Gluster Users,<br><div><br></div>When I put some heavy IO load onto my cluster (a rsync operation, <br>~600MB/s), one of the node instantly get inode locked and teared down <br>the whole cluster. I've already turned on "features.lock-heal" but it <br>didn't help.<br><div><br></div>My clients is using a round-robin tactic to mount servers, hoping to <br>average the pressure. Could it be caused by a race between NFS servers <br>on different nodes? Should I instead create a dedicated NFS Server with <br>huge memory, no brick, and multiple Ethernet cables?<br><div><br></div>I really appreciate any help from you guys.<br><div><br></div>Best wishes,<br>Chen<br><div><br></div>PS. Don't know why the native fuse client is 5 times inferior than the <br>old good NFSv3.<br><div><br></div>On 4/4/2016 6:11 PM, Ashish Pandey wrote:<br>> Hi Chen,<br>><br>> As I suspected, there are many blocked call for inodelk in sm11/mnt-disk1-mainvol.31115.dump.1459760675.<br>><br>> =============================================<br>> [xlator.features.locks.mainvol-locks.inode]<br>> path=/home/analyzer/softs/bin/GenomeAnalysisTK.jar<br>> mandatory=0<br>> inodelk-count=4<br>> lock-dump.domain.domain=mainvol-disperse-0:self-heal<br>> lock-dump.domain.domain=mainvol-disperse-0<br>> inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 1, owner=dc2d3dfcc57f0000, client=0x7ff03435d5f0, connection-id=sm12-8063-2016/04/01-07:51:46:892384-mainvol-client-0-0-0, blocked at 2016-04-01 16:52:58, granted at 2016-04-01 16:52:58<br>> inodelk.inodelk[1](BLOCKED)=type=WRITE, whence=0, start=0, len=0, pid = 1, owner=1414371e1a7f0000, client=0x7ff034204490, connection-id=hw10-17315-2016/04/01-07:51:44:421807-mainvol-client-0-0-0, blocked at 2016-04-01 16:58:51<br>> inodelk.inodelk[2](BLOCKED)=type=WRITE, whence=0, start=0, len=0, pid = 1, owner=a8eb14cd9b7f0000, client=0x7ff01400dbd0, connection-id=sm14-879-2016/04/01-07:51:56:133106-mainvol-client-0-0-0, blocked at 2016-04-01 17:03:41<br>> inodelk.inodelk[3](BLOCKED)=type=WRITE, whence=0, start=0, len=0, pid = 1, owner=b41a0482867f0000, client=0x7ff01800e670, connection-id=sm15-30906-2016/04/01-07:51:45:711474-mainvol-client-0-0-0, blocked at 2016-04-01 17:05:09<br>> =============================================<br>><br>> This could be the cause of hang.<br>> Possible Workaround -<br>> If there is no IO going on for this volume, we can restart the volume using - gluster v start <volume-name> force. This will restart the nfs process too which will release the locks and<br>> we could come out of this issue.<br>><br>> Ashish<br><div><br></div>-- <br>Chen Chen<br>Shanghai SmartQuerier Biotechnology Co., Ltd.<br>Add: Add: 3F, 1278 Keyuan Road, Shanghai 201203, P. R. China<br>Mob: +86 15221885893<br>Email: chenchen@smartquerier.com<br>Web: www.smartquerier.com<br><div><br></div><br>_______________________________________________<br>Gluster-users mailing list<br>Gluster-users@gluster.org<br>http://www.gluster.org/mailman/listinfo/gluster-users</div><div><br></div></div></body></html>