<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div><br></div><div><br></div><div>I think this is the statesump of only one brick. </div><div>We would required statedump from all the bricks and client process in case of fuse or nfs process if it is mounted through nfs.<br></div><div><br></div><div>----<br></div><div>Ashish<br></div><div><br></div><hr id="zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Serkan Çoban" <cobanserkan@gmail.com><br><b>To: </b>"Ashish Pandey" <aspandey@redhat.com><br><b>Cc: </b>"Gluster Users" <gluster-users@gluster.org>, "Gluster Devel" <gluster-devel@gluster.org><br><b>Sent: </b>Friday, April 15, 2016 2:11:57 PM<br><b>Subject: </b>Re: [Gluster-users] Assertion failed: ec_get_inode_size<br><div><br></div>Sorry for typo, brick state dump file.<br><div><br></div>On Fri, Apr 15, 2016 at 11:41 AM, Serkan Çoban <cobanserkan@gmail.com> wrote:<br>> Hi I reproduce the problem, brick log file is in below link:<br>> https://www.dropbox.com/s/iy09j7mm2hrsf03/bricks-02.5677.dump.1460705370.gz?dl=0<br>><br>><br>> On Thu, Apr 14, 2016 at 8:07 PM, Ashish Pandey <aspandey@redhat.com> wrote:<br>>> Hi Serkan,<br>>><br>>> Could you also provide us the statedump of all the brick processes and<br>>> clients?<br>>><br>>> Commands to generate statedumps for brick processes/nfs server/quotad<br>>><br>>> For bricks: gluster volume statedump <volname><br>>><br>>> For nfs server: gluster volume statedump <volname> nfs<br>>><br>>><br>>> We can find the directory where statedump files are created using 'gluster<br>>> --print-statedumpdir'<br>>> Also, the mount logs would help us to debug the issue.<br>>><br>>> Ashish<br>>><br>>> ________________________________<br>>> From: "Serkan Çoban" <cobanserkan@gmail.com><br>>> To: "Gluster Users" <gluster-users@gluster.org>, "Gluster Devel"<br>>> <gluster-devel@gluster.org><br>>> Sent: Thursday, April 14, 2016 6:27:10 PM<br>>> Subject: Re: [Gluster-users] Assertion failed: ec_get_inode_size<br>>><br>>><br>>> Here is the related brick log:<br>>><br>>> /var/log/glusterfs/bricks/bricks-02.log:[2016-04-14 11:31:25.700556] E<br>>> [inodelk.c:309:__inode_unlock_lock] 0-v0-locks: Matching lock not<br>>> found for unlock 0-9223372036854775807, by 94d29e885e7f0000 on<br>>> 0x7f037413b990<br>>> /var/log/glusterfs/bricks/bricks-02.log:[2016-04-14 11:31:25.700639] E<br>>> [MSGID: 115053] [server-rpc-fops.c:276:server_inodelk_cbk]<br>>> 0-v0-server: 712984: INODELK<br>>> /workdir/raw_output/xxx/yyy/zzz.dat.gz.snappy1460474606605<br>>> (1191e32e-44ba-4e20-87ca-35ace8519c19) ==> (Invalid argument) [Invalid<br>>> argument]<br>>><br>>> On Thu, Apr 14, 2016 at 3:25 PM, Serkan Çoban <cobanserkan@gmail.com> wrote:<br>>>> Hi,<br>>>><br>>>> During read/write tests to a 78x(16+4) distributed disperse volume<br>>>> from 50 clients, One clients hangs on read/write with the following<br>>>> logs:<br>>>><br>>>> [2016-04-14 11:11:04.728580] W [MSGID: 122056]<br>>>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-6: Mismatching xdata<br>>>> in answers of 'LOOKUP'<br>>>> [2016-04-14 11:11:04.728624] W [MSGID: 122053]<br>>>> [ec-common.c:116:ec_check_status] 0-v0-disperse-6: Operation failed on<br>>>> some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=DFFFF,<br>>>> bad=20000)<br>>>> [2016-04-14 11:11:04.736689] I [MSGID: 122058]<br>>>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-6: /workdir/raw_output2:<br>>>> name heal successful on FFFFF<br>>>> [2016-04-14 11:29:26.718036] W [MSGID: 122056]<br>>>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-1: Mismatching xdata<br>>>> in answers of 'LOOKUP'<br>>>> [2016-04-14 11:29:26.718121] W [MSGID: 122053]<br>>>> [ec-common.c:116:ec_check_status] 0-v0-disperse-1: Operation failed on<br>>>> some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=EFFFF,<br>>>> bad=10000)<br>>>> [2016-04-14 11:29:42.501760] I [MSGID: 122058]<br>>>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-1: /workdir/raw_output2:<br>>>> name heal successful on FFFFF<br>>>> [2016-04-14 11:31:25.714812] E [ec-inode-read.c:1612:ec_manager_stat]<br>>>> (-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_resume+0x91)<br>>>> [0x7f5ec9f942b1]<br>>>><br>>>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(__ec_manager+0x57)<br>>>> [0x7f5ec9f94497]<br>>>><br>>>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_stat+0x2c4)<br>>>> [0x7f5ec9faaed4] ) 0-: Assertion failed: ec_get_inode_size(fop,<br>>>> fop->locks[0].lock->loc.inode, &cbk->iatt[0].ia_size)<br>>>> [2016-04-14 11:31:25.722372] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-40: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.722411] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-41: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.722450] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-44: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.722477] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-42: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.722503] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-43: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.722577] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-45: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.722605] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-46: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.722742] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-49: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.722794] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-47: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.722818] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-48: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.722840] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-50: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.722883] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-52: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.722906] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-54: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.722958] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-51: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.722983] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-55: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.723037] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-56: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.723045] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-53: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.725044] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-59: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.741338] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-58: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.746602] E [MSGID: 114031]<br>>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-57: remote<br>>>> operation failed [Invalid argument]<br>>>> [2016-04-14 11:31:25.746629] W [MSGID: 122015]<br>>>> [ec-common.c:1675:ec_unlocked] 0-v0-disperse-2: entry/inode unlocking<br>>>> failed (FSTAT) [Invalid argument]<br>>>> [2016-04-14 11:31:25.746687] E [ec-common.c:1639:ec_lock_unfreeze]<br>>>><br>>>> (-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_inodelk+0x2ae)<br>>>> [0x7f5ec9fa009e]<br>>>><br>>>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_unlocked+0x40)<br>>>> [0x7f5ec9f97fc0]<br>>>><br>>>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_lock_unfreeze+0x91)<br>>>> [0x7f5ec9f975d1] ) 0-: Assertion failed: lock->refs == lock->inserted<br>>>> [2016-04-14 11:31:25.746719] E [ec-common.c:1641:ec_lock_unfreeze]<br>>>><br>>>> (-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_inodelk+0x2ae)<br>>>> [0x7f5ec9fa009e]<br>>>><br>>>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_unlocked+0x40)<br>>>> [0x7f5ec9f97fc0]<br>>>><br>>>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_lock_unfreeze+0xf2)<br>>>> [0x7f5ec9f97632] ) 0-: Assertion failed: list_empty(&lock->waiting) &&<br>>>> list_empty(&lock->owners)<br>>>> [2016-04-14 11:40:54.139418] W [MSGID: 122056]<br>>>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-0: Mismatching xdata<br>>>> in answers of 'LOOKUP'<br>>>> [2016-04-14 11:40:54.139554] W [MSGID: 122053]<br>>>> [ec-common.c:116:ec_check_status] 0-v0-disperse-0: Operation failed on<br>>>> some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=2FFFF,<br>>>> bad=D0000)<br>>>> [2016-04-14 11:40:54.143400] I [MSGID: 122058]<br>>>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-0: /workdir/raw_output2:<br>>>> name heal successful on FFFFF<br>>>> The message "W [MSGID: 122056] [ec-combine.c:866:ec_combine_check]<br>>>> 0-v0-disperse-0: Mismatching xdata in answers of 'LOOKUP'" repeated 2<br>>>> times between [2016-04-14 11:40:54.139418] and [2016-04-14<br>>>> 11:40:54.139547]<br>>>> [2016-04-14 11:42:31.471317] W [MSGID: 122056]<br>>>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-45: Mismatching<br>>>> xdata in answers of 'LOOKUP'<br>>>> The message "W [MSGID: 122056] [ec-combine.c:866:ec_combine_check]<br>>>> 0-v0-disperse-45: Mismatching xdata in answers of 'LOOKUP'" repeated 8<br>>>> times between [2016-04-14 11:42:31.471317] and [2016-04-14<br>>>> 11:42:31.471557]<br>>>><br>>>> Gluster version is 3.7.10<br>>>> Any thoughts?<br>>>><br>>>> Serkan<br>>> _______________________________________________<br>>> Gluster-users mailing list<br>>> Gluster-users@gluster.org<br>>> http://www.gluster.org/mailman/listinfo/gluster-users<br>>><br>_______________________________________________<br>Gluster-users mailing list<br>Gluster-users@gluster.org<br>http://www.gluster.org/mailman/listinfo/gluster-users</div><div><br></div></div></body></html>