<div dir="ltr">Hi, i have im having issues with gluster nfs, it keep crashing after few hours under medium load.<div><br></div><div>OS: CentOS 7.2</div><div>Gluster version 3.7.13</div><div><br></div><div>Gluster info;</div><div><div>Volume Name: vlm01</div><div>Type: Distributed-Replicate</div><div>Volume ID: eacd8248-dca3-4530-9aed-7714a5a114f2</div><div>Status: Started</div><div>Number of Bricks: 7 x 3 = 21</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: gfs01:/bricks/b01/vlm01</div><div>Brick2: gfs02:/bricks/b01/vlm01</div><div>Brick3: gfs03:/bricks/b01/vlm01</div><div>Brick4: gfs01:/bricks/b02/vlm01</div><div>Brick5: gfs02:/bricks/b02/vlm01</div><div>Brick6: gfs03:/bricks/b02/vlm01</div><div>Brick7: gfs01:/bricks/b03/vlm01</div><div>Brick8: gfs02:/bricks/b03/vlm01</div><div>Brick9: gfs03:/bricks/b03/vlm01</div><div>Brick10: gfs01:/bricks/b04/vlm01</div><div>Brick11: gfs02:/bricks/b04/vlm01</div><div>Brick12: gfs03:/bricks/b04/vlm01</div><div>Brick13: gfs01:/bricks/b05/vlm01</div><div>Brick14: gfs02:/bricks/b05/vlm01</div><div>Brick15: gfs03:/bricks/b05/vlm01</div><div>Brick16: gfs01:/bricks/b06/vlm01</div><div>Brick17: gfs02:/bricks/b06/vlm01</div><div>Brick18: gfs03:/bricks/b06/vlm01</div><div>Brick19: gfs01:/bricks/b07/vlm01</div><div>Brick20: gfs02:/bricks/b07/vlm01</div><div>Brick21: gfs03:/bricks/b07/vlm01</div><div>Options Reconfigured:</div><div>auth.allow: 192.168.221.50,192.168.221.51,192.168.221.52,192.168.221.56</div><div>features.shard: on</div><div>features.shard-block-size: 16MB</div><div>cluster.self-heal-window-size: 128</div><div>cluster.data-self-heal-algorithm: full</div><div>performance.write-behind: off</div><div>performance.strict-write-ordering: on</div><div>cluster.server-quorum-type: server</div><div>cluster.quorum-type: auto</div><div>network.remote-dio: enable</div><div>cluster.eager-lock: enable</div><div>performance.stat-prefetch: off</div><div>performance.io-cache: off</div><div>performance.read-ahead: off</div><div>performance.quick-read: off</div><div>performance.readdir-ahead: on</div><div>network.ping-timeout: 10</div></div><div>#####</div><div><br></div><div><br></div><div>Gluster status:</div><div><div>Status of volume: vlm01</div><div>Gluster process TCP Port RDMA Port Online Pid</div><div>------------------------------------------------------------------------------</div><div>Brick gfs01:/bricks/b01/vlm01 49159 0 Y 3050 </div><div>Brick gfs02:/bricks/b01/vlm01 49158 0 Y 3012 </div><div>Brick gfs03:/bricks/b01/vlm01 49158 0 Y 3889 </div><div>Brick gfs01:/bricks/b02/vlm01 49160 0 Y 3058 </div><div>Brick gfs02:/bricks/b02/vlm01 49159 0 Y 3011 </div><div>Brick gfs03:/bricks/b02/vlm01 49159 0 Y 3888 </div><div>Brick gfs01:/bricks/b03/vlm01 49161 0 Y 3067 </div><div>Brick gfs02:/bricks/b03/vlm01 49160 0 Y 3024 </div><div>Brick gfs03:/bricks/b03/vlm01 49160 0 Y 3899 </div><div>Brick gfs01:/bricks/b04/vlm01 49162 0 Y 3057 </div><div>Brick gfs02:/bricks/b04/vlm01 49161 0 Y 3035 </div><div>Brick gfs03:/bricks/b04/vlm01 49161 0 Y 3898 </div><div>Brick gfs01:/bricks/b05/vlm01 49163 0 Y 3075 </div><div>Brick gfs02:/bricks/b05/vlm01 49162 0 Y 3030 </div><div>Brick gfs03:/bricks/b05/vlm01 49162 0 Y 3914 </div><div>Brick gfs01:/bricks/b06/vlm01 49164 0 Y 3091 </div><div>Brick gfs02:/bricks/b06/vlm01 49163 0 Y 3048 </div><div>Brick gfs03:/bricks/b06/vlm01 49163 0 Y 3913 </div><div>Brick gfs01:/bricks/b07/vlm01 49165 0 Y 3080 </div><div>Brick gfs02:/bricks/b07/vlm01 49164 0 Y 3042 </div><div>Brick gfs03:/bricks/b07/vlm01 49164 0 Y 3908 </div><div>NFS Server on localhost 2049 0 Y 28926</div><div>Self-heal Daemon on localhost N/A N/A Y 28934</div><div>NFS Server on gfs02 2049 0 Y 9944 </div><div>Self-heal Daemon on gfs02 N/A N/A Y 9953 </div><div>NFS Server on gfs01 2049 0 Y 46993</div><div>Self-heal Daemon on gfs01 N/A N/A Y 47003</div><div> </div><div>Task Status of Volume vlm01</div><div>------------------------------------------------------------------------------</div><div>There are no active volume tasks</div></div><div>#####<br></div><div><br></div><div><br></div><div>dmesg;</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: frame : type(0) op(0)<br></div><div><div>Jul 23 09:53:07 gfs03 nfs[31243]: frame : type(0) op(0)</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: frame : type(0) op(0)</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: frame : type(0) op(0)</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: frame : type(0) op(0)</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: frame : type(0) op(0)</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: frame : type(0) op(0)</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: frame : type(0) op(0)</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: patchset: git://<a href="http://git.gluster.com/glusterfs.git">git.gluster.com/glusterfs.git</a></div><div>Jul 23 09:53:07 gfs03 nfs[31243]: signal received: 11</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: time of crash:</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: 2016-07-23 06:53:07</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: configuration details:</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: argp 1</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: backtrace 1</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: dlfcn 1</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: libpthread 1</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: llistxattr 1</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: setfsid 1</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: spinlock 1</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: epoll.h 1</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: xattr.h 1</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: st_atim.tv_nsec 1</div><div>Jul 23 09:53:07 gfs03 nfs[31243]: package-string: glusterfs 3.7.13</div><div>#####</div><div><br></div><div><br></div><div>nfs.log;</div><div><div><div>[2016-07-23 05:59:19.961654] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vlm01-client-18: Connected to vlm01-client-18, attached to remote volume '/bricks/b07/vlm01'.</div><div>[2016-07-23 05:59:19.961670] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vlm01-client-18: Server and Client lk-version numbers are not same, reopening the fds</div><div>[2016-07-23 05:59:19.961717] I [MSGID: 108005] [afr-common.c:4142:afr_notify] 0-vlm01-replicate-6: Subvolume 'vlm01-client-18' came back up; going online.</div><div>[2016-07-23 05:59:19.961854] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vlm01-client-18: Server lk version = 1</div><div>[2016-07-23 05:59:19.962027] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-vlm01-client-20: changing port to 49164 (from 0)</div><div>[2016-07-23 05:59:19.964637] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vlm01-client-19: Using Program GlusterFS 3.3, Num (1298437), Version (330)</div><div>[2016-07-23 05:59:19.965956] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vlm01-client-19: Connected to vlm01-client-19, attached to remote volume '/bricks/b07/vlm01'.</div><div>[2016-07-23 05:59:19.965989] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vlm01-client-19: Server and Client lk-version numbers are not same, reopening the fds</div><div>[2016-07-23 05:59:19.966140] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vlm01-client-19: Server lk version = 1</div><div>[2016-07-23 05:59:19.967605] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vlm01-client-20: Using Program GlusterFS 3.3, Num (1298437), Version (330)</div><div>[2016-07-23 05:59:19.967919] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vlm01-client-20: Connected to vlm01-client-20, attached to remote volume '/bricks/b07/vlm01'.</div><div>[2016-07-23 05:59:19.967943] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vlm01-client-20: Server and Client lk-version numbers are not same, reopening the fds</div><div>[2016-07-23 05:59:19.968107] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vlm01-client-20: Server lk version = 1</div><div>[2016-07-23 05:59:19.973053] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vlm01-client-17: Connected to vlm01-client-17, attached to remote volume '/bricks/b06/vlm01'.</div><div>[2016-07-23 05:59:19.973081] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vlm01-client-17: Server and Client lk-version numbers are not same, reopening the fds</div><div>[2016-07-23 05:59:19.973582] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vlm01-client-17: Server lk version = 1</div><div>[2016-07-23 05:59:19.974557] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-vlm01-replicate-0: selecting local read_child vlm01-client-2</div><div>[2016-07-23 05:59:19.976100] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-vlm01-replicate-1: selecting local read_child vlm01-client-5</div><div>[2016-07-23 05:59:19.976161] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-vlm01-replicate-2: selecting local read_child vlm01-client-8</div><div>[2016-07-23 05:59:19.976583] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-vlm01-replicate-3: selecting local read_child vlm01-client-11</div><div>[2016-07-23 05:59:19.976640] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-vlm01-replicate-4: selecting local read_child vlm01-client-14</div><div>[2016-07-23 05:59:19.976676] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-vlm01-replicate-5: selecting local read_child vlm01-client-17</div><div>[2016-07-23 05:59:19.976879] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-vlm01-replicate-6: selecting local read_child vlm01-client-20</div><div>[2016-07-23 05:59:36.360646] I [MSGID: 109066] [dht-rename.c:1568:dht_rename] 0-vlm01-dht: renaming <gfid:d61aff47-e715-489d-9908-f824789e27b3>/PRTG 15/PRTG 15.vmx~ (hash=vlm01-replicate-0/cache=vlm01-replicate-0) => <gfid:d61aff47-e715-489d-9908-f824789e27b3>/PRTG 15/PRTG 15.vmx (hash=vlm01-replicate-2/cache=vlm01-replicate-0)</div><div>[2016-07-23 05:59:36.962314] I [MSGID: 109066] [dht-rename.c:1568:dht_rename] 0-vlm01-dht: renaming <gfid:d61aff47-e715-489d-9908-f824789e27b3>/PRTG 15/PRTG 15_1.vmdk~ (hash=vlm01-replicate-6/cache=vlm01-replicate-6) => <gfid:d61aff47-e715-489d-9908-f824789e27b3>/PRTG 15/PRTG 15_1.vmdk (hash=vlm01-replicate-3/cache=vlm01-replicate-6)</div><div>[2016-07-23 05:59:37.019564] I [MSGID: 109066] [dht-rename.c:1568:dht_rename] 0-vlm01-dht: renaming <gfid:d61aff47-e715-489d-9908-f824789e27b3>/PRTG 40/PRTG 40.vmx~ (hash=vlm01-replicate-0/cache=vlm01-replicate-0) => <gfid:d61aff47-e715-489d-9908-f824789e27b3>/PRTG 40/PRTG 40.vmx (hash=vlm01-replicate-2/cache=vlm01-replicate-0)</div><div>The message "I [MSGID: 109066] [dht-rename.c:1568:dht_rename] 0-vlm01-dht: renaming <gfid:d61aff47-e715-489d-9908-f824789e27b3>/PRTG 40/PRTG 40.vmx~ (hash=vlm01-replicate-0/cache=vlm01-replicate-0) => <gfid:d61aff47-e715-489d-9908-f824789e27b3>/PRTG 40/PRTG 40.vmx (hash=vlm01-replicate-2/cache=vlm01-replicate-0)" repeated 2 times between [2016-07-23 05:59:37.019564] and [2016-07-23 05:59:37.421227]</div><div>[2016-07-23 05:59:38.757822] I [MSGID: 109066] [dht-rename.c:1568:dht_rename] 0-vlm01-dht: renaming <gfid:d61aff47-e715-489d-9908-f824789e27b3>/PRTG 15/PRTG 15.vmx~ (hash=vlm01-replicate-0/cache=vlm01-replicate-0) => <gfid:d61aff47-e715-489d-9908-f824789e27b3>/PRTG 15/PRTG 15.vmx (hash=vlm01-replicate-2/cache=vlm01-replicate-0)</div><div>[2016-07-23 05:59:39.950960] I [MSGID: 109066] [dht-rename.c:1568:dht_rename] 0-vlm01-dht: renaming <gfid:d61aff47-e715-489d-9908-f824789e27b3>/PRTG 40/PRTG 40.vmdk~ (hash=vlm01-replicate-5/cache=vlm01-replicate-5) => <gfid:d61aff47-e715-489d-9908-f824789e27b3>/PRTG 40/PRTG 40.vmdk (hash=vlm01-replicate-5/cache=vlm01-replicate-5)</div><div>[2016-07-23 06:00:03.048266] I [MSGID: 109066] [dht-rename.c:1568:dht_rename] 0-vlm01-dht: renaming <gfid:d61aff47-e715-489d-9908-f824789e27b3>/PRTG 15/PRTG 15.vmdk~ (hash=vlm01-replicate-2/cache=vlm01-replicate-2) => <gfid:d61aff47-e715-489d-9908-f824789e27b3>/PRTG 15/PRTG 15.vmdk (hash=vlm01-replicate-5/cache=vlm01-replicate-2)</div><div>[2016-07-23 06:00:07.994953] W [MSGID: 112199] [nfs3-helpers.c:3520:nfs3_log_newfh_res] 0-nfs-nfsv3: <gfid:d61aff47-e715-489d-9908-f824789e27b3>/PRTG 4/PRTG 4.vmx => (XID: 8439cb9c, LOOKUP: NFS: 70(Invalid file handle), POSIX: 116(Stale file handle)), FH: exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000, mountid 00000000-0000-0000-0000-000000000000</div><div>[2016-07-23 06:01:02.831132] E [MSGID: 112069] [nfs3.c:3483:nfs3_remove_resume] 0-nfs-nfsv3: No such file or directory: (<a href="http://192.168.208.85:676">192.168.208.85:676</a>) vlm01 : a0d6a061-866e-4b75-b3ab-4005e52ed364</div><div>[2016-07-23 06:16:48.221237] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-vlm01-client-12: remote operation failed. Path: <gfid:d61aff47-e715-489d-9908-f824789e27b3>/.lck-c7607978ef6c5b99 [File exists]</div><div>[2016-07-23 06:16:48.221231] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-vlm01-client-13: remote operation failed. Path: <gfid:d61aff47-e715-489d-9908-f824789e27b3>/.lck-c7607978ef6c5b99 [File exists]</div><div>[2016-07-23 06:16:48.221382] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-vlm01-client-14: remote operation failed. Path: <gfid:d61aff47-e715-489d-9908-f824789e27b3>/.lck-c7607978ef6c5b99 [File exists]</div><div>[2016-07-23 06:16:48.221878] W [MSGID: 112199] [nfs3-helpers.c:3520:nfs3_log_newfh_res] 0-nfs-nfsv3: <gfid:d61aff47-e715-489d-9908-f824789e27b3>/.lck-c7607978ef6c5b99 => (XID: 8441a50a, CREATE: NFS: 17(File exists), POSIX: 17(File exists)), FH: exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000, mountid 00000000-0000-0000-0000-000000000000</div><div>[2016-07-23 06:17:11.343148] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-vlm01-client-18: remote operation failed. Path: <gfid:d61aff47-e715-489d-9908-f824789e27b3>/.iorm.sf/.lck-aa088f801cb21f94 [File exists]</div><div>[2016-07-23 06:17:11.343170] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-vlm01-client-20: remote operation failed. Path: <gfid:d61aff47-e715-489d-9908-f824789e27b3>/.iorm.sf/.lck-aa088f801cb21f94 [File exists]</div><div>[2016-07-23 06:17:11.343234] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-vlm01-client-19: remote operation failed. Path: <gfid:d61aff47-e715-489d-9908-f824789e27b3>/.iorm.sf/.lck-aa088f801cb21f94 [File exists]</div><div>[2016-07-23 06:17:11.343596] W [MSGID: 112199] [nfs3-helpers.c:3520:nfs3_log_newfh_res] 0-nfs-nfsv3: <gfid:d61aff47-e715-489d-9908-f824789e27b3>/.iorm.sf/.lck-aa088f801cb21f94 => (XID: 51e43a2f, CREATE: NFS: 17(File exists), POSIX: 17(File exists)), FH: exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000, mountid 00000000-0000-0000-0000-000000000000 [Invalid argument]</div><div>[2016-07-23 06:17:21.393996] E [MSGID: 112069] [nfs3.c:3483:nfs3_remove_resume] 0-nfs-nfsv3: No such file or directory: (<a href="http://192.168.208.86:906">192.168.208.86:906</a>) vlm01 : a0d6a061-866e-4b75-b3ab-4005e52ed364</div><div>[2016-07-23 06:50:11.441462] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-vlm01-client-19: remote operation failed. Path: <gfid:d61aff47-e715-489d-9908-f824789e27b3>/.iorm.sf/.lck-7a24a36f7bded3b0 [File exists]</div><div>[2016-07-23 06:50:11.441471] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-vlm01-client-18: remote operation failed. Path: <gfid:d61aff47-e715-489d-9908-f824789e27b3>/.iorm.sf/.lck-7a24a36f7bded3b0 [File exists]</div><div>[2016-07-23 06:50:11.441530] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-vlm01-client-20: remote operation failed. Path: <gfid:d61aff47-e715-489d-9908-f824789e27b3>/.iorm.sf/.lck-7a24a36f7bded3b0 [File exists]</div><div>[2016-07-23 06:50:11.441959] W [MSGID: 112199] [nfs3-helpers.c:3520:nfs3_log_newfh_res] 0-nfs-nfsv3: <gfid:d61aff47-e715-489d-9908-f824789e27b3>/.iorm.sf/.lck-7a24a36f7bded3b0 => (XID: 51ea9a6e, CREATE: NFS: 17(File exists), POSIX: 17(File exists)), FH: exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000, mountid 00000000-0000-0000-0000-000000000000</div><div>[2016-07-23 06:50:21.712570] E [MSGID: 112069] [nfs3.c:3483:nfs3_remove_resume] 0-nfs-nfsv3: No such file or directory: (<a href="http://192.168.208.86:906">192.168.208.86:906</a>) vlm01 : a0d6a061-866e-4b75-b3ab-4005e52ed364</div></div><div>frame : type(0) op(0)<br></div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>frame : type(0) op(0)</div><div>patchset: git://<a href="http://git.gluster.com/glusterfs.git">git.gluster.com/glusterfs.git</a></div><div>signal received: 11</div><div>time of crash: </div><div>2016-07-23 06:53:07</div><div>configuration details:</div><div>argp 1</div><div>backtrace 1</div><div>dlfcn 1</div><div>libpthread 1</div><div>llistxattr 1</div><div>setfsid 1</div><div>spinlock 1</div><div>epoll.h 1</div><div>xattr.h 1</div><div>st_atim.tv_nsec 1</div><div>package-string: glusterfs 3.7.13</div><div>/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xc2)[0x7f74cbde32f2]</div><div>/lib64/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f74cbe08aad]</div><div>/lib64/libc.so.6(+0x35670)[0x7f74ca4cf670]</div><div>/lib64/libpthread.so.0(pthread_spin_lock+0x0)[0x7f74cac50210]</div></div><div><br></div><div><br></div><div>I appreciate your help guys.</div><div><br></div><div><div class="gmail_signature"><div dir="ltr"><font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br></font><br></div></div></div>
</div></div>