<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">try restarting glusterd.<div class=""># service glistered restart</div><div class=""><br class=""></div><div class="">if it didn’t work, try killing glusterfsd PID(s)</div><div class=""># kill $(ps -ef | grep glusterfsd | awk '{print $2}’)</div><div class="">t</div><div class="">hen, restart glusterd</div><div class=""># service glusterd restart</div><div class=""><br class=""></div><div class="">PS: killing glusterfsd that way will kill all the bricks on that node, but restarting glusterd should return the bricks back online.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><b class="">—Bishoy</b></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Apr 19, 2016, at 11:07 PM, Lindsay Mathieson <<a href="mailto:lindsay.mathieson@gmail.com" class="">lindsay.mathieson@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">A brick has died on node vnb of my cluster. Unfortnately it has left a<br class="">zombie glusterfsd process which is holding the brick socket so I can't<br class="">restart it. Any advice on how to work round that asap would be<br class="">appreciated.<br class=""><br class="">Tail of brick logging:<br class=""><br class="">2016-04-20 05:41:37.325846] I [dict.c:473:dict_get]<br class="">(-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_getxattr_cbk+0xab)<br class="">[0x7f5ff77d239b]<br class="">-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/features/marker.so(marker_getxattr_cbk+0xa7)<br class="">[0x7f5feb9c88e7]<br class="">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get+0x93)<br class="">[0x7f5ff77c30f3] ) 0-dict: !this || key=() [Invalid argument]<br class="">[2016-04-20 05:41:37.328255] I [dict.c:473:dict_get]<br class="">(-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_getxattr_cbk+0xab)<br class="">[0x7f5ff77d239b]<br class="">-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/features/marker.so(marker_getxattr_cbk+0xa7)<br class="">[0x7f5feb9c88e7]<br class="">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get+0x93)<br class="">[0x7f5ff77c30f3] ) 0-dict: !this || key=() [Invalid argument]<br class="">[2016-04-20 05:41:37.599402] I [dict.c:473:dict_get]<br class="">(-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_getxattr_cbk+0xab)<br class="">[0x7f5ff77d239b]<br class="">-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/features/marker.so(marker_getxattr_cbk+0xa7)<br class="">[0x7f5feb9c88e7]<br class="">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get+0x93)<br class="">[0x7f5ff77c30f3] ) 0-dict: !this || key=() [Invalid argument]<br class="">[2016-04-20 05:41:37.601843] I [dict.c:473:dict_get]<br class="">(-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_getxattr_cbk+0xab)<br class="">[0x7f5ff77d239b]<br class="">-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/features/marker.so(marker_getxattr_cbk+0xa7)<br class="">[0x7f5feb9c88e7]<br class="">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get+0x93)<br class="">[0x7f5ff77c30f3] ) 0-dict: !this || key=() [Invalid argument]<br class="">[2016-04-20 05:41:37.604164] I [dict.c:473:dict_get]<br class="">(-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_getxattr_cbk+0xab)<br class="">[0x7f5ff77d239b]<br class="">-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/features/marker.so(marker_getxattr_cbk+0xa7)<br class="">[0x7f5feb9c88e7]<br class="">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get+0x93)<br class="">[0x7f5ff77c30f3] ) 0-dict: !this || key=() [Invalid argument]<br class="">[2016-04-20 05:41:37.682886] I [dict.c:473:dict_get]<br class="">(-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_getxattr_cbk+0xab)<br class="">[0x7f5ff77d239b]<br class="">-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/features/marker.so(marker_getxattr_cbk+0xa7)<br class="">[0x7f5feb9c88e7]<br class="">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get+0x93)<br class="">[0x7f5ff77c30f3] ) 0-dict: !this || key=() [Invalid argument]<br class="">[2016-04-20 05:55:16.203806] W [glusterfsd.c:1251:cleanup_and_exit]<br class="">(-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4) [0x7f5ff6a4f0a4]<br class="">-->/usr/sbin/glusterfsd(glusterfs_sigwaiter+0xe5) [0x5629ffed26f5]<br class="">-->/usr/sbin/glusterfsd(cleanup_and_exit+0x59) [0x5629ffed2569] ) 0-:<br class="">received signum (15), shutting down<br class="">[2016-04-20 05:55:35.536514] I [MSGID: 100030]<br class="">[glusterfsd.c:2332:main] 0-/usr/sbin/glusterfsd: Started running<br class="">/usr/sbin/glusterfsd version 3.7.11 (args: /usr/sbin/glusterfsd -s<br class="">vnb.proxmox.softlog --volfile-id<br class="">datastore4.vnb.proxmox.softlog.tank-vmdata-datastore4 -p<br class="">/var/lib/glusterd/vols/datastore4/run/vnb.proxmox.softlog-tank-vmdata-datastore4.pid<br class="">-S /var/run/gluster/5ca23018ece7b94960f0580687e60650.socket<br class="">--brick-name /tank/vmdata/datastore4 -l<br class="">/var/log/glusterfs/bricks/tank-vmdata-datastore4.log --xlator-option<br class="">*-posix.glusterd-uuid=43a1bf8c-3e69-4581-8e16-f2e1462cfc36<br class="">--brick-port 49156 --xlator-option<br class="">datastore4-server.listen-port=49156)<br class="">[2016-04-20 05:55:35.541739] E [socket.c:770:__socket_server_bind]<br class="">0-socket.glusterfsd: binding to failed: Address already in use<br class="">[2016-04-20 05:55:35.541777] E [socket.c:773:__socket_server_bind]<br class="">0-socket.glusterfsd: Port is already in use<br class="">[2016-04-20 05:55:35.541794] W [rpcsvc.c:1604:rpcsvc_transport_create]<br class="">0-rpc-service: listening on transport failed<br class="">[2016-04-20 05:55:35.547990] I [MSGID: 100030]<br class="">[glusterfsd.c:2332:main] 0-/usr/sbin/glusterfsd: Started running<br class="">/usr/sbin/glusterfsd version 3.7.11 (args: /usr/sbin/glusterfsd -s<br class="">vnb.proxmox.softlog --volfile-id<br class="">datastore4.vnb.proxmox.softlog.tank-vmdata-datastore4 -p<br class="">/var/lib/glusterd/vols/datastore4/run/vnb.proxmox.softlog-tank-vmdata-datastore4.pid<br class="">-S /var/run/gluster/5ca23018ece7b94960f0580687e60650.socket<br class="">--brick-name /tank/vmdata/datastore4 -l<br class="">/var/log/glusterfs/bricks/tank-vmdata-datastore4.log --xlator-option<br class="">*-posix.glusterd-uuid=43a1bf8c-3e69-4581-8e16-f2e1462cfc36<br class="">--brick-port 49156 --xlator-option<br class="">datastore4-server.listen-port=49156)<br class=""><br class=""><br class="">I did a quick check of the other bricks, they are all filled with "I<br class="">[dict.c:473:dict_get]<br class="">(-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_getxattr_cbk+0xab)<br class="">[0x7f5ff77d239b]<br class="">-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/features/marker.so(marker_getxattr_cbk+0xa7)<br class="">[0x7f5feb9c88e7]<br class="">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get+0x93)<br class="">[0x7f5ff77c30f3] ) 0-dict: !this || key=() [Invalid argument]"<br class=""><br class=""><br class="">Thanks,<br class="">-- <br class="">Lindsay<br class="">_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">http://www.gluster.org/mailman/listinfo/gluster-users<br class=""></div></div></blockquote></div><br class=""></div></body></html>