<div dir="ltr"><div>Would apprecaite any insight into this issue:<br>replica 3 volume, it is showing a number of files on two of the bricks as needing healed, when you examine the files on the fuse mounts they generate I/O errors.<br></div>No files listed in split brain, but if I look at one of the files it looks to me like they have been updated on gluster-2 and gluster0 but not on gluster1 (see below).<br><div>I see  errors in /va/log/gluster/glustershd.log<br><br></div><div>-Thanks Alastair<br><br></div><div><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">[2016-12-20 07:25:06.018829] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1<br>[2016-12-20 07:25:06.018901] E [socket.c:2309:socket_connect_finish] 0-glusterfs: connection to ::1:24007 failed (Connection refused)<br>[2016-12-20 07:25:06.018944] E [glusterfsd-mgmt.c:1902:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: localhost (Transport endpoint is not connected)<br>[2016-12-20 07:25:07.187710] W [glusterfsd.c:1327:cleanup_and_exit] (--&gt;/lib64/libpthread.so.0(+0x7dc5) [0x7fd93f669dc5] --&gt;/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fd940cfbcd5] --&gt;/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fd940cfbb4b] ) 0-: received signum (15), shutting down<br>[2016-12-20 07:25:08.197959] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.5 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/3fe0b238bd46c38a95636f25cb5b9d8a.socket --xlator-option *replicate*.node-uuid=bcff5245-ea86-4384-a1bf-9219c8be8001)<br>[2016-12-20 07:25:08.216336] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1<br>[2016-12-20 07:25:08.216419] E [socket.c:2309:socket_connect_finish] 0-glusterfs: connection to ::1:24007 failed (Connection refused)<br>[2016-12-20 07:25:08.216464] E [glusterfsd-mgmt.c:1902:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: localhost (Transport endpoint is not connected)<br>[2016-12-20 07:25:12.208092] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-digitalcorpora-replicate-0: adding option &#39;node-uuid&#39; for volume &#39;digitalcorpora-replicate-0&#39; with value &#39;bcff5245-ea86-4384-a1bf-9219c8be8001&#39;<br>[2016-12-20 07:25:12.208122] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-gluster_shared_storage-replicate-0: adding option &#39;node-uuid&#39; for volume &#39;gluster_shared_storage-replicate-0&#39; with value &#39;bcff5245-ea86-4384-a1bf-9219c8be8001&#39;<br>[2016-12-20 07:25:12.208140] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-homes-replicate-0: adding option &#39;node-uuid&#39; for volume &#39;homes-replicate-0&#39; with value &#39;bcff5245-ea86-4384-a1bf-9219c8be8001&#39;<br>[2016-12-20 07:25:12.208155] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-public-replicate-0: adding option &#39;node-uuid&#39; for volume &#39;public-replicate-0&#39; with value &#39;bcff5245-ea86-4384-a1bf-9219c8be8001&#39;<br>[2016-12-20 07:25:12.208173] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-static-web-replicate-0: adding option &#39;node-uuid&#39; for volume &#39;static-web-replicate-0&#39; with value &#39;bcff5245-ea86-4384-a1bf-9219c8be8001&#39;<br>[2016-12-20 07:25:12.208199] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-tmp-replicate-0: adding option &#39;node-uuid&#39; for volume &#39;tmp-replicate-0&#39; with value &#39;bcff5245-ea86-4384-a1bf-9219c8be8001&#39;<br>[2016-12-20 07:25:12.208215] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-usr-local-replicate-0: adding option &#39;node-uuid&#39; for volume &#39;usr-local-replicate-0&#39; with value &#39;bcff5245-ea86-4384-a1bf-9219c8be8001&#39;<br>[2016-12-20 18:32:06.121734] E [client-common.c:526:client_pre_getxattr] (--&gt;/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xb5d8) [0x7f6bc4ba65d8] --&gt;/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x26ebd) [0x7f6bc4bc1ebd] --&gt;/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x393e3) [0x7f6bc4bd43e3] ) 0-: Assertion failed: 0<br>[2016-12-20 18:32:06.121809] E [client-common.c:587:client_pre_opendir] (--&gt;/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xa9d5) [0x7f6bc4ba59d5] --&gt;/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x25a65) [0x7f6bc4bc0a65] --&gt;/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x396b7) [0x7f6bc4bd46b7] ) 0-: Assertion failed: 0<br>[2016-12-20 18:46:51.764776] E [client-common.c:526:client_pre_getxattr] (--&gt;/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xb5d8) [0x7f6bc4ba65d8] --&gt;/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x26ebd) [0x7f6bc4bc1ebd] --&gt;/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x393e3) [0x7f6bc4bd43e3] ) 0-: Assertion failed: 0<br>[2016-12-20 18:46:51.764850] E [client-common.c:587:client_pre_opendir] (--&gt;/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xa9d5) [0x7f6bc4ba59d5] --&gt;/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x25a65) [0x7f6bc4bc0a65] --&gt;/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x396b7) [0x7f6bc4bd46b7] ) 0-: Assertion failed: 0<br>[2016-12-20 18:49:29.657568] E [client-common.c:526:client_pre_getxattr] (--&gt;/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xb5d8) [0x7f6bc4ba65d8] --&gt;/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x26ebd) [0x7f6bc4bc1ebd] --&gt;/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x393e3) [0x7f6bc4bd43e3] ) 0-: Assertion failed: 0<br>[2016-12-20 18:49:29.657645] E [client-common.c:587:client_pre_opendir] (--&gt;/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xa9d5) [0x7f6bc4ba59d5] --&gt;/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x25a65) [0x7f6bc4bc0a65] --&gt;/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x396b7) [0x7f6bc4bd46b7] ) 0-: Assertion failed: 0<br></blockquote><br>gluster2:<br><br># getfattr -d -m. -e hex /export/brick2/home/a/j/ajn/.Xauthority<br>getfattr: Removing leading &#39;/&#39; from absolute path names<br># file: export/brick2/home/a/j/ajn/.Xauthority<br>trusted.afr.dirty=0x000000000000000000000000<br>trusted.afr.homes-client-5=0x000000020000000100000000<br>trusted.bit-rot.version=0x020000000000000058589e6b0005bdac<br>trusted.gfid=0xb8b156b764304fd1bf7e692649bcecc5<br></div><div><br>gluster1:<br></div><div><br># getfattr -d -m. -e hex /export/brick2/home/a/j/ajn/.Xauthority<br>getfattr: Removing leading &#39;/&#39; from absolute path names<br># file: export/brick2/home/a/j/ajn/.Xauthority<br>trusted.afr.dirty=0x000000000000000000000000<br>trusted.bit-rot.version=0x0200000000000000583f45c20008d152<br>trusted.gfid=0x6c278b5c94ae436bb669b5f5dd21777e<br></div><div><br>gluster0:<br><br></div><div># getfattr -d -m. -e hex /export/brick2/home/a/j/ajn/.Xauthority <br>getfattr: Removing leading &#39;/&#39; from absolute path names<br># file: export/brick2/home/a/j/ajn/.Xauthority<br>trusted.afr.dirty=0x000000000000000000000000<br>trusted.afr.homes-client-5=0x000000020000000100000000<br>trusted.bit-rot.version=0x0200000000000000583f3fbb000b5b01<br>trusted.gfid=0xb8b156b764304fd1bf7e692649bcecc5<br><br><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">[root@gluster0 Project3]# glv heal homes info <br>Brick gluster-2:/export/brick2/home<br>/s/a/sadams25/pp2.txt <br>/s/a/sadams25/.viminfo <br>/a/v/avakil/.Xauthority <br>/j/m/jmurra17/fork <br>/c/f/cferris2/.viminfo <br>/c/s/cs367/bomblab/S001/log-status.txt <br>/c/s/cs367/bomblab/S001/bomblab-scoreboard.html <br>/c/s/cs367/bomblab/S001/scores.txt <br>/c/s/cs367/bomblab/S003/bomblab-scoreboard.html <br>/c/s/cs367/bomblab/S003/scores.txt <br>/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/libsupport.a <br>/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/Makefile <br>/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.c <br>/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.h <br>/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/caching.c <br>/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.o <br>/j/m/jmurra17/fork/fork.c <br>/j/m/jmurra17/.viminfo <br>/a/j/ajn/.Xauthority <br>/a/v/avakil/source_code/rm_setup/common_setup.tcl <br>/a/v/avakil/source_code/rm_setup/dc_setup_filenames.tcl <br>/a/v/avakil/source_code/rm_setup/dc_setup.tcl <br>/j/d/jdenton3/.viminfo <br>/s/a/sadams25/x.txt <br>/j/d/jdenton3/Project3/Project3.c <br>/j/m/jmurra17/fork/fork <br>/j/d/jdenton3/Project3/p5 <br>Status: Connected<br>Number of entries: 27<br><br>Brick gluster1.vsnet.gmu.edu:/export/brick2/home<br>Status: Connected<br>Number of entries: 0<br><br>Brick gluster0:/export/brick2/home<br>/s/a/sadams25/pp2.txt <br>/s/a/sadams25/.viminfo <br>/c/s/cs367/bomblab/S003/scores.txt <br>/a/v/avakil/.Xauthority <br>/c/s/cs367/bomblab/S001/scores.txt <br>/c/f/cferris2/.viminfo <br>/c/s/cs367/bomblab/S001/log-status.txt <br>/c/s/cs367/bomblab/S003/tmpwebpage.14635 <br>/c/s/cs367/bomblab/S001/bomblab-scoreboard.html <br>/c/s/cs367/bomblab/S003/bomblab-scoreboard.html <br>/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/libsupport.a <br>/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/Makefile <br>/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.c <br>/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.h <br>/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/caching.c <br>/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.o <br>/j/m/jmurra17/fork <br>&lt;gfid:310211c2-aeec-4906-894f-023d0ad7d5cc&gt;/#<a href="http://affiliate.nagios.com/settings.sol">affiliate.nagios.com/settings.sol</a> <br>/a/v/avakil/source_code/rm_setup/common_setup.tcl <br>/a/j/ajn/.Xauthority <br>/j/m/jmurra17/.viminfo <br>/a/v/avakil/source_code/rm_setup/dc_setup.tcl <br>/j/m/jmurra17/fork/fork.c <br>/a/v/avakil/source_code/rm_setup/dc_setup_filenames.tcl <br>/j/d/jdenton3/Project3/Project3.c <br>/j/d/jdenton3/.viminfo <br>/s/a/sadams25/x.txt <br>/j/m/jmurra17/fork/fork <br>/j/d/jdenton3/Project3/p5 <br>Status: Connected<br>Number of entries: 29<br><br>[<br>[root@gluster0 .bad]# cd /mnt/home/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/<br>[root@gluster0 mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh]# ls -al <br>ls: cannot access libsupport.a: Input/output error<br>ls: cannot access Makefile: Input/output error<br>ls: cannot access memory_system.c: Input/output error<br>ls: cannot access memory_system.h: Input/output error<br>ls: cannot access caching.c: Input/output error<br>ls: cannot access memory_system.o: Input/output error<br>total 626<br>drwxrwxr-x 2 1735 users   4096 Dec 20 11:38 .<br>drwxr-xr-x 3 root root    4096 Dec 20 13:53 ..<br>-????????? ? ?    ?          ?            ? caching.c<br>-rw-rw-r-- 1 1735 users   9056 Dec 20 11:36 caching.o<br>-rwxrwxr-x 1 1735 users 147855 Dec 20 11:36 lab4<br>-rw-r--r-- 1 1735 users 307200 Dec 13 07:04 Lab 4 - 12 9_mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh.tar<br>-rw-rw-r-- 1 1735 users   8254 Dec 20 11:38 lab4_logfile<br>-rw-r--r-- 1 1735 users 153600 Dec 20 11:32 lab4_mchehreh.tar<br>-????????? ? ?    ?          ?            ? libsupport.a<br>-????????? ? ?    ?          ?            ? Makefile<br>-????????? ? ?    ?          ?            ? memory_system.c<br>-????????? ? ?    ?          ?            ? memory_system.h<br>-????????? ? ?    ?          ?            ? memory_system.o<br>-rw-rw-r-- 1 1735 users    449 Dec 20 11:38 t1<br>-rw-rw-r-- 1 1735 users    453 Dec 20 11:38 t2<br>-rw-rw-r-- 1 1735 users   2185 Dec 20 11:38 t3<br>-rw-rw-r-- 1 1735 users   2195 Dec 20 11:38 t4<br></blockquote><br></div></div>