<font size=2 face="sans-serif">I was checking on the client fuse mount
and saw that</font><br><br><br><font size=2 face="sans-serif">[2015-09-26 14:41:29.417267] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
0-devstatic-replicate-1: Unable to self-heal contents of '/' (possible
split-brain). Please delete the file from all but the preferred subvolume.-
Pending matrix: [ [ 0 15 ] [ 1 0 ] ]</font><br><font size=2 face="sans-serif">[2015-09-26 14:41:29.418063] E [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-devstatic-replicate-1: metadata self heal failed,
on /</font><br><br><font size=2 face="sans-serif">After reviewing your doc to build understanding,
</font><a href="https://github.com/gluster/glusterdocs/blob/master/Troubleshooting/split-brain.md"><font size=2 color=blue face="sans-serif">https://github.com/gluster/glusterdocs/blob/master/Troubleshooting/split-brain.md</font></a><br><font size=2 face="sans-serif"> I found </font><a href="http://thr3ads.net/gluster-users/2013/11/2710016-Unable-to-self-heal-contents-of-gfid-00000000-0000-0000-0000-000000000001"><font size=3 color=blue><u>http://thr3ads.net/gluster-users/2013/11/2710016-Unable-to-self-heal-contents-of-gfid-00000000-0000-0000-0000-000000000001</u></font></a><p><font size=2 face="sans-serif">also had the same issue of '/'. </font><br><br><font size=2 face="sans-serif">Now I believe all is clear. </font><br><br><font size=2 face="sans-serif">Client log:</font><br><font size=2 face="sans-serif">[2015-09-26 14:53:35.662325] I [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-devstatic-replicate-1: metadata self heal is successfully
completed, metadata self heal from source devstatic-client-2 to
devstatic-client-3, metadata - Pending matrix: [ [ 0 0 ] [
0 0 ] ], on /</font><br><font size=2 face="sans-serif">[2015-09-26 14:53:35.667537] I [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-devstatic-replicate-0: metadata self heal is successfully
completed, metadata self heal from source devstatic-client-0 to
devstatic-client-1, metadata - Pending matrix: [ [ 0 0 ] [
0 0 ] ], on /</font><br><br><font size=2 face="sans-serif">gluster storage cli output.</font><br><font size=2 face="sans-serif">Now the gluster volume heal devstatic
info split-brain is clean</font><br><font size=2 face="sans-serif">[root@omdx1b51 ~]# gluster volume
heal devstatic info split-brain</font><br><font size=2 face="sans-serif">Gathering list of split brain entries
on volume devstatic has been successful</font><br><br><font size=2 face="sans-serif">Brick omhq1b4e:/static/content</font><br><font size=2 face="sans-serif">Number of entries: 0</font><br><br><font size=2 face="sans-serif">Brick omdx1b50:/static/content</font><br><font size=2 face="sans-serif">Number of entries: 0</font><br><br><font size=2 face="sans-serif">Brick omhq1b4f:/static/content</font><br><font size=2 face="sans-serif">Number of entries: 0</font><br><br><font size=2 face="sans-serif">Brick omdx1b51:/static/content</font><br><font size=2 face="sans-serif">Number of entries: 0</font><br><br><font size=2 face="sans-serif">Please let me know if there were any
steps I've missed or additional areas to look at.</font><br><br><font size=2 face="sans-serif">Thank you,<br><br>Khoi Mai<br>Union Pacific Railroad<br>Distributed Engineering & Architecture<br>Senior Project Engineer<br><br></font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Ravishankar N <ravishankar@redhat.com></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">Khoi Mai <KHOIMAI@up.com></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">09/26/2015 12:59 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [Gluster-users]
glusterfs3.4.2-1 split-brain question</font><br><hr noshade><br><br><br><font size=3>This email originated from outside of the company. Please
use discretion if opening attachments or clicking on links.</font><hr><font size=3> </font><p><font size=3><br></font><br><font size=3>On 09/26/2015 10:37 AM, Khoi Mai wrote:</font><br><font size=2 face="sans-serif">I'd like to run the afr attr reset on
omdx1b51, does that make omhq1b4f the winning source?</font><br><font size=2>Yes. Resetting </font><font size=2 face="sans-serif">trusted.afr.devstatic-client-2
on omdx1b51 makes omhq1b4f the source because it blames omdx1b51 via trusted.afr.devstatic-client-3.</font><font size=3></font><br><font size=2 face="sans-serif"> Or do I run the commands on the
server I want to be the source? For example below?</font><font size=3><br></font><font size=2 face="sans-serif"><br>So, the intended changes are:<br>On omdx1b51<br>For trusted.afr.devstatic-client-2<br>0x000000000000000600000000 to 0x000000000000000000000000<br>Hence execute setfattr -n trusted.afr.vol-client-0 -v 0x000000000000000100000000
/static/content/<br>then<br>gluster volume heal devstatic </font><font size=3><br></font><font size=2 face="sans-serif"><br>Thank you for your help!<br><br>Khoi Mai<br>Union Pacific Railroad<br>Distributed Engineering & Architecture<br>Senior Project Engineer<br></font><font size=3><br><br><br><br></font><font size=1 color=#5f5f5f face="sans-serif"><br>From: </font><font size=1 face="sans-serif">Ravishankar
N </font><a href=mailto:ravishankar@redhat.com><font size=1 color=blue face="sans-serif"><u><ravishankar@redhat.com></u></font></a><font size=1 color=#5f5f5f face="sans-serif"><br>To: </font><font size=1 face="sans-serif">Khoi
Mai </font><a href=mailto:KHOIMAI@up.com><font size=1 color=blue face="sans-serif"><u><KHOIMAI@up.com></u></font></a><font size=1 color=#5f5f5f face="sans-serif"><br>Cc: </font><a href="mailto:gluster-users@gluster.org"><font size=1 color=blue face="sans-serif"><u>gluster-users@gluster.org</u></font></a><font size=1 color=#5f5f5f face="sans-serif"><br>Date: </font><font size=1 face="sans-serif">09/25/2015
09:04 PM</font><font size=1 color=#5f5f5f face="sans-serif"><br>Subject: </font><font size=1 face="sans-serif">Re:
[Gluster-users] glusterfs3.4.2-1 split-brain question</font><font size=3><br></font><hr noshade><font size=3><br><br><br>This email originated from outside of the company. Please use discretion
if opening attachments or clicking on links. </font><hr><font size=3> </font><p><font size=3><br><br>On 09/25/2015 07:40 PM, Khoi Mai wrote:</font><font size=2 face="sans-serif"><br>I think I found it from your github doc. the quota size does not
match with the replicate pair. I don't know if that would make
the difference. I apologize, i cannot use fpaste.org, or pastebin.com
due to policies at my company.</font><font size=2><br><br>I'm not sure quota xattrs are handled in AFR in glusterfs-3.4. There
doesn't seem to be any split-brain in the first replica pair since the
afr xattrs all seem to be zero. For the second replica pair, they are in
metadata split-brain (but unlikely due to the quota-size xattr).
You can pick one brick as source reset the appropriate afr xattr and run
`gluster v heal volname` once. </font><font size=3><br></font><font size=2 face="sans-serif"><br><br>[root@omhq1b4e ~]# getfattr -d -m . -e hex /static/content/<br>getfattr: Removing leading '/' from absolute path names<br># file: static/content/<br>trusted.afr.devstatic-client-0=0x000000000000000000000000<br>trusted.afr.devstatic-client-1=0x000000000000000000000000<br>trusted.gfid=0x00000000000000000000000000000001<br>trusted.glusterfs.dht=0x0000000100000000000000007ffffffe<br>trusted.glusterfs.quota.dirty=0x3000<br>trusted.glusterfs.quota.limit-set=0x0000018000000000ffffffffffffffff<b><br>trusted.glusterfs.quota.size=0x0000006f303e4e00</b><br>trusted.glusterfs.volume-id=0x75832afbf20e40188d748550a92233fc<br><br>[root@omdx1b50 ~]# getfattr -d -m . -e hex /static/content/<br>getfattr: Removing leading '/' from absolute path names<br># file: static/content/<br>trusted.afr.devstatic-client-0=0x000000000000000000000000<br>trusted.afr.devstatic-client-1=0x000000000000000000000000<br>trusted.gfid=0x00000000000000000000000000000001<br>trusted.glusterfs.dht=0x0000000100000000000000007ffffffe<br>trusted.glusterfs.quota.dirty=0x3000<br>trusted.glusterfs.quota.limit-set=0x0000018000000000ffffffffffffffff<b><br>trusted.glusterfs.quota.size=0x00000081bfca4e00</b><br>trusted.glusterfs.volume-id=0x75832afbf20e40188d748550a92233fc</font><font size=3><br></font><font size=2 face="sans-serif"><br><br>[root@omhq1b4f ~]# getfattr -d -m . -e hex /static/content/<br>getfattr: Removing leading '/' from absolute path names<br># file: static/content/<br>trusted.afr.devstatic-client-2=0x000000000000000000000000<br>trusted.afr.devstatic-client-3=0x000000000000000900000000<br>trusted.gfid=0x00000000000000000000000000000001<br>trusted.glusterfs.dht=0x00000001000000007fffffffffffffff<br>trusted.glusterfs.quota.dirty=0x3000<br>trusted.glusterfs.quota.limit-set=0x0000018000000000ffffffffffffffff<b><br>trusted.glusterfs.quota.size=0x00000076b9b20800</b><br>trusted.glusterfs.volume-id=0x75832afbf20e40188d748550a92233fc<br><br>[root@omdx1b51 ~]# getfattr -d -m . -e hex /static/content/<br>getfattr: Removing leading '/' from absolute path names<br># file: static/content/<br>trusted.afr.devstatic-client-2=0x000000000000000600000000<br>trusted.afr.devstatic-client-3=0x000000000000000000000000<br>trusted.gfid=0x00000000000000000000000000000001<br>trusted.glusterfs.dht=0x00000001000000007fffffffffffffff<br>trusted.glusterfs.quota.dirty=0x3000<br>trusted.glusterfs.quota.limit-set=0x0000018000000000ffffffffffffffff<b><br>trusted.glusterfs.quota.size=0x0000006eb4e0b000</b><br>trusted.glusterfs.volume-id=0x75832afbf20e40188d748550a92233fc</font><font size=3><br><br></font><font size=2 face="sans-serif"><br><br><br><br>Khoi Mai<br>Union Pacific Railroad<br>Distributed Engineering & Architecture<br>Senior Project Engineer</font><font size=3><br><br><br><br></font><font size=1 color=#5f5f5f face="sans-serif"><br><br>From: </font><font size=1 face="sans-serif">Khoi
Mai/UPC</font><font size=1 color=#5f5f5f face="sans-serif"><br>To: </font><font size=1 face="sans-serif">Ravishankar
N </font><a href=mailto:ravishankar@redhat.com><font size=1 color=blue face="sans-serif"><u><ravishankar@redhat.com></u></font></a><font size=1 color=#5f5f5f face="sans-serif"><br>Cc: </font><a href="mailto:gluster-users@gluster.org"><font size=1 color=blue face="sans-serif"><u>gluster-users@gluster.org</u></font></a><font size=1 color=#5f5f5f face="sans-serif"><br>Date: </font><font size=1 face="sans-serif">09/25/2015
09:01 AM</font><font size=1 color=#5f5f5f face="sans-serif"><br>Subject: </font><font size=1 face="sans-serif">Re:
[Gluster-users] glusterfs3.4.2-1 split-brain question</font><p><hr noshade><font size=2 face="sans-serif"><br><br>the gfid looks the same. I'm not sure what gluster volume heal info
split-brain is reporting when the GFID matches, and for all 4 nodes in
the devstatic volume.<br><br>[root@omhq1b4f ~]# getfattr -h -d -m trusted.gfid -e hex /static/content/<br>getfattr: Removing leading '/' from absolute path names<br># file: static/content/<br>trusted.gfid=0x00000000000000000000000000000001<br><br>[root@omhq1b4f ~]# stat /static/content/<br> File: `/static/content/'<br> Size: 4096 Blocks: 8
IO Block: 4096 directory<br>Device: fd02h/64770d Inode: 536871040 Links: 90<br>Access: (0775/drwxrwxr-x) Uid: ( 0/ root)
Gid: ( 0/ root)<br>Access: 2014-02-02 09:06:27.073528000 -0600<br>Modify: 2014-12-23 10:13:00.823641000 -0600<br>Change: 2015-09-25 08:42:44.524336543 -0500<br>[root@omhq1b4f ~]#<br><br>[root@omdx1b51 ~]# getfattr -h -d -m trusted.gfid -e hex /static/content/<br>getfattr: Removing leading '/' from absolute path names<br># file: static/content/<br>trusted.gfid=0x00000000000000000000000000000001</font><font size=3><br></font><font size=2 face="sans-serif"><br><br>[root@omdx1b51 ~]# stat /static/content/<br> File: `/static/content/'<br> Size: 4096 Blocks: 8
IO Block: 4096 directory<br>Device: fd02h/64770d Inode: 536871040 Links: 90<br>Access: (0775/drwxrwxr-x) Uid: ( 0/ root)
Gid: ( 0/ root)<br>Access: 2014-02-02 09:06:27.073528000 -0600<br>Modify: 2014-12-23 10:13:00.823641000 -0600<br>Change: 2015-09-25 08:42:44.526287950 -0500</font><font size=3><br><br></font><font size=2 face="sans-serif"><br><br><br><br>Khoi Mai<br>Union Pacific Railroad<br>Distributed Engineering & Architecture<br>Senior Project Engineer</font><font size=3><br><br><br><br><br></font><font size=1 color=#5f5f5f face="sans-serif"><br><br>From: </font><font size=1 face="sans-serif">Ravishankar
N </font><a href=mailto:ravishankar@redhat.com><font size=1 color=blue face="sans-serif"><u><ravishankar@redhat.com></u></font></a><font size=1 color=#5f5f5f face="sans-serif"><br>To: </font><font size=1 face="sans-serif">Khoi
Mai </font><a href=mailto:KHOIMAI@UP.COM><font size=1 color=blue face="sans-serif"><u><KHOIMAI@UP.COM></u></font></a><font size=1 face="sans-serif">,
</font><a href="mailto:gluster-users@gluster.org"><font size=1 color=blue face="sans-serif"><u>gluster-users@gluster.org</u></font></a><font size=1 color=#5f5f5f face="sans-serif"><br>Date: </font><font size=1 face="sans-serif">09/25/2015
03:13 AM</font><font size=1 color=#5f5f5f face="sans-serif"><br>Subject: </font><font size=1 face="sans-serif">Re:
[Gluster-users] glusterfs3.4.2-1 split-brain question</font><font size=3><br></font><hr noshade><font size=3><br><br><br>This email originated from outside of the company. Please use discretion
if opening attachments or clicking on links. </font><hr><font size=3> </font><p><font size=3><br><br>On 09/25/2015 07:48 AM, Khoi Mai wrote:</font><font size=2 face="sans-serif"><br>I have a 4 node distributed-replicated gluster farm.<br><br>Volume Name: devstatic<br>Type: Distributed-Replicate<br>Volume ID: 75832afb-f20e-4017-8d74-8550a92233fd<br>Status: Started<br>Number of Bricks: 2 x 2 = 4<br>Transport-type: tcp<br>Bricks:<br>Brick1: omhq1b4e:/static/content<br>Brick2: omdx1b50:/static/content<br>Brick3: omhq1b4f:/static/content<br>Brick4: omdx1b51:/static/content<br>Options Reconfigured:<br>features.quota-deem-statfs: on<br>server.allow-insecure: on<br>network.ping-timeout: 10<br>performance.lazy-open: off<br>performance.write-behind: on<br>features.quota: on<br>geo-replication.indexing: off<br>server.statedump-path: /tmp/<br>diagnostics.brick-log-level: CRITICAL<br><br><br>When I query heal split-brain info I get the following.<br><br>[root@omhq1b4e ~]# gluster volume heal devstatic info split-brain<br>Gathering list of split brain entries on volume devstatic has been successful<br><br>Brick omhq1b4e:/static/content<br>Number of entries: 0<br><br>Brick omdx1b50:/static/content<br>Number of entries: 0<br><br>Brick omhq1b4f:/static/content<br>Number of entries: 43<br>at path
on brick<br>-----------------------------------<br>2015-09-24 18:50:20 /<br>2015-09-24 18:50:20 /<br>2015-09-24 18:52:01 /<br>2015-09-24 19:10:22 /<br><br>Brick omdx1b51:/static/content<br>Number of entries: 42<br>at path
on brick<br>-----------------------------------<br>2015-09-24 18:51:58 /<br>2015-09-24 18:51:59 /<br>2015-09-24 19:01:59 /<br>2015-09-24 19:11:59 /<br><br><br>Being / on the same replicate, how would I safely resolve this issue? Is
it really going to require me to delete the root of each node and heal?
I hope not, the entire volume is about 1TB.</font><font size=3><br><br>No, it is likely that the root is only in metadata split-brain. What does
the getfattr output of '/' show on the bricks? </font><a href="https://github.com/gluster/glusterdocs/blob/master/Troubleshooting/split-brain.md"></a><a href="https://github.com/gluster/glusterdocs/blob/master/Troubleshooting/split-brain.md"><font size=3 color=blue><u>https://github.com/gluster/glusterdocs/blob/master/Troubleshooting/split-brain.md</u></font></a><font size=3>should
tell you how to resolve split-brains.</font><font size=2 face="sans-serif"><br>Thank you,</font><font size=3><br></font><font size=2 face="sans-serif"><br><br><br><br><br>Khoi Mai<br>Union Pacific Railroad<br>Distributed Engineering & Architecture<br>Senior Project Engineer</font><font size=3><br><br><br><br><br>**<br><br><br><br>This email and any attachments may contain information that is confidential
and/or privileged for the sole use of the intended recipient. Any use,
review, disclosure, copying, distribution or reliance by others, and any
forwarding of this email or its contents, without the express permission
of the sender is strictly prohibited by law. If you are not the intended
recipient, please contact the sender immediately, delete the e-mail and
destroy all copies.<br><br>**<br></font><tt><font size=3><br><br>_______________________________________________<br>Gluster-users mailing list</font></tt><font size=3 color=blue><u><br></u></font><a href="mailto:Gluster-users@gluster.org"><tt><font size=3 color=blue><u>Gluster-users@gluster.org</u></font></tt></a><font size=3 color=blue><u><br></u></font><a href="http://www.gluster.org/mailman/listinfo/gluster-users"><tt><font size=3 color=blue><u>http://www.gluster.org/mailman/listinfo/gluster-users</u></font></tt></a><font size=3><br><br><br><br><br>**<br><br><br><br>This email and any attachments may contain information that is confidential
and/or privileged for the sole use of the intended recipient. Any use,
review, disclosure, copying, distribution or reliance by others, and any
forwarding of this email or its contents, without the express permission
of the sender is strictly prohibited by law. If you are not the intended
recipient, please contact the sender immediately, delete the e-mail and
destroy all copies.<br><br>**</font><p><font size=3><br><br><br>**<br><br><br><br>This email and any attachments may contain information that is confidential
and/or privileged for the sole use of the intended recipient. Any use,
review, disclosure, copying, distribution or reliance by others, and any
forwarding of this email or its contents, without the express permission
of the sender is strictly prohibited by law. If you are not the intended
recipient, please contact the sender immediately, delete the e-mail and
destroy all copies.<br><br>**</font><p><p><BR>
<BR>
<BR>
**<BR>
<BR>
<BR>
<BR>
This email and any attachments may contain information that is confidential and/or privileged for the sole use of the intended recipient. Any use, review, disclosure, copying, distribution or reliance by others, and any forwarding of this email or its contents, without the express permission of the sender is strictly prohibited by law. If you are not the intended recipient, please contact the sender immediately, delete the e-mail and destroy all copies.<BR>
<BR>
**<BR>