<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<br>
<div class="moz-cite-prefix">On 09/25/2015 07:40 PM, Khoi Mai wrote:<br>
</div>
<blockquote
cite="mid:OFD2BC48C7.D6EF40A5-ON86257ECB.004D6318-86257ECB.004DE8D3@LocalDomain"
type="cite"><font face="sans-serif" size="2">I think I found it
from your github doc.
the quota size does not match with the replicate pair. I don't
know if that would make the difference. I apologize, i cannot
use fpaste.org, or pastebin.com due to policies at my company.</font><br>
</blockquote>
<br>
<font size="2">I'm not sure quota xattrs are handled in AFR in
glusterfs-3.4. There doesn't seem to be any split-brain in the
first replica pair since the afr xattrs all seem to be zero. For
the second replica pair, they are in metadata split-brain (but
unlikely due to the quota-size xattr). You can pick one brick as
source reset the appropriate afr xattr and run `gluster v heal
volname` once. <br>
<br>
</font>
<blockquote
cite="mid:OFD2BC48C7.D6EF40A5-ON86257ECB.004D6318-86257ECB.004DE8D3@LocalDomain"
type="cite"><br>
<font face="sans-serif" size="2">[root@omhq1b4e ~]# getfattr -d -m
.
-e hex /static/content/</font><br>
<font face="sans-serif" size="2">getfattr: Removing leading '/'
from
absolute path names</font><br>
<font face="sans-serif" size="2"># file: static/content/</font><br>
<font face="sans-serif" size="2">trusted.afr.devstatic-client-0=0x000000000000000000000000</font><br>
<font face="sans-serif" size="2">trusted.afr.devstatic-client-1=0x000000000000000000000000</font><br>
<font face="sans-serif" size="2">trusted.gfid=0x00000000000000000000000000000001</font><br>
<font face="sans-serif" size="2">trusted.glusterfs.dht=0x0000000100000000000000007ffffffe</font><br>
<font face="sans-serif" size="2">trusted.glusterfs.quota.dirty=0x3000</font><br>
<font face="sans-serif" size="2">trusted.glusterfs.quota.limit-set=0x0000018000000000ffffffffffffffff</font><br>
<font face="sans-serif" size="2"><b>trusted.glusterfs.quota.size=0x0000006f303e4e00</b></font><br>
<font face="sans-serif" size="2">trusted.glusterfs.volume-id=0x75832afbf20e40188d748550a92233fc</font><br>
<br>
<font face="sans-serif" size="2">[root@omdx1b50 ~]# getfattr -d -m
.
-e hex /static/content/</font><br>
<font face="sans-serif" size="2">getfattr: Removing leading '/'
from
absolute path names</font><br>
<font face="sans-serif" size="2"># file: static/content/</font><br>
<font face="sans-serif" size="2">trusted.afr.devstatic-client-0=0x000000000000000000000000</font><br>
<font face="sans-serif" size="2">trusted.afr.devstatic-client-1=0x000000000000000000000000</font><br>
<font face="sans-serif" size="2">trusted.gfid=0x00000000000000000000000000000001</font><br>
<font face="sans-serif" size="2">trusted.glusterfs.dht=0x0000000100000000000000007ffffffe</font><br>
<font face="sans-serif" size="2">trusted.glusterfs.quota.dirty=0x3000</font><br>
<font face="sans-serif" size="2">trusted.glusterfs.quota.limit-set=0x0000018000000000ffffffffffffffff</font><br>
<font face="sans-serif" size="2"><b>trusted.glusterfs.quota.size=0x00000081bfca4e00</b></font><br>
<font face="sans-serif" size="2">trusted.glusterfs.volume-id=0x75832afbf20e40188d748550a92233fc</font><br>
<br>
<br>
<font face="sans-serif" size="2">[root@omhq1b4f ~]# getfattr -d -m
.
-e hex /static/content/</font><br>
<font face="sans-serif" size="2">getfattr: Removing leading '/'
from
absolute path names</font><br>
<font face="sans-serif" size="2"># file: static/content/</font><br>
<font face="sans-serif" size="2">trusted.afr.devstatic-client-2=0x000000000000000000000000</font><br>
<font face="sans-serif" size="2">trusted.afr.devstatic-client-3=0x000000000000000900000000</font><br>
<font face="sans-serif" size="2">trusted.gfid=0x00000000000000000000000000000001</font><br>
<font face="sans-serif" size="2">trusted.glusterfs.dht=0x00000001000000007fffffffffffffff</font><br>
<font face="sans-serif" size="2">trusted.glusterfs.quota.dirty=0x3000</font><br>
<font face="sans-serif" size="2">trusted.glusterfs.quota.limit-set=0x0000018000000000ffffffffffffffff</font><br>
<font face="sans-serif" size="2"><b>trusted.glusterfs.quota.size=0x00000076b9b20800</b></font><br>
<font face="sans-serif" size="2">trusted.glusterfs.volume-id=0x75832afbf20e40188d748550a92233fc</font><br>
<br>
<font face="sans-serif" size="2">[root@omdx1b51 ~]# getfattr -d -m
.
-e hex /static/content/</font><br>
<font face="sans-serif" size="2">getfattr: Removing leading '/'
from
absolute path names</font><br>
<font face="sans-serif" size="2"># file: static/content/</font><br>
<font face="sans-serif" size="2">trusted.afr.devstatic-client-2=0x000000000000000600000000</font><br>
<font face="sans-serif" size="2">trusted.afr.devstatic-client-3=0x000000000000000000000000</font><br>
<font face="sans-serif" size="2">trusted.gfid=0x00000000000000000000000000000001</font><br>
<font face="sans-serif" size="2">trusted.glusterfs.dht=0x00000001000000007fffffffffffffff</font><br>
<font face="sans-serif" size="2">trusted.glusterfs.quota.dirty=0x3000</font><br>
<font face="sans-serif" size="2">trusted.glusterfs.quota.limit-set=0x0000018000000000ffffffffffffffff</font><br>
<font face="sans-serif" size="2"><b>trusted.glusterfs.quota.size=0x0000006eb4e0b000</b></font><br>
<font face="sans-serif" size="2">trusted.glusterfs.volume-id=0x75832afbf20e40188d748550a92233fc</font><br>
<br>
<br>
<br>
<font face="sans-serif" size="2"><br>
<br>
Khoi Mai<br>
Union Pacific Railroad<br>
Distributed Engineering & Architecture<br>
Senior Project Engineer<br>
<br>
</font><br>
<br>
<br>
<br>
<font face="sans-serif" size="1" color="#5f5f5f">From:
</font><font face="sans-serif" size="1">Khoi Mai/UPC</font><br>
<font face="sans-serif" size="1" color="#5f5f5f">To:
</font><font face="sans-serif" size="1">Ravishankar N
<a class="moz-txt-link-rfc2396E" href="mailto:ravishankar@redhat.com"><ravishankar@redhat.com></a></font><br>
<font face="sans-serif" size="1" color="#5f5f5f">Cc:
</font><font face="sans-serif" size="1"><a class="moz-txt-link-abbreviated" href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a></font><br>
<font face="sans-serif" size="1" color="#5f5f5f">Date:
</font><font face="sans-serif" size="1">09/25/2015 09:01 AM</font><br>
<font face="sans-serif" size="1" color="#5f5f5f">Subject:
</font><font face="sans-serif" size="1">Re: [Gluster-users]
glusterfs3.4.2-1 split-brain question</font><br>
<hr noshade="noshade"><br>
<br>
<font face="sans-serif" size="2">the gfid looks the same. I'm not
sure what gluster volume heal info split-brain is reporting when
the GFID
matches, and for all 4 nodes in the devstatic volume.</font><br>
<br>
<font face="sans-serif" size="2">[root@omhq1b4f ~]# getfattr -h -d
-m
trusted.gfid -e hex /static/content/</font><br>
<font face="sans-serif" size="2">getfattr: Removing leading '/'
from
absolute path names</font><br>
<font face="sans-serif" size="2"># file: static/content/</font><br>
<font face="sans-serif" size="2">trusted.gfid=0x00000000000000000000000000000001</font><br>
<br>
<font face="sans-serif" size="2">[root@omhq1b4f ~]# stat
/static/content/</font><br>
<font face="sans-serif" size="2"> File: `/static/content/'</font><br>
<font face="sans-serif" size="2"> Size: 4096
Blocks: 8 IO Block:
4096 directory</font><br>
<font face="sans-serif" size="2">Device: fd02h/64770d Inode:
536871040 Links: 90</font><br>
<font face="sans-serif" size="2">Access: (0775/drwxrwxr-x) Uid:
( 0/ root) Gid: ( 0/
root)</font><br>
<font face="sans-serif" size="2">Access: 2014-02-02
09:06:27.073528000
-0600</font><br>
<font face="sans-serif" size="2">Modify: 2014-12-23
10:13:00.823641000
-0600</font><br>
<font face="sans-serif" size="2">Change: 2015-09-25
08:42:44.524336543
-0500</font><br>
<font face="sans-serif" size="2">[root@omhq1b4f ~]#</font><br>
<br>
<font face="sans-serif" size="2">[root@omdx1b51 ~]# getfattr -h -d
-m
trusted.gfid -e hex /static/content/</font><br>
<font face="sans-serif" size="2">getfattr: Removing leading '/'
from
absolute path names</font><br>
<font face="sans-serif" size="2"># file: static/content/</font><br>
<font face="sans-serif" size="2">trusted.gfid=0x00000000000000000000000000000001</font><br>
<br>
<br>
<font face="sans-serif" size="2">[root@omdx1b51 ~]# stat
/static/content/</font><br>
<font face="sans-serif" size="2"> File: `/static/content/'</font><br>
<font face="sans-serif" size="2"> Size: 4096
Blocks: 8 IO Block:
4096 directory</font><br>
<font face="sans-serif" size="2">Device: fd02h/64770d Inode:
536871040 Links: 90</font><br>
<font face="sans-serif" size="2">Access: (0775/drwxrwxr-x) Uid:
( 0/ root) Gid: ( 0/
root)</font><br>
<font face="sans-serif" size="2">Access: 2014-02-02
09:06:27.073528000
-0600</font><br>
<font face="sans-serif" size="2">Modify: 2014-12-23
10:13:00.823641000
-0600</font><br>
<font face="sans-serif" size="2">Change: 2015-09-25
08:42:44.526287950
-0500</font><br>
<br>
<br>
<br>
<font face="sans-serif" size="2"><br>
<br>
Khoi Mai<br>
Union Pacific Railroad<br>
Distributed Engineering & Architecture<br>
Senior Project Engineer<br>
<br>
</font><br>
<br>
<br>
<br>
<br>
<font face="sans-serif" size="1" color="#5f5f5f">From:
</font><font face="sans-serif" size="1">Ravishankar N
<a class="moz-txt-link-rfc2396E" href="mailto:ravishankar@redhat.com"><ravishankar@redhat.com></a></font><br>
<font face="sans-serif" size="1" color="#5f5f5f">To:
</font><font face="sans-serif" size="1">Khoi Mai
<a class="moz-txt-link-rfc2396E" href="mailto:KHOIMAI@UP.COM"><KHOIMAI@UP.COM></a>,
<a class="moz-txt-link-abbreviated" href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a></font><br>
<font face="sans-serif" size="1" color="#5f5f5f">Date:
</font><font face="sans-serif" size="1">09/25/2015 03:13 AM</font><br>
<font face="sans-serif" size="1" color="#5f5f5f">Subject:
</font><font face="sans-serif" size="1">Re: [Gluster-users]
glusterfs3.4.2-1 split-brain question</font><br>
<hr noshade="noshade"><br>
<br>
<br>
<font size="3">This email originated from outside of the company.
Please
use discretion if opening attachments or clicking on links.</font>
<hr><font size="3"> </font>
<p><font size="3"><br>
</font><br>
<font size="3">On 09/25/2015 07:48 AM, Khoi Mai wrote:</font><br>
<font face="sans-serif" size="2">I have a 4 node
distributed-replicated
gluster farm.</font><font size="3"><br>
</font><font face="sans-serif" size="2"><br>
Volume Name: devstatic<br>
Type: Distributed-Replicate<br>
Volume ID: 75832afb-f20e-4017-8d74-8550a92233fd<br>
Status: Started<br>
Number of Bricks: 2 x 2 = 4<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: omhq1b4e:/static/content<br>
Brick2: omdx1b50:/static/content<br>
Brick3: omhq1b4f:/static/content<br>
Brick4: omdx1b51:/static/content<br>
Options Reconfigured:<br>
features.quota-deem-statfs: on<br>
server.allow-insecure: on<br>
network.ping-timeout: 10<br>
performance.lazy-open: off<br>
performance.write-behind: on<br>
features.quota: on<br>
geo-replication.indexing: off<br>
server.statedump-path: /tmp/<br>
diagnostics.brick-log-level: CRITICAL</font><font size="3"><br>
<br>
</font><font face="sans-serif" size="2"><br>
When I query heal split-brain info I get the following.</font><font
size="3"><br>
</font><font face="sans-serif" size="2"><br>
[root@omhq1b4e ~]# gluster volume heal devstatic info
split-brain<br>
Gathering list of split brain entries on volume devstatic has
been successful</font><font size="3"><br>
</font><font face="sans-serif" size="2"><br>
Brick omhq1b4e:/static/content<br>
Number of entries: 0</font><font size="3"><br>
</font><font face="sans-serif" size="2"><br>
Brick omdx1b50:/static/content<br>
Number of entries: 0</font><font size="3"><br>
</font><font face="sans-serif" size="2"><br>
Brick omhq1b4f:/static/content<br>
Number of entries: 43<br>
at path
on brick<br>
-----------------------------------<br>
2015-09-24 18:50:20 /<br>
2015-09-24 18:50:20 /<br>
2015-09-24 18:52:01 /<br>
2015-09-24 19:10:22 /</font><font size="3"><br>
</font><font face="sans-serif" size="2"><br>
Brick omdx1b51:/static/content<br>
Number of entries: 42<br>
at path
on brick<br>
-----------------------------------<br>
2015-09-24 18:51:58 /<br>
2015-09-24 18:51:59 /<br>
2015-09-24 19:01:59 /<br>
2015-09-24 19:11:59 /</font><font size="3"><br>
<br>
</font><font face="sans-serif" size="2"><br>
Being / on the same replicate, how would I safely resolve this
issue? Is
it really going to require me to delete the root of each node
and heal?
I hope not, the entire volume is about 1TB.</font><font
size="3"><br>
</font><br>
<font size="3">No, it is likely that the root is only in
metadata split-brain.
What does the getfattr output of '/' show on the bricks? </font><a
moz-do-not-send="true"
href="https://github.com/gluster/glusterdocs/blob/master/Troubleshooting/split-brain.md"><font
size="3" color="blue"><u><a class="moz-txt-link-freetext" href="https://github.com/gluster/glusterdocs/blob/master/Troubleshooting/split-brain.md">https://github.com/gluster/glusterdocs/blob/master/Troubleshooting/split-brain.md</a></u></font></a><font
size="3">should tell you how to resolve split-brains.</font><br>
<font face="sans-serif" size="2">Thank you,</font><font size="3"><br>
<br>
<br>
</font><font face="sans-serif" size="2"><br>
<br>
<br>
Khoi Mai<br>
Union Pacific Railroad<br>
Distributed Engineering & Architecture<br>
Senior Project Engineer<br>
</font><font size="3"><br>
<br>
<br>
<br>
**<br>
<br>
<br>
<br>
This email and any attachments may contain information that is
confidential
and/or privileged for the sole use of the intended recipient.
Any use,
review, disclosure, copying, distribution or reliance by
others, and any
forwarding of this email or its contents, without the express
permission
of the sender is strictly prohibited by law. If you are not
the intended
recipient, please contact the sender immediately, delete the
e-mail and
destroy all copies.<br>
<br>
**<br>
<br>
</font><br>
<tt><font size="3">_______________________________________________<br>
Gluster-users mailing list<br>
</font></tt><a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org"><tt><font size="3"
color="blue"><u>Gluster-users@gluster.org</u></font></tt></a><tt><font
size="3"><br>
</font></tt><a moz-do-not-send="true"
href="http://www.gluster.org/mailman/listinfo/gluster-users"><tt><font
size="3" color="blue"><u>http://www.gluster.org/mailman/listinfo/gluster-users</u></font></tt></a><br>
<br>
<br>
<br>
<br>
**<br>
<br>
<br>
<br>
This email and any attachments may contain information that is
confidential and/or privileged for the sole use of the intended
recipient. Any use, review, disclosure, copying, distribution or
reliance by others, and any forwarding of this email or its
contents, without the express permission of the sender is
strictly prohibited by law. If you are not the intended
recipient, please contact the sender immediately, delete the
e-mail and destroy all copies.<br>
<br>
**<br>
</p>
</blockquote>
<br>
</body>
</html>