<div dir="ltr">Ah, that's really weird. I'm pretty sure that nothing ever made write changes to /export on either machine, so I wonder how the hard links ended up being split. I'll indeed clean up the .glusterfs directory and keep close tabs on Gluster's repair.<br><div><br></div><div>Glustershd.log and the client mount logs (data.log and gluster.log at least) on the client are empty and nothing appears when I read the mismatching studies.dat file.</div><div><br></div><div>Thanks for your help!</div><div>Sjors</div></div><br><div class="gmail_quote"><div dir="ltr">Op zo 7 jun. 2015 om 22:10 schreef Joe Julian <<a href="mailto:joe@julianfamily.org">joe@julianfamily.org</a>>:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
(oops... I hate when I reply off-list)<br>
<br>
That warning should, imho, be an error. That's saying that the
handle, which should be a hardlink to the file, doesn't have a
matching inode. It should if it's a hardlink.<br>
<br>
If it were me, I would:<br>
<br>
find /export/sdb1/data/.glusterfs -type f -links 1 -print0 |
xargs /bin/rm<br>
<br>
This would clean up any handles that are not hardlinked where they
should be and will allow gluster to repair them.<br>
<br>
Btw, the self-heal errors would be in glustershd.log and/or the
client mount log(s), not (usually) the brick logs.</div><div bgcolor="#FFFFFF" text="#000000"><br>
<br>
<div>On 06/07/2015 12:21 PM, Sjors Gielen
wrote:<br>
</div>
</div><div bgcolor="#FFFFFF" text="#000000"><blockquote type="cite">
<div dir="ltr">Oops! Accidentally ran the command as non-root on
Curacao, that's why there was no output. The actual output is:<br>
<div><br>
</div>
<div>curacao#<span style="line-height:1.5;font-size:13.1999998092651px"> getfattr
-m . -d -e hex /export/sdb1/data/Case/21000355/studies.dat</span></div>
<div><span style="line-height:1.5;font-size:13.1999998092651px">getfattr:
Removing leading '/' from absolute path names</span></div>
<div><span style="line-height:1.5;font-size:13.1999998092651px">#
file: export/sdb1/data/Case/21000355/studies.dat</span></div>
<div><span style="line-height:1.5;font-size:13.1999998092651px">trusted.afr.data-client-0=0x000000000000000000000000</span></div>
<div><span style="line-height:1.5;font-size:13.1999998092651px">trusted.afr.data-client-1=0x000000000000000000000000</span></div>
<div><span style="line-height:1.5;font-size:13.1999998092651px">trusted.afr.dirty=0x000000000000000000000000</span></div>
<div><span style="line-height:1.5;font-size:13.1999998092651px">trusted.gfid=0xfb34574974cf4804b8b80789738c0f81</span></div>
<div><span style="line-height:1.5;font-size:13.1999998092651px"><br>
</span></div>
<div><span style="line-height:1.5;font-size:13.1999998092651px">For
reference, the output on bonaire:</span></div>
<div><span style="line-height:1.5;font-size:13.1999998092651px"><br>
</span></div>
<div><span style="line-height:1.5;font-size:13.1999998092651px">bonaire# </span><span style="line-height:1.5;font-size:13.1999998092651px">getfattr
-m . -d -e hex /export/sdb1/data/Case/21000355/studies.dat</span></div>
<div><span style="font-size:13.1999998092651px;line-height:1.5">getfattr:
Removing leading '/' from absolute path names</span></div>
<div><span style="line-height:1.5;font-size:13.1999998092651px">#
file: export/sdb1/data/Case/21000355/studies.dat</span></div>
<div><span style="line-height:1.5;font-size:13.1999998092651px">trusted.gfid=0xfb34574974cf4804b8b80789738c0f81</span></div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">Op zo 7 jun. 2015 om 21:13 schreef Sjors Gielen
<<a href="mailto:sjors@sjorsgielen.nl" target="_blank">sjors@sjorsgielen.nl</a>>:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>I'm reading about quorums, I haven't set up anything
like that yet.</div>
<div><br>
</div>
<div>(In reply to Joe Julian, who responded off-list)</div>
<div><br>
</div>
The output of getfattr on bonaire:
<div><br>
</div>
<div>bonaire# getfattr -m . -d -e hex
/export/sdb1/data/Case/21000355/studies.dat</div>
<div>getfattr: Removing leading '/' from absolute path names</div>
<div># file: export/sdb1/data/Case/21000355/studies.dat</div>
<div>trusted.gfid=0xfb34574974cf4804b8b80789738c0f81</div>
<div><br>
</div>
<div>On curacao, the command gives no output.</div>
<div><br>
</div>
<div>From `gluster volume status`, it seems that while the
"brick curacao:/export/sdb1/data" is online, it has no
associated port number. Curacao can connect to the port
number provided by Bonaire just fine. There are no
firewalls on/between the two machines, they are on the
same subnet connected by Ethernet cables and two switches.</div>
<div><br>
</div>
<div>By the way, warning messages just started appearing to
/var/log/glusterfs/bricks/export-sdb1-data.log on Bonaire
saying "mismatching ino/dev between file X and handle Y",
though, maybe only just now even though I started the full
self-heal hours ago.</div>
<div><br>
</div>
<div>[2015-06-07 19:10:39.624393] W
[posix-handle.c:727:posix_handle_hard] 0-data-posix:
mismatching ino/dev between file
/export/sdb1/data/Archive/S21/21008971/studies.dat
(9127104621/2065) and handle
/export/sdb1/data/.glusterfs/97/c2/97c2a65d-36e0-4566-a5c1-5925f97af1fd
(9190215976/2065)</div>
<div><br>
</div>
<div>Thanks again!</div>
<div>Sjors</div>
<div><br>
<div class="gmail_quote">
<div dir="ltr">Op zo 7 jun. 2015 om 19:13 schreef Sjors
Gielen <<a href="mailto:sjors@sjorsgielen.nl" target="_blank">sjors@sjorsgielen.nl</a>>:<br>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi all,
<div><br>
</div>
<div>I work at a small, 8-person company that uses
Gluster for its primary data storage. We have a
volume called "data" that is replicated over two
servers (details below). This worked perfectly for
over a year, but lately we've been noticing some
mismatches between the two bricks, so it seems
there has been some split-brain situation that is
not being detected or resolved. I have two
questions about this:</div>
<div><br>
</div>
<div>1) I expected Gluster to (eventually) detect a
situation like this; why doesn't it?</div>
<div>2) How do I fix this situation? I've tried an
explicit 'heal', but that didn't seem to change
anything.</div>
<div><br>
</div>
<div>Thanks a lot for your help!</div>
<div>Sjors</div>
<div><br>
</div>
<div>------8<------</div>
<div><br>
</div>
<div>Volume & peer info: <a href="http://pastebin.com/PN7tRXdU" target="_blank">http://pastebin.com/PN7tRXdU</a></div>
<div>curacao# md5sum
/export/sdb1/data/Case/21000355/studies.dat</div>
<div>7bc2daec6be953ffae920d81fe6fa25c</div>
<div>/export/sdb1/data/Case/21000355/studies.dat<br>
</div>
<div>bonaire# md5sum
/export/sdb1/data/Case/21000355/studies.dat</div>
<div>28c950a1e2a5f33c53a725bf8cd72681
/export/sdb1/data/Case/21000355/studies.dat<br>
</div>
<div><br>
</div>
<div># mallorca is one of the clients</div>
<div>mallorca# md5sum
/data/Case/21000355/studies.dat</div>
<div>7bc2daec6be953ffae920d81fe6fa25c
/data/Case/21000355/studies.dat</div>
<div><br>
</div>
<div>I expected an input/output error after reading
this file, because of the split-brain situation,
but got none. There are no entries in the
GlusterFS logs of either bonaire or curacao.</div>
<div><br>
</div>
<div>bonaire# gluster volume heal data full</div>
<div>Launching heal operation to perform full self
heal on volume data has been successful</div>
<div>Use heal info commands to check status</div>
<div>bonaire# gluster volume heal data info</div>
<div>Brick bonaire:/export/sdb1/data/</div>
<div>Number of entries: 0<br>
</div>
<div><br>
</div>
<div>Brick curacao:/export/sdb1/data/</div>
<div>Number of entries: 0</div>
<div><br>
</div>
<div>(Same output on curacao, and hours after this,
the md5sums on both bricks still differ.)</div>
<div><br>
</div>
<div>curacao# gluster --version</div>
<div>glusterfs 3.6.2 built on Mar 2 2015 14:05:34</div>
<div>Repository revision: git://<a href="http://git.gluster.com/glusterfs.git" target="_blank">git.gluster.com/glusterfs.git</a></div>
<div>(Same version on Bonaire)</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<fieldset></fieldset>
<br>
</blockquote></div><div bgcolor="#FFFFFF" text="#000000"><blockquote type="cite"><pre>_______________________________________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div>