<div dir="ltr">Oops! Accidentally ran the command as non-root on Curacao, that's why there was no output. The actual output is:<br><div><br></div><div>curacao#<span class="s2" style="line-height:1.5;font-size:13.1999998092651px"> getfattr -m . -d -e hex /export/sdb1/data/Case/21000355/studies.dat</span></div><div><span style="line-height:1.5;font-size:13.1999998092651px">getfattr: Removing leading '/' from absolute path names</span></div><div><span style="line-height:1.5;font-size:13.1999998092651px"># file: export/sdb1/data/Case/21000355/studies.dat</span></div><div><span style="line-height:1.5;font-size:13.1999998092651px">trusted.afr.data-client-0=0x000000000000000000000000</span></div><div><span style="line-height:1.5;font-size:13.1999998092651px">trusted.afr.data-client-1=0x000000000000000000000000</span></div><div><span style="line-height:1.5;font-size:13.1999998092651px">trusted.afr.dirty=0x000000000000000000000000</span></div><div><span style="line-height:1.5;font-size:13.1999998092651px">trusted.gfid=0xfb34574974cf4804b8b80789738c0f81</span></div><div><span style="line-height:1.5;font-size:13.1999998092651px"><br></span></div><div><span style="line-height:1.5;font-size:13.1999998092651px">For reference, the output on bonaire:</span></div><div><span style="line-height:1.5;font-size:13.1999998092651px"><br></span></div><div><span style="line-height:1.5;font-size:13.1999998092651px">bonaire# </span><span style="line-height:1.5;font-size:13.1999998092651px">getfattr -m . -d -e hex /export/sdb1/data/Case/21000355/studies.dat</span></div>
<div><span style="font-size:13.1999998092651px;line-height:1.5">getfattr: Removing leading '/' from absolute path names</span></div><div><span style="line-height:1.5;font-size:13.1999998092651px"># file: export/sdb1/data/Case/21000355/studies.dat</span></div><div><span style="line-height:1.5;font-size:13.1999998092651px">trusted.gfid=0xfb34574974cf4804b8b80789738c0f81</span></div></div><br><div class="gmail_quote"><div dir="ltr">Op zo 7 jun. 2015 om 21:13 schreef Sjors Gielen <<a href="mailto:sjors@sjorsgielen.nl">sjors@sjorsgielen.nl</a>>:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I'm reading about quorums, I haven't set up anything like that yet.</div><div><br></div><div>(In reply to Joe Julian, who responded off-list)</div><div><br></div>The output of getfattr on bonaire:<div><br></div><div>bonaire# getfattr -m . -d -e hex /export/sdb1/data/Case/21000355/studies.dat</div><div>getfattr: Removing leading '/' from absolute path names</div><div># file: export/sdb1/data/Case/21000355/studies.dat</div><div>trusted.gfid=0xfb34574974cf4804b8b80789738c0f81</div><div><br></div><div>On curacao, the command gives no output.</div><div><br></div><div>From `gluster volume status`, it seems that while the "brick curacao:/export/sdb1/data" is online, it has no associated port number. Curacao can connect to the port number provided by Bonaire just fine. There are no firewalls on/between the two machines, they are on the same subnet connected by Ethernet cables and two switches.</div><div><br></div><div>By the way, warning messages just started appearing to /var/log/glusterfs/bricks/export-sdb1-data.log on Bonaire saying "mismatching ino/dev between file X and handle Y", though, maybe only just now even though I started the full self-heal hours ago.</div><div><br></div><div>[2015-06-07 19:10:39.624393] W [posix-handle.c:727:posix_handle_hard] 0-data-posix: mismatching ino/dev between file /export/sdb1/data/Archive/S21/21008971/studies.dat (9127104621/2065) and handle /export/sdb1/data/.glusterfs/97/c2/97c2a65d-36e0-4566-a5c1-5925f97af1fd (9190215976/2065)</div><div><br></div><div>Thanks again!</div><div>Sjors</div>
<div><br><div class="gmail_quote"><div dir="ltr">Op zo 7 jun. 2015 om 19:13 schreef Sjors Gielen <<a href="mailto:sjors@sjorsgielen.nl" target="_blank">sjors@sjorsgielen.nl</a>>:<br></div></div></div></div><div dir="ltr"><div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi all,<div><br></div><div>I work at a small, 8-person company that uses Gluster for its primary data storage. We have a volume called "data" that is replicated over two servers (details below). This worked perfectly for over a year, but lately we've been noticing some mismatches between the two bricks, so it seems there has been some split-brain situation that is not being detected or resolved. I have two questions about this:</div><div><br></div><div>1) I expected Gluster to (eventually) detect a situation like this; why doesn't it?</div><div>2) How do I fix this situation? I've tried an explicit 'heal', but that didn't seem to change anything.</div><div><br></div><div>Thanks a lot for your help!</div><div>Sjors</div><div><br></div><div>------8<------</div><div><br></div><div>Volume & peer info: <a href="http://pastebin.com/PN7tRXdU" target="_blank">http://pastebin.com/PN7tRXdU</a></div><div>curacao# md5sum /export/sdb1/data/Case/21000355/studies.dat</div><div>7bc2daec6be953ffae920d81fe6fa25c</div><div>/export/sdb1/data/Case/21000355/studies.dat<br></div><div>bonaire# md5sum /export/sdb1/data/Case/21000355/studies.dat</div>
<div>28c950a1e2a5f33c53a725bf8cd72681 /export/sdb1/data/Case/21000355/studies.dat<br></div><div><br></div><div># mallorca is one of the clients</div><div>mallorca# md5sum /data/Case/21000355/studies.dat</div><div>7bc2daec6be953ffae920d81fe6fa25c /data/Case/21000355/studies.dat</div><div><br></div><div>I expected an input/output error after reading this file, because of the split-brain situation, but got none. There are no entries in the GlusterFS logs of either bonaire or curacao.</div><div><br></div><div>bonaire# gluster volume heal data full</div><div>Launching heal operation to perform full self heal on volume data has been successful</div><div>Use heal info commands to check status</div><div>bonaire# gluster volume heal data info</div><div>Brick bonaire:/export/sdb1/data/</div><div>Number of entries: 0<br></div><div><br></div><div>Brick curacao:/export/sdb1/data/</div><div>Number of entries: 0</div><div><br></div><div>(Same output on curacao, and hours after this, the md5sums on both bricks still differ.)</div><div><br></div><div>curacao# gluster --version</div><div>glusterfs 3.6.2 built on Mar 2 2015 14:05:34</div><div>Repository revision: git://<a href="http://git.gluster.com/glusterfs.git" target="_blank">git.gluster.com/glusterfs.git</a></div><div>(Same version on Bonaire)</div>
</div></blockquote></div></div></div></blockquote></div>