<div dir="ltr">Hi all,<div><br></div><div>I work at a small, 8-person company that uses Gluster for its primary data storage. We have a volume called "data" that is replicated over two servers (details below). This worked perfectly for over a year, but lately we've been noticing some mismatches between the two bricks, so it seems there has been some split-brain situation that is not being detected or resolved. I have two questions about this:</div><div><br></div><div>1) I expected Gluster to (eventually) detect a situation like this; why doesn't it?</div><div>2) How do I fix this situation? I've tried an explicit 'heal', but that didn't seem to change anything.</div><div><br></div><div>Thanks a lot for your help!</div><div>Sjors</div><div><br></div><div>------8<------</div><div><br></div><div>Volume & peer info: <a href="http://pastebin.com/PN7tRXdU">http://pastebin.com/PN7tRXdU</a></div><div>curacao# md5sum /export/sdb1/data/Case/21000355/studies.dat</div><div>7bc2daec6be953ffae920d81fe6fa25c</div><div>/export/sdb1/data/Case/21000355/studies.dat<br></div><div>bonaire# md5sum /export/sdb1/data/Case/21000355/studies.dat</div>
<div>28c950a1e2a5f33c53a725bf8cd72681 /export/sdb1/data/Case/21000355/studies.dat<br></div><div><br></div><div># mallorca is one of the clients</div><div>mallorca# md5sum /data/Case/21000355/studies.dat</div><div>7bc2daec6be953ffae920d81fe6fa25c /data/Case/21000355/studies.dat</div><div><br></div><div>I expected an input/output error after reading this file, because of the split-brain situation, but got none. There are no entries in the GlusterFS logs of either bonaire or curacao.</div><div><br></div><div>bonaire# gluster volume heal data full</div><div>Launching heal operation to perform full self heal on volume data has been successful</div><div>Use heal info commands to check status</div><div>bonaire# gluster volume heal data info</div><div>Brick bonaire:/export/sdb1/data/</div><div>Number of entries: 0<br></div><div><br></div><div>Brick curacao:/export/sdb1/data/</div><div>Number of entries: 0</div><div><br></div><div>(Same output on curacao, and hours after this, the md5sums on both bricks still differ.)</div><div><br></div><div>curacao# gluster --version</div><div>glusterfs 3.6.2 built on Mar 2 2015 14:05:34</div><div>Repository revision: git://<a href="http://git.gluster.com/glusterfs.git">git.gluster.com/glusterfs.git</a></div><div>(Same version on Bonaire)</div>
</div>