<div dir="ltr"><div><div>Lindsay,<br><br></div>Could you send logs from your setup: bricks, shds and client logs. Also send in the gfids of these 8 shards.<br><br></div>-Krutika<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 25, 2016 at 3:27 AM, Paul Cuzner <span dir="ltr"><<a href="mailto:pcuzner@redhat.com" target="_blank">pcuzner@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><br></div><br></div>Just wondering how shards can silently be different across bricks in a replica? Lindsay caught this issue due to her due diligence taking on 'new' tech - and resolved the inconsistency, but tbh this shouldn't be an admin's job :(<br><br><br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Sun, Apr 24, 2016 at 7:06 PM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr"><div>OK. Under normal circumstances it should have been possible to heal a single file by issuing a lookup on it (==> stat on the file from the mountpoint). But with shards this won't work. We take care not to expose /.shard on the mountpoint, and as a result any attempt to issue lookup on a shard from the mountpoint will be met with an 'operation not permitted' error.<span><font color="#888888"><br><br></font></span></div><span><font color="#888888">-Krutika<br></font></span></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Apr 24, 2016 at 11:42 AM, Lindsay Mathieson <span dir="ltr"><<a href="mailto:lindsay.mathieson@gmail.com" target="_blank">lindsay.mathieson@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On 24/04/2016 2:56 PM, Krutika Dhananjay wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Nope, it's not necessary for them to all have the xattr.<br>
</blockquote>
<br></span>
Thats good :)<span><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Do you see anything at least in .glusterfs/indices/dirty on all bricks?<br>
</blockquote>
<br></span>
I checked, dirty dir empty on all bricks<br>
<br>
I used diff3 to compare the checksums of the shards and it revealed that seven of the shards were the same on two bricks (vna & vng) and one of the shards was the same on two other bricks (vna & vnb). Fortunately none were different on all 3 bricks :)<br>
<br>
Using the checksum as a quorum I deleted all the singleton shards (7 on vnb, 1 on vng), touched the file owner and issule a "heal full". All 8 shards were restored with matching checksums for the other two bricks. A rechack of the entire set of shards for the vm showed all 3 copies as identical and the VM itself is functioning normally.<br>
<br>
Its one way to manually heal up shard mismatches which gluster hasn't detected, if somewhat tedious. Its a method which lends itself to automation though.<br>
<br>
<br>
Cheers,<span><font color="#888888"><br>
<br>
<br>
-- <br>
Lindsay Mathieson<br>
<br>
</font></span></blockquote></div><br></div>
</div></div><br></div></div><span class="">_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></span></blockquote></div><br></div>
</blockquote></div><br></div>