<div dir="ltr"><div>OK. Under normal circumstances it should have been possible to heal a single file by issuing a lookup on it (==> stat on the file from the mountpoint). But with shards this won't work. We take care not to expose /.shard on the mountpoint, and as a result any attempt to issue lookup on a shard from the mountpoint will be met with an 'operation not permitted' error.<br><br></div>-Krutika<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Apr 24, 2016 at 11:42 AM, Lindsay Mathieson <span dir="ltr"><<a href="mailto:lindsay.mathieson@gmail.com" target="_blank">lindsay.mathieson@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 24/04/2016 2:56 PM, Krutika Dhananjay wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Nope, it's not necessary for them to all have the xattr.<br>
</blockquote>
<br></span>
Thats good :)<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Do you see anything at least in .glusterfs/indices/dirty on all bricks?<br>
</blockquote>
<br></span>
I checked, dirty dir empty on all bricks<br>
<br>
I used diff3 to compare the checksums of the shards and it revealed that seven of the shards were the same on two bricks (vna & vng) and one of the shards was the same on two other bricks (vna & vnb). Fortunately none were different on all 3 bricks :)<br>
<br>
Using the checksum as a quorum I deleted all the singleton shards (7 on vnb, 1 on vng), touched the file owner and issule a "heal full". All 8 shards were restored with matching checksums for the other two bricks. A rechack of the entire set of shards for the vm showed all 3 copies as identical and the VM itself is functioning normally.<br>
<br>
Its one way to manually heal up shard mismatches which gluster hasn't detected, if somewhat tedious. Its a method which lends itself to automation though.<br>
<br>
<br>
Cheers,<span class="HOEnZb"><font color="#888888"><br>
<br>
<br>
-- <br>
Lindsay Mathieson<br>
<br>
</font></span></blockquote></div><br></div>