<html><body><div style="font-family: garamond,new york,times,serif; font-size: 12pt; color: #000000"><div>Hi,</div><div><br></div><div>So I looked at the glustershd logs and there are no log messages that indicate that there was a reverse heal.<br></div><div>For instance, see this:<br></div><div>[2016-01-22 06:13:50.268068] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-datastore1-replicate-0: Completed data selfheal on 5a3dcdd1-8866-47a5-8432-7b625a2806c3. source=0 sinks=2 </div><div><br></div><div>The above implies that the sink was "2" (index of the sink brick - vng - counting from 0), and the source was "0" (vnb in this case), which means the heal happened into the correct brick (last brick).<br></div><div><br></div><div>Could you do the following:<br></div><div>1) Disable client-side healing:<br></div><div> For this, you need to execute<br></div><div> # gluster volume set datastore1 entry-self-heal off<br></div><div> # gluster volume set datastore1 data-self-heal off<br></div><div> # gluster volume set datastore1 metadata-self-heal off<br></div><div><br></div><div>2) Run your add-brick/remove-brick tests again (you know it best).<br></div><div>3) Share the resultant glustershd.log from all three machines.<br></div><div><br></div><div>-Krutika<br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Lindsay Mathieson" <lindsay.mathieson@gmail.com><br><b>To: </b>"Krutika Dhananjay" <kdhananj@redhat.com>, "gluster-users" <gluster-users@gluster.org><br><b>Sent: </b>Friday, January 22, 2016 11:55:27 AM<br><b>Subject: </b>Re: [Gluster-users] File Corruption when adding bricks to live replica volumes<br><div><br></div>
<div class="moz-cite-prefix">On 21/01/16 20:03, Krutika Dhananjay
wrote:<br>
</div>
<blockquote cite="mid:1986181672.7299488.1453370613211.JavaMail.zimbra@redhat.com">
<div>Also could you please attach glustershd.log files and the
client logs?<br>
</div>
<div><br>
</div>
</blockquote>
<br>
Ok, I ran the procedure again just to be sure. This time I got 81
shards being healed from the other bricks, but still got 2 bricks
being "reverse healed" from the new brick. A image check on the vm
file failed with:<br>
<br>
<blockquote><tt>ERROR: I/O error in check_refcounts_l2</tt><tt><br>
</tt><tt>qemu-img: Check failed: Input/output error</tt><tt><br>
</tt></blockquote>
<br>
removing the brick resolved the problems.<br>
<br>
glustershd.log from node vng attached (the new brick node), the fuse
client log was empty.<br>
<br>
<br>
thanks,<br>
<pre class="moz-signature">--
Lindsay Mathieson</pre>
</blockquote><div><br></div></div></body></html>