<html><body><div style="font-family: garamond,new york,times,serif; font-size: 12pt; color: #000000"><div><br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Lindsay Mathieson" <lindsay.mathieson@gmail.com><br><b>To: </b>"gluster-users" <gluster-users@gluster.org><br><b>Sent: </b>Monday, January 18, 2016 11:19:22 AM<br><b>Subject: </b>[Gluster-users] Issues removing then adding a brick to a replica volume (Gluster 3.7.6)<br><div><br></div>
Been running through my eternal testing regime ... and experimenting
with removing/adding bricks - to me, a necessary part of volume
maintenance for dealing with failed disks. The datastore is a VM
host and all the following is done live. Sharding is active with a
512MB shard size.<br><br>
So I started off with a replica 3 volume<br><br><blockquote><tt>// recreated from memory</tt><br><tt>Volume Name: datastore1</tt><br><tt>Type: Replicate</tt><br><tt>Volume ID: bf882533-f1a9-40bf-a13e-d26d934bfa8b</tt><br><tt>Status: Started</tt><br><tt>Number of Bricks: 1 x 3 = 3</tt><br><tt>Transport-type: tcp</tt><br><tt>Bricks:</tt><br><tt>Brick1: vnb.proxmox.softlog:/vmdata/datastore1</tt><br><tt>Brick2: vng.proxmox.softlog:/vmdata/datastore1</tt><br><tt>Brick3: vna.proxmox.softlog:/vmdata/datastore1</tt><br></blockquote><br><br>
I remove a brick with:<br><br><tt>gluster volume remove-brick datastore1 replica 2
vng.proxmox.softlog:/vmdata/datastore1 force</tt><br><br>
so we end up with:<br><br><blockquote><tt>Volume Name: datastore1</tt><br><tt>Type: Replicate</tt><br><tt>Volume ID: bf882533-f1a9-40bf-a13e-d26d934bfa8b</tt><br><tt>Status: Started</tt><br><tt>Number of Bricks: 1 x 2 = 2</tt><br><tt>Transport-type: tcp</tt><br><tt>Bricks:</tt><br><tt>Brick1: vna.proxmox.softlog:/vmdata/datastore1</tt><br><tt>Brick2: vnb.proxmox.softlog:/vmdata/datastore1</tt><br></blockquote><br><br>
All well and good. No heal issues, VM's running ok.<br><br>
Then I clean the brick off the vng host:<br><br><tt>rm -rf /vmdata/datastore1<br>
<br>
<br>
</tt>I then add the brick back with:<br><br><blockquote><tt>gluster volume add-brick datastore1 replica 3
vng.proxmox.softlog:/vmdata/datastore1 <br>
<br>
Volume Name: datastore1<br>
Type: Replicate<br>
Volume ID: bf882533-f1a9-40bf-a13e-d26d934bfa8b<br>
Status: Started<br>
Number of Bricks: 1 x 3 = 3<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: vna.proxmox.softlog:/vmdata/datastore1<br>
Brick2: vnb.proxmox.softlog:/vmdata/datastore1<br>
Brick3: vng.proxmox.softlog:/vmdata/datastore1<br>
</tt></blockquote><tt><br>
<br>
</tt>This recreates the brick directory "datastore1". Unfortunately
this is where things start to go wrong :( Heal info:<br><br><blockquote><tt>gluster volume heal datastore1 info</tt><br><tt>Brick vna.proxmox.softlog:/vmdata/datastore1</tt><br><tt>/.shard/d6aad699-d71d-4b35-b021-d35e5ff297c4.57 </tt><br><tt>/.shard/d6aad699-d71d-4b35-b021-d35e5ff297c4.5 </tt><br><tt>Number of entries: 2</tt><br><br><tt>Brick vnb.proxmox.softlog:/vmdata/datastore1</tt><br><tt>/.shard/d6aad699-d71d-4b35-b021-d35e5ff297c4.5 </tt><br><tt>/.shard/d6aad699-d71d-4b35-b021-d35e5ff297c4.57 </tt><br><tt>Number of entries: 2</tt><br><br><tt>Brick vng.proxmox.softlog:/vmdata/datastore1</tt><br><tt>/.shard/d6aad699-d71d-4b35-b021-d35e5ff297c4.1 </tt><br><tt>/.shard/d6aad699-d71d-4b35-b021-d35e5ff297c4.6 </tt><br><tt>/.shard/d6aad699-d71d-4b35-b021-d35e5ff297c4.15 </tt><br><tt>/.shard/d6aad699-d71d-4b35-b021-d35e5ff297c4.18 </tt><br><tt>/.shard/d6aad699-d71d-4b35-b021-d35e5ff297c4.5 </tt><br></blockquote><br>
Its my understanding that there shouldn't be any heal entries on vng
as it that is where all the shards should be sent *to*</blockquote><div><br></div><div>Lindsay,<br></div><div><br></div><div>Heal _is_ necessary when you add a brick that changes the replica count from n to (n+1). Now the new brick that is also part of the existing replica set is lagging with respect to the existing bricks<br></div><div>and needs to be brought in sync with these. All files and directories in vna and/or vnb will be healed to vng in your case.<br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><br><br>
also running qemu-img check on the hosted VM images results in a I/O
error. Eventually the VM's themselves crash - I suspect this is due
to individual shards being unreadable.<br><br>
Another odd behaviour I get is if I run a full heal on vnb I get the
following error:<br><br><blockquote><tt>Launching heal operation to perform full self heal
on volume datastore1 has been unsuccessful</tt><br></blockquote><br>
However if I run it on VNA, it succeeds.</blockquote><div><br></div><div>Yes, there is a bug report for this @ <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1112158">https://bugzilla.redhat.com/show_bug.cgi?id=1112158</a>.</div><div>The workaround, like you yourself figured, is to run the command on the node with the highest uuid.</div><div>Steps:</div><div>1) Collect output of `cat /var/lib/glusterd/glusterd.info | grep UUID` from each of the nodes<span style="font-family: garamond,new york,times,serif; font-size: small;" data-mce-style="font-family: garamond,new york,times,serif; font-size: small;"><span style="color: #000000; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; display: inline ! important; float: none; background-color: #fdfdfd;" data-mce-style="color: #000000; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; display: inline ! important; float: none; background-color: #fdfdfd;"></span><span class="Object" id="OBJ_PREFIX_DWT125_com_zimbra_url" style="color: #336699; text-decoration: none; cursor: pointer; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; background-color: #fdfdfd;" data-mce-style="color: #336699; text-decoration: none; cursor: pointer; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; background-color: #fdfdfd;"></span><span style="color: #000000; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; display: inline ! important; float: none; background-color: #fdfdfd;" data-mce-style="color: #000000; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; display: inline ! important; float: none; background-color: #fdfdfd;"><span class="Apple-converted-space"></span></span></span>, perhaps into a file named 'uuid.txt'.<br></div><div>2) cat uuid.txt | sort<br></div><div>3) Pick the last gfid.<br></div><div>4) find out which of the glusterd.info files has the same uuid as this selected uuid.<br></div><div>5) Run 'heal info full' on that same node.<br></div><div><br></div><div>Let me know if this works for you.<br></div><div><br></div><div>-Krutika<br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><br><br><br>
Lastly - if I remove the brick everythign returns to normal
immediately. Heal Info shows no issues and qemu-img check returns no
errors.<br><br><br><br><tt><br>
</tt><pre class="moz-signature">--
Lindsay Mathieson</pre><br>_______________________________________________<br>Gluster-users mailing list<br>Gluster-users@gluster.org<br>http://www.gluster.org/mailman/listinfo/gluster-users</blockquote><div><br></div></div></body></html>