<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Am testing replacing the brick in a replica 3 test volume.
Gluster 3.7.11. Volume hosts two VM's. 3 Nodes, vna, vnb and vng.<br>
</p>
<p><br>
</p>
<p><b>First off I tried removing/adding a brick.</b></p>
<p><tt> gluster v remove-brick replica 2
vng.proxmox.softlog:/tank/vmdata/test1 force.</tt></p>
<p>That worked fine, VM's (on another node) kept running without a
hiccup<br>
</p>
<br>
I deleted /tank/vmdata/test1, then<br>
<br>
<tt> gluster v add-brick replica 3
vng.proxmox.softlog:/tank/vmdata/test1 force.</tt><br>
<br>
<br>
Succeeded and heal statistics immediatly showed 3000+ shards being
healed on vna and vnb<br>
<br>
Unfortunately it also show 100's of sharded being healed on vng,
which should not be happening as it had no data on it. Reverse heal
basically.<br>
<br>
Eventually all the heals completed, but the VM's were hopeless
ccorrupted.<br>
<br>
<b>Then I retried the above, but with all VM's shutdown</b><br>
i.e, no writes or reads happening on the volume.<br>
<br>
This worked - i.e all the shards on vna & vnb healed, nothing in
reverse. Once completed the data (VM's) was fine.<br>
<br>
Unfortunately this isn't practical in production - can' bring all
the VM's down for the 1-2 days it would take to heal.<br>
<br>
<br>
<b>Replacing the brick<br>
<br>
</b>I tried<br>
<br>
killed the glusterfsd process on vng, then<br>
<tt> gluster v replace-brick test1 </tt><tt>vng.proxmox.softlog:/tank/vmdata/test1
</tt><tt>vng.proxmox.softlog:/tank/vmdata/test1.1 commit force</tt><br>
<b><br>
</b>vna & vnb shards started healing, but vng showed 5 reverse
heals happening.<br>
Eventually it got down to 4-5 shards needing healing on each brick
and stopped. They didn't go away till I removed the test1.1 brick.<br>
<b><br>
<br>
</b>Currently the replace brick processes seems to be unusable
except when the volume is not being used.<br>
<br>
<pre class="moz-signature" cols="72">--
Lindsay Mathieson</pre>
</body>
</html>