<div dir="ltr"><div class="gmail_default" style="font-size:small">To answer my own question. I now realize the importance of creating specifically directories and not files because it appears that directories are created on each pair regardless of the hash. And so, if a host is down, changes will be marked as pending for that host. With files it's different because in that case the name of the file <b>IS </b>important resulting in the file being placed on only 1 pair.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 10, 2016 at 3:47 PM, Sergei Gerasenko <span dir="ltr"><<a href="mailto:sgerasenko74@gmail.com" target="_blank">sgerasenko74@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-size:small">The guide here: <a href="https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Managing%20Volumes/#replace-faulty-brick" target="_blank">https://gluster.readthed<wbr>ocs.io/en/latest/Administrator<wbr>%20Guide/Managing%20Volumes/#<wbr>replace-faulty-brick</a> suggests running the following while the partner host is down:</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small"><span class="m_-1905138536765705426gmail-hljs-keyword" style="box-sizing:border-box;color:rgb(51,51,51);font-weight:bold;font-family:consolas,"andale mono wt","andale mono","lucida console","lucida sans typewriter","dejavu sans mono","bitstream vera sans mono","liberation mono","nimbus mono l",monaco,"courier new",courier,monospace;font-size:10.8px;white-space:pre-wrap">mkdir</span><span style="color:rgb(51,51,51);font-family:consolas,"andale mono wt","andale mono","lucida console","lucida sans typewriter","dejavu sans mono","bitstream vera sans mono","liberation mono","nimbus mono l",monaco,"courier new",courier,monospace;font-size:10.8px;white-space:pre-wrap"> /mnt/r2/<name-of-nonexistent-</span><span class="m_-1905138536765705426gmail-hljs-keyword" style="box-sizing:border-box;color:rgb(51,51,51);font-weight:bold;font-family:consolas,"andale mono wt","andale mono","lucida console","lucida sans typewriter","dejavu sans mono","bitstream vera sans mono","liberation mono","nimbus mono l",monaco,"courier new",courier,monospace;font-size:10.8px;white-space:pre-wrap">d<wbr>ir</span><span style="color:rgb(51,51,51);font-family:consolas,"andale mono wt","andale mono","lucida console","lucida sans typewriter","dejavu sans mono","bitstream vera sans mono","liberation mono","nimbus mono l",monaco,"courier new",courier,monospace;font-size:10.8px;white-space:pre-wrap">>
</span><span class="m_-1905138536765705426gmail-hljs-keyword" style="box-sizing:border-box;color:rgb(51,51,51);font-weight:bold;font-family:consolas,"andale mono wt","andale mono","lucida console","lucida sans typewriter","dejavu sans mono","bitstream vera sans mono","liberation mono","nimbus mono l",monaco,"courier new",courier,monospace;font-size:10.8px;white-space:pre-wrap">rmdir</span><span style="color:rgb(51,51,51);font-family:consolas,"andale mono wt","andale mono","lucida console","lucida sans typewriter","dejavu sans mono","bitstream vera sans mono","liberation mono","nimbus mono l",monaco,"courier new",courier,monospace;font-size:10.8px;white-space:pre-wrap"> /mnt/r2/<name-of-nonexistent-</span><span class="m_-1905138536765705426gmail-hljs-keyword" style="box-sizing:border-box;color:rgb(51,51,51);font-weight:bold;font-family:consolas,"andale mono wt","andale mono","lucida console","lucida sans typewriter","dejavu sans mono","bitstream vera sans mono","liberation mono","nimbus mono l",monaco,"courier new",courier,monospace;font-size:10.8px;white-space:pre-wrap">d<wbr>ir</span><span style="color:rgb(51,51,51);font-family:consolas,"andale mono wt","andale mono","lucida console","lucida sans typewriter","dejavu sans mono","bitstream vera sans mono","liberation mono","nimbus mono l",monaco,"courier new",courier,monospace;font-size:10.8px;white-space:pre-wrap">>
setfattr -</span><span class="m_-1905138536765705426gmail-hljs-keyword" style="box-sizing:border-box;color:rgb(51,51,51);font-weight:bold;font-family:consolas,"andale mono wt","andale mono","lucida console","lucida sans typewriter","dejavu sans mono","bitstream vera sans mono","liberation mono","nimbus mono l",monaco,"courier new",courier,monospace;font-size:10.8px;white-space:pre-wrap">n</span><span style="color:rgb(51,51,51);font-family:consolas,"andale mono wt","andale mono","lucida console","lucida sans typewriter","dejavu sans mono","bitstream vera sans mono","liberation mono","nimbus mono l",monaco,"courier new",courier,monospace;font-size:10.8px;white-space:pre-wrap"> trusted.non-existent-key -v abc /mnt/r2
setfattr -x trusted.non-existent-key /mnt/r2</span><br></div><div class="gmail_default" style="font-size:small"><span style="color:rgb(51,51,51);font-family:consolas,"andale mono wt","andale mono","lucida console","lucida sans typewriter","dejavu sans mono","bitstream vera sans mono","liberation mono","nimbus mono l",monaco,"courier new",courier,monospace;font-size:10.8px;white-space:pre-wrap"><br></span></div><div class="gmail_default"><font color="#333333" face="arial, helvetica, sans-serif"><span style="white-space:pre-wrap">That should set an extended attribute on the healthy replica partner indicating that there are pending changes for the partner host.</span></font></div><div class="gmail_default"><font color="#333333" face="arial, helvetica, sans-serif"><span style="white-space:pre-wrap"><br></span></font></div><div class="gmail_default">Remembering that we're in a distributed, replicated situation, I don't quite understand because the created directories can be created on any pair, not necessarily the one we're fixing. I think the name of the directory should chosen such that its dht value lands the file on the affected brick (the healthy of the two replicate hosts). That's not easy to do.</div><div class="gmail_default"><br></div><div class="gmail_default">Does somebody have any suggestions?</div><div class="gmail_default"><font color="#333333" face="arial, helvetica, sans-serif"><span style="white-space:pre-wrap"><br></span></font></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 6, 2016 at 10:47 PM, Sergei Gerasenko <span dir="ltr"><<a href="mailto:sgerasenko74@gmail.com" target="_blank">sgerasenko74@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-size:small">Step 10 isn't really necessary. The changes should probably be monitored under the brick directory.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 6, 2016 at 10:25 PM, Sergei Gerasenko <span dir="ltr"><<a href="mailto:sgerasenko74@gmail.com" target="_blank">sgerasenko74@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default">I've simulated the problem on 4 VMs in a distributed replicated setup with a 2 replica-factor. I've repeatedly torn down and brought up a VM from a snapshot in each of my tests.</div><div class="gmail_default"><br></div><div class="gmail_default">What has worked so far is this:</div><div class="gmail_default"><br></div><div class="gmail_default" style="font-size:12.8px"><ol><li style="margin-left:15px">Make a copy of /var/lib/glusterd from the affected machine, save it elsewhere.</li><li style="margin-left:15px">Configure your new machine (in my case I reverted to a VM snapshot). Assign the same ip and hostname!</li><li style="margin-left:15px">Install gluster.</li><li style="margin-left:15px">Stop the daemons if they are running.</li><li style="margin-left:15px">Nuke the /var/lib/glusterd directory and replace it with the saved copy in step 1.</li><li style="margin-left:15px">Create the brick directory.</li><li style="margin-left:15px">Get the extended volume attribute from a healthy node like so: <font face="monospace, monospace">getfattr -e base64 -n trusted.glusterfs.volume-id /data/brick_dir</font></li><li style="margin-left:15px">Apply the extended attribute volume id attribute like so: <font face="monospace, monospace">setfattr -n trusted.glusterfs.volume-id -v 'the_value_you_got_in_7==' /data/brick_dir</font></li><li style="margin-left:15px">Start the daemons.</li><li style="margin-left:15px">FUSE mount the gluster partition through the daemons running locally. So the /etc/fstab would contain something like: <font face="monospace, monospace">localhost:/gluster_volum<wbr>e /mnt/gluster glusterfs _netdev,defaults 0 0</font></li><li style="margin-left:15px">On the healthy partner machine with another fuse mount point to the same volume do something like:<font face="monospace, monospace"> find /mnt/fuse | xargs stat</font>.</li><li style="margin-left:15px">Step 8 will make files appear under the mount point on the new box but the files are not going to be physically in the brick directory -- yet. See 10.</li><li style="margin-left:15px">Run the <font face="monospace, monospace">heal</font> command from the same host where you ran <font face="monospace, monospace">find. </font><font face="arial, helvetica, sans-serif">That will finally sync the files to the brick. Run the heal info command periodically and the number of files being healed should eventually go down to 0. </font></li></ol><div><font face="arial, helvetica, sans-serif">That's my experience with the VMs today.</font></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Oct 5, 2016 at 4:46 PM, Joe Julian <span dir="ltr"><<a href="mailto:joe@julianfamily.org" target="_blank">joe@julianfamily.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>What I always do is just shut it down, repair (or replace) the brick, then start it up again with "... start $volname force". <br><br><div class="gmail_quote">On October 5, 2016 11:27:36 PM GMT+02:00, Sergei Gerasenko <<a href="mailto:sgerasenko74@gmail.com" target="_blank">sgerasenko74@gmail.com</a>> wrote:<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div class="gmail_default" style="font-size:small">Hi, sorry if this has been asked before but the documentation is a bit conflicting in various sources on what to do exactly.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">I have an 6-node, distributed replicated cluster with a replica factor of 2. So it's 3 pairs of servers. I need to remove a server from one of those replica sets, rebuild it and put it back in.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">What's the tried and proven sequence of steps for this? Any pointers would be very useful.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Thanks!</div><div class="gmail_default" style="font-size:small"> Sergei</div></div>
<p style="margin-top:2.5em;margin-bottom:1em;border-bottom:1px solid #000"></p><pre class="m_-1905138536765705426m_-4359921232482362757m_-866053349708398838m_6342889941002355632k9mail"><hr><br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a></pre></blockquote></div><span class="m_-1905138536765705426m_-4359921232482362757m_-866053349708398838HOEnZb"><font color="#888888"><br><span class="HOEnZb"><font color="#888888"><span class="m_-1905138536765705426HOEnZb"><font color="#888888"><span class="m_-1905138536765705426m_-4359921232482362757HOEnZb"><font color="#888888">
-- <br>
Sent from my Android device with K-9 Mail. Please excuse my brevity.</font></span></font></span></font></span></font></span></div><span class="HOEnZb"><font color="#888888"><span class="m_-1905138536765705426HOEnZb"><font color="#888888"><span class="m_-1905138536765705426m_-4359921232482362757HOEnZb"><font color="#888888"><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a><br></font></span></font></span></font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><span class="m_-1905138536765705426HOEnZb"><font color="#888888"><br></font></span></font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br></font></span></div>
</blockquote></div><br></div>
</blockquote></div><br></div>