<html><head></head><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:10px"><div dir="ltr" id="yui_3_16_0_ym19_1_1470150454423_25794"><span id="yui_3_16_0_ym19_1_1470150454423_25793">if you don't want any downtime (in the case that your node 2 really die), you have to create a new gluster san (if you have the resources of course, 3 nodes as much as possible this time), and then just migrate your vms (or files), therefore no downtime but you have to cross your finger that the only node will not die too... also without sharding the vm migration especially an rdp one, will be slow access from users till it migrated.</span></div><div dir="ltr" id="yui_3_16_0_ym19_1_1470150454423_25794"><span><br></span></div><div dir="ltr" id="yui_3_16_0_ym19_1_1470150454423_25794">you have to start testing sharding, it's fast and cool...</div><div dir="ltr" id="yui_3_16_0_ym19_1_1470150454423_25794"><span><br></span></div><div dir="ltr" id="yui_3_16_0_ym19_1_1470150454423_25794"><span><br></span></div> <div class="qtdSeparateBR"><br><br></div><div class="yahoo_quoted" style="display: block;"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 10px;"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;"> <div dir="ltr"><font size="2" face="Arial"> On Tuesday, August 2, 2016 2:51 PM, Andres E. Moya <amoya@moyasolutions.com> wrote:<br></font></div> <br><br> <div class="y_msg_container"><div id="yiv8603185647"><div><div style="font-family:arial, helvetica, sans-serif;font-size:12pt;color:#000000;"><div></div><div>couldnt we just add a new server by</div><div><br clear="none"></div><div>gluster peer probe</div><div>gluster volume add-brick replica 3 (will this command succeed with 1 current failed brick?)</div><div><br clear="none"></div><div>let it heal, then </div><div><span style="color:#ffffff;font-family:'Source Code Pro', 'Courier New', monospace;font-size:13px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:20px;text-indent:0px;text-transform:none;white-space:pre;widows:1;word-spacing:0px;display:inline;float:none;background-color:#3b3f45;"></span><div style="clear:both;"><br clear="none"></div></div><div>gluster volume remove remove-brick</div><hr id="yiv8603185647zwchr"><div class="yiv8603185647yqt2377873150" id="yiv8603185647yqt55410"><div><b>From: </b>"Leno Vo" <lenovolastname@yahoo.com><br clear="none"><b>To: </b>"Andres E. Moya" <amoya@moyasolutions.com>, "gluster-users" <gluster-users@gluster.org><br clear="none"><b>Sent: </b>Tuesday, August 2, 2016 1:26:42 PM<br clear="none"><b>Subject: </b>Re: [Gluster-users] Failed file system<br clear="none"></div><div><br clear="none"></div><div><div style="color:#000;background-color:#fff;font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:10px;"><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7604"><span id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7694">you need to have a downtime to recreate the second node, two nodes is actually not good for production and you should have put raid 1 or raid 5 as your gluster storage, when you recreate the second node you might try running some VMs that need to be up and rest of vm need to be down but stop all backup and if you have replication, stop it too. if you have 1G nic, 2cpu and less 8Gram, then i suggest all turn off the VMs during recreation of second node. someone said if you have sharding with 3.7.x, maybe some vip vm can be up...</span></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7604"><span><br clear="none"></span></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7604"><span id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7882">if it just a filesystem, then just turn off the backup service until you recreate the second node. depending on your resources and how big is your storage, it might be hours to recreate it and even days...</span></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7604"><span><br clear="none"></span></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7604"><span id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7883">here's my process on recreating the second or third node (copied and modifed from the net),</span></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7604"><span><br clear="none"></span></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7974">#make sure partition is already added!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7975">This procedure is for replacing a failed server, IF your newly installed server has the same hostname as the failed one:</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7976"><br clear="none" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7977"></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7978">(If your new server will have a different hostname, see this article instead.)</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7979"><br clear="none" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7980"></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7981">For purposes of this example, the server that crashed will be server3 and the other servers will be server1 and server2</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7982"><br clear="none" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7983"></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7984">On both server1 and server2, make sure hostname server3 resolves to the correct IP address of the new replacement server.</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7985">#On either server1 or server2, do</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7986">grep server3 /var/lib/glusterd/peers/*</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7987"><br clear="none" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7988"></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7989">This will return a uuid followed by ":hostname1=server3"</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7990"><br clear="none" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7991"></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7992">#On server3, make sure glusterd is stopped, then do</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7993">echo UUID={uuid from previous step}>/var/lib/glusterd/glusterd.info</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7994"><br clear="none" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7995"></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7996">#actual testing below,</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7997">[root@node1 ~]# cat /var/lib/glusterd/glusterd.info</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7998">UUID=4b9d153c-5958-4dbe-8f91-7b5002882aac</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7999">operating-version=30710</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8000">#the second line is new......... maybe not needed...</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8001"><br clear="none" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8002"></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8003">On server3:</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8004">make sure that all brick directories are created/mounted</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8005">start glusterd</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8006">peer probe one of the existing servers</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8007"><br clear="none" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8008"></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8009">#restart glusterd, check that full peer list has been populated using</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8010"> gluster peer status</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8011"><br clear="none" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8012"></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8013">(if peers are missing, probe them explicitly, then restart glusterd again)</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8014">#check that full volume configuration has been populated using</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8015"> gluster volume info</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8016"><br clear="none" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8017"></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8018">if volume configuration is missing, do </div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8019">#on the other node</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8020">gluster volume sync "replace-node" all</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8021"><br clear="none" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8022"></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8023">#on the node to be replaced</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8024">setfattr -n trusted.glusterfs.volume-id -v 0x$(grep volume-id /var/lib/glusterd/vols/v1/info | cut -d= -f2 | sed 's/-//g') /gfs/b1/v1</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8025">setfattr -n trusted.glusterfs.volume-id -v 0x$(grep volume-id /var/lib/glusterd/vols/v2/info | cut -d= -f2 | sed 's/-//g') /gfs/b2/v2</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8026">setfattr -n trusted.glusterfs.volume-id -v 0x$(grep volume-id /var/lib/glusterd/vols/config/info | cut -d= -f2 | sed 's/-//g') /gfs/b1/config/c1</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8027"><br clear="none" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8028"></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8029">mount -t glusterfs localhost:config /data/data1</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8030"><br clear="none" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8031"></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8032">#install ctdb if not yet installed and put it back online, use the step on creating the ctdb config but </div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8033">#use your common sense not to deleted or modify current one.</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8034"><br clear="none" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8035"></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8036">gluster vol heal v1 full</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8037">gluster vol heal v2 full</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_7604"></div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8038">gluster vol heal config full</div><div dir="ltr" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8039"><br clear="none" id="yiv8603185647yui_3_16_0_ym19_1_1470150454423_8040"></div> <div class="yiv8603185647qtdSeparateBR"><br clear="none"><br clear="none"></div><div class="yiv8603185647yahoo_quoted" style="display:block;"> <div style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:10px;"> <div style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;"> <div dir="ltr"><span style="font-family:Arial;font-size:small;"> On Tuesday, August 2, 2016 11:57 AM, Andres E. Moya <amoya@moyasolutions.com> wrote:<br clear="none"></span></div> <br clear="none"><br clear="none"> <div class="yiv8603185647y_msg_container">Hi, we have a 2 node replica setup<br clear="none">on 1 of the nodes the file system that had the brick on it failed, not the OS<br clear="none">can we re create a file system and mount the bricks on the same mount point<br clear="none"><br clear="none">what will happen, will the data from the other node sync over, or will the failed node wipe out the data on the other mode?<br clear="none"><br clear="none">what would be the correct process?<br clear="none"><br clear="none">Thanks in advance for any help<br clear="none">_______________________________________________<br clear="none">Gluster-users mailing list<br clear="none"><a rel="nofollow" shape="rect" ymailto="mailto:Gluster-users@gluster.org" target="_blank" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br clear="none"><a rel="nofollow" shape="rect" target="_blank" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><br clear="none"><br clear="none"><br clear="none"></div> </div> </div> </div></div><br clear="none"></div></div></div></div></div><br><br></div> </div> </div> </div></div></body></html>