<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=iso-8859-1"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-GB link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Greetings,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I have a problem in replicating an existing gluster volume from single brick setup to two brick setup. Background of the problem is following:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>OS: Ubuntu 14.04<o:p></o:p></p><p class=MsoNormal>Gluster version (from gluster repos): glusterfs 3.7.14 built on Aug 1 2016 16:57:28<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>1. I had a replication setup consisting of two Gluster bricks (srv100, srv102), and three volumes (gv0, gv100). <o:p></o:p></p><p class=MsoNormal>2. I had to completely rebuild raid/disks of one of the bricks (srv100) due to hardware failure. I did it by doing following on the faulty node:<o:p></o:p></p><p class=MsoNormal>2.1 Removed the failed brick from replication setup (reduced replica count to 1 from 2, and detached the node). I executed following commands on the *<b>good</b>* brick.<o:p></o:p></p><p class=MsoNormal> sudo gluster volume remove-brick gv100 replica 1 srv100:/pool01/gfs/brick1/gv100 force<o:p></o:p></p><p class=MsoNormal> sudo gluster volume remove-brick gv0 replica 1 srv100:/pool01/gfs/brick1/gv0 force<o:p></o:p></p><p class=MsoNormal> sudo gluster vol info #make sure the faulty node bricks are not listed, and brick count is 1 for each volume<o:p></o:p></p><p class=MsoNormal> sudo gluster peer detach srv100 force<o:p></o:p></p><p class=MsoNormal> sudo gluster peer status # --> OK, only one node/brick<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>2.2 Stopped glusterd, killed all gluster processes<o:p></o:p></p><p class=MsoNormal>2.3 Replaced HDs, and recreated raid. This means all GlusterFS data relevant directories were lost on the faulty-brick (srv100), while GlusterFS service installation and config files were untouched (including host name and IP address).<o:p></o:p></p><p class=MsoNormal>2.4 After rebuilding, I created volume directories on the rebuilt-node<o:p></o:p></p><p class=MsoNormal>2.5 Then I started gluster service, and added the node back to gluster cluster. Peer status is ok (in cluster)<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>2.6 Then I attempted to replicate one of the existing volume (gv0), and *<b>there</b>* came the problem. The replication could not be setup properly, and gave following error<o:p></o:p></p><p class=MsoNormal> sudo gluster volume add-brick gv0 replica 2 srv100:/pool01/gfs/brick1/gv0<o:p></o:p></p><p class=MsoNormal> volume add-brick: failed: Staging failed on srv100. Please check log file for details.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal> The relevant gluster log file says<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'><o:p> </o:p></p><p class=MsoNormal style='margin-left:36.0pt'>[2016-08-25 12:32:29.499708] I [MSGID: 106499] [glusterd-handler.c:4267:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv-temp<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>[2016-08-25 12:32:29.501881] E [MSGID: 106301] [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gv-temp is not started<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>[2016-08-25 12:32:29.505033] I [MSGID: 106499] [glusterd-handler.c:4267:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>[2016-08-25 12:32:29.508585] E [MSGID: 106153] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on srv100. Error: Volume gv0 does not exist<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>[2016-08-25 12:32:29.511062] I [MSGID: 106499] [glusterd-handler.c:4267:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv100<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>[2016-08-25 12:32:29.514556] E [MSGID: 106153] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on srv100. Error: Volume gv100 does not exist<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>[2016-08-25 12:33:15.865773] I [MSGID: 106499] [glusterd-handler.c:4267:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>[2016-08-25 12:33:15.869441] E [MSGID: 106153] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on srv100. Error: Volume gv0 does not exist<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>[2016-08-25 12:33:15.872630] I [MSGID: 106499] [glusterd-handler.c:4267:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv100<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>[2016-08-25 12:33:15.876199] E [MSGID: 106153] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on srv100. Error: Volume gv100 does not exist<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>[2016-08-25 12:34:14.716735] I [MSGID: 106482] [glusterd-brick-ops.c:442:__glusterd_handle_add_brick] 0-management: Received add brick req<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>[2016-08-25 12:34:14.716787] I [MSGID: 106062] [glusterd-brick-ops.c:494:__glusterd_handle_add_brick] 0-management: replica-count is 2<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>[2016-08-25 12:34:14.716809] I [MSGID: 106447] [glusterd-brick-ops.c:240:gd_addbr_validate_replica_count] 0-management: Changing the type of volume gv0 from 'distribute' to 'replica'<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>[2016-08-25 12:34:14.720133] E [MSGID: 106153] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on srv100. Please check log file for details.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>3. I tried to create a new replicated volume (gv-temp) over the nodes <span style='font-family:Wingdings'>à</span> it is created and replicated. It is only that the existing volume I cannot replicate again!<o:p></o:p></p><p class=MsoNormal>4. I also observed that /var/lib/glusterd/vols directory on the rebuilt node contains directory for the newly created volume (gv-temp), and no existing volumes (gv100, gv0)<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>*Question<span style='color:#1F497D'>s</span>:* <o:p></o:p></p><p class=MsoNormal>a. How to re-replicate the exiting volume, for which I set the replica count to 1 (see point 2.1)?<o:p></o:p></p><p class=MsoNormal>b. Is there a “glusterfs” way to create missing volume directories (under /var/lib/glusterd/vols) on the re-built node (see point 4)?<o:p></o:p></p><p class=MsoNormal>c. Any other pointers, hints?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Kind regards,<o:p></o:p></p><p class=MsoNormal>JAsghar<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>