<div dir="ltr"><div><div><div>Please check the cmd-history file in attached logs of the Board A in which we are removing-brick at 11:56:48 because at this time after waiting for 1 minute brick was not online.<br><br></div>So, We need to identify near to this time why brick is not online.<br><br></div><div>Because If brick will be online we will not remove-brick and if we remove brick, volume become distributed which will also cause problem in our case.<br></div><div><br>Regards,<br></div></div>Abhishek<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 1, 2016 at 4:08 PM, Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><br>
<br>
On 04/01/2016 03:57 PM, ABHISHEK PALIWAL wrote:<br>
><br>
><br>
> On Fri, Apr 1, 2016 at 3:39 PM, Atin Mukherjee <<a href="mailto:amukherj@redhat.com">amukherj@redhat.com</a><br>
</span><span class="">> <mailto:<a href="mailto:amukherj@redhat.com">amukherj@redhat.com</a>>> wrote:<br>
><br>
><br>
><br>
> On 04/01/2016 03:06 PM, ABHISHEK PALIWAL wrote:<br>
> ><br>
> > On Fri, Apr 1, 2016 at 2:59 PM, Atin Mukherjee <<a href="mailto:amukherj@redhat.com">amukherj@redhat.com</a> <mailto:<a href="mailto:amukherj@redhat.com">amukherj@redhat.com</a>><br>
</span><span class="">> > <mailto:<a href="mailto:amukherj@redhat.com">amukherj@redhat.com</a> <mailto:<a href="mailto:amukherj@redhat.com">amukherj@redhat.com</a>>>> wrote:<br>
> ><br>
> ><br>
> ><br>
</span><div><div class="h5">> > On 04/01/2016 02:55 PM, ABHISHEK PALIWAL wrote:<br>
> > > Hi Atin,<br>
> > ><br>
> > > Thanks for reply.<br>
> > ><br>
> > > Could you please help me to identify the error log in the respective<br>
> > > brick log file. I tried but not able to identified where the problem is<br>
> > > occuring.<br>
> > ><br>
> > > I am attaching the brick log file which is not coming online even after<br>
> > > waiting for 1 minute.<br>
> > What time did you reboot B? Could you also attach glusterd log file<br>
> > (complete log) for board B?<br>
> > ><br>
> ><br>
> > it is hard to say at what time we rebooted the B board because we are<br>
> > continuously rebooting the board B.<br>
> ><br>
> > Here I am attaching the glusterd and glsuterfs log for board B.<br>
> I can see the last restart of glusterd was at 13:11:27 and as per the<br>
> brick log there was no restart. This doesn't look like a reboot of the<br>
> board as in that case brick process should have also died and restarted.<br>
> Brick log indicates that the process is still running and there is no<br>
> interruption.<br>
><br>
> In brick log file at 13:11:27 we have the logs showing reboot of board<br>
><br>
> [2016-03-31 13:11:27.408512] I [MSGID: 115036]<br>
> [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting<br>
> connection from<br>
> 002500-10939-2016/03/31-11:56:52:528771-c_glusterfs-client-11-0-0<br>
> [2016-03-31 13:11:27.408603] I [MSGID: 101055]<br>
> [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down<br>
> connection<br>
> 002500-10939-2016/03/31-11:56:52:528771-c_glusterfs-client-11-0-0<br>
</div></div>FYI..this log indicates client disconnection.<br>
<div class="HOEnZb"><div class="h5">><br>
> ><br>
> > > Regards,<br>
> > > Abhishek<br>
> > ><br>
> > > On Fri, Apr 1, 2016 at 1:04 PM, Atin Mukherjee <<a href="mailto:amukherj@redhat.com">amukherj@redhat.com</a> <mailto:<a href="mailto:amukherj@redhat.com">amukherj@redhat.com</a>><br>
> <mailto:<a href="mailto:amukherj@redhat.com">amukherj@redhat.com</a> <mailto:<a href="mailto:amukherj@redhat.com">amukherj@redhat.com</a>>><br>
> > > <mailto:<a href="mailto:amukherj@redhat.com">amukherj@redhat.com</a> <mailto:<a href="mailto:amukherj@redhat.com">amukherj@redhat.com</a>><br>
> <mailto:<a href="mailto:amukherj@redhat.com">amukherj@redhat.com</a> <mailto:<a href="mailto:amukherj@redhat.com">amukherj@redhat.com</a>>>>> wrote:<br>
> > ><br>
> > ><br>
> > ><br>
> > > On 04/01/2016 12:10 PM, ABHISHEK PALIWAL wrote:<br>
> > > > Hi,<br>
> > > ><br>
> > > ><br>
> > > > I have the setup of two boards A and B with two bricks in<br>
> > replica mode.<br>
> > > ><br>
> > > > There is one test scenario<br>
> > > ><br>
> > > > 1. A acts as an active board and having the glusterfs<br>
> mount<br>
> > point on it.<br>
> > > > 2. B acts as Passive board.<br>
> > > > 3. We are repetitively rebooting the B board (In this time<br>
> > period peer<br>
> > > > status on A board will be "peer in cluster (Disconnected)"<br>
> > and brick is<br>
> > > > not present in "gluster volume status") and when Board B<br>
> > comes up,<br>
> > > > starts the gluster daemon.<br>
> > > > 4. if Gluster daemon starts successfully it will make<br>
> "peer in<br>
> > > > cluster(Connected)"<br>
> > > > 5. At the same with the immediate effect "gluster volume<br>
> > status" command<br>
> > > > should show the brick is available in online.<br>
> > > ><br>
> > > ><br>
> > > > But in my case sometime step 5 takes immediate reflection<br>
> > sometime 10-15<br>
> > > > second and sometime doesn't show brick is online even<br>
> after<br>
> > the 1minute.<br>
> > > ><br>
> > > > Could you please confirm why this type of unpredictable<br>
> > behavior is<br>
> > > > occuring. It should be reflect with immediate effect in<br>
> > "gluster volume<br>
> > > > status" command.<br>
> > > When glusterd restarts bricks processes are brought up<br>
> > asynchronously<br>
> > > and hence you may not see the brick processes reflecting in<br>
> > gluster<br>
> > > volume status output immediately after restart. 5-10<br>
> seconds is an<br>
> > > accepted time frame. However if it doesn't come back online<br>
> > post that<br>
> > > then probably brick fails to start in that case.<br>
> > ><br>
> > > Please check the respective brick log file and if you<br>
> can find<br>
> > any error<br>
> > > logs in it.<br>
> > > ><br>
> > > ><br>
> > > > --<br>
> > > > Regards<br>
> > > > Abhishek Paliwal<br>
> > > ><br>
> > > ><br>
> > > > _______________________________________________<br>
> > > > Gluster-devel mailing list<br>
> > > > <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> <mailto:<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>> <mailto:<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> <mailto:<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>>><br>
> > <mailto:<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> <mailto:<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>> <mailto:<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> <mailto:<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>>>><br>
> > > > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
> > > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > Regards<br>
> > > Abhishek Paliwal<br>
> ><br>
> ><br>
><br>
><br>
><br>
><br>
> --<br>
><br>
><br>
><br>
><br>
> Regards<br>
> Abhishek Paliwal<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr"><br><br><br><br>Regards<br>
Abhishek Paliwal<br>
</div></div>
</div>