<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 26, 2016 at 12:34 AM, Niels de Vos <span dir="ltr"><<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span>On Mon, Jul 25, 2016 at 04:34:17PM +0530, Avra Sengupta wrote:<br>
> The crux of the problem is that as of today, brick processes on restart try<br>
> to reuse the old port they were using (assuming that no other process will<br>
> be using it, and not consulting pmap_registry_alloc() before using it). With<br>
> a recent change, pmap_registry_alloc (), reassigns older ports that were<br>
> used, but are now free. Hence snapd now gets a port that was previously used<br>
> by a brick and tries to bind to it, whereas the older brick process without<br>
> consulting pmap table blindly tries to connect to it, and hence we see this<br>
> problem.<br>
><br>
> Now coming to the fix, I feel brick process should not try to get the older<br>
> port and should just take a new port every time it comes up. We will not run<br>
> out of ports with this change coz, now pmap allocates old ports again, and<br>
> the previous port being used by the brick process will eventually be reused.<br>
> If anyone sees any concern with this approach, please feel free to raise so<br>
> now.<br>
<br>
</span>I wonder how this is handled with reconnecting clients. If a client<br>
thinks it was connected to a brick, but the connection was lost, does it<br>
try to connect to the same port again? I dont know if it really connects<br>
to the pmap service in GlusterD to find the new/updated port...<br></blockquote><div><br></div><div> client does query for the portmap every time using client_query_portmap () in reconnect logic. So in case there is a change in the port it goes for rpc_clnt_reconfig which will ensure client talk to the brick process(es) on the new port.<br><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Niels<br>
<div><div><br>
<br>
><br>
> While awaiting feedback from you guys, I have sent this patch<br>
> (<a href="http://review.gluster.org/15001" rel="noreferrer" target="_blank">http://review.gluster.org/15001</a>), which moves the said test case to bad<br>
> tests for now, and after we collectively reach to a conclusion on the fix,<br>
> we will remove this from bad test.<br>
><br>
> Regards,<br>
> Avra<br>
><br>
> On 07/25/2016 02:33 PM, Avra Sengupta wrote:<br>
> > The failure suggests that the port snapd is trying to bind to is already<br>
> > in use. But snapd has been modified to use a new port everytime. I am<br>
> > looking into this.<br>
> ><br>
> > On 07/25/2016 02:23 PM, Nithya Balachandran wrote:<br>
> > > More failures:<br>
> > > <a href="https://build.gluster.org/job/rackspace-regression-2GB-triggered/22452/console" rel="noreferrer" target="_blank">https://build.gluster.org/job/rackspace-regression-2GB-triggered/22452/console</a><br>
> > ><br>
> > > I see these messages in the snapd.log:<br>
> > ><br>
> > > [2016-07-22 05:31:52.482282] I<br>
> > > [rpcsvc.c:2199:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service:<br>
> > > Configured rpc.outstanding-rpc-limit with value 64<br>
> > > [2016-07-22 05:31:52.482352] W [MSGID: 101002]<br>
> > > [options.c:954:xl_opt_validate] 0-patchy-server: option<br>
> > > 'listen-port' is deprecated, preferred is<br>
> > > 'transport.socket.listen-port', continuing with correction<br>
> > > [2016-07-22 05:31:52.482436] E [socket.c:771:__socket_server_bind]<br>
> > > 0-tcp.patchy-server: binding to failed: Address already in use<br>
> > > [2016-07-22 05:31:52.482447] E [socket.c:774:__socket_server_bind]<br>
> > > 0-tcp.patchy-server: Port is already in use<br>
> > > [2016-07-22 05:31:52.482459] W<br>
> > > [rpcsvc.c:1630:rpcsvc_create_listener] 0-rpc-service: listening on<br>
> > > transport failed<br>
> > > [2016-07-22 05:31:52.482469] W [MSGID: 115045] [server.c:1061:init]<br>
> > > 0-patchy-server: creation of listener failed<br>
> > > [2016-07-22 05:31:52.482481] E [MSGID: 101019]<br>
> > > [xlator.c:433:xlator_init] 0-patchy-server: Initialization of volume<br>
> > > 'patchy-server' failed, review your volfile again<br>
> > > [2016-07-22 05:31:52.482491] E [MSGID: 101066]<br>
> > > [graph.c:324:glusterfs_graph_init] 0-patchy-server: initializing<br>
> > > translator failed<br>
> > > [2016-07-22 05:31:52.482499] E [MSGID: 101176]<br>
> > > [graph.c:670:glusterfs_graph_activate] 0-graph: init failed<br>
> > ><br>
> > > On Mon, Jul 25, 2016 at 12:00 PM, Ashish Pandey <<a href="mailto:aspandey@redhat.com" target="_blank">aspandey@redhat.com</a><br>
</div></div><span>> > > <mailto:<a href="mailto:aspandey@redhat.com" target="_blank">aspandey@redhat.com</a>>> wrote:<br>
> > ><br>
> > > Hi,<br>
> > ><br>
> > > Following test has failed 3 times in last two days -<br>
> > ><br>
> > > ./tests/bugs/snapshot/bug-1316437.t<br>
> > > <a href="https://build.gluster.org/job/rackspace-regression-2GB-triggered/22445/consoleFull" rel="noreferrer" target="_blank">https://build.gluster.org/job/rackspace-regression-2GB-triggered/22445/consoleFull</a><br>
> > > <a href="https://build.gluster.org/job/rackspace-regression-2GB-triggered/22445/consoleFull" rel="noreferrer" target="_blank">https://build.gluster.org/job/rackspace-regression-2GB-triggered/22445/consoleFull</a><br>
> > > <a href="https://build.gluster.org/job/rackspace-regression-2GB-triggered/22470/consoleFull" rel="noreferrer" target="_blank">https://build.gluster.org/job/rackspace-regression-2GB-triggered/22470/consoleFull</a><br>
> > ><br>
> > > Please take a look at it and check if it spurious failure or not.<br>
> > ><br>
> > > Ashish<br>
> > ><br>
> > > _______________________________________________<br>
> > > Gluster-devel mailing list<br>
</span>> > > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a> <mailto:<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>><br>
> > > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
> > ><br>
> > ><br>
> ><br>
><br>
</blockquote></div><br><br clear="all"><br>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div><br></div>--Atin<br></div></div>
</div></div>