<p dir="ltr">Thanks Prasanna for the patches :)</p>
<p dir="ltr">-Atin<br>
Sent from one plus one</p>
<div class="gmail_quote">On Jul 2, 2015 9:19 PM, "Prasanna Kalever" <<a href="mailto:pkalever@redhat.com">pkalever@redhat.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
This is caused because when bind-insecure is turned on (which is the default now), it may happen<br>
that brick is not able to bind to port assigned by Glusterd for example 49192-49195...<br>
It seems to occur because the rpc_clnt connections are binding to ports in the same range.<br>
so brick fails to bind to a port which is already used by someone else.<br>
<br>
This bug already exist before <a href="http://review.gluster.org/#/c/11039/" rel="noreferrer" target="_blank">http://review.gluster.org/#/c/11039/</a> when use rdma, i.e. even<br>
previously rdma binds to port >= 1024 if it cannot find a free port < 1024,<br>
even when bind insecure was turned off (ref to commit '0e3fd04e').<br>
Since we don't have tests related to rdma we did not discover this issue previously.<br>
<br>
<a href="http://review.gluster.org/#/c/11039/" rel="noreferrer" target="_blank">http://review.gluster.org/#/c/11039/</a> discovers the bug we encountered, however now the bug can be fixed by<br>
<a href="http://review.gluster.org/#/c/11512/" rel="noreferrer" target="_blank">http://review.gluster.org/#/c/11512/</a> by making rpc_clnt to get port numbers from 65535 in a descending<br>
order, as a result port clash is minimized, also it fixes issues in rdma too<br>
<br>
Thanks to Raghavendra Talur for help in discovering the real cause<br>
<br>
<br>
Regards,<br>
Prasanna Kalever<br>
<br>
<br>
<br>
----- Original Message -----<br>
From: "Raghavendra Talur" <<a href="mailto:raghavendra.talur@gmail.com">raghavendra.talur@gmail.com</a>><br>
To: "Krishnan Parthasarathi" <<a href="mailto:kparthas@redhat.com">kparthas@redhat.com</a>><br>
Cc: "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a>><br>
Sent: Thursday, July 2, 2015 6:45:17 PM<br>
Subject: Re: [Gluster-devel] spurious failures tests/bugs/tier/bug-1205545-CTR-and-trash-integration.t<br>
<br>
<br>
<br>
On Thu, Jul 2, 2015 at 4:40 PM, Raghavendra Talur < <a href="mailto:raghavendra.talur@gmail.com">raghavendra.talur@gmail.com</a> > wrote:<br>
<br>
<br>
<br>
<br>
<br>
On Thu, Jul 2, 2015 at 10:52 AM, Krishnan Parthasarathi < <a href="mailto:kparthas@redhat.com">kparthas@redhat.com</a> > wrote:<br>
<br>
<br>
<br>
> ><br>
> > A port assigned by Glusterd for a brick is found to be in use already by<br>
> > the brick. Any changes in Glusterd recently which can cause this?<br>
> ><br>
> > Or is it a test infra problem?<br>
<br>
This issue is likely to be caused by <a href="http://review.gluster.org/11039" rel="noreferrer" target="_blank">http://review.gluster.org/11039</a><br>
This patch changes the port allocation that happens for rpc_clnt based<br>
connections. Previously, ports allocated where < 1024. With this change,<br>
these connections, typically mount process, gluster-nfs server processes<br>
etc could end up using ports that bricks are being assigned to.<br>
<br>
IIUC, the intention of the patch was to make server processes lenient to<br>
inbound messages from ports > 1024. If we don't require to use ports > 1024<br>
we could leave the port allocation for rpc_clnt connections as before.<br>
Alternately, we could reserve the range of ports starting from 49152 for bricks<br>
by setting net.ipv4.ip_local_reserved_ports using sysctl(8). This is specific to Linux.<br>
I'm not aware of how this could be done in NetBSD for instance though.<br>
<br>
<br>
It seems this is exactly whats happening.<br>
<br>
I have a question, I get the following data from netstat and grep<br>
<br>
tcp 0 0 f6be17c0fbf5:1023 f6be17c0fbf5:24007 ESTABLISHED 31516/glusterfsd<br>
tcp 0 0 f6be17c0fbf5:49152 f6be17c0fbf5:490 ESTABLISHED 31516/glusterfsd<br>
unix 3 [ ] STREAM CONNECTED 988353 31516/glusterfsd /var/run/gluster/4878d6e905c5f6032140a00cc584df8a.socket<br>
<br>
Here 31516 is the brick pid.<br>
<br>
Looking at the data, line 2 is very clear, it shows connection between brick and glusterfs client.<br>
unix socket on line 3 is also clear, it is the unix socket connection that glusterd and brick process use for communication.<br>
<br>
I am not able to understand line 1; which part of brick process established a tcp connection with glusterd using port 1023?<br>
Note: this data is from a build which does not have the above mentioned patch.<br>
<br>
<br>
The patch which exposed this bug is being reverted till the underlying bug is also fixed.<br>
You can monitor revert patches here<br>
master: <a href="http://review.gluster.org/11507" rel="noreferrer" target="_blank">http://review.gluster.org/11507</a><br>
3.7 branch: <a href="http://review.gluster.org/11508" rel="noreferrer" target="_blank">http://review.gluster.org/11508</a><br>
<br>
Please rebase your patches after the above patches are merged to ensure that you patches pass regression.<br>
<br>
<br>
<br>
<br>
<br>
--<br>
Raghavendra Talur<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
</blockquote></div>