<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, May 6, 2016 at 4:04 PM, Kaushal M <span dir="ltr"><<a href="mailto:kshlmster@gmail.com" target="_blank">kshlmster@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I'm currently trying to straighten out the encrypted transport<br>
(SSL/TLS socket) code, and make it more robust, and work well with<br>
IPv6 in particular [1]. When testing the changes, the mount.glusterfs<br>
script cause some troubles.<br>
<br>
The mount script tries to check if the mount is online by performing a<br>
stat on the mount point after the glusterfs command returns, and<br>
umounts if the stat fails. This is a check is racey and doesn't always<br>
do the right thing.<br>
<br>
The check is racey because it could be run before the client<br>
translators have been able to connect to the bricks. The following<br>
sequence of events happen when the mount happens, which help explain<br>
the race.<br>
<br>
- mount script runs the glusterfs command<br>
- mount process fetches the volfile<br>
- mount process initalizes the graph. The client xlator is also<br>
initialized now, but the connections aren't started.<br>
- mount process sends a PARENT_UP event to the graph. The client now<br>
begins the connection process (portmap first, followed by connecting<br>
to the brick). It's not guaranteed yet if the connection happened.<br>
- mount process returns<br>
- mount script does a stat on mount point to check health<br>
<br>
In an environment (like the on I'm testing in) the connection couldn't<br>
be completed by the time the health check is done. In my environment,<br>
the client connection sequence is as follows,<br>
- the portmap connection is started<br>
- the first address returned for the hostname is a IPv6 address. With<br>
the IPv6 change that was merged recently name lookups are done with<br>
AF_UNSPEC, which return IPv6. My envrionment returns v6 addresses<br>
first for getaddrinfo calls (which I think is the default for a lot of<br>
environments)<br>
- the connection fails as glusterd doesn't listen on IPv6 addresses<br>
(it listens on 0.0.0.0 which v4 only)<br>
- a reconnection is made with the next address. This takes a while<br>
because of the encrypted transports.<br>
- portmap query is done after connection is established and the port<br>
is obtained<br>
- the client xlator now reconnects to the obtained port.<br>
- (same above cycle of connection/reconnection happens)<br>
- once connection is established, handshakes are done<br>
- CHILD_UP event is sent<br>
<br>
After this point the client xlator becomes usable.<br>
<br>
But this is not reached before the mount script does the health check<br>
in my environment. So the mount ends up being terminated.<br>
<br>
Now the simplest solution would be to sleep for some time before doing<br>
the check to give the xlators time to get ready. But this is<br>
non-deterministic and isn't something I'm very fond of.<br>
<br></blockquote><div><br></div><div><br></div><div>Have you tried the wait builtin?</div><div><br></div><div>-sac</div></div></div></div>