<font size=3 color=blue face="Times New Roman">Sorry for spamming you
guys, but this is kind of important for me to debug, so if you saw anything
like this before, do let me know. Here's an update:</font>
<br>
<br><font size=3 color=blue face="Times New Roman">It seems the mount client
is attempting connection with an invalid port number. 49175 is NOT the
port number of glusterfsd on serv1 (192.168.24.8).</font>
<br>
<br><font size=3 color=blue face="Times New Roman">I got me an strace:</font>
<br>
<br><font size=2 face="Courier New">[pid 31026] open("/proc/sys/net/ipv4/ip_local_reserved_ports",
O_RDONLY) = -1 ENOENT (No such file or directory)</font>
<br><font size=2 face="Courier New">[pid 31026] write(4, "[2015-02-04
20:39:02.793154] W ["..., 215) = 215</font>
<br><font size=2 face="Courier New">[pid 31026] write(4, "[2015-02-04
20:39:02.793289] W ["..., 194) = 194</font>
<br><font size=2 face="Courier New">[pid 31026] bind(10, {sa_family=AF_INET,
sin_port=htons(1023), sin_addr=inet_addr("192.168.24.80")}, 16)
= 0</font>
<br><font size=2 face="Courier New">[pid 31026] fcntl(10, F_GETFL)
= 0x2 (flags O_RDWR)</font>
<br><font size=2 face="Courier New">[pid 31026] fcntl(10, F_SETFL, O_RDWR|O_NONBLOCK)
= 0</font>
<br><font size=2 face="Courier New">[pid 31026] connect(10, {sa_family=AF_INET,
sin_port=htons(49175), sin_addr=inet_addr("192.168.24.81")},
16) = -1 EINPROGRESS (Operation now in progress)</font>
<br><font size=2 face="Courier New">[pid 31026] fcntl(10, F_GETFL)
= 0x802 (flags O_RDWR|O_NONBLOCK)</font>
<br><font size=2 face="Courier New">[pid 31026] fcntl(10, F_SETFL, O_RDWR|O_NONBLOCK)
= 0</font>
<br><font size=2 face="Courier New">[pid 31026] epoll_ctl(3, EPOLL_CTL_ADD,
10, {EPOLLIN|EPOLLPRI|EPOLLOUT, {u32=10, u64=8589934602}}) = 0</font>
<br><font size=2 face="Courier New">[pid 31026] nanosleep({1, 0}, <unfinished
...></font>
<br><font size=2 face="Courier New">[pid 31021] <... epoll_wait resumed>
{{EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=10, u64=8589934602}}}, 257,
4294967295) = 1</font>
<br><font size=2 face="Courier New">[pid 31021] getsockopt(10, SOL_SOCKET,
SO_ERROR, [29422518842425455], [4]) = 0</font>
<br><font size=2 face="Courier New">[pid 31021] shutdown(10, 2 /* send
and receive */) = -1 ENOTCONN (Transport endpoint is not connected)</font>
<br>
<br><font size=3 color=blue face="Times New Roman">Which basically told
me that connection is attempted via a non-blocking socket at port 49175.
The errno from the failure is -ECONNREFUSED, which is what is expected.
</font>
<br>
<br><font size=2 face="Courier New">807 in socket.c</font>
<br><font size=2 face="Courier New">(gdb) bt</font>
<br><font size=2 face="Courier New">#0 __socket_connect_finish (this=0x6887a0)
at socket.c:807</font>
<br><font size=2 face="Courier New">#1 socket_connect_finish (this=0x6887a0)
at socket.c:2147</font>
<br><font size=2 face="Courier New">#2 0x00007fc863de4c04 in socket_event_handler
(fd=<value optimized out>, idx=<value optimized out>, data=0x6887a0,</font>
<br><font size=2 face="Courier New"> poll_in=1, poll_out=4,
poll_err=<value optimized out>) at socket.c:2223</font>
<br><font size=2 face="Courier New">#3 0x00007fc867f7919f in event_dispatch_epoll_handler
(event_pool=0x62db70) at event-epoll.c:384</font>
<br><font size=2 face="Courier New">#4 event_dispatch_epoll (event_pool=0x62db70)
at event-epoll.c:445</font>
<br><font size=2 face="Courier New">#5 0x0000000000406b06 in main
(argc=4, argv=0x7fff25302c38) at glusterfsd.c:1934</font>
<br><font size=2 face="Courier New">(gdb) print *optval</font>
<br><font size=2 face="Courier New">Cannot access memory at address 0x6f</font>
<br><font size=2 face="Courier New">(gdb) print optval</font>
<br><font size=2 face="Courier New">$1 = 111</font>
<br>
<br><font size=3 color=blue face="Times New Roman">Note that this agrees
with the following debug log:</font>
<br>
<br><font size=2 face="Courier New">[2015-02-03 12:11:33.833647] D [socket.c:1962:__socket_proto_state_machine]
0-replicated_vol-1: reading from socket failed. Error (No data available),
peer (192.168.24.81:49175)</font>
<br>
<br><font size=3 color=blue face="Times New Roman">There is, of course
no service running on port 49175. In fact, listen port for corresponding
glusterd on serv1 is 49206. Where does the mount client pick this port
number from? I know that if I kill and restart the mount client on serv0
from command line, then the problem will disappear. So, it's not something
that is up with the processes on serv1...</font>
<br>
<br><font size=3 color=blue face="Times New Roman">Thanks,</font>
<br><font size=3 color=blue face="Times New Roman">Anirban</font>
<br>
<br>
<br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">A Ghoshal/MUM/TCS</font>
<br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">A Ghoshal <a.ghoshal@tcs.com></font>
<br><font size=1 color=#5f5f5f face="sans-serif">Cc:
</font><font size=1 face="sans-serif">gluster-users@gluster.org,
gluster-users-bounces@gluster.org, Pranith Kumar Karampuri <pkarampu@redhat.com></font>
<br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">02/05/2015 02:03 AM</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [Gluster-users]
In a replica 2 server, file-updates on one server missing on the other
server #Personal#</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">A Ghoshal</font>
<br>
<hr noshade>
<br>
<br><font size=3 color=blue face="Times New Roman">Ok, more updates here:</font>
<br>
<br><font size=3 color=blue face="Times New Roman">I turned on trace and
it seems bind to secure port on the mount client with the remote brick
is successful - afterwards the connect() fails to complete. I saw these
logs:</font>
<br>
<br><font size=2 face="Courier New">[2015-02-03 12:11:33.832615] T [rpc-clnt.c:422:rpc_clnt_reconnect]
0-replicated_vol-1: attempting reconnect</font>
<br><font size=2 face="Courier New">[2015-02-03 12:11:33.832666] D [name.c:155:client_fill_address_family]
0-replicated_vol-1: address-family not specified, guessing it to be inet
from (remote-host: serv1)</font>
<br><font size=2 face="Courier New">[2015-02-03 12:11:33.832683] T [name.c:225:af_inet_client_get_remote_sockaddr]
0-replicated_vol-1: option remote-port missing in volume replicated_vol-1.
Defaulting to 24007</font>
<br><font size=2 face="Courier New">[2015-02-03 12:11:33.833083] D [common-utils.c:237:gf_resolve_ip6]
0-resolver: returning ip-192.168.24.81 (port-24007) for hostname: serv1
and port: 24007</font>
<br><font size=2 face="Courier New">[2015-02-03 12:11:33.833113] T [socket.c:731:__socket_nodelay]
0-replicated_vol-1: NODELAY enabled for socket 10</font>
<br><font size=2 face="Courier New">[2015-02-03 12:11:33.833128] T [socket.c:790:__socket_keepalive]
0-replicated_vol-1: Keep-alive enabled for socket 10, interval 2, idle:
20</font>
<br><font size=2 face="Courier New">[2015-02-03 12:11:33.833188] W [common-utils.c:2247:gf_get_reserved_ports]
0-glusterfs: could not open the file /proc/sys/net/ipv4/ip_local_reserved_ports
for getting reserved ports info (No such file or directory)</font>
<br><font size=2 face="Courier New">[2015-02-03 12:11:33.833204] W [common-utils.c:2280:gf_process_reserved_ports]
0-glusterfs: Not able to get reserved ports, hence there is a possibility
that glusterfs may consume reserved port</font>
<br><font size=2 face="Courier New">[2015-02-03 12:11:33.833560] D [socket.c:605:__socket_shutdown]
0-replicated_vol-1: shutdown() returned -1. Transport endpoint is not connected</font>
<br><font size=2 face="Courier New">[2015-02-03 12:11:33.833594] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup]
0-replicated_vol-1: cleaning up state in transport object 0x6887a0</font>
<br><font size=2 face="Courier New">[2015-02-03 12:11:33.833622] D [socket.c:486:__socket_rwv]
0-replicated_vol-1: EOF on socket</font>
<br><font size=2 face="Courier New">[2015-02-03 12:11:33.833635] W [socket.c:514:__socket_rwv]
0-replicated_vol-1: readv failed (No data available)</font>
<br><font size=2 face="Courier New">[2015-02-03 12:11:33.833647] D [socket.c:1962:__socket_proto_state_machine]
0-replicated_vol-1: reading from socket failed. Error (No data available),
peer (192.168.24.81:49175)</font>
<br><font size=2 face="Courier New">[2015-02-03 12:11:33.833659] D [socket.c:2236:socket_event_handler]
0-transport: disconnecting now</font>
<br><font size=2 face="Courier New">[2015-02-03 12:11:33.833684] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup]
0-replicated_vol-1: cleaning up state in transport object 0x6887a0</font>
<br>
<br><font size=3 color=blue face="Times New Roman">After that, I attached
the mount process to my gdb session and attached a breakpoint at __socket_shutdown().
It gave me the following traceback </font>
<br>
<br><font size=2 face="Courier New">Breakpoint 1, __socket_shutdown (this=0x68a7f0)
at socket.c:594</font>
<br><font size=2 face="Courier New">594 socket.c: No such
file or directory.</font>
<br><font size=2 face="Courier New"> in socket.c</font>
<br><font size=2 face="Courier New">(gdb) bt</font>
<br><font size=2 face="Courier New">#0 __socket_shutdown (this=0x68a7f0)
at socket.c:594</font>
<br><font size=2 face="Courier New">#1 0x00007f3391906553 in __socket_disconnect
(this=0x68a7f0) at socket.c:623</font>
<br><font size=2 face="Courier New">#2 0x00007f339190856a in socket_connect_finish
(this=0x68a7f0) at socket.c:2160</font>
<br><font size=2 face="Courier New">#3 0x00007f339190dc04 in socket_event_handler
(fd=<value optimized out>, idx=<value optimized out>, data=0x68a7f0,</font>
<br><font size=2 face="Courier New"> poll_in=1, poll_out=4,
poll_err=<value optimized out>) at socket.c:2223</font>
<br><font size=2 face="Courier New">#4 0x00007f3395aa219f in event_dispatch_epoll_handler
(event_pool=0x62d8b0) at event-epoll.c:384</font>
<br><font size=2 face="Courier New">#5 event_dispatch_epoll (event_pool=0x62d8b0)
at event-epoll.c:445</font>
<br><font size=2 face="Courier New">#6 0x0000000000406b06 in main
(argc=4, argv=0x7fff71108c38) at glusterfsd.c:1934</font>
<br><font size=2 face="Courier New">(gdb)</font>
<br>
<br><font size=3 color=blue face="Times New Roman">In case somebody could
give me a clue or two on my next move (I'm looking at the code in the meantime),
I'd be very grateful.</font>
<br>
<br><font size=3 color=blue face="Times New Roman">Thanks,</font>
<br><font size=3 color=blue face="Times New Roman">Anirban</font>
<br>
<br>
<br>
<br>
<br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">A Ghoshal <a.ghoshal@tcs.com></font>
<br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">Pranith Kumar Karampuri
<pkarampu@redhat.com></font>
<br><font size=1 color=#5f5f5f face="sans-serif">Cc:
</font><font size=1 face="sans-serif">gluster-users@gluster.org</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">02/03/2015 05:24 AM</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [Gluster-users]
In a replica 2 server, file-updates on one server missing on the other
server #Personal#</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gluster-users-bounces@gluster.org</font>
<br>
<hr noshade>
<br>
<br>
<br><font size=3 color=blue face="Times New Roman">Hi Pranith,</font><font size=3>
<br>
</font><font size=3 color=blue face="Times New Roman"><br>
I finally understood what you meant the secure ports, because the issue
occurred in one of our setups once more. It seems one of the clients on
serv1 could not open a connection to the glusterfsd running on serv0. I'd
actually started a mail trail about it (believing it might be something
else, initially) here: </font><a href="http://www.gluster.org/pipermail/gluster-users/2015-February/020465.html"><font size=3 color=blue face="Times New Roman"><u>http://www.gluster.org/pipermail/gluster-users/2015-February/020465.html</u></font></a><font size=3>
<br>
</font><font size=3 color=blue face="Times New Roman"><br>
I think I can write me a rudimentary kind of patch altering <i>af_inet_bind_to_port_lt_ceiling()</i>
to get it to call bind with port 0, rather than specify a port explicitly
when the client.bind-insecure is specified ... </font><font size=3><br>
</font><font size=3 color=blue face="Times New Roman"><br>
Then I'd need to create a way to set server.allow-insecure using the cli
(or if you already sent around the patch to do that like you said in the
earlier mail, do let me know). I'll keep you posted about it round here
or @ [gluster-devel] if I can get it to work. </font><font size=3><br>
</font><font size=3 color=blue face="Times New Roman"><br>
Thanks a lot,</font><font size=3> <br>
</font><font size=3 color=blue face="Times New Roman"><br>
Anirban</font><font size=3> <br>
<br>
<br>
</font><font size=1 color=#5f5f5f face="sans-serif"><br>
From: </font><font size=1 face="sans-serif">A
Ghoshal/MUM/TCS</font><font size=3> </font><font size=1 color=#5f5f5f face="sans-serif"><br>
To: </font><font size=1 face="sans-serif">Pranith
Kumar Karampuri <pkarampu@redhat.com></font><font size=3> </font><font size=1 color=#5f5f5f face="sans-serif"><br>
Cc: </font><font size=1 face="sans-serif">gluster-users@gluster.org,
Niels de Vos <ndevos@redhat.com></font><font size=3> </font><font size=1 color=#5f5f5f face="sans-serif"><br>
Date: </font><font size=1 face="sans-serif">01/23/2015
02:45 PM</font><font size=3> </font><font size=1 color=#5f5f5f face="sans-serif"><br>
Subject: </font><font size=1 face="sans-serif">Re:
[Gluster-users] In a replica 2 server, file-updates on one server missing
on the other server #Personal#</font><font size=3> </font><font size=1 color=#5f5f5f face="sans-serif"><br>
Sent by: </font><font size=1 face="sans-serif">A
Ghoshal</font><font size=3> <br>
</font>
<hr noshade><font size=3><br>
</font><font size=3 color=blue face="Times New Roman"><br>
Oh, I didn't I only read a fragment of the IRC log and assumed --xlator-option
would be enough. Apparently it's a lot more work.... </font><font size=3><br>
</font><font size=3 color=blue face="Times New Roman"><br>
I do have a query, though. These connections, from one of our setups, are
these on secure ports? Or, maybe I didn't get it the first time.....</font><font size=3>
<br>
</font><font size=2 face="Courier New"><br>
root@serv0:/root> ps -ef | grep replicated_vol</font><font size=3> </font><font size=2 face="Courier New"><br>
root 8851 25307 0 10:03 pts/2 00:00:00
grep replicated_vol</font><font size=3> </font><font size=2 face="Courier New"><br>
root 29751 1 4 Jan21 ?
01:47:20 /usr/sbin/glusterfsd -s serv0 --volfile-id replicated_vol.serv0.mnt-bricks-replicated_vol-brick
-p /var/lib/glusterd/vols/_replicated_vol/run/serv0-mnt-bricks-replicated_vol-brick.pid
-S /var/run/dff9fa3c93e82f20103f2a3d91adc4a8.socket --brick-name /mnt/bricks/replicated_vol/brick
-l /var/log/glusterfs/bricks/mnt-bricks-replicated_vol-brick.log --xlator-option
*-posix.glusterd-uuid=1a1d1ebc-4b92-428f-b66b-9c5efa49574d --brick-port
49185 --xlator-option replicated_vol-server.listen-port=49185</font><font size=3>
</font><font size=2 face="Courier New"><br>
root 30399 1 0 Jan21 ?
00:19:06 /usr/sbin/glusterfs --volfile-id=replicated_vol --volfile-server=serv0
/mnt/replicated_vol</font><font size=3> <br>
</font><font size=2 face="Courier New"><br>
root@serv0:/root> netstat -p | grep 30399</font><font size=3> </font><font size=2 face="Courier New"><br>
tcp 0 0 serv0:969
serv0:49185 ESTABLISHED
30399/glusterfs</font><font size=3> </font><font size=2 face="Courier New"><br>
tcp 0 0 serv0:999
serv1:49159 ESTABLISHED
30399/glusterfs</font><font size=3> </font><font size=2 face="Courier New"><br>
tcp 0 0 serv0:1023
serv0:24007 ESTABLISHED
30399/glusterfs</font><font size=3> </font><font size=2 face="Courier New"><br>
root@serv0:/root></font><font size=3> <br>
</font><font size=3 color=blue face="Times New Roman"><br>
Thanks again,</font><font size=3> </font><font size=3 color=blue face="Times New Roman"><br>
Anirban</font><font size=3> <br>
<br>
<br>
<br>
</font><font size=1 color=#5f5f5f face="sans-serif"><br>
From: </font><font size=1 face="sans-serif">Pranith
Kumar Karampuri <pkarampu@redhat.com></font><font size=3> </font><font size=1 color=#5f5f5f face="sans-serif"><br>
To: </font><font size=1 face="sans-serif">A
Ghoshal <a.ghoshal@tcs.com></font><font size=3> </font><font size=1 color=#5f5f5f face="sans-serif"><br>
Cc: </font><font size=1 face="sans-serif">gluster-users@gluster.org,
Niels de Vos <ndevos@redhat.com></font><font size=3> </font><font size=1 color=#5f5f5f face="sans-serif"><br>
Date: </font><font size=1 face="sans-serif">01/23/2015
01:58 PM</font><font size=3> </font><font size=1 color=#5f5f5f face="sans-serif"><br>
Subject: </font><font size=1 face="sans-serif">Re:
[Gluster-users] In a replica 2 server, file-updates on one server missing
on the other server #Personal#</font><font size=3> <br>
</font>
<hr noshade><font size=3><br>
<br>
<br>
<br>
On 01/23/2015 01:54 PM, A Ghoshal wrote: </font><font size=3 color=blue face="Times New Roman"><br>
Thanks a lot, Pranith. <br>
<br>
We'll set this option on our test servers and keep the setup under observation.</font><font size=3>
<br>
How did you get the bind-insecure option working?<br>
I guess I will send a patch to make it 'volume set option'<br>
<br>
Pranith </font><font size=3 color=blue face="Times New Roman"><br>
<br>
Thanks,</font><font size=3> </font><font size=3 color=blue face="Times New Roman"><br>
Anirban</font><font size=3> <br>
<br>
</font><font size=1 color=#5f5f5f face="sans-serif"><br>
<br>
From: </font><font size=1 face="sans-serif">Pranith
Kumar Karampuri </font><a href=mailto:pkarampu@redhat.com><font size=1 color=blue face="sans-serif"><u><pkarampu@redhat.com></u></font></a><font size=3>
</font><font size=1 color=#5f5f5f face="sans-serif"><br>
To: </font><font size=1 face="sans-serif">A
Ghoshal </font><a href=mailto:a.ghoshal@tcs.com><font size=1 color=blue face="sans-serif"><u><a.ghoshal@tcs.com></u></font></a><font size=3>
</font><font size=1 color=#5f5f5f face="sans-serif"><br>
Cc: </font><a href="mailto:gluster-users@gluster.org"><font size=1 color=blue face="sans-serif"><u>gluster-users@gluster.org</u></font></a><font size=1 face="sans-serif">,
Niels de Vos </font><a href=mailto:ndevos@redhat.com><font size=1 color=blue face="sans-serif"><u><ndevos@redhat.com></u></font></a><font size=3>
</font><font size=1 color=#5f5f5f face="sans-serif"><br>
Date: </font><font size=1 face="sans-serif">01/23/2015
01:28 PM</font><font size=3> </font><font size=1 color=#5f5f5f face="sans-serif"><br>
Subject: </font><font size=1 face="sans-serif">Re:
[Gluster-users] In a replica 2 server, file-updates on one server missing
on the other server #Personal#</font><font size=3> <br>
</font>
<hr noshade><font size=3><br>
<br>
<br>
<br>
On 01/22/2015 02:07 PM, A Ghoshal wrote: </font><font size=3 color=blue face="Times New Roman"><br>
Hi Pranith,</font><font size=3> </font><font size=3 color=blue face="Times New Roman"><br>
<br>
Yes, the very same (</font><a href=mailto:chalcogen_eg_oxygen@yahoo.com><font size=3 color=blue face="Times New Roman"><u>chalcogen_eg_oxygen@yahoo.com</u></font></a><font size=3 color=blue face="Times New Roman">).
Justin Clift sent me a mail a while back telling me that it is better if
we all use our business email addresses so I made me a new profile. <br>
<br>
Glusterfs complains about /proc/sys/net/ipv4/ip_local_reserved_ports because
we use a really old Linux kernel (2.6.34) wherein this feature is not present.
We plan to upgrade our Linux so often but each time we are dissuaded from
it by some compatibility issue or the other. So, we get this log every
time - on both good volumes and bad ones. What bothered me was this (on
serv1):</font><font size=3> <br>
Basically to make the connections to servers i.e. bricks clients need to
choose secure ports i.e. port less than 1024. Since this file is not present,
it is not binding to any port as per the code I just checked. There is
an option called client-bind-insecure which bypasses this check. I feel
that is one (probably only way) to get around this. <br>
You have to "volume set server.allow-insecure on" option and
bind-insecure option.<br>
CC ndevos who seemed to have helped someone set bind-insecure option correctly
here (</font><a href="http://irclog.perlgeek.de/gluster/2014-04-09/text"><font size=3 color=blue><u>http://irclog.perlgeek.de/gluster/2014-04-09/text</u></font></a><font size=3>)<br>
<br>
Pranith </font><font size=2 face="Courier New"><br>
<br>
[2015-01-20 09:37:49.151744] T [rpc-clnt.c:1182:rpc_clnt_record_build_header]
0-rpc-clnt: Request fraglen 456, payload: 360, rpc hdr: 96</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.151780] T [rpc-clnt.c:1499:rpc_clnt_submit] 0-rpc-clnt:
submitted request (XID: 0x39620x Program: GlusterFS 3.3, ProgVers: 330,
Proc: 27) to rpc-transport (replicated_vol-client-0)</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.151810] T [rpc-clnt.c:1302:rpc_clnt_record] 0-replicated_vol-client-1:
Auth Info: pid: 7599, uid: 0, gid: 0, owner: 0000000000000000</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.151824] T [rpc-clnt.c:1182:rpc_clnt_record_build_header]
0-rpc-clnt: Request fraglen 456, payload: 360, rpc hdr: 96</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.151889] T [rpc-clnt.c:1499:rpc_clnt_submit] 0-rpc-clnt:
submitted request (XID: 0x39563x Program: GlusterFS 3.3, ProgVers: 330,
Proc: 27) to rpc-transport (replicated_vol-client-1)</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.152239] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-replicated_vol-client-1:
received rpc message (RPC XID: 0x39563x Program: GlusterFS 3.3, ProgVers:
330, Proc: 27) from rpc-transport (replicated_vol-client-1)</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.152484] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-replicated_vol-client-0:
received rpc message (RPC XID: 0x39620x Program: GlusterFS 3.3, ProgVers:
330, Proc: 27) from rpc-transport (replicated_vol-client-0)</font><font size=3>
</font><font size=3 color=blue face="Times New Roman"><br>
<br>
When I write on the good server (serv1), we see that an RPC request is
sent to both client-0 and client-1. While, when I write on the bad server
(serv0), the RPC request is sent only to client-0, which is why it is no
wonder that the writes are not synced over to serv1. Somehow I could not
make the daemon on serv0 understand that there are two up-children and
not just one. <br>
<br>
One additional detail - since we are using a kernel that's too old, we
do not have the (Anand Avati's?) FUse readdirplus patches, either. I've
noticed that the fixes in the readdirplus version of glusterfs aren't always
guaranteed to be present on the non-readdirplus version of the patches.
I'd filed a bug around one such anomaly back, but never got around to writing
a patch for it (sorry!) Here it is: </font><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1062287"><font size=3 color=blue face="Times New Roman"><u>https://bugzilla.redhat.com/show_bug.cgi?id=1062287</u></font></a><font size=3>
<br>
I don't this has anything to do with readdirplus. </font><font size=3 color=blue face="Times New Roman"><br>
<br>
Maybe something on similar lines here?</font><font size=3> </font><font size=3 color=blue face="Times New Roman"><br>
<br>
Thanks,</font><font size=3> </font><font size=3 color=blue face="Times New Roman"><br>
Anirban</font><font size=3> </font><font size=3 color=blue face="Times New Roman"><br>
<br>
P.s. Please ignore the #Personal# in the subject line - we need to do that
to push mails to the public domain past the email filter safely.</font><font size=3>
<br>
</font><font size=1 color=#5f5f5f face="sans-serif"><br>
<br>
<br>
From: </font><font size=1 face="sans-serif">Pranith
Kumar Karampuri </font><a href=mailto:pkarampu@redhat.com><font size=1 color=blue face="sans-serif"><u><pkarampu@redhat.com></u></font></a><font size=3>
</font><font size=1 color=#5f5f5f face="sans-serif"><br>
To: </font><font size=1 face="sans-serif">A
Ghoshal </font><a href=mailto:a.ghoshal@tcs.com><font size=1 color=blue face="sans-serif"><u><a.ghoshal@tcs.com></u></font></a><font size=1 face="sans-serif">,
</font><a href="mailto:gluster-users@gluster.org"><font size=1 color=blue face="sans-serif"><u>gluster-users@gluster.org</u></font></a><font size=3>
</font><font size=1 color=#5f5f5f face="sans-serif"><br>
Date: </font><font size=1 face="sans-serif">01/22/2015
12:09 AM</font><font size=3> </font><font size=1 color=#5f5f5f face="sans-serif"><br>
Subject: </font><font size=1 face="sans-serif">Re:
[Gluster-users] In a replica 2 server, file-updates on one server missing
on the other server</font><font size=3> <br>
</font>
<hr noshade><font size=3><br>
<br>
<br>
hi,<br>
Responses inline.<br>
<br>
PS: You are chalkogen_oxygen?<br>
<br>
Pranith <br>
On 01/20/2015 05:34 PM, A Ghoshal wrote: </font><font size=3 color=blue face="Times New Roman"><br>
Hello,</font><font size=3> </font><font size=3 color=blue face="Times New Roman"><br>
<br>
I am using the following replicated volume:</font><font size=3> </font><font size=2 face="Courier New"><br>
<br>
root@serv0:~> gluster v info replicated_vol</font><font size=3> </font><font size=2 face="Courier New"><br>
<br>
Volume Name: replicated_vol</font><font size=3> </font><font size=2 face="Courier New"><br>
Type: Replicate</font><font size=3> </font><font size=2 face="Courier New"><br>
Volume ID: 26d111e3-7e4c-479e-9355-91635ab7f1c2</font><font size=3> </font><font size=2 face="Courier New"><br>
Status: Started</font><font size=3> </font><font size=2 face="Courier New"><br>
Number of Bricks: 1 x 2 = 2</font><font size=3> </font><font size=2 face="Courier New"><br>
Transport-type: tcp</font><font size=3> </font><font size=2 face="Courier New"><br>
Bricks:</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick1: serv0:/mnt/bricks/replicated_vol/brick</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick2: serv1:/mnt/bricks/replicated_vol/brick</font><font size=3> </font><font size=2 face="Courier New"><br>
Options Reconfigured:</font><font size=3> </font><font size=2 face="Courier New"><br>
diagnostics.client-log-level: INFO</font><font size=3> </font><font size=2 face="Courier New"><br>
network.ping-timeout: 10</font><font size=3> </font><font size=2 face="Courier New"><br>
nfs.enable-ino32: on</font><font size=3> </font><font size=2 face="Courier New"><br>
cluster.self-heal-daemon: on</font><font size=3> </font><font size=2 face="Courier New"><br>
nfs.disable: off</font><font size=3> </font><font size=3 color=blue face="Times New Roman"><br>
<br>
replicated_vol is mounted at /mnt/replicated_vol on both serv0 and serv1.
If I do the following on serv0:</font><font size=3> </font><font size=2 face="Courier New"><br>
<br>
root@serv0:~>echo "cranberries" > /mnt/replicated_vol/testfile</font><font size=3>
</font><font size=2 face="Courier New"><br>
root@serv0:~>echo "tangerines" >> /mnt/replicated_vol/testfile</font><font size=3>
</font><font size=3 color=blue face="Times New Roman"><br>
<br>
And then I check for the state of the replicas in the bricks, then I find
that</font><font size=3> </font><font size=2><br>
<br>
root@serv0:~>cat /mnt/bricks/replicated_vol/brick/testfile</font><font size=3>
</font><font size=2 face="Courier New"><br>
cranberries</font><font size=3> </font><font size=2 face="Courier New"><br>
tangerines</font><font size=3> </font><font size=2 face="Courier New"><br>
root@serv0:~></font><font size=3> </font><font size=2 face="Courier New"><br>
<br>
root@serv1:~>cat /mnt/bricks/replicated_vol/brick/testfile</font><font size=3>
</font><font size=2 face="Courier New"><br>
root@serv1:~></font><font size=3> </font><font size=3 color=blue face="Times New Roman"><br>
<br>
As may be seen, the replica on serv1 is blank, when I write into testfile
from serv0 (even though the file is created on both bricks). Interestingly,
if I write something to the file at serv1, then the two replicas become
identical.</font><font size=3> </font><font size=2 face="Courier New"><br>
<br>
root@serv1:~>echo "artichokes" >> /mnt/replicated_vol/testfile</font><font size=3>
</font><font size=2 face="Courier New"><br>
<br>
root@serv1:~>cat /mnt/bricks/replicated_vol/brick/testfile</font><font size=3>
</font><font size=2 face="Courier New"><br>
cranberries</font><font size=3> </font><font size=2 face="Courier New"><br>
tangerines</font><font size=3> </font><font size=2 face="Courier New"><br>
artichokes</font><font size=3> </font><font size=2 face="Courier New"><br>
root@serv1:~></font><font size=3> </font><font size=2 face="Courier New"><br>
<br>
root@serv0:~>cat /mnt/bricks/replicated_vol/brick/testfile</font><font size=3>
</font><font size=2 face="Courier New"><br>
cranberries</font><font size=3> </font><font size=2 face="Courier New"><br>
tangerines</font><font size=3> </font><font size=2 face="Courier New"><br>
artichokes</font><font size=3> </font><font size=2 face="Courier New"><br>
root@serv0:~></font><font size=3> </font><font size=3 color=blue face="Times New Roman"><br>
<br>
So, I dabbled into the logs a little bit, after upping the diagnostic level,
and this is what I saw:</font><font size=3> </font><font size=3 face="Times New Roman"><b><u><br>
<br>
When I write on serv0 (bad case):</u></b></font><font size=3> </font><font size=2 face="Courier New"><br>
<br>
[2015-01-20 09:21:52.197704] T [fuse-bridge.c:546:fuse_lookup_resume] 0-glusterfs-fuse:
53027: LOOKUP /testfl(f0a76987-8a42-47a2-b027-a823254b736b)</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:52.197959] D [afr-common.c:131:afr_lookup_xattr_req_prepare]
0-replicated_vol-replicate-0: /testfl: failed to get the gfid from dict</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:52.198006] T [rpc-clnt.c:1302:rpc_clnt_record] 0-replicated_vol-client-0:
Auth Info: pid: 28151, uid: 0, gid: 0, owner: 0000000000000000</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:52.198024] T [rpc-clnt.c:1182:rpc_clnt_record_build_header]
0-rpc-clnt: Request fraglen 456, payload: 360, rpc hdr: 96</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:52.198108] T [rpc-clnt.c:1499:rpc_clnt_submit] 0-rpc-clnt:
submitted request (XID: 0x78163x Program: GlusterFS 3.3, ProgVers: 330,
Proc: 27) to rpc-transport (replicated_vol-client-0)</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:52.198565] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-replicated_vol-client-0:
received rpc message (RPC XID: 0x78163x Program: GlusterFS 3.3, ProgVers:
330, Proc: 27) from rpc-transport (replicated_vol-client-0)</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:52.198640] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix]
0-replicated_vol-replicate-0: pending_matrix: [ 0 3 ]</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:52.198669] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix]
0-replicated_vol-replicate-0: pending_matrix: [ 0 0 ]</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:52.198681] D [afr-self-heal-common.c:887:afr_mark_sources]
0-replicated_vol-replicate-0: Number of sources: 1</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:52.198694] D [afr-self-heal-data.c:825:afr_lookup_select_read_child_by_txn_type]
0-replicated_vol-replicate-0: returning read_child: 0</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:52.198705] D [afr-common.c:1380:afr_lookup_select_read_child]
0-replicated_vol-replicate-0: Source selected as 0 for /testfl</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:52.198720] D [afr-common.c:1117:afr_lookup_build_response_params]
0-replicated_vol-replicate-0: Building lookup response from 0</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:52.198732] D [afr-common.c:1732:afr_lookup_perform_self_heal]
0-replicated_vol-replicate-0: Only 1 child up - do not attempt to detect
self heal</font><font size=3> </font><font size=3 face="Times New Roman"><b><u><br>
<br>
When I write on serv1 (good case):</u></b></font><font size=3> </font><font size=2 face="Courier New"><br>
<br>
[2015-01-20 09:37:49.151506] T [fuse-bridge.c:546:fuse_lookup_resume] 0-glusterfs-fuse:
31212: LOOKUP /testfl(f0a76987-8a42-47a2-b027-a823254b736b)</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.151683] D [afr-common.c:131:afr_lookup_xattr_req_prepare]
0-replicated_vol-replicate-0: /testfl: failed to get the gfid from dict</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.151726] T [rpc-clnt.c:1302:rpc_clnt_record] 0-replicated_vol-client-0:
Auth Info: pid: 7599, uid: 0, gid: 0, owner: 0000000000000000</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.151744] T [rpc-clnt.c:1182:rpc_clnt_record_build_header]
0-rpc-clnt: Request fraglen 456, payload: 360, rpc hdr: 96</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.151780] T [rpc-clnt.c:1499:rpc_clnt_submit] 0-rpc-clnt:
submitted request (XID: 0x39620x Program: GlusterFS 3.3, ProgVers: 330,
Proc: 27) to rpc-transport (replicated_vol-client-0)</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.151810] T [rpc-clnt.c:1302:rpc_clnt_record] 0-replicated_vol-client-1:
Auth Info: pid: 7599, uid: 0, gid: 0, owner: 0000000000000000</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.151824] T [rpc-clnt.c:1182:rpc_clnt_record_build_header]
0-rpc-clnt: Request fraglen 456, payload: 360, rpc hdr: 96</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.151889] T [rpc-clnt.c:1499:rpc_clnt_submit] 0-rpc-clnt:
submitted request (XID: 0x39563x Program: GlusterFS 3.3, ProgVers: 330,
Proc: 27) to rpc-transport (replicated_vol-client-1)</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.152239] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-replicated_vol-client-1:
received rpc message (RPC XID: 0x39563x Program: GlusterFS 3.3, ProgVers:
330, Proc: 27) from rpc-transport (replicated_vol-client-1)</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.152484] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-replicated_vol-client-0:
received rpc message (RPC XID: 0x39620x Program: GlusterFS 3.3, ProgVers:
330, Proc: 27) from rpc-transport (replicated_vol-client-0)</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.152582] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix]
0-replicated_vol-replicate-0: pending_matrix: [ 0 3 ]</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.152596] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix]
0-replicated_vol-replicate-0: pending_matrix: [ 0 0 ]</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.152621] D [afr-self-heal-common.c:887:afr_mark_sources]
0-replicated_vol-replicate-0: Number of sources: 1</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.152633] D [afr-self-heal-data.c:825:afr_lookup_select_read_child_by_txn_type]
0-replicated_vol-replicate-0: returning read_child: 0</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.152644] D [afr-common.c:1380:afr_lookup_select_read_child]
0-replicated_vol-replicate-0: Source selected as 0 for /testfl</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:37:49.152657] D [afr-common.c:1117:afr_lookup_build_response_params]
0-replicated_vol-replicate-0: Building lookup response from 0</font><font size=3>
</font><font size=3 color=blue face="Times New Roman"><br>
<br>
We see that when you write on serv1, the RPC request is sent to both replicated_vol-client-0
and replicated_vol-client-1, while when we write on serv0, the request
is sent only to replicated_vol-client-0, and the FUse client is unaware
of the presence of client-1 in the latter case.</font><font size=3> </font><font size=3 color=blue><br>
<br>
I checked a bit more in the logs. When I turn on my trace, I found many
instances of these logs on serv0 but NOT on serv1:</font><font size=3>
</font><font size=2 face="Courier New"><br>
<br>
[2015-01-20 09:21:15.520784] T [fuse-bridge.c:681:fuse_attr_cbk] 0-glusterfs-fuse:
53011: LOOKUP() / => 1</font><font size=3> </font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:17.683088] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-replicated_vol-client-1:
attempting reconnect</font><font size=3> </font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:17.683159] D [name.c:155:client_fill_address_family]
0-replicated_vol-client-1: address-family not specified, guessing it to
be inet from (remote-host: serv1)</font><font size=3> </font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:17.683178] T [name.c:225:af_inet_client_get_remote_sockaddr]
0-replicated_vol-client-1: option remote-port missing in volume replicated_vol-client-1.
Defaulting to 24007</font><font size=3> </font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:17.683191] T [common-utils.c:188:gf_resolve_ip6] 0-resolver:
flushing DNS cache</font><font size=3> </font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:17.683202] T [common-utils.c:195:gf_resolve_ip6] 0-resolver:
DNS cache not present, freshly probing hostname: serv1</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:17.683814] D [common-utils.c:237:gf_resolve_ip6] 0-resolver:
returning ip-192.168.24.81 (port-24007) for hostname: serv1 and port: 24007</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:17.684139] D [common-utils.c:257:gf_resolve_ip6] 0-resolver:
next DNS query will return: ip-192.168.24.81 port-24007</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:17.684164] T [socket.c:731:__socket_nodelay] 0-replicated_vol-client-1:
NODELAY enabled for socket 10</font><font size=3> </font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:17.684177] T [socket.c:790:__socket_keepalive] 0-replicated_vol-client-1:
Keep-alive enabled for socket 10, interval 2, idle: 20</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:17.684236] W [common-utils.c:2247:gf_get_reserved_ports]
0-glusterfs: could not open the file /proc/sys/net/ipv4/ip_local_reserved_ports
for getting reserved ports info (No such file or directory)</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:17.684253] W [common-utils.c:2280:gf_process_reserved_ports]
0-glusterfs: Not able to get reserved ports, hence there is a possibility
that glusterfs may consume reserved port</font><font size=3> <br>
Logs above suggest that mount process couldn't assign a reserved port because
it couldn't find the file </font><font size=2>/proc/sys/net/ipv4/ip_local_reserved_ports<br>
<br>
I guess reboot of the machine fixed it. Wonder why it was not found in
the first place.<br>
<br>
Pranith.</font><font size=3> </font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:17.684660] D [socket.c:605:__socket_shutdown] 0-replicated_vol-client-1:
shutdown() returned -1. Transport endpoint is not connected</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:17.684699] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup]
0-replicated_vol-client-1: cleaning up state in transport object 0x68a630</font><font size=3>
</font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:17.684731] D [socket.c:486:__socket_rwv] 0-replicated_vol-client-1:
EOF on socket</font><font size=3> </font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:17.684750] W [socket.c:514:__socket_rwv] 0-replicated_vol-client-1:
readv failed (No data available)</font><font size=3> </font><font size=2 face="Courier New"><br>
[2015-01-20 09:21:17.684766] D [socket.c:1962:__socket_proto_state_machine]
0-replicated_vol-client-1: reading from socket failed. Error (No data available),
peer (192.168.24.81:49198)</font><font size=3> </font><font size=3 color=blue face="Times New Roman"><br>
<br>
I could not find a 'remote-port' option in /var/lib/glusterd on either
peer. Could somebody tell me where this configuration is looked up from?
Also, sometime later, I rebooted serv0 and that seemed to solve the problem.
However, stop+start of replicated_vol and restart of /etc/init.d/glusterd
did NOT solve the problem.</font><font size=3> <br>
Ignore that log. If no port is given in that volfile, it picks 24007 as
the port, which is the default port where glusterd 'listens'</font><font size=3 color=blue face="Times New Roman"><br>
<br>
<br>
Any help on this matter will be greatly appreciated as I need to provide
robustness assurances for our setup. <br>
<br>
Thanks a lot,</font><font size=3> </font><font size=3 color=blue face="Times New Roman"><br>
Anirban</font><font size=3> </font><font size=3 color=blue face="Times New Roman"><br>
<br>
P.s. Additional details:</font><font size=3> </font><font size=3 color=blue face="Times New Roman"><i><br>
glusterfs version: 3.4.2</i></font><font size=3> </font><font size=3 color=blue face="Times New Roman"><i><br>
Linux kernel version: 2.6.34</i></font><font size=3> </font>
<p><font size=3>=====-----=====-----=====<br>
Notice: The information contained in this e-mail<br>
message and/or attachments to it may contain <br>
confidential or privileged information. If you are <br>
not the intended recipient, any dissemination, use, <br>
review, distribution, printing or copying of the <br>
information contained in this e-mail message <br>
and/or attachments to it are strictly prohibited. If <br>
you have received this communication in error, <br>
please notify us by reply e-mail or telephone and <br>
immediately and permanently delete the message <br>
and any attachments. Thank you </font>
<p><tt><font size=3><br>
<br>
_______________________________________________<br>
Gluster-users mailing list</font></tt><font size=3 color=blue><u><br>
</u></font><a href="mailto:Gluster-users@gluster.org"><tt><font size=3 color=blue><u>Gluster-users@gluster.org</u></font></tt></a><font size=3 color=blue><u><br>
</u></font><a href="http://www.gluster.org/mailman/listinfo/gluster-users"><tt><font size=3 color=blue><u>http://www.gluster.org/mailman/listinfo/gluster-users</u></font></tt></a><font size=3>
</font><tt><font size=2>_______________________________________________<br>
Gluster-users mailing list<br>
Gluster-users@gluster.org<br>
</font></tt><a href="http://www.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://www.gluster.org/mailman/listinfo/gluster-users</font></tt></a>
<p>