<font size=3 color=blue face="Times New Roman">CC gluster-users.</font>
<br>
<br><font size=3 color=blue face="Times New Roman">No, there aren't any
firewall rules in our server. As I write in one of my earlier emails, if
I kill the mount client, and remount the volume, then the problem disappears.
That is to say, this causes the client to refresh remote port data and
from there everything's fine. Also, we dont' use gfapi - and bind() is
always good.</font>
<br>
<br>
<br>
<br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Ben England <bengland@redhat.com></font>
<br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">A Ghoshal <a.ghoshal@tcs.com></font>
<br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">02/05/2015 04:40 AM</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [Gluster-devel]
[Gluster-users] In a replica 2 server, file-updates on one server missing
on the other server #Personal#</font>
<br>
<hr noshade>
<br>
<br>
<br><tt><font size=2>could it be a problem with iptables blocking connections?
DO iptables --list and make sure gluster ports are allowed through,
at both ends. Also, if you are using libgfapi, be sure you use rpc-auth-allow-insecure
if you have a lot of gfapi instances, or else you'll run into problems.<br>
<br>
----- Original Message -----<br>
> From: "A Ghoshal" <a.ghoshal@tcs.com><br>
> To: "Ben England" <bengland@redhat.com><br>
> Sent: Wednesday, February 4, 2015 6:07:10 PM<br>
> Subject: Re: [Gluster-devel] [Gluster-users] In a replica 2 server,
file-updates on one server missing on the other<br>
> server #Personal#<br>
> <br>
> Thanks, Ben, same here :/ I actually get port numbers for glusterfsd
in<br>
> any of the three ways:<br>
> <br>
> 1. gluster volume status <VOLNAME><br>
> 2. command line for glusterfsd on target server.<br>
> 3. if you're really paranoid, get the glusterfsd PID and use netstat.<br>
> <br>
> Looking at the code it seems to me that the whole thing operates on
a<br>
> statd-notify paradigm. Your local mount client registers for notify
on all<br>
> remote glusterfsd's. When remote brick goes down and comes back up,
you<br>
> are notified and then it calls portmap to obtain remote glusterfsd
port.<br>
> <br>
> I see here that both glusterd are up. But somehow the port number
of the<br>
> remote glusterfsd with the mount client is now stale - not sure how
it<br>
> happens. Now, the client keeps trying to connect on the stale port
every 3<br>
> seconds. It gets the return errno of -111 (-ECONNREFUSED) which is
clearly<br>
> indicating that there is not listener on the remote host's IP at this<br>
> port.<br>
> <br>
> Design-wise, could it indicate to the mount client that the port number<br>
> information needs to be refreshed? Would you say this is a bug of
sorts?<br>
> <br>
> <br>
> <br>
> <br>
> From: Ben England <bengland@redhat.com><br>
> To: A Ghoshal <a.ghoshal@tcs.com><br>
> Date: 02/05/2015 03:59 AM<br>
> Subject: Re: [Gluster-devel] [Gluster-users]
In a replica 2 server,<br>
> file-updates on one server missing on the other server #Personal#<br>
> <br>
> <br>
> <br>
> I thought Gluster was based on ONC RPC, which means there are no fixed<br>
> port numbers except for glusterd (24007). The client connects
to<br>
> Glusterd, reads the volfile, and gets the port numbers of the registered<br>
> glusterfsd processes at that time, then it connects to glusterfsd.
Make<br>
> sense? What you need to know is whether glusterfsd is running
or not, and<br>
> whether glusterd is finding out current state of glusterfsd.<br>
> /var/log/glusterfsd/bricks/*log has log files for each glusterfsd
process,<br>
> you might be able to see from that what glusterfsd port number is.<br>
> /var/log/glusterfs/etc*log is glusterd's log file, it might say whether<br>
> glusterd knows about glusterfsd. I'm not as good at troubleshooting
as<br>
> some of the other people are so don't take my word for it.<br>
> -ben<br>
> <br>
> <br>
> ----- Original Message -----<br>
> > From: "A Ghoshal" <a.ghoshal@tcs.com><br>
> > To: gluster-devel@gluster.org<br>
> > Cc: gluster-users@gluster.org, gluster-users-bounces@gluster.org<br>
> > Sent: Wednesday, February 4, 2015 4:36:02 PM<br>
> > Subject: Re: [Gluster-devel] [Gluster-users] In a replica 2 server,<br>
> file-updates on one server missing on the other<br>
> > server #Personal#<br>
> > <br>
> > Sorry for spamming you guys, but this is kind of important for
me to<br>
> debug,<br>
> > so if you saw anything like this before, do let me know. Here's
an<br>
> update:<br>
> > <br>
> > It seems the mount client is attempting connection with an invalid
port<br>
> > number. 49175 is NOT the port number of glusterfsd on serv1<br>
> (192.168.24.8).<br>
> > <br>
> > I got me an strace:<br>
> > <br>
> > [pid 31026] open("/proc/sys/net/ipv4/ip_local_reserved_ports",
O_RDONLY)<br>
> = -1<br>
> > ENOENT (No such file or directory)<br>
> > [pid 31026] write(4, "[2015-02-04 20:39:02.793154] W ["...,
215) = 215<br>
> > [pid 31026] write(4, "[2015-02-04 20:39:02.793289] W ["...,
194) = 194<br>
> > [pid 31026] bind(10, {sa_family=AF_INET, sin_port=htons(1023),<br>
> > sin_addr=inet_addr("192.168.24.80")}, 16) = 0<br>
> > [pid 31026] fcntl(10, F_GETFL) = 0x2 (flags O_RDWR)<br>
> > [pid 31026] fcntl(10, F_SETFL, O_RDWR|O_NONBLOCK) = 0<br>
> > [pid 31026] connect(10, {sa_family=AF_INET, sin_port=htons(49175),<br>
> > sin_addr=inet_addr("192.168.24.81")}, 16) = -1 EINPROGRESS
(Operation<br>
> now in<br>
> > progress)<br>
> > [pid 31026] fcntl(10, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)<br>
> > [pid 31026] fcntl(10, F_SETFL, O_RDWR|O_NONBLOCK) = 0<br>
> > [pid 31026] epoll_ctl(3, EPOLL_CTL_ADD, 10, {EPOLLIN|EPOLLPRI|EPOLLOUT,<br>
> > {u32=10, u64=8589934602}}) = 0<br>
> > [pid 31026] nanosleep({1, 0}, <unfinished ...><br>
> > [pid 31021] <... epoll_wait resumed><br>
> {{EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP,<br>
> > {u32=10, u64=8589934602}}}, 257, 4294967295) = 1<br>
> > [pid 31021] getsockopt(10, SOL_SOCKET, SO_ERROR, [29422518842425455],<br>
> [4]) =<br>
> > 0<br>
> > [pid 31021] shutdown(10, 2 /* send and receive */) = -1 ENOTCONN<br>
> (Transport<br>
> > endpoint is not connected)<br>
> > <br>
> > Which basically told me that connection is attempted via a non-blocking<br>
> > socket at port 49175. The errno from the failure is -ECONNREFUSED,
which<br>
> is<br>
> > what is expected.<br>
> > <br>
> > 807 in socket.c<br>
> > (gdb) bt<br>
> > #0 __socket_connect_finish (this=0x6887a0) at socket.c:807<br>
> > #1 socket_connect_finish (this=0x6887a0) at socket.c:2147<br>
> > #2 0x00007fc863de4c04 in socket_event_handler (fd=<value optimized
out>,<br>
> > idx=<value optimized out>, data=0x6887a0,<br>
> > poll_in=1, poll_out=4, poll_err=<value optimized out>)
at socket.c:2223<br>
> > #3 0x00007fc867f7919f in event_dispatch_epoll_handler<br>
> (event_pool=0x62db70)<br>
> > at event-epoll.c:384<br>
> > #4 event_dispatch_epoll (event_pool=0x62db70) at event-epoll.c:445<br>
> > #5 0x0000000000406b06 in main (argc=4, argv=0x7fff25302c38) at<br>
> > glusterfsd.c:1934<br>
> > (gdb) print *optval<br>
> > Cannot access memory at address 0x6f<br>
> > (gdb) print optval<br>
> > $1 = 111<br>
> > <br>
> > Note that this agrees with the following debug log:<br>
> > <br>
> > [2015-02-03 12:11:33.833647] D<br>
> [socket.c:1962:__socket_proto_state_machine]<br>
> > 0-replicated_vol-1: reading from socket failed. Error (No data<br>
> available),<br>
> > peer (192.168.24.81:49175)<br>
> > <br>
> > There is, of course no service running on port 49175. In fact,
listen<br>
> port<br>
> > for corresponding glusterd on serv1 is 49206. Where does the
mount<br>
> client<br>
> > pick this port number from? I know that if I kill and restart
the mount<br>
> > client on serv0 from command line, then the problem will disappear.
So,<br>
> it's<br>
> > not something that is up with the processes on serv1...<br>
> > <br>
> > Thanks,<br>
> > Anirban<br>
> > <br>
> > <br>
> > From: A Ghoshal/MUM/TCS<br>
> > To: A Ghoshal <a.ghoshal@tcs.com><br>
> > Cc: gluster-users@gluster.org, gluster-users-bounces@gluster.org,<br>
> Pranith<br>
> > Kumar Karampuri <pkarampu@redhat.com><br>
> > Date: 02/05/2015 02:03 AM<br>
> > Subject: Re: [Gluster-users] In a replica 2 server, file-updates
on one<br>
> > server missing on the other server #Personal#<br>
> > Sent by: A Ghoshal<br>
> > <br>
> > <br>
> > <br>
> > Ok, more updates here:<br>
> > <br>
> > I turned on trace and it seems bind to secure port on the mount
client<br>
> with<br>
> > the remote brick is successful - afterwards the connect() fails
to<br>
> complete.<br>
> > I saw these logs:<br>
> > <br>
> > [2015-02-03 12:11:33.832615] T [rpc-clnt.c:422:rpc_clnt_reconnect]<br>
> > 0-replicated_vol-1: attempting reconnect<br>
> > [2015-02-03 12:11:33.832666] D [name.c:155:client_fill_address_family]<br>
> > 0-replicated_vol-1: address-family not specified, guessing it
to be inet<br>
> > from (remote-host: serv1)<br>
> > [2015-02-03 12:11:33.832683] T<br>
> > [name.c:225:af_inet_client_get_remote_sockaddr] 0-replicated_vol-1:<br>
> option<br>
> > remote-port missing in volume replicated_vol-1. Defaulting to
24007<br>
> > [2015-02-03 12:11:33.833083] D [common-utils.c:237:gf_resolve_ip6]<br>
> > 0-resolver: returning ip-192.168.24.81 (port-24007) for hostname:
serv1<br>
> and<br>
> > port: 24007<br>
> > [2015-02-03 12:11:33.833113] T [socket.c:731:__socket_nodelay]<br>
> > 0-replicated_vol-1: NODELAY enabled for socket 10<br>
> > [2015-02-03 12:11:33.833128] T [socket.c:790:__socket_keepalive]<br>
> > 0-replicated_vol-1: Keep-alive enabled for socket 10, interval
2, idle:<br>
> 20<br>
> > [2015-02-03 12:11:33.833188] W<br>
> [common-utils.c:2247:gf_get_reserved_ports]<br>
> > 0-glusterfs: could not open the file<br>
> > /proc/sys/net/ipv4/ip_local_reserved_ports for getting reserved
ports<br>
> info<br>
> > (No such file or directory)<br>
> > [2015-02-03 12:11:33.833204] W<br>
> > [common-utils.c:2280:gf_process_reserved_ports] 0-glusterfs:
Not able to<br>
> get<br>
> > reserved ports, hence there is a possibility that glusterfs may
consume<br>
> > reserved port<br>
> > [2015-02-03 12:11:33.833560] D [socket.c:605:__socket_shutdown]<br>
> > 0-replicated_vol-1: shutdown() returned -1. Transport endpoint
is not<br>
> > connected<br>
> > [2015-02-03 12:11:33.833594] T<br>
> [rpc-clnt.c:519:rpc_clnt_connection_cleanup]<br>
> > 0-replicated_vol-1: cleaning up state in transport object 0x6887a0<br>
> > [2015-02-03 12:11:33.833622] D [socket.c:486:__socket_rwv]<br>
> > 0-replicated_vol-1: EOF on socket<br>
> > [2015-02-03 12:11:33.833635] W [socket.c:514:__socket_rwv]<br>
> > 0-replicated_vol-1: readv failed (No data available)<br>
> > [2015-02-03 12:11:33.833647] D<br>
> [socket.c:1962:__socket_proto_state_machine]<br>
> > 0-replicated_vol-1: reading from socket failed. Error (No data<br>
> available),<br>
> > peer (192.168.24.81:49175)<br>
> > [2015-02-03 12:11:33.833659] D [socket.c:2236:socket_event_handler]<br>
> > 0-transport: disconnecting now<br>
> > [2015-02-03 12:11:33.833684] T<br>
> [rpc-clnt.c:519:rpc_clnt_connection_cleanup]<br>
> > 0-replicated_vol-1: cleaning up state in transport object 0x6887a0<br>
> > <br>
> > After that, I attached the mount process to my gdb session and
attached<br>
> a<br>
> > breakpoint at __socket_shutdown(). It gave me the following traceback<br>
> > <br>
> > Breakpoint 1, __socket_shutdown (this=0x68a7f0) at socket.c:594<br>
> > 594 socket.c: No such file or directory.<br>
> > in socket.c<br>
> > (gdb) bt<br>
> > #0 __socket_shutdown (this=0x68a7f0) at socket.c:594<br>
> > #1 0x00007f3391906553 in __socket_disconnect (this=0x68a7f0)
at<br>
> socket.c:623<br>
> > #2 0x00007f339190856a in socket_connect_finish (this=0x68a7f0)
at<br>
> > socket.c:2160<br>
> > #3 0x00007f339190dc04 in socket_event_handler (fd=<value optimized
out>,<br>
> > idx=<value optimized out>, data=0x68a7f0,<br>
> > poll_in=1, poll_out=4, poll_err=<value optimized out>)
at socket.c:2223<br>
> > #4 0x00007f3395aa219f in event_dispatch_epoll_handler<br>
> (event_pool=0x62d8b0)<br>
> > at event-epoll.c:384<br>
> > #5 event_dispatch_epoll (event_pool=0x62d8b0) at event-epoll.c:445<br>
> > #6 0x0000000000406b06 in main (argc=4, argv=0x7fff71108c38) at<br>
> > glusterfsd.c:1934<br>
> > (gdb)<br>
> > <br>
> > In case somebody could give me a clue or two on my next move
(I'm<br>
> looking at<br>
> > the code in the meantime), I'd be very grateful.<br>
> > <br>
> > Thanks,<br>
> > Anirban<br>
> > <br>
> > <br>
> > <br>
> > <br>
> > From: A Ghoshal <a.ghoshal@tcs.com><br>
> > To: Pranith Kumar Karampuri <pkarampu@redhat.com><br>
> > Cc: gluster-users@gluster.org<br>
> > Date: 02/03/2015 05:24 AM<br>
> > Subject: Re: [Gluster-users] In a replica 2 server, file-updates
on one<br>
> > server missing on the other server #Personal#<br>
> > Sent by: gluster-users-bounces@gluster.org<br>
> > <br>
> > <br>
> > <br>
> > <br>
> > Hi Pranith,<br>
> > <br>
> > I finally understood what you meant the secure ports, because
the issue<br>
> > occurred in one of our setups once more. It seems one of the
clients on<br>
> > serv1 could not open a connection to the glusterfsd running on
serv0.<br>
> I'd<br>
> > actually started a mail trail about it (believing it might be
something<br>
> > else, initially) here:<br>
> > </font></tt><a href="http://www.gluster.org/pipermail/gluster-users/2015-February/020465.html"><tt><font size=2>http://www.gluster.org/pipermail/gluster-users/2015-February/020465.html</font></tt></a><tt><font size=2><br>
> > <br>
> > I think I can write me a rudimentary kind of patch altering<br>
> > af_inet_bind_to_port_lt_ceiling() to get it to call bind with
port 0,<br>
> rather<br>
> > than specify a port explicitly when the client.bind-insecure
is<br>
> specified<br>
> > ...<br>
> > <br>
> > Then I'd need to create a way to set server.allow-insecure using
the cli<br>
> (or<br>
> > if you already sent around the patch to do that like you said
in the<br>
> earlier<br>
> > mail, do let me know). I'll keep you posted about it round here
or @<br>
> > [gluster-devel] if I can get it to work.<br>
> > <br>
> > Thanks a lot,<br>
> > <br>
> > Anirban<br>
> > <br>
> > <br>
> > <br>
> > From: A Ghoshal/MUM/TCS<br>
> > To: Pranith Kumar Karampuri <pkarampu@redhat.com><br>
> > Cc: gluster-users@gluster.org, Niels de Vos <ndevos@redhat.com><br>
> > Date: 01/23/2015 02:45 PM<br>
> > Subject: Re: [Gluster-users] In a replica 2 server, file-updates
on one<br>
> > server missing on the other server #Personal#<br>
> > Sent by: A Ghoshal<br>
> > <br>
> > <br>
> > <br>
> > Oh, I didn't I only read a fragment of the IRC log and assumed<br>
> > --xlator-option would be enough. Apparently it's a lot more work....<br>
> > <br>
> > I do have a query, though. These connections, from one of our
setups,<br>
> are<br>
> > these on secure ports? Or, maybe I didn't get it the first time.....<br>
> > <br>
> > root@serv0:/root> ps -ef | grep replicated_vol<br>
> > root 8851 25307 0 10:03 pts/2 00:00:00 grep replicated_vol<br>
> > root 29751 1 4 Jan21 ? 01:47:20 /usr/sbin/glusterfsd -s serv0<br>
> --volfile-id<br>
> > replicated_vol.serv0.mnt-bricks-replicated_vol-brick -p<br>
> > <br>
> /var/lib/glusterd/vols/_replicated_vol/run/serv0-mnt-bricks-replicated_vol-brick.pid<br>
> > -S /var/run/dff9fa3c93e82f20103f2a3d91adc4a8.socket --brick-name<br>
> > /mnt/bricks/replicated_vol/brick -l<br>
> > /var/log/glusterfs/bricks/mnt-bricks-replicated_vol-brick.log<br>
> > --xlator-option<br>
> *-posix.glusterd-uuid=1a1d1ebc-4b92-428f-b66b-9c5efa49574d<br>
> > --brick-port 49185 --xlator-option<br>
> replicated_vol-server.listen-port=49185<br>
> > root 30399 1 0 Jan21 ? 00:19:06 /usr/sbin/glusterfs<br>
> > --volfile-id=replicated_vol --volfile-server=serv0 /mnt/replicated_vol<br>
> > <br>
> > root@serv0:/root> netstat -p | grep 30399<br>
> > tcp 0 0 serv0:969 serv0:49185 ESTABLISHED 30399/glusterfs<br>
> > tcp 0 0 serv0:999 serv1:49159 ESTABLISHED 30399/glusterfs<br>
> > tcp 0 0 serv0:1023 serv0:24007 ESTABLISHED 30399/glusterfs<br>
> > root@serv0:/root><br>
> > <br>
> > Thanks again,<br>
> > Anirban<br>
> > <br>
> > <br>
> > <br>
> > <br>
> > From: Pranith Kumar Karampuri <pkarampu@redhat.com><br>
> > To: A Ghoshal <a.ghoshal@tcs.com><br>
> > Cc: gluster-users@gluster.org, Niels de Vos <ndevos@redhat.com><br>
> > Date: 01/23/2015 01:58 PM<br>
> > Subject: Re: [Gluster-users] In a replica 2 server, file-updates
on one<br>
> > server missing on the other server #Personal#<br>
> > <br>
> > <br>
> > <br>
> > <br>
> > <br>
> > On 01/23/2015 01:54 PM, A Ghoshal wrote:<br>
> > Thanks a lot, Pranith.<br>
> > <br>
> > We'll set this option on our test servers and keep the setup
under<br>
> > observation.<br>
> > How did you get the bind-insecure option working?<br>
> > I guess I will send a patch to make it 'volume set option'<br>
> > <br>
> > Pranith<br>
> > <br>
> > Thanks,<br>
> > Anirban<br>
> > <br>
> > <br>
> > <br>
> > From: Pranith Kumar Karampuri <pkarampu@redhat.com><br>
> > To: A Ghoshal <a.ghoshal@tcs.com><br>
> > Cc: gluster-users@gluster.org , Niels de Vos <ndevos@redhat.com><br>
> > Date: 01/23/2015 01:28 PM<br>
> > Subject: Re: [Gluster-users] In a replica 2 server, file-updates
on one<br>
> > server missing on the other server #Personal#<br>
> > <br>
> > <br>
> > <br>
> > <br>
> > <br>
> > On 01/22/2015 02:07 PM, A Ghoshal wrote:<br>
> > Hi Pranith,<br>
> > <br>
> > Yes, the very same ( chalcogen_eg_oxygen@yahoo.com ). Justin
Clift sent<br>
> me a<br>
> > mail a while back telling me that it is better if we all use
our<br>
> business<br>
> > email addresses so I made me a new profile.<br>
> > <br>
> > Glusterfs complains about /proc/sys/net/ipv4/ip_local_reserved_ports<br>
> because<br>
> > we use a really old Linux kernel (2.6.34) wherein this feature
is not<br>
> > present. We plan to upgrade our Linux so often but each time
we are<br>
> > dissuaded from it by some compatibility issue or the other. So,
we get<br>
> this<br>
> > log every time - on both good volumes and bad ones. What bothered
me was<br>
> > this (on serv1):<br>
> > Basically to make the connections to servers i.e. bricks clients
need to<br>
> > choose secure ports i.e. port less than 1024. Since this file
is not<br>
> > present, it is not binding to any port as per the code I just
checked.<br>
> There<br>
> > is an option called client-bind-insecure which bypasses this
check. I<br>
> feel<br>
> > that is one (probably only way) to get around this.<br>
> > You have to "volume set server.allow-insecure on" option
and<br>
> bind-insecure<br>
> > option.<br>
> > CC ndevos who seemed to have helped someone set bind-insecure
option<br>
> > correctly here ( </font></tt><a href="http://irclog.perlgeek.de/gluster/2014-04-09/text"><tt><font size=2>http://irclog.perlgeek.de/gluster/2014-04-09/text</font></tt></a><tt><font size=2>
)<br>
> > <br>
> > Pranith<br>
> > <br>
> > [2015-01-20 09:37:49.151744] T<br>
> [rpc-clnt.c:1182:rpc_clnt_record_build_header]<br>
> > 0-rpc-clnt: Request fraglen 456, payload: 360, rpc hdr: 96<br>
> > [2015-01-20 09:37:49.151780] T [rpc-clnt.c:1499:rpc_clnt_submit]<br>
> 0-rpc-clnt:<br>
> > submitted request (XID: 0x39620x Program: GlusterFS 3.3, ProgVers:
330,<br>
> > Proc: 27) to rpc-transport (replicated_vol-client-0)<br>
> > [2015-01-20 09:37:49.151810] T [rpc-clnt.c:1302:rpc_clnt_record]<br>
> > 0-replicated_vol-client-1: Auth Info: pid: 7599, uid: 0, gid:
0, owner:<br>
> > 0000000000000000<br>
> > [2015-01-20 09:37:49.151824] T<br>
> [rpc-clnt.c:1182:rpc_clnt_record_build_header]<br>
> > 0-rpc-clnt: Request fraglen 456, payload: 360, rpc hdr: 96<br>
> > [2015-01-20 09:37:49.151889] T [rpc-clnt.c:1499:rpc_clnt_submit]<br>
> 0-rpc-clnt:<br>
> > submitted request (XID: 0x39563x Program: GlusterFS 3.3, ProgVers:
330,<br>
> > Proc: 27) to rpc-transport (replicated_vol-client-1)<br>
> > [2015-01-20 09:37:49.152239] T [rpc-clnt.c:669:rpc_clnt_reply_init]<br>
> > 0-replicated_vol-client-1: received rpc message (RPC XID: 0x39563x<br>
> Program:<br>
> > GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport<br>
> > (replicated_vol-client-1)<br>
> > [2015-01-20 09:37:49.152484] T [rpc-clnt.c:669:rpc_clnt_reply_init]<br>
> > 0-replicated_vol-client-0: received rpc message (RPC XID: 0x39620x<br>
> Program:<br>
> > GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport<br>
> > (replicated_vol-client-0)<br>
> > <br>
> > When I write on the good server (serv1), we see that an RPC request
is<br>
> sent<br>
> > to both client-0 and client-1. While, when I write on the bad
server<br>
> > (serv0), the RPC request is sent only to client-0, which is why
it is no<br>
> > wonder that the writes are not synced over to serv1. Somehow
I could not<br>
> > make the daemon on serv0 understand that there are two up-children
and<br>
> not<br>
> > just one.<br>
> > <br>
> > One additional detail - since we are using a kernel that's too
old, we<br>
> do not<br>
> > have the (Anand Avati's?) FUse readdirplus patches, either. I've
noticed<br>
> > that the fixes in the readdirplus version of glusterfs aren't
always<br>
> > guaranteed to be present on the non-readdirplus version of the
patches.<br>
> I'd<br>
> > filed a bug around one such anomaly back, but never got around
to<br>
> writing a<br>
> > patch for it (sorry!) Here it is:<br>
> > </font></tt><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1062287"><tt><font size=2>https://bugzilla.redhat.com/show_bug.cgi?id=1062287</font></tt></a><tt><font size=2><br>
> > I don't this has anything to do with readdirplus.<br>
> > <br>
> > Maybe something on similar lines here?<br>
> > <br>
> > Thanks,<br>
> > Anirban<br>
> > <br>
> > P.s. Please ignore the #Personal# in the subject line - we need
to do<br>
> that to<br>
> > push mails to the public domain past the email filter safely.<br>
> > <br>
> > <br>
> > <br>
> > From: Pranith Kumar Karampuri <pkarampu@redhat.com><br>
> > To: A Ghoshal <a.ghoshal@tcs.com> , gluster-users@gluster.org<br>
> > Date: 01/22/2015 12:09 AM<br>
> > Subject: Re: [Gluster-users] In a replica 2 server, file-updates
on one<br>
> > server missing on the other server<br>
> > <br>
> > <br>
> > <br>
> > <br>
> > hi,<br>
> > Responses inline.<br>
> > <br>
> > PS: You are chalkogen_oxygen?<br>
> > <br>
> > Pranith<br>
> > On 01/20/2015 05:34 PM, A Ghoshal wrote:<br>
> > Hello,<br>
> > <br>
> > I am using the following replicated volume:<br>
> > <br>
> > root@serv0:~> gluster v info replicated_vol<br>
> > <br>
> > Volume Name: replicated_vol<br>
> > Type: Replicate<br>
> > Volume ID: 26d111e3-7e4c-479e-9355-91635ab7f1c2<br>
> > Status: Started<br>
> > Number of Bricks: 1 x 2 = 2<br>
> > Transport-type: tcp<br>
> > Bricks:<br>
> > Brick1: serv0:/mnt/bricks/replicated_vol/brick<br>
> > Brick2: serv1:/mnt/bricks/replicated_vol/brick<br>
> > Options Reconfigured:<br>
> > diagnostics.client-log-level: INFO<br>
> > network.ping-timeout: 10<br>
> > nfs.enable-ino32: on<br>
> > cluster.self-heal-daemon: on<br>
> > nfs.disable: off<br>
> > <br>
> > replicated_vol is mounted at /mnt/replicated_vol on both serv0
and<br>
> serv1. If<br>
> > I do the following on serv0:<br>
> > <br>
> > root@serv0:~>echo "cranberries" > /mnt/replicated_vol/testfile<br>
> > root@serv0:~>echo "tangerines" >> /mnt/replicated_vol/testfile<br>
> > <br>
> > And then I check for the state of the replicas in the bricks,
then I<br>
> find<br>
> > that<br>
> > <br>
> > root@serv0:~>cat /mnt/bricks/replicated_vol/brick/testfile<br>
> > cranberries<br>
> > tangerines<br>
> > root@serv0:~><br>
> > <br>
> > root@serv1:~>cat /mnt/bricks/replicated_vol/brick/testfile<br>
> > root@serv1:~><br>
> > <br>
> > As may be seen, the replica on serv1 is blank, when I write into<br>
> testfile<br>
> > from serv0 (even though the file is created on both bricks).<br>
> Interestingly,<br>
> > if I write something to the file at serv1, then the two replicas
become<br>
> > identical.<br>
> > <br>
> > root@serv1:~>echo "artichokes" >> /mnt/replicated_vol/testfile<br>
> > <br>
> > root@serv1:~>cat /mnt/bricks/replicated_vol/brick/testfile<br>
> > cranberries<br>
> > tangerines<br>
> > artichokes<br>
> > root@serv1:~><br>
> > <br>
> > root@serv0:~>cat /mnt/bricks/replicated_vol/brick/testfile<br>
> > cranberries<br>
> > tangerines<br>
> > artichokes<br>
> > root@serv0:~><br>
> > <br>
> > So, I dabbled into the logs a little bit, after upping the diagnostic<br>
> level,<br>
> > and this is what I saw:<br>
> > <br>
> > When I write on serv0 (bad case):<br>
> > <br>
> > [2015-01-20 09:21:52.197704] T [fuse-bridge.c:546:fuse_lookup_resume]<br>
> > 0-glusterfs-fuse: 53027: LOOKUP<br>
> > /testfl(f0a76987-8a42-47a2-b027-a823254b736b)<br>
> > [2015-01-20 09:21:52.197959] D<br>
> > [afr-common.c:131:afr_lookup_xattr_req_prepare]<br>
> > 0-replicated_vol-replicate-0: /testfl: failed to get the gfid
from dict<br>
> > [2015-01-20 09:21:52.198006] T [rpc-clnt.c:1302:rpc_clnt_record]<br>
> > 0-replicated_vol-client-0: Auth Info: pid: 28151, uid: 0, gid:
0, owner:<br>
> > 0000000000000000<br>
> > [2015-01-20 09:21:52.198024] T<br>
> [rpc-clnt.c:1182:rpc_clnt_record_build_header]<br>
> > 0-rpc-clnt: Request fraglen 456, payload: 360, rpc hdr: 96<br>
> > [2015-01-20 09:21:52.198108] T [rpc-clnt.c:1499:rpc_clnt_submit]<br>
> 0-rpc-clnt:<br>
> > submitted request (XID: 0x78163x Program: GlusterFS 3.3, ProgVers:
330,<br>
> > Proc: 27) to rpc-transport (replicated_vol-client-0)<br>
> > [2015-01-20 09:21:52.198565] T [rpc-clnt.c:669:rpc_clnt_reply_init]<br>
> > 0-replicated_vol-client-0: received rpc message (RPC XID: 0x78163x<br>
> Program:<br>
> > GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport<br>
> > (replicated_vol-client-0)<br>
> > [2015-01-20 09:21:52.198640] D<br>
> > [afr-self-heal-common.c:138:afr_sh_print_pending_matrix]<br>
> > 0-replicated_vol-replicate-0: pending_matrix: [ 0 3 ]<br>
> > [2015-01-20 09:21:52.198669] D<br>
> > [afr-self-heal-common.c:138:afr_sh_print_pending_matrix]<br>
> > 0-replicated_vol-replicate-0: pending_matrix: [ 0 0 ]<br>
> > [2015-01-20 09:21:52.198681] D<br>
> [afr-self-heal-common.c:887:afr_mark_sources]<br>
> > 0-replicated_vol-replicate-0: Number of sources: 1<br>
> > [2015-01-20 09:21:52.198694] D<br>
> > [afr-self-heal-data.c:825:afr_lookup_select_read_child_by_txn_type]<br>
> > 0-replicated_vol-replicate-0: returning read_child: 0<br>
> > [2015-01-20 09:21:52.198705] D<br>
> > [afr-common.c:1380:afr_lookup_select_read_child]<br>
> > 0-replicated_vol-replicate-0: Source selected as 0 for /testfl<br>
> > [2015-01-20 09:21:52.198720] D<br>
> > [afr-common.c:1117:afr_lookup_build_response_params]<br>
> > 0-replicated_vol-replicate-0: Building lookup response from 0<br>
> > [2015-01-20 09:21:52.198732] D<br>
> > [afr-common.c:1732:afr_lookup_perform_self_heal]<br>
> > 0-replicated_vol-replicate-0: Only 1 child up - do not attempt
to detect<br>
> > self heal<br>
> > <br>
> > When I write on serv1 (good case):<br>
> > <br>
> > [2015-01-20 09:37:49.151506] T [fuse-bridge.c:546:fuse_lookup_resume]<br>
> > 0-glusterfs-fuse: 31212: LOOKUP</font></tt>
<br><tt><font size=2>> > /testfl(f0a76987-8a42-47a2-b027-a823254b736b)<br>
> > [2015-01-20 09:37:49.151683] D<br>
> > [afr-common.c:131:afr_lookup_xattr_req_prepare]<br>
> > 0-replicated_vol-replicate-0: /testfl: failed to get the gfid
from dict<br>
> > [2015-01-20 09:37:49.151726] T [rpc-clnt.c:1302:rpc_clnt_record]<br>
> > 0-replicated_vol-client-0: Auth Info: pid: 7599, uid: 0, gid:
0, owner:<br>
> > 0000000000000000<br>
> > [2015-01-20 09:37:49.151744] T<br>
> [rpc-clnt.c:1182:rpc_clnt_record_build_header]<br>
> > 0-rpc-clnt: Request fraglen 456, payload: 360, rpc hdr: 96<br>
> > [2015-01-20 09:37:49.151780] T [rpc-clnt.c:1499:rpc_clnt_submit]<br>
> 0-rpc-clnt:<br>
> > submitted request (XID: 0x39620x Program: GlusterFS 3.3, ProgVers:
330,<br>
> > Proc: 27) to rpc-transport (replicated_vol-client-0)<br>
> > [2015-01-20 09:37:49.151810] T [rpc-clnt.c:1302:rpc_clnt_record]<br>
> > 0-replicated_vol-client-1: Auth Info: pid: 7599, uid: 0, gid:
0, owner:<br>
> > 0000000000000000<br>
> > [2015-01-20 09:37:49.151824] T<br>
> [rpc-clnt.c:1182:rpc_clnt_record_build_header]<br>
> > 0-rpc-clnt: Request fraglen 456, payload: 360, rpc hdr: 96<br>
> > [2015-01-20 09:37:49.151889] T [rpc-clnt.c:1499:rpc_clnt_submit]<br>
> 0-rpc-clnt:<br>
> > submitted request (XID: 0x39563x Program: GlusterFS 3.3, ProgVers:
330,<br>
> > Proc: 27) to rpc-transport (replicated_vol-client-1)<br>
> > [2015-01-20 09:37:49.152239] T [rpc-clnt.c:669:rpc_clnt_reply_init]<br>
> > 0-replicated_vol-client-1: received rpc message (RPC XID: 0x39563x<br>
> Program:<br>
> > GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport<br>
> > (replicated_vol-client-1)<br>
> > [2015-01-20 09:37:49.152484] T [rpc-clnt.c:669:rpc_clnt_reply_init]<br>
> > 0-replicated_vol-client-0: received rpc message (RPC XID: 0x39620x<br>
> Program:<br>
> > GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport<br>
> > (replicated_vol-client-0)<br>
> > [2015-01-20 09:37:49.152582] D<br>
> > [afr-self-heal-common.c:138:afr_sh_print_pending_matrix]<br>
> > 0-replicated_vol-replicate-0: pending_matrix: [ 0 3 ]<br>
> > [2015-01-20 09:37:49.152596] D<br>
> > [afr-self-heal-common.c:138:afr_sh_print_pending_matrix]<br>
> > 0-replicated_vol-replicate-0: pending_matrix: [ 0 0 ]<br>
> > [2015-01-20 09:37:49.152621] D<br>
> [afr-self-heal-common.c:887:afr_mark_sources]<br>
> > 0-replicated_vol-replicate-0: Number of sources: 1<br>
> > [2015-01-20 09:37:49.152633] D<br>
> > [afr-self-heal-data.c:825:afr_lookup_select_read_child_by_txn_type]<br>
> > 0-replicated_vol-replicate-0: returning read_child: 0<br>
> > [2015-01-20 09:37:49.152644] D<br>
> > [afr-common.c:1380:afr_lookup_select_read_child]<br>
> > 0-replicated_vol-replicate-0: Source selected as 0 for /testfl<br>
> > [2015-01-20 09:37:49.152657] D<br>
> > [afr-common.c:1117:afr_lookup_build_response_params]<br>
> > 0-replicated_vol-replicate-0: Building lookup response from 0<br>
> > <br>
> > We see that when you write on serv1, the RPC request is sent
to both<br>
> > replicated_vol-client-0 and replicated_vol-client-1, while when
we write<br>
> on<br>
> > serv0, the request is sent only to replicated_vol-client-0, and
the FUse<br>
> > client is unaware of the presence of client-1 in the latter case.<br>
> > <br>
> > I checked a bit more in the logs. When I turn on my trace, I
found many<br>
> > instances of these logs on serv0 but NOT on serv1:<br>
> > <br>
> > [2015-01-20 09:21:15.520784] T [fuse-bridge.c:681:fuse_attr_cbk]<br>
> > 0-glusterfs-fuse: 53011: LOOKUP() / => 1<br>
> > [2015-01-20 09:21:17.683088] T [rpc-clnt.c:422:rpc_clnt_reconnect]<br>
> > 0-replicated_vol-client-1: attempting reconnect<br>
> > [2015-01-20 09:21:17.683159] D [name.c:155:client_fill_address_family]<br>
> > 0-replicated_vol-client-1: address-family not specified, guessing
it to<br>
> be<br>
> > inet from (remote-host: serv1)<br>
> > [2015-01-20 09:21:17.683178] T<br>
> > [name.c:225:af_inet_client_get_remote_sockaddr]<br>
> 0-replicated_vol-client-1:<br>
> > option remote-port missing in volume replicated_vol-client-1.
Defaulting<br>
> to<br>
> > 24007<br>
> > [2015-01-20 09:21:17.683191] T [common-utils.c:188:gf_resolve_ip6]<br>
> > 0-resolver: flushing DNS cache<br>
> > [2015-01-20 09:21:17.683202] T [common-utils.c:195:gf_resolve_ip6]<br>
> > 0-resolver: DNS cache not present, freshly probing hostname:
serv1<br>
> > [2015-01-20 09:21:17.683814] D [common-utils.c:237:gf_resolve_ip6]<br>
> > 0-resolver: returning ip-192.168.24.81 (port-24007) for hostname:
serv1<br>
> and<br>
> > port: 24007<br>
> > [2015-01-20 09:21:17.684139] D [common-utils.c:257:gf_resolve_ip6]<br>
> > 0-resolver: next DNS query will return: ip-192.168.24.81 port-24007<br>
> > [2015-01-20 09:21:17.684164] T [socket.c:731:__socket_nodelay]<br>
> > 0-replicated_vol-client-1: NODELAY enabled for socket 10<br>
> > [2015-01-20 09:21:17.684177] T [socket.c:790:__socket_keepalive]<br>
> > 0-replicated_vol-client-1: Keep-alive enabled for socket 10,
interval 2,<br>
> > idle: 20<br>
> > [2015-01-20 09:21:17.684236] W<br>
> [common-utils.c:2247:gf_get_reserved_ports]<br>
> > 0-glusterfs: could not open the file<br>
> > /proc/sys/net/ipv4/ip_local_reserved_ports for getting reserved
ports<br>
> info<br>
> > (No such file or directory)<br>
> > [2015-01-20 09:21:17.684253] W<br>
> > [common-utils.c:2280:gf_process_reserved_ports] 0-glusterfs:
Not able to<br>
> get<br>
> > reserved ports, hence there is a possibility that glusterfs may
consume<br>
> > reserved port<br>
> > Logs above suggest that mount process couldn't assign a reserved
port<br>
> because<br>
> > it couldn't find the file /proc/sys/net/ipv4/ip_local_reserved_ports<br>
> > <br>
> > I guess reboot of the machine fixed it. Wonder why it was not
found in<br>
> the<br>
> > first place.<br>
> > <br>
> > Pranith.<br>
> > [2015-01-20 09:21:17.684660] D [socket.c:605:__socket_shutdown]<br>
> > 0-replicated_vol-client-1: shutdown() returned -1. Transport
endpoint is<br>
> not<br>
> > connected<br>
> > [2015-01-20 09:21:17.684699] T<br>
> [rpc-clnt.c:519:rpc_clnt_connection_cleanup]<br>
> > 0-replicated_vol-client-1: cleaning up state in transport object<br>
> 0x68a630<br>
> > [2015-01-20 09:21:17.684731] D [socket.c:486:__socket_rwv]<br>
> > 0-replicated_vol-client-1: EOF on socket<br>
> > [2015-01-20 09:21:17.684750] W [socket.c:514:__socket_rwv]<br>
> > 0-replicated_vol-client-1: readv failed (No data available)<br>
> > [2015-01-20 09:21:17.684766] D<br>
> [socket.c:1962:__socket_proto_state_machine]<br>
> > 0-replicated_vol-client-1: reading from socket failed. Error
(No data<br>
> > available), peer (192.168.24.81:49198)<br>
> > <br>
> > I could not find a 'remote-port' option in /var/lib/glusterd
on either<br>
> peer.<br>
> > Could somebody tell me where this configuration is looked up
from? Also,<br>
> > sometime later, I rebooted serv0 and that seemed to solve the
problem.<br>
> > However, stop+start of replicated_vol and restart of<br>
> /etc/init.d/glusterd<br>
> > did NOT solve the problem.<br>
> > Ignore that log. If no port is given in that volfile, it picks
24007 as<br>
> the<br>
> > port, which is the default port where glusterd 'listens'<br>
> > <br>
> > <br>
> > Any help on this matter will be greatly appreciated as I need
to provide<br>
> > robustness assurances for our setup.<br>
> > <br>
> > Thanks a lot,<br>
> > Anirban<br>
> > <br>
> > P.s. Additional details:<br>
> > glusterfs version: 3.4.2<br>
> > Linux kernel version: 2.6.34<br>
> > <br>
> > =====-----=====-----=====<br>
> > Notice: The information contained in this e-mail<br>
> > message and/or attachments to it may contain<br>
> > confidential or privileged information. If you are<br>
> > not the intended recipient, any dissemination, use,<br>
> > review, distribution, printing or copying of the<br>
> > information contained in this e-mail message<br>
> > and/or attachments to it are strictly prohibited. If<br>
> > you have received this communication in error,<br>
> > please notify us by reply e-mail or telephone and<br>
> > immediately and permanently delete the message<br>
> > and any attachments. Thank you<br>
> > <br>
> > <br>
> > <br>
> > _______________________________________________<br>
> > Gluster-users mailing list<br>
> > Gluster-users@gluster.org<br>
> > </font></tt><a href="http://www.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://www.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>
> > _______________________________________________<br>
> > Gluster-users mailing list<br>
> > Gluster-users@gluster.org<br>
> > </font></tt><a href="http://www.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://www.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>
> > <br>
> > <br>
> > <br>
> > _______________________________________________<br>
> > Gluster-devel mailing list<br>
> > Gluster-devel@gluster.org<br>
> > </font></tt><a href="http://www.gluster.org/mailman/listinfo/gluster-devel"><tt><font size=2>http://www.gluster.org/mailman/listinfo/gluster-devel</font></tt></a><tt><font size=2><br>
> > <br>
> <br>
> <br>
</font></tt>
<br>