<font size=3 color=blue face="Times New Roman">Hello,</font>
<br>
<br><font size=3 color=blue face="Times New Roman">I am using the following
replicated volume:</font>
<br>
<br><font size=2 face="Courier New">root@serv0:~> gluster v info replicated_vol</font>
<br>
<br><font size=2 face="Courier New">Volume Name: replicated_vol</font>
<br><font size=2 face="Courier New">Type: Replicate</font>
<br><font size=2 face="Courier New">Volume ID: 26d111e3-7e4c-479e-9355-91635ab7f1c2</font>
<br><font size=2 face="Courier New">Status: Started</font>
<br><font size=2 face="Courier New">Number of Bricks: 1 x 2 = 2</font>
<br><font size=2 face="Courier New">Transport-type: tcp</font>
<br><font size=2 face="Courier New">Bricks:</font>
<br><font size=2 face="Courier New">Brick1: serv0:/mnt/bricks/replicated_vol/brick</font>
<br><font size=2 face="Courier New">Brick2: serv1:/mnt/bricks/replicated_vol/brick</font>
<br><font size=2 face="Courier New">Options Reconfigured:</font>
<br><font size=2 face="Courier New">diagnostics.client-log-level: INFO</font>
<br><font size=2 face="Courier New">network.ping-timeout: 10</font>
<br><font size=2 face="Courier New">nfs.enable-ino32: on</font>
<br><font size=2 face="Courier New">cluster.self-heal-daemon: on</font>
<br><font size=2 face="Courier New">nfs.disable: off</font>
<br>
<br><font size=3 color=blue face="Times New Roman">replicated_vol is mounted
at /mnt/replicated_vol on both serv0 and serv1. If I do the following on
serv0:</font>
<br>
<br><font size=2 face="Courier New">root@serv0:~>echo "cranberries"
> /mnt/replicated_vol/testfile</font>
<br><font size=2 face="Courier New">root@serv0:~>echo "tangerines"
>> /mnt/replicated_vol/testfile</font>
<br>
<br><font size=3 color=blue face="Times New Roman">And then I check for
the state of the replicas in the bricks, then I find that</font>
<br>
<br><font size=2 face="Courier New">root@serv0:~>cat /mnt/bricks/replicated_vol/brick/testfile</font>
<br><font size=2 face="Courier New">cranberries</font>
<br><font size=2 face="Courier New">tangerines</font>
<br><font size=2 face="Courier New">root@serv0:~></font>
<br>
<br><font size=2 face="Courier New">root@serv1:~>cat /mnt/bricks/replicated_vol/brick/testfile</font>
<br><font size=2 face="Courier New">root@serv1:~></font>
<br>
<br><font size=3 color=blue face="Times New Roman">As may be seen, the
replica on serv1 is blank, when I write into testfile from serv0 (even
though the file is created on both bricks). Interestingly, if I write something
to the file at serv1, then the two replicas become identical.</font>
<br>
<br><font size=2 face="Courier New">root@serv1:~>echo "artichokes"
>> /mnt/replicated_vol/testfile</font>
<br>
<br><font size=2 face="Courier New">root@serv1:~>cat /mnt/bricks/replicated_vol/brick/testfile</font>
<br><font size=2 face="Courier New">cranberries</font>
<br><font size=2 face="Courier New">tangerines</font>
<br><font size=2 face="Courier New">artichokes</font>
<br><font size=2 face="Courier New">root@serv1:~></font>
<br>
<br><font size=2 face="Courier New">root@serv0:~>cat /mnt/bricks/replicated_vol/brick/testfile</font>
<br><font size=2 face="Courier New">cranberries</font>
<br><font size=2 face="Courier New">tangerines</font>
<br><font size=2 face="Courier New">artichokes</font>
<br><font size=2 face="Courier New">root@serv0:~></font>
<br>
<br><font size=3 color=blue face="Times New Roman">So, I dabbled into the
logs a little bit, after upping the diagnostic level, and this is what
I saw:</font>
<br>
<br><font size=3 face="Times New Roman"><b><u>When I write on serv0 (bad
case):</u></b></font>
<br>
<br><font size=2 face="Courier New">[2015-01-20 09:21:52.197704] T [fuse-bridge.c:546:fuse_lookup_resume]
0-glusterfs-fuse: 53027: LOOKUP /testfl(f0a76987-8a42-47a2-b027-a823254b736b)</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:52.197959] D [afr-common.c:131:afr_lookup_xattr_req_prepare]
0-replicated_vol-replicate-0: /testfl: failed to get the gfid from dict</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:52.198006] T [rpc-clnt.c:1302:rpc_clnt_record]
0-replicated_vol-client-0: Auth Info: pid: 28151, uid: 0, gid: 0, owner:
0000000000000000</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:52.198024] T [rpc-clnt.c:1182:rpc_clnt_record_build_header]
0-rpc-clnt: Request fraglen 456, payload: 360, rpc hdr: 96</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:52.198108] T [rpc-clnt.c:1499:rpc_clnt_submit]
0-rpc-clnt: submitted request (XID: 0x78163x Program: GlusterFS 3.3, ProgVers:
330, Proc: 27) to rpc-transport (replicated_vol-client-0)</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:52.198565] T [rpc-clnt.c:669:rpc_clnt_reply_init]
0-replicated_vol-client-0: received rpc message (RPC XID: 0x78163x Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (replicated_vol-client-0)</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:52.198640] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix]
0-replicated_vol-replicate-0: pending_matrix: [ 0 3 ]</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:52.198669] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix]
0-replicated_vol-replicate-0: pending_matrix: [ 0 0 ]</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:52.198681] D [afr-self-heal-common.c:887:afr_mark_sources]
0-replicated_vol-replicate-0: Number of sources: 1</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:52.198694] D [afr-self-heal-data.c:825:afr_lookup_select_read_child_by_txn_type]
0-replicated_vol-replicate-0: returning read_child: 0</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:52.198705] D [afr-common.c:1380:afr_lookup_select_read_child]
0-replicated_vol-replicate-0: Source selected as 0 for /testfl</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:52.198720] D [afr-common.c:1117:afr_lookup_build_response_params]
0-replicated_vol-replicate-0: Building lookup response from 0</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:52.198732] D [afr-common.c:1732:afr_lookup_perform_self_heal]
0-replicated_vol-replicate-0: Only 1 child up - do not attempt to detect
self heal</font>
<br>
<br><font size=3 face="Times New Roman"><b><u>When I write on serv1 (good
case):</u></b></font>
<br>
<br><font size=2 face="Courier New">[2015-01-20 09:37:49.151506] T [fuse-bridge.c:546:fuse_lookup_resume]
0-glusterfs-fuse: 31212: LOOKUP /testfl(f0a76987-8a42-47a2-b027-a823254b736b)</font>
<br><font size=2 face="Courier New">[2015-01-20 09:37:49.151683] D [afr-common.c:131:afr_lookup_xattr_req_prepare]
0-replicated_vol-replicate-0: /testfl: failed to get the gfid from dict</font>
<br><font size=2 face="Courier New">[2015-01-20 09:37:49.151726] T [rpc-clnt.c:1302:rpc_clnt_record]
0-replicated_vol-client-0: Auth Info: pid: 7599, uid: 0, gid: 0, owner:
0000000000000000</font>
<br><font size=2 face="Courier New">[2015-01-20 09:37:49.151744] T [rpc-clnt.c:1182:rpc_clnt_record_build_header]
0-rpc-clnt: Request fraglen 456, payload: 360, rpc hdr: 96</font>
<br><font size=2 face="Courier New">[2015-01-20 09:37:49.151780] T [rpc-clnt.c:1499:rpc_clnt_submit]
0-rpc-clnt: submitted request (XID: 0x39620x Program: GlusterFS 3.3, ProgVers:
330, Proc: 27) to rpc-transport (replicated_vol-client-0)</font>
<br><font size=2 face="Courier New">[2015-01-20 09:37:49.151810] T [rpc-clnt.c:1302:rpc_clnt_record]
0-replicated_vol-client-1: Auth Info: pid: 7599, uid: 0, gid: 0, owner:
0000000000000000</font>
<br><font size=2 face="Courier New">[2015-01-20 09:37:49.151824] T [rpc-clnt.c:1182:rpc_clnt_record_build_header]
0-rpc-clnt: Request fraglen 456, payload: 360, rpc hdr: 96</font>
<br><font size=2 face="Courier New">[2015-01-20 09:37:49.151889] T [rpc-clnt.c:1499:rpc_clnt_submit]
0-rpc-clnt: submitted request (XID: 0x39563x Program: GlusterFS 3.3, ProgVers:
330, Proc: 27) to rpc-transport (replicated_vol-client-1)</font>
<br><font size=2 face="Courier New">[2015-01-20 09:37:49.152239] T [rpc-clnt.c:669:rpc_clnt_reply_init]
0-replicated_vol-client-1: received rpc message (RPC XID: 0x39563x Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (replicated_vol-client-1)</font>
<br><font size=2 face="Courier New">[2015-01-20 09:37:49.152484] T [rpc-clnt.c:669:rpc_clnt_reply_init]
0-replicated_vol-client-0: received rpc message (RPC XID: 0x39620x Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (replicated_vol-client-0)</font>
<br><font size=2 face="Courier New">[2015-01-20 09:37:49.152582] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix]
0-replicated_vol-replicate-0: pending_matrix: [ 0 3 ]</font>
<br><font size=2 face="Courier New">[2015-01-20 09:37:49.152596] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix]
0-replicated_vol-replicate-0: pending_matrix: [ 0 0 ]</font>
<br><font size=2 face="Courier New">[2015-01-20 09:37:49.152621] D [afr-self-heal-common.c:887:afr_mark_sources]
0-replicated_vol-replicate-0: Number of sources: 1</font>
<br><font size=2 face="Courier New">[2015-01-20 09:37:49.152633] D [afr-self-heal-data.c:825:afr_lookup_select_read_child_by_txn_type]
0-replicated_vol-replicate-0: returning read_child: 0</font>
<br><font size=2 face="Courier New">[2015-01-20 09:37:49.152644] D [afr-common.c:1380:afr_lookup_select_read_child]
0-replicated_vol-replicate-0: Source selected as 0 for /testfl</font>
<br><font size=2 face="Courier New">[2015-01-20 09:37:49.152657] D [afr-common.c:1117:afr_lookup_build_response_params]
0-replicated_vol-replicate-0: Building lookup response from 0</font>
<br>
<br><font size=3 color=blue face="Times New Roman">We see that when you
write on serv1, the RPC request is sent to both replicated_vol-client-0
and replicated_vol-client-1, while when we write on serv0, the request
is sent only to replicated_vol-client-0, and the FUse client is unaware
of the presence of client-1 in the latter case.</font>
<br>
<br><font size=3 color=blue face="Times New Roman">I checked a bit more
in the logs. When I turn on my trace, I found many instances of these logs
on serv0 but NOT on serv1:</font>
<br>
<br><font size=2 face="Courier New">[2015-01-20 09:21:15.520784] T [fuse-bridge.c:681:fuse_attr_cbk]
0-glusterfs-fuse: 53011: LOOKUP() / => 1</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:17.683088] T [rpc-clnt.c:422:rpc_clnt_reconnect]
0-replicated_vol-client-1: attempting reconnect</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:17.683159] D [name.c:155:client_fill_address_family]
0-replicated_vol-client-1: address-family not specified, guessing it to
be inet from (remote-host: serv1)</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:17.683178] T [name.c:225:af_inet_client_get_remote_sockaddr]
0-replicated_vol-client-1: option remote-port missing in volume replicated_vol-client-1.
Defaulting to 24007</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:17.683191] T [common-utils.c:188:gf_resolve_ip6]
0-resolver: flushing DNS cache</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:17.683202] T [common-utils.c:195:gf_resolve_ip6]
0-resolver: DNS cache not present, freshly probing hostname: serv1</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:17.683814] D [common-utils.c:237:gf_resolve_ip6]
0-resolver: returning ip-192.168.24.81 (port-24007) for hostname: serv1
and port: 24007</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:17.684139] D [common-utils.c:257:gf_resolve_ip6]
0-resolver: next DNS query will return: ip-192.168.24.81 port-24007</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:17.684164] T [socket.c:731:__socket_nodelay]
0-replicated_vol-client-1: NODELAY enabled for socket 10</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:17.684177] T [socket.c:790:__socket_keepalive]
0-replicated_vol-client-1: Keep-alive enabled for socket 10, interval 2,
idle: 20</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:17.684236] W [common-utils.c:2247:gf_get_reserved_ports]
0-glusterfs: could not open the file /proc/sys/net/ipv4/ip_local_reserved_ports
for getting reserved ports info (No such file or directory)</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:17.684253] W [common-utils.c:2280:gf_process_reserved_ports]
0-glusterfs: Not able to get reserved ports, hence there is a possibility
that glusterfs may consume reserved port</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:17.684660] D [socket.c:605:__socket_shutdown]
0-replicated_vol-client-1: shutdown() returned -1. Transport endpoint is
not connected</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:17.684699] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup]
0-replicated_vol-client-1: cleaning up state in transport object 0x68a630</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:17.684731] D [socket.c:486:__socket_rwv]
0-replicated_vol-client-1: EOF on socket</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:17.684750] W [socket.c:514:__socket_rwv]
0-replicated_vol-client-1: readv failed (No data available)</font>
<br><font size=2 face="Courier New">[2015-01-20 09:21:17.684766] D [socket.c:1962:__socket_proto_state_machine]
0-replicated_vol-client-1: reading from socket failed. Error (No data available),
peer (192.168.24.81:49198)</font>
<br>
<br><font size=3 color=blue face="Times New Roman">I could not find a 'remote-port'
option in /var/lib/glusterd on either peer. Could somebody tell me where
this configuration is looked up from? Also, sometime later, I rebooted
serv0 and that seemed to solve the problem. However, stop+start of replicated_vol
and restart of /etc/init.d/glusterd did NOT solve the problem.</font>
<br>
<br><font size=3 color=blue face="Times New Roman">Any help on this matter
will be greatly appreciated as I need to provide robustness assurances
for our setup. </font>
<br>
<br><font size=3 color=blue face="Times New Roman">Thanks a lot,</font>
<br><font size=3 color=blue face="Times New Roman">Anirban</font>
<br>
<br><font size=3 color=blue face="Times New Roman">P.s. Additional details:</font>
<br><font size=3 color=blue face="Times New Roman"><i>glusterfs version:
3.4.2</i></font>
<br><font size=3 color=blue face="Times New Roman"><i>Linux kernel version:
2.6.34</i></font><p>=====-----=====-----=====<br>
Notice: The information contained in this e-mail<br>
message and/or attachments to it may contain <br>
confidential or privileged information. If you are <br>
not the intended recipient, any dissemination, use, <br>
review, distribution, printing or copying of the <br>
information contained in this e-mail message <br>
and/or attachments to it are strictly prohibited. If <br>
you have received this communication in error, <br>
please notify us by reply e-mail or telephone and <br>
immediately and permanently delete the message <br>
and any attachments. Thank you</p>
<p></p>