<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi All,<div class=""><br class=""></div><div class="">I’ve a three nodes replica 3 cluster.</div><div class="">A network split happened which marked one of the three nodes offline on two nodes. And this very node set itself as RO.</div><div class="">After the network split was fixed, the who cluster became healthy again, and all the three peers status is connected on the three nodes. But the very node where the network split happened is still RO, it didn’t return back again to RW.</div><div class=""><br class=""></div><div class="">From the node where the network split happened, I see the following in /var/log/message:</div><div class=""><br class=""></div><div class=""><div class=""><span class="" style="font-size: 11px;"><font class="" color="#454545" face="Menlo">Dec 15 06:37:32 SN02 nfs[19840]: [2016-12-15 06:37:32.626927] C [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 0-gvol001-client-0: server SN01:49152 has not responded in the last 42 seconds, disconnecting.</font></span></div><div class=""><span class="" style="font-size: 11px;"><font class="" color="#454545" face="Menlo">Dec 15 06:40:27 SN02 nfs[19840]: [2016-12-15 06:40:27.645780] C [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 0-gvol001-client-2: server SN03:49152 has not responded in the last 42 seconds, disconnecting.</font></span></div><div class=""><span class="" style="font-size: 11px;"><font class="" color="#454545" face="Menlo">Dec 15 06:42:05 SN02 nfs[19840]: [2016-12-15 06:42:05.656223] C [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 0-gvol001-client-2: server SN03:49152 has not responded in the last 42 seconds, disconnecting.</font></span></div></div><div class=""><b class=""><font color="#ff4013" class=""><br class=""></font></b></div><div class=""><b class=""><font color="#ff4013" class=""><span style="color: rgb(0, 0, 0); font-weight: normal;" class="">log from /var/log/glusterfs/nfs.log</span></font></b></div><div class=""><b class=""><font color="#ff4013" class=""><span style="color: rgb(0, 0, 0); font-weight: normal;" class=""><br class=""></span></font></b></div><div class=""><span class="" style="font-size: 11px;"><font class="" color="#454545" face="Menlo"><div class="">[2016-12-15 06:40:41.222456] I [MSGID: 108002] [afr-common.c:4086:afr_notify] 0-gvol001-replicate-0: Client-quorum is met</div><div class="">[2016-12-15 06:40:41.222604] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gvol001-client-2: Server lk version = 1</div><div class="">[2016-12-15 06:40:41.225404] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 0-gvol001-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid argument]</div><div class="">[2016-12-15 06:40:41.230934] W [MSGID: 112199] [nfs3-helpers.c:3582:nfs3_log_commit_res] 0-nfs-nfsv3: /test2/tools/statistics/user_connections/last.summary => (XID: 65af65c, COMMIT: NFS: 30(<span class=""><font class="">Read-only file system</font></span>), POSIX: 30(Read-only file system)), wverf: 1477951811</div><div class="">[2016-12-15 06:40:41.231068] W [MSGID: 112199] [nfs3-helpers.c:3498:nfs3_log_write_res] 0-nfs-nfsv3: /test/datdiffs/diffs.client12.1481782734/5.base.client11 => (XID: 17e4dc6a, WRITE: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)), count: 0, UNSTABLE,wverf: 1477951811</div><div class=""><br class=""></div></font></span></div><br class=""><br class=""><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><b class="">— Bishoy</b></div></div></div></div></div>
</div>
<br class=""></body></html>