<html><head><style id="-x-evo-style-a" type="text/css">a { cursor: text; }</style><style id="-x-evo-style-a" type="text/css">a { cursor: text; }</style><style id="-x-evo-style-a" type="text/css">a { cursor: text; }</style><style id="-x-evo-style-a" type="text/css">a { cursor: text; }</style></head><body style="font-family: Monospace;"><div>Hello,</div><div><br></div><div>i tried with a 4-node setup but the effect is the same, it takes down the cluster when one of the node is offline. I thought even in a 3-node setup when 2 nodes are online and only one is gone the majority of <b>2 nodes up vs. 1 node down</b> should not result in a lost quorum?</div><div><br></div><div>I have created the gluster volume with the following command:</div><div><br></div><div class="-x-evo-indented"><div> #> gluster volume create scratch replica 4 transport tcp kaukasus:/tank/brick1 altai:/tank/brick1 rnas2:/tank/brick1 bbk1:/scratch/brick1 force</div></div><div><br></div><div>The following is the log during the takedown of one node (altai):</div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div class="-x-evo-indented"><div> Nov 30 11:23:43 rnas2 corosync[16869]: [TOTEM ] A new membership (129.132.145.5:1120) was formed. Members left: 2</div><div> Nov 30 11:23:43 rnas2 cib[16088]: notice: crm_update_peer_proc: Node altai[2] - state is now lost (was member)</div><div> Nov 30 11:23:43 rnas2 cib[16088]: notice: Removing altai/2 from the membership list</div><div> Nov 30 11:23:43 rnas2 cib[16088]: notice: Purged 1 peers with id=2 and/or uname=altai from the membership cache</div><div> Nov 30 11:23:43 rnas2 crmd[16093]: notice: Our peer on the DC (altai) is dead</div><div> Nov 30 11:23:43 rnas2 attrd[16091]: notice: crm_update_peer_proc: Node altai[2] - state is now lost (was member)</div><div> Nov 30 11:23:43 rnas2 attrd[16091]: notice: Removing all altai attributes for attrd_peer_change_cb</div><div> Nov 30 11:23:43 rnas2 crmd[16093]: notice: State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_CRMD_STATUS_CALLBACK...llback ]</div><div> Nov 30 11:23:43 rnas2 attrd[16091]: notice: Lost attribute writer altai</div><div> Nov 30 11:23:43 rnas2 attrd[16091]: notice: Removing altai/2 from the membership list</div><div> Nov 30 11:23:43 rnas2 attrd[16091]: notice: Purged 1 peers with id=2 and/or uname=altai from the membership cache</div><div> Nov 30 11:23:43 rnas2 pacemakerd[16085]: notice: crm_update_peer_proc: Node altai[2] - state is now lost (was member)</div><div> Nov 30 11:23:43 rnas2 pacemakerd[16085]: notice: Removing altai/2 from the membership list</div><div> Nov 30 11:23:43 rnas2 stonith-ng[16089]: notice: crm_update_peer_proc: Node altai[2] - state is now lost (was member)</div><div> Nov 30 11:23:43 rnas2 pacemakerd[16085]: notice: Purged 1 peers with id=2 and/or uname=altai from the membership cache</div><div> Nov 30 11:23:43 rnas2 stonith-ng[16089]: notice: Removing altai/2 from the membership list</div><div> Nov 30 11:23:43 rnas2 corosync[16869]: [QUORUM] Members[3]: 1 3 4</div><div> Nov 30 11:23:43 rnas2 crmd[16093]: notice: Node altai[2] - state is now lost (was member)</div><div> Nov 30 11:23:43 rnas2 stonith-ng[16089]: notice: Purged 1 peers with id=2 and/or uname=altai from the membership cache</div><div> Nov 30 11:23:43 rnas2 corosync[16869]: [MAIN ] Completed service synchronization, ready to provide service.</div><div> Nov 30 11:23:43 rnas2 crmd[16093]: notice: State transition S_ELECTION -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=...t_vote ]</div><div> Nov 30 11:23:44 rnas2 crmd[16093]: notice: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl...espond ]</div><div> Nov 30 11:23:44 rnas2 IPaddr(rnas2-cluster_ip-1)[10934]: INFO: IP status = ok, IP_CIP=</div><div> Nov 30 11:23:44 rnas2 crmd[16093]: notice: Operation rnas2-cluster_ip-1_stop_0: ok (node=rnas2, call=53, rc=0, cib-update=36, confirmed=true)</div><div> Nov 30 11:23:44 rnas2 crmd[16093]: notice: Operation nfs-grace_stop_0: ok (node=rnas2, call=55, rc=0, cib-update=37, confirmed=true)</div><div> Nov 30 11:23:44 rnas2 attrd[16091]: notice: Processing sync-response from bbk1</div><div><b> Nov 30 11:23:45 rnas2 ntpd[1700]: Deleting interface #47 bond0, 129.132.145.23#123, interface stats: received=0, sent=0, dropped=0...783 secs</b></div><div>Nov 30 11:24:24 rnas2 lrmd[16090]: warning: nfs-grace_start_0 process (PID 10947) timed out</div><div>Nov 30 11:24:24 rnas2 lrmd[16090]: warning: nfs-grace_start_0:10947 - timed out after 40000ms</div><div><b>Nov 30 11:24:24 rnas2 crmd[16093]: error: Operation nfs-grace_start_0: Timed Out (node=rnas2, call=56, timeout=40000ms)</b></div><div>Nov 30 11:24:24 rnas2 crmd[16093]: notice: Operation nfs-grace_stop_0: ok (node=rnas2, call=57, rc=0, cib-update=39, confirmed=true)</div></div><div><br></div><div>I discovered, when i restart the pacemaker service on one of the running nodes, it can successfully take the cluster online again:</div><div><br></div><div><a href="mailto:root@kaukasus">root@kaukasus</a> ~# systemctl restart pacemaker</div><div><br></div><div><br></div><div class="-x-evo-indented"><div> Nov 30 11:45:36 rnas2 crmd[16093]: notice: Our peer on the DC (kaukasus) is dead</div><div> Nov 30 11:45:36 rnas2 crmd[16093]: notice: State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_CRMD_STATUS_CALLBACK...llback ]</div><div> Nov 30 11:45:36 rnas2 crmd[16093]: notice: State transition S_ELECTION -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=...t_vote ]</div><div> Nov 30 11:45:36 rnas2 attrd[16091]: notice: crm_update_peer_proc: Node kaukasus[1] - state is now lost (was member)</div><div> Nov 30 11:45:36 rnas2 attrd[16091]: notice: Removing all kaukasus attributes for attrd_peer_change_cb</div><div> Nov 30 11:45:36 rnas2 attrd[16091]: notice: Removing kaukasus/1 from the membership list</div><div> Nov 30 11:45:36 rnas2 attrd[16091]: notice: Purged 1 peers with id=1 and/or uname=kaukasus from the membership cache</div><div> Nov 30 11:45:36 rnas2 stonith-ng[16089]: notice: crm_update_peer_proc: Node kaukasus[1] - state is now lost (was member)</div><div> Nov 30 11:45:36 rnas2 stonith-ng[16089]: notice: Removing kaukasus/1 from the membership list</div><div> Nov 30 11:45:36 rnas2 stonith-ng[16089]: notice: Purged 1 peers with id=1 and/or uname=kaukasus from the membership cache</div><div> Nov 30 11:45:36 rnas2 cib[16088]: notice: crm_update_peer_proc: Node kaukasus[1] - state is now lost (was member)</div><div> Nov 30 11:45:36 rnas2 cib[16088]: notice: Removing kaukasus/1 from the membership list</div><div> Nov 30 11:45:36 rnas2 cib[16088]: notice: Purged 1 peers with id=1 and/or uname=kaukasus from the membership cache</div><div> Nov 30 11:45:36 rnas2 pacemakerd[16085]: notice: crm_update_peer_proc: Node kaukasus[1] - state is now lost (was member)</div><div> Nov 30 11:45:36 rnas2 pacemakerd[16085]: notice: Removing kaukasus/1 from the membership list</div><div> Nov 30 11:45:36 rnas2 pacemakerd[16085]: notice: Purged 1 peers with id=1 and/or uname=kaukasus from the membership cache</div><div> Nov 30 11:45:36 rnas2 pacemakerd[16085]: notice: crm_update_peer_proc: Node kaukasus[1] - state is now member (was (null))</div><div> Nov 30 11:45:36 rnas2 crmd[16093]: notice: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl...espond ]</div><div> Nov 30 11:45:36 rnas2 stonith-ng[16089]: notice: crm_update_peer_proc: Node kaukasus[1] - state is now member (was (null))</div><div> Nov 30 11:45:36 rnas2 attrd[16091]: notice: crm_update_peer_proc: Node kaukasus[1] - state is now member (was (null))</div><div> Nov 30 11:45:36 rnas2 cib[16088]: notice: crm_update_peer_proc: Node kaukasus[1] - state is now member (was (null))</div><div> Nov 30 11:45:46 rnas2 IPaddr(rnas2-cluster_ip-1)[16591]: INFO: Adding inet address 129.132.145.23/32 with broadcast address 129.132.... bond0</div><div> Nov 30 11:45:46 rnas2 IPaddr(rnas2-cluster_ip-1)[16600]: INFO: Bringing device bond0 up</div><div> Nov 30 11:45:46 rnas2 IPaddr(rnas2-cluster_ip-1)[16609]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agen...t_used</div><div><b> Nov 30 11:45:46 rnas2 crmd[16093]: notice: Operation rnas2-cluster_ip-1_start_0: ok (node=rnas2, call=58, rc=0, cib-update=44, con...ed=true)</b></div><div>Nov 30 11:45:48 rnas2 ntpd[1700]: Listen normally on 48 bond0 129.132.145.23 UDP 123</div><div>Nov 30 11:45:48 rnas2 ntpd[1700]: new interface(s) found: waking up resolver</div></div><div><br></div><div><br></div><div>Yours,</div><div>Rigi</div><div><br></div><div>On Mon, 2015-11-30 at 15:26 +0530, Soumya Koduri wrote:</div><blockquote type="cite"><div>Hi,</div><blockquote type="cite"><div>But are you telling me that in a 3-node cluster,</div><div>quorum is lost when one of the nodes ip is down?</div></blockquote><div><br></div><div>yes. Its the limitation with Pacemaker/Corosync. If the nodes </div><div>participating in cluster cannot communicate with majority of them </div><div>(quorum is lost), then the cluster is shut down.</div><div><br></div><blockquote type="cite"><div><br></div><div>However i am setting up a additional node to test a 4-node setup, but</div><div>even then if i put down one node and nfs-grace_start</div><div>(/usr/lib/ocf/resource.d/heartbeat/ganesha_grace) did not run properly</div><div>on the other nodes, could it be that the whole cluster goes down as</div><div>quorum lost again?</div></blockquote><div><br></div><div>That's strange. We have tested quite a few times such configuration but </div><div>haven't hit this issue. (CCin Saurabh who has been testing many such </div><div>configurations).</div><div><br></div><div>Recently we have observed resource agents (nfs-grace_*) timing out </div><div>sometimes esp when any node is taken down. But that shouldn't cause the </div><div>entire cluster to shutdown.</div><div>Could you check the logs (/var/log/messages, /var/log/pacemaker.log) for </div><div>any error/warning reported when one node is taken down in case of 4-node </div><div>setup.</div><div><br></div><div>Thanks,</div><div>Soumya</div></blockquote></body></html>