<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Soumya,<div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><span class="" style="float: none; display: inline !important;">Did you check 'pcs status' output that time? Maybe the *-ClusterIP* resources would have gone to Stopped state, making VIPs unavailable.</span><br class=""></blockquote><br class=""></div><div class="">Yes, I did check the ‘pcs status’ and everything was good at the time. </div><div class=""><br class=""></div><div class="">I just hit the issue again with VIP mounting and df output yesterday. </div><div class=""><br class=""></div><div class="">On the client 1, DF output was hung . I also could NOT mount the gluster volume via VIP x.x.x.001, but I could mount the gluster volume via VIP x.x.x.002 & x.x.x.003.</div><div class="">On the client 2, I could mount the gluster volume via VIP x.x.x.001 & x.x.x.002 & x.x.x.003.</div><div class=""><br class=""></div><div class="">Since I did configure pacemaker VIP ip x.x.x.001 for SN1, so I went ahead to stop pcs service on SN1 ‘pcs cluster stop’. The VIP ip x.x.x.001 failover to SN2 as my configuration, afterward I could mount the gluster volume via VIP’s IP x.x.x.001 on the client 1.</div><div class=""><br class=""></div><div class="">Any idea ??</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">~ Vic Le</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Sep 23, 2016, at 1:33 AM, Soumya Koduri <<a href="mailto:skoduri@redhat.com" class="">skoduri@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">On 09/23/2016 02:34 AM, Dung Le wrote:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Hello,<br class=""><br class="">I have a pretty straight forward configuration as below:<br class=""><br class="">3 storage nodes running version 3.7.11 with replica of 3 and it using<br class="">native gluster NFS.<br class="">corosync version 1.4.7 and pacemaker version 1.1.12<br class="">I have DNS round-robin on 3 VIPs living on the 3 storage nodes.<br class=""><br class="">*_Here is how I configure my corosync:_*<br class=""><br class="">SN1 with x.x.x.001<br class="">SN2 with x.x.x.002<br class="">SN3 with x.x.x.003<br class=""><br class=""><br class="">******************************************************************************************************************<br class="">*_Below is pcs config output:_*<br class=""><br class="">Cluster Name: dfs_cluster<br class="">Corosync Nodes:<br class="">SN1 SN2 SN3<br class="">Pacemaker Nodes:<br class="">SN1 SN2 SN3<br class=""><br class="">Resources:<br class="">Clone: Gluster-clone<br class=""> Meta Attrs: clone-max=3 clone-node-max=3 globally-unique=false<br class=""> Resource: Gluster (class=ocf provider=glusterfs type=glusterd)<br class=""> Operations: start interval=0s timeout=20 (Gluster-start-interval-0s)<br class=""> stop interval=0s timeout=20 (Gluster-stop-interval-0s)<br class=""> monitor interval=10s (Gluster-monitor-interval-10s)<br class="">Resource: SN1-ClusterIP (class=ocf provider=heartbeat type=IPaddr2)<br class=""> Attributes: ip=x.x.x.001 cidr_netmask=32<br class=""> Operations: start interval=0s timeout=20s<br class="">(SN1-ClusterIP-start-interval-0s)<br class=""> stop interval=0s timeout=20s (SN1-ClusterIP-stop-interval-0s)<br class=""> monitor interval=10s (SN1-ClusterIP-monitor-interval-10s)<br class="">Resource: SN2-ClusterIP (class=ocf provider=heartbeat type=IPaddr2)<br class=""> Attributes: ip=x.x.x.002 cidr_netmask=32<br class=""> Operations: start interval=0s timeout=20s<br class="">(SN2-ClusterIP-start-interval-0s)<br class=""> stop interval=0s timeout=20s (SN2-ClusterIP-stop-interval-0s)<br class=""> monitor interval=10s (SN2-ClusterIP-monitor-interval-10s)<br class="">Resource: SN3-ClusterIP (class=ocf provider=heartbeat type=IPaddr2)<br class=""> Attributes: ip=x.x.x.003 cidr_netmask=32<br class=""> Operations: start interval=0s timeout=20s<br class="">(SN3-ClusterIP-start-interval-0s)<br class=""> stop interval=0s timeout=20s (SN3-ClusterIP-stop-interval-0s)<br class=""> monitor interval=10s (SN3-ClusterIP-monitor-interval-10s)<br class=""><br class="">Stonith Devices:<br class="">Fencing Levels:<br class=""><br class="">Location Constraints:<br class=""> Resource: SN1-ClusterIP<br class=""> Enabled on: SN1 (score:3000) (id:location-SN1-ClusterIP-SN1-3000)<br class=""> Enabled on: SN2 (score:2000) (id:location-SN1-ClusterIP-SN2-2000)<br class=""> Enabled on: SN3 (score:1000) (id:location-SN1-ClusterIP-SN3-1000)<br class=""> Resource: SN2-ClusterIP<br class=""> Enabled on: SN2 (score:3000) (id:location-SN2-ClusterIP-SN2-3000)<br class=""> Enabled on: SN3 (score:2000) (id:location-SN2-ClusterIP-SN3-2000)<br class=""> Enabled on: SN1 (score:1000) (id:location-SN2-ClusterIP-SN1-1000)<br class=""> Resource: SN3-ClusterIP<br class=""> Enabled on: SN3 (score:3000) (id:location-SN3-ClusterIP-SN3-3000)<br class=""> Enabled on: SN1 (score:2000) (id:location-SN3-ClusterIP-SN1-2000)<br class=""> Enabled on: SN2 (score:1000) (id:location-SN3-ClusterIP-SN2-1000)<br class="">Ordering Constraints:<br class=""> start Gluster-clone then start SN1-ClusterIP (kind:Mandatory)<br class="">(id:order-Gluster-clone-SN1-ClusterIP-mandatory)<br class=""> start Gluster-clone then start SN2-ClusterIP (kind:Mandatory)<br class="">(id:order-Gluster-clone-SN2-ClusterIP-mandatory)<br class=""> start Gluster-clone then start SN3-ClusterIP (kind:Mandatory)<br class="">(id:order-Gluster-clone-SN3-ClusterIP-mandatory)<br class="">Colocation Constraints:<br class=""><br class="">Resources Defaults:<br class="">is-managed: true<br class="">target-role: Started<br class="">requires: nothing<br class="">multiple-active: stop_nkart<br class="">Operations Defaults:<br class="">No defaults set<br class=""><br class="">Cluster Properties:<br class="">cluster-infrastructure: cman<br class="">dc-version: 1.1.11-97629de<br class="">no-quorum-policy: ignore<br class="">stonith-enabled: false<br class=""><br class="">******************************************************************************************************************<br class="">*_pcs status output:_*<br class=""><br class="">Cluster name: dfs_cluster<br class="">Last updated: Thu Sep 22 16:57:35 2016<br class="">Last change: Mon Aug 29 18:02:44 2016<br class="">Stack: cman<br class="">Current DC: SN1 - partition with quorum<br class="">Version: 1.1.11-97629de<br class="">3 Nodes configured<br class="">6 Resources configured<br class=""><br class=""><br class="">Online: [ SN1 SN2 SN3 ]<br class=""><br class="">Full list of resources:<br class=""><br class="">Clone Set: Gluster-clone [Gluster]<br class=""> Started: [ SN1 SN2 SN3 ]<br class="">SN1-ClusterIP(ocf::heartbeat:IPaddr2):Started SN1<br class="">SN2-ClusterIP(ocf::heartbeat:IPaddr2):Started SN2<br class="">SN3-ClusterIP(ocf::heartbeat:IPaddr2):Started SN3<br class=""><br class="">******************************************************************************************************************<br class=""><br class=""><br class="">When I mount the gluster volume, I'm using the VIP name. It will choose<br class="">one of the storage nodes to establish NFS.<br class=""><br class="">*_My issue is:_*<br class="">*_<br class="">_*<br class="">After mounted gluster volume for 1 - 2 hrs, all the clients are<br class="">reporting not getting df output as df got hung. I did check the dmessage<br class="">log from client side and getting the following error :<br class=""><br class="">/Sep 20 05:46:45 xxxxx kernel: nfs: server nfsserver001 not responding,<br class="">still trying/<br class="">/Sep 20 05:49:45 xxxxx kernel: nfs: server nfsserver001 not responding,<br class="">still trying/<br class=""><br class="">I did try to mount the gluster volume using the DNS round-robin to<br class="">different mountpoint but the mount process was not successful.<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Did you check 'pcs status' output that time? Maybe the *-ClusterIP* resources would have gone to Stopped state, making VIPs unavailable.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Thanks,</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Soumya</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Then I</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">tried to mount the gluster volume using storage node IP itself (not VIP<br class="">ip), and I was able to mount the gluster volume. Afterward, I flipped<br class="">all the clients to mount storage node IP directly and they have been up<br class="">for more than 12hrs without any issue.<br class=""><br class="">Any idea what might cause this issue?<br class=""><br class="">Thanks a lot,<br class=""><br class="">~ Vic Le<br class=""><br class=""><br class="">_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">http://www.gluster.org/mailman/listinfo/gluster-users</blockquote></div></blockquote></div><br class=""></div></body></html>