<div dir="ltr">Unfortunately, no success. I did the following:<div><br></div><div>- gluster nfs-ganesha disable </div><div>first time the request timed out, after a reboot of the server tried the same command again and it succeeded</div><div>- /usr/libexec/ganesha/ganesha-ha.sh --cleanup /etc/ganesha</div><div>no output</div><div>- gluster nfs-ganesha enable</div><div>Again timeout and corosync is unresponsive and using 100% CPU. Had to do kill -9 on the process.</div><div>Same messages in the log as previous (Corosync in failed state).</div><div><br></div><div>Does the ganesha-ha.sh script handle multiple network interfaces? There are two interfaces on both servers and corosync/pacemaker should use only one of them.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 22 September 2015 at 21:44, Tiemen Ruiten <span dir="ltr"><<a href="mailto:t.ruiten@rdmedia.com" target="_blank">t.ruiten@rdmedia.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Allright, thank you Soumya. I actually did do the cleanup every time (gluster nfs-ganesha disable), but it didn't always finish succesfully. Sometimes it would just time out. I'll try with the second command tomorrow.<div><br></div><div>Good to know that it should work with two nodes as well.</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On 22 September 2015 at 19:26, Soumya Koduri <span dir="ltr"><<a href="mailto:skoduri@redhat.com" target="_blank">skoduri@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><br>
<br>
On 09/22/2015 05:06 PM, Tiemen Ruiten wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
That's correct and my original question was actually if a two node +<br>
arbiter setup is possible. The documentation provided by Soumya only<br>
mentions two servers in the example ganesha-ha.sh script. Perhaps that<br>
could be updated as well then, to not give the wrong impression.<br>
<br>
</blockquote></span>
It does work with 2-node as well. In the script, there is already a check to verify if the number of servers < 3, it automatically disables quorum.<br>
Quorum cannot be enabled for a 2-node setup for obvious reasons. If one node fails, other node just takes over the IP.<br>
<br>
Thanks,<br>
Soumya<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>
I could try to change the script to disable quorum, but wouldn't that<br>
defeat the purpose? What will happen in case one node goes down<br>
unexpectedly?<br>
<br>
On 22 September 2015 at 12:47, Kaleb Keithley <<a href="mailto:kkeithle@redhat.com" target="_blank">kkeithle@redhat.com</a><br></span><div><div>
<mailto:<a href="mailto:kkeithle@redhat.com" target="_blank">kkeithle@redhat.com</a>>> wrote:<br>
<br>
<br>
Hi,<br>
<br>
IIRC, the setup is two nodes gluster+ganesha nodes plus the arbiter<br>
node for gluster quorum.<br>
<br>
Have I remembered that correctly?<br>
<br>
The Ganesha HA in 3.7 requires a minimum of three servers running<br>
ganesha and pacemaker. Two might work if you change the<br>
ganesha-ha.sh to not enable pacemaker quorum, but I haven't tried<br>
that myself. I'll try and find time in the next couple of days to<br>
update the documentation or write a blog post.<br>
<br>
<br>
<br>
----- Original Message ----<br>
><br>
><br>
><br>
> On 21/09/15 21:21, Tiemen Ruiten wrote:<br>
> > Whoops, replied off-list.<br>
> ><br>
> > Additionally I noticed that the generated corosync config is not<br>
> > valid, as there is no interface section:<br>
> ><br>
> > /etc/corosync/corosync.conf<br>
> ><br>
> > totem {<br>
> > version: 2<br>
> > secauth: off<br>
> > cluster_name: rd-ganesha-ha<br>
> > transport: udpu<br>
> > }<br>
> ><br>
> > nodelist {<br>
> > node {<br>
> > ring0_addr: cobalt<br>
> > nodeid: 1<br>
> > }<br>
> > node {<br>
> > ring0_addr: iron<br>
> > nodeid: 2<br>
> > }<br>
> > }<br>
> ><br>
> > quorum {<br>
> > provider: corosync_votequorum<br>
> > two_node: 1<br>
> > }<br>
> ><br>
> > logging {<br>
> > to_syslog: yes<br>
> > }<br>
> ><br>
> ><br>
> ><br>
><br>
> May be Kaleb can help you out.<br>
> ><br>
> > ---------- Forwarded message ----------<br>
> > From: *Tiemen Ruiten* <<a href="mailto:t.ruiten@rdmedia.com" target="_blank">t.ruiten@rdmedia.com</a> <mailto:<a href="mailto:t.ruiten@rdmedia.com" target="_blank">t.ruiten@rdmedia.com</a>><br></div></div><span>
<mailto:<a href="mailto:t.ruiten@rdmedia.com" target="_blank">t.ruiten@rdmedia.com</a> <mailto:<a href="mailto:t.ruiten@rdmedia.com" target="_blank">t.ruiten@rdmedia.com</a>>>><br>
> > Date: 21 September 2015 at 17:16<br>
> > Subject: Re: [Gluster-users] nfs-ganesha HA with arbiter volume<br>
> > To: Jiffin Tony Thottan <<a href="mailto:jthottan@redhat.com" target="_blank">jthottan@redhat.com</a><br></span><div><div>
<mailto:<a href="mailto:jthottan@redhat.com" target="_blank">jthottan@redhat.com</a>> <mailto:<a href="mailto:jthottan@redhat.com" target="_blank">jthottan@redhat.com</a><br>
<mailto:<a href="mailto:jthottan@redhat.com" target="_blank">jthottan@redhat.com</a>>>><br>
> ><br>
> ><br>
> > Could you point me to the latest documentation? I've been<br>
struggling<br>
> > to find something up-to-date. I believe I have all the<br>
prerequisites:<br>
> ><br>
> > - shared storage volume exists and is mounted<br>
> > - all nodes in hosts files<br>
> > - Gluster-NFS disabled<br>
> > - corosync, pacemaker and nfs-ganesha rpm's installed<br>
> ><br>
> > Anything I missed?<br>
> ><br>
> > Everything has been installed by RPM so is in the default<br>
locations:<br>
> > /usr/libexec/ganesha/ganesha-ha.sh<br>
> > /etc/ganesha/ganesha.conf (empty)<br>
> > /etc/ganesha/ganesha-ha.conf<br>
> ><br>
><br>
> Looks fine for me.<br>
><br>
> > After I started the pcsd service manually, nfs-ganesha could be<br>
> > enabled successfully, but there was no virtual IP present on the<br>
> > interfaces and looking at the system log, I noticed corosync<br>
failed to<br>
> > start:<br>
> ><br>
> > - on the host where I issued the gluster nfs-ganesha enable<br>
command:<br>
> ><br>
> > Sep 21 17:07:18 iron systemd: Starting NFS-Ganesha file server...<br>
> > Sep 21 17:07:19 iron systemd: Started NFS-Ganesha file server.<br>
> > Sep 21 17:07:19 iron rpc.statd[2409]: Received SM_UNMON_ALL request<br>
> > from <a href="http://iron.int.rdmedia.com" rel="noreferrer" target="_blank">iron.int.rdmedia.com</a> <<a href="http://iron.int.rdmedia.com" rel="noreferrer" target="_blank">http://iron.int.rdmedia.com</a>><br>
<<a href="http://iron.int.rdmedia.com" rel="noreferrer" target="_blank">http://iron.int.rdmedia.com</a>> while not<br>
> > monitoring any hosts<br>
> > Sep 21 17:07:20 iron systemd: Starting Corosync Cluster Engine...<br>
> > Sep 21 17:07:20 iron corosync[3426]: [MAIN ] Corosync Cluster<br>
Engine<br>
> > ('2.3.4'): started and ready to provide service.<br>
> > Sep 21 17:07:20 iron corosync[3426]: [MAIN ] Corosync built-in<br>
> > features: dbus systemd xmlconf snmp pie relro bindnow<br>
> > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] Initializing<br>
transport<br>
> > (UDP/IP Unicast).<br>
> > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] Initializing<br>
> > transmit/receive security (NSS) crypto: none hash: none<br>
> > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] The network interface<br>
> > [10.100.30.38] is now up.<br>
> > Sep 21 17:07:20 iron corosync[3427]: [SERV ] Service engine<br>
loaded:<br>
> > corosync configuration map access [0]<br>
> > Sep 21 17:07:20 iron corosync[3427]: [QB ] server name: cmap<br>
> > Sep 21 17:07:20 iron corosync[3427]: [SERV ] Service engine<br>
loaded:<br>
> > corosync configuration service [1]<br>
> > Sep 21 17:07:20 iron corosync[3427]: [QB ] server name: cfg<br>
> > Sep 21 17:07:20 iron corosync[3427]: [SERV ] Service engine<br>
loaded:<br>
> > corosync cluster closed process group service v1.01 [2]<br>
> > Sep 21 17:07:20 iron corosync[3427]: [QB ] server name: cpg<br>
> > Sep 21 17:07:20 iron corosync[3427]: [SERV ] Service engine<br>
loaded:<br>
> > corosync profile loading service [4]<br>
> > Sep 21 17:07:20 iron corosync[3427]: [QUORUM] Using quorum provider<br>
> > corosync_votequorum<br>
> > Sep 21 17:07:20 iron corosync[3427]: [VOTEQ ] Waiting for all<br>
cluster<br>
> > members. Current votes: 1 expected_votes: 2<br>
> > Sep 21 17:07:20 iron corosync[3427]: [SERV ] Service engine<br>
loaded:<br>
> > corosync vote quorum service v1.0 [5]<br>
> > Sep 21 17:07:20 iron corosync[3427]: [QB ] server name:<br>
votequorum<br>
> > Sep 21 17:07:20 iron corosync[3427]: [SERV ] Service engine<br>
loaded:<br>
> > corosync cluster quorum service v0.1 [3]<br>
> > Sep 21 17:07:20 iron corosync[3427]: [QB ] server name: quorum<br>
> > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] adding new UDPU<br>
member<br>
> > {10.100.30.38}<br>
> > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] adding new UDPU<br>
member<br>
> > {10.100.30.37}<br>
> > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] A new membership<br>
> > (<a href="http://10.100.30.38:104" rel="noreferrer" target="_blank">10.100.30.38:104</a> <<a href="http://10.100.30.38:104" rel="noreferrer" target="_blank">http://10.100.30.38:104</a>><br>
<<a href="http://10.100.30.38:104" rel="noreferrer" target="_blank">http://10.100.30.38:104</a>>) was formed. Members joined: 1<br>
> > Sep 21 17:07:20 iron corosync[3427]: [VOTEQ ] Waiting for all cluster<br>
> > members. Current votes: 1 expected_votes: 2<br>
> > Sep 21 17:07:20 iron corosync[3427]: [VOTEQ ] Waiting for all cluster<br>
> > members. Current votes: 1 expected_votes: 2<br>
> > Sep 21 17:07:20 iron corosync[3427]: [VOTEQ ] Waiting for all cluster<br>
> > members. Current votes: 1 expected_votes: 2<br>
> > Sep 21 17:07:20 iron corosync[3427]: [QUORUM] Members[1]: 1<br>
> > Sep 21 17:07:20 iron corosync[3427]: [MAIN ] Completed service<br>
> > synchronization, ready to provide service.<br>
> > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] A new membership<br>
> > (<a href="http://10.100.30.37:108" rel="noreferrer" target="_blank">10.100.30.37:108</a> <<a href="http://10.100.30.37:108" rel="noreferrer" target="_blank">http://10.100.30.37:108</a>><br>
<<a href="http://10.100.30.37:108" rel="noreferrer" target="_blank">http://10.100.30.37:108</a>>) was formed. Members joined: 1<br>
> > Sep 21 17:08:21 iron corosync: Starting Corosync Cluster Engine<br>
> > (corosync): [FAILED]<br>
> > Sep 21 17:08:21 iron systemd: corosync.service: control process<br>
> > exited, code=exited status=1<br>
> > Sep 21 17:08:21 iron systemd: Failed to start Corosync Cluster<br>
Engine.<br>
> > Sep 21 17:08:21 iron systemd: Unit corosync.service entered<br>
failed state.<br>
> ><br>
> ><br>
> > - on the other host:<br>
> ><br>
> > Sep 21 17:07:19 cobalt systemd: Starting Preprocess NFS<br>
configuration...<br>
> > Sep 21 17:07:19 cobalt systemd: Starting RPC Port Mapper.<br>
> > Sep 21 17:07:19 cobalt systemd: Reached target RPC Port Mapper.<br>
> > Sep 21 17:07:19 cobalt systemd: Starting Host and Network Name<br>
Lookups.<br>
> > Sep 21 17:07:19 cobalt systemd: Reached target Host and Network<br>
Name<br>
> > Lookups.<br>
> > Sep 21 17:07:19 cobalt systemd: Starting RPC bind service...<br>
> > Sep 21 17:07:19 cobalt systemd: Started Preprocess NFS<br>
configuration.<br>
> > Sep 21 17:07:19 cobalt systemd: Started RPC bind service.<br>
> > Sep 21 17:07:19 cobalt systemd: Starting NFS status monitor for<br>
> > NFSv2/3 locking....<br>
> > Sep 21 17:07:19 cobalt rpc.statd[2662]: Version 1.3.0 starting<br>
> > Sep 21 17:07:19 cobalt rpc.statd[2662]: Flags: TI-RPC<br>
> > Sep 21 17:07:19 cobalt systemd: Started NFS status monitor for<br>
NFSv2/3<br>
> > locking..<br>
> > Sep 21 17:07:19 cobalt systemd: Starting NFS-Ganesha file server...<br>
> > Sep 21 17:07:19 cobalt systemd: Started NFS-Ganesha file server.<br>
> > Sep 21 17:07:19 cobalt kernel: warning: `ganesha.nfsd' uses 32-bit<br>
> > capabilities (legacy support in use)<br>
> > Sep 21 17:07:19 cobalt logger: setting up rd-ganesha-ha<br>
> > Sep 21 17:07:19 cobalt rpc.statd[2662]: Received SM_UNMON_ALL<br>
request<br>
> > from <a href="http://cobalt.int.rdmedia.com" rel="noreferrer" target="_blank">cobalt.int.rdmedia.com</a> <<a href="http://cobalt.int.rdmedia.com" rel="noreferrer" target="_blank">http://cobalt.int.rdmedia.com</a>><br>
<<a href="http://cobalt.int.rdmedia.com" rel="noreferrer" target="_blank">http://cobalt.int.rdmedia.com</a>> while not<br>
> > monitoring any hosts<br>
> > Sep 21 17:07:19 cobalt logger: setting up cluster rd-ganesha-ha<br>
with<br>
> > the following cobalt iron<br>
> > Sep 21 17:07:20 cobalt systemd: Stopped Pacemaker High Availability<br>
> > Cluster Manager.<br>
> > Sep 21 17:07:20 cobalt systemd: Stopped Corosync Cluster Engine.<br>
> > Sep 21 17:07:20 cobalt systemd: Reloading.<br>
> > Sep 21 17:07:20 cobalt systemd:<br>
> > [/usr/lib/systemd/system/dm-event.socket:10] Unknown lvalue<br>
> > 'RemoveOnStop' in section 'Socket'<br>
> > Sep 21 17:07:20 cobalt systemd:<br>
> > [/usr/lib/systemd/system/lvm2-lvmetad.socket:9] Unknown lvalue<br>
> > 'RemoveOnStop' in section 'Socket'<br>
> > Sep 21 17:07:20 cobalt systemd: Reloading.<br>
> > Sep 21 17:07:20 cobalt systemd:<br>
> > [/usr/lib/systemd/system/dm-event.socket:10] Unknown lvalue<br>
> > 'RemoveOnStop' in section 'Socket'<br>
> > Sep 21 17:07:20 cobalt systemd:<br>
> > [/usr/lib/systemd/system/lvm2-lvmetad.socket:9] Unknown lvalue<br>
> > 'RemoveOnStop' in section 'Socket'<br>
> > Sep 21 17:07:20 cobalt systemd: Starting Corosync Cluster Engine...<br>
> > Sep 21 17:07:20 cobalt corosync[2816]: [MAIN ] Corosync Cluster<br>
> > Engine ('2.3.4'): started and ready to provide service.<br>
> > Sep 21 17:07:20 cobalt corosync[2816]: [MAIN ] Corosync built-in<br>
> > features: dbus systemd xmlconf snmp pie relro bindnow<br>
> > Sep 21 17:07:20 cobalt corosync[2817]: [TOTEM ] Initializing<br>
transport<br>
> > (UDP/IP Unicast).<br>
> > Sep 21 17:07:20 cobalt corosync[2817]: [TOTEM ] Initializing<br>
> > transmit/receive security (NSS) crypto: none hash: none<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ] The network<br>
interface<br>
> > [10.100.30.37] is now up.<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [SERV ] Service engine<br>
loaded:<br>
> > corosync configuration map access [0]<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [QB ] server name: cmap<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [SERV ] Service engine<br>
loaded:<br>
> > corosync configuration service [1]<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [QB ] server name: cfg<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [SERV ] Service engine<br>
loaded:<br>
> > corosync cluster closed process group service v1.01 [2]<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [QB ] server name: cpg<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [SERV ] Service engine<br>
loaded:<br>
> > corosync profile loading service [4]<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [QUORUM] Using quorum<br>
provider<br>
> > corosync_votequorum<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ] Waiting for all<br>
> > cluster members. Current votes: 1 expected_votes: 2<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [SERV ] Service engine<br>
loaded:<br>
> > corosync vote quorum service v1.0 [5]<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [QB ] server name:<br>
votequorum<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [SERV ] Service engine<br>
loaded:<br>
> > corosync cluster quorum service v0.1 [3]<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [QB ] server name: quorum<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ] adding new UDPU<br>
member<br>
> > {10.100.30.37}<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ] adding new UDPU<br>
member<br>
> > {10.100.30.38}<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ] A new membership<br>
> > (<a href="http://10.100.30.37:100" rel="noreferrer" target="_blank">10.100.30.37:100</a> <<a href="http://10.100.30.37:100" rel="noreferrer" target="_blank">http://10.100.30.37:100</a>><br>
<<a href="http://10.100.30.37:100" rel="noreferrer" target="_blank">http://10.100.30.37:100</a>>) was formed. Members joined: 1<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ] Waiting for all<br>
> > cluster members. Current votes: 1 expected_votes: 2<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ] Waiting for all<br>
> > cluster members. Current votes: 1 expected_votes: 2<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ] Waiting for all<br>
> > cluster members. Current votes: 1 expected_votes: 2<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [QUORUM] Members[1]: 1<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [MAIN ] Completed service<br>
> > synchronization, ready to provide service.<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ] A new membership<br>
> > (<a href="http://10.100.30.37:108" rel="noreferrer" target="_blank">10.100.30.37:108</a> <<a href="http://10.100.30.37:108" rel="noreferrer" target="_blank">http://10.100.30.37:108</a>><br>
<<a href="http://10.100.30.37:108" rel="noreferrer" target="_blank">http://10.100.30.37:108</a>>) was formed. Members joined: 1<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ] Waiting for all<br>
> > cluster members. Current votes: 1 expected_votes: 2<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [QUORUM] Members[1]: 1<br>
> > Sep 21 17:07:21 cobalt corosync[2817]: [MAIN ] Completed service<br>
> > synchronization, ready to provide service.<br>
> > Sep 21 17:08:50 cobalt systemd: corosync.service operation<br>
timed out.<br>
> > Terminating.<br>
> > Sep 21 17:08:50 cobalt corosync: Starting Corosync Cluster Engine<br>
> > (corosync):<br>
> > Sep 21 17:08:50 cobalt systemd: Failed to start Corosync<br>
Cluster Engine.<br>
> > Sep 21 17:08:50 cobalt systemd: Unit corosync.service entered<br>
failed<br>
> > state.<br>
> > Sep 21 17:08:55 cobalt logger: warning: pcs property set<br>
> > no-quorum-policy=ignore failed<br>
> > Sep 21 17:08:55 cobalt logger: warning: pcs property set<br>
> > stonith-enabled=false failed<br>
> > Sep 21 17:08:55 cobalt logger: warning: pcs resource create<br>
nfs_start<br>
> > ganesha_nfsd ha_vol_mnt=/var/run/gluster/shared_storage --clone<br>
failed<br>
> > Sep 21 17:08:56 cobalt logger: warning: pcs resource delete<br>
> > nfs_start-clone failed<br>
> > Sep 21 17:08:56 cobalt logger: warning: pcs resource create nfs-mon<br>
> > ganesha_mon --clone failed<br>
> > Sep 21 17:08:56 cobalt logger: warning: pcs resource create<br>
nfs-grace<br>
> > ganesha_grace --clone failed<br>
> > Sep 21 17:08:57 cobalt logger: warning pcs resource create<br>
> > cobalt-cluster_ip-1 ocf:heartbeat:IPaddr ip= cidr_netmask=32 op<br>
> > monitor interval=15s failed<br>
> > Sep 21 17:08:57 cobalt logger: warning: pcs resource create<br>
> > cobalt-trigger_ip-1 ocf:heartbeat:Dummy failed<br>
> > Sep 21 17:08:57 cobalt logger: warning: pcs constraint<br>
colocation add<br>
> > cobalt-cluster_ip-1 with cobalt-trigger_ip-1 failed<br>
> > Sep 21 17:08:57 cobalt logger: warning: pcs constraint order<br>
> > cobalt-trigger_ip-1 then nfs-grace-clone failed<br>
> > Sep 21 17:08:57 cobalt logger: warning: pcs constraint order<br>
> > nfs-grace-clone then cobalt-cluster_ip-1 failed<br>
> > Sep 21 17:08:57 cobalt logger: warning pcs resource create<br>
> > iron-cluster_ip-1 ocf:heartbeat:IPaddr ip= cidr_netmask=32 op<br>
monitor<br>
> > interval=15s failed<br>
> > Sep 21 17:08:57 cobalt logger: warning: pcs resource create<br>
> > iron-trigger_ip-1 ocf:heartbeat:Dummy failed<br>
> > Sep 21 17:08:57 cobalt logger: warning: pcs constraint<br>
colocation add<br>
> > iron-cluster_ip-1 with iron-trigger_ip-1 failed<br>
> > Sep 21 17:08:57 cobalt logger: warning: pcs constraint order<br>
> > iron-trigger_ip-1 then nfs-grace-clone failed<br>
> > Sep 21 17:08:58 cobalt logger: warning: pcs constraint order<br>
> > nfs-grace-clone then iron-cluster_ip-1 failed<br>
> > Sep 21 17:08:58 cobalt logger: warning: pcs constraint location<br>
> > cobalt-cluster_ip-1 rule score=-INFINITY ganesha-active ne 1 failed<br>
> > Sep 21 17:08:58 cobalt logger: warning: pcs constraint location<br>
> > cobalt-cluster_ip-1 prefers iron=1000 failed<br>
> > Sep 21 17:08:58 cobalt logger: warning: pcs constraint location<br>
> > cobalt-cluster_ip-1 prefers cobalt=2000 failed<br>
> > Sep 21 17:08:58 cobalt logger: warning: pcs constraint location<br>
> > iron-cluster_ip-1 rule score=-INFINITY ganesha-active ne 1 failed<br>
> > Sep 21 17:08:58 cobalt logger: warning: pcs constraint location<br>
> > iron-cluster_ip-1 prefers cobalt=1000 failed<br>
> > Sep 21 17:08:58 cobalt logger: warning: pcs constraint location<br>
> > iron-cluster_ip-1 prefers iron=2000 failed<br>
> > Sep 21 17:08:58 cobalt logger: warning pcs cluster cib-push<br>
> > /tmp/tmp.nXTfyA1GMR failed<br>
> > Sep 21 17:08:58 cobalt logger: warning: scp ganesha-ha.conf to<br>
cobalt<br>
> > failed<br>
> ><br>
> > BTW, I'm using CentOS 7. There are multiple network interfaces<br>
on the<br>
> > servers, could that be a problem?<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On 21 September 2015 at 11:48, Jiffin Tony Thottan<br>
> > <<a href="mailto:jthottan@redhat.com" target="_blank">jthottan@redhat.com</a> <mailto:<a href="mailto:jthottan@redhat.com" target="_blank">jthottan@redhat.com</a>><br></div></div><div><div>
<mailto:<a href="mailto:jthottan@redhat.com" target="_blank">jthottan@redhat.com</a> <mailto:<a href="mailto:jthottan@redhat.com" target="_blank">jthottan@redhat.com</a>>>> wrote:<br>
> ><br>
> ><br>
> ><br>
> > On 21/09/15 13:56, Tiemen Ruiten wrote:<br>
> >> Hello Soumya, Kaleb, list,<br>
> >><br>
> >> This Friday I created the gluster_shared_storage volume<br>
manually,<br>
> >> I just tried it with the command you supplied, but both<br>
have the<br>
> >> same result:<br>
> >><br>
> >> from etc-glusterfs-glusterd.vol.log on the node where I issued<br>
> >> the command:<br>
> >><br>
> >> [2015-09-21 07:59:47.756845] I [MSGID: 106474]<br>
> >> [glusterd-ganesha.c:403:check_host_list] 0-management: ganesha<br>
> >> host found Hostname is cobalt<br>
> >> [2015-09-21 07:59:48.071755] I [MSGID: 106474]<br>
> >> [glusterd-ganesha.c:349:is_ganesha_host] 0-management: ganesha<br>
> >> host found Hostname is cobalt<br>
> >> [2015-09-21 07:59:48.653879] E [MSGID: 106470]<br>
> >> [glusterd-ganesha.c:264:glusterd_op_set_ganesha] 0-management:<br>
> >> Initial NFS-Ganesha set up failed<br>
> ><br>
> > As far as what I understand from the logs, it called<br>
> > setup_cluser()[calls `ganesha-ha.sh` script ] but script<br>
failed.<br>
> > Can u please provide following details :<br>
> > -Location of ganesha.sh file??<br>
> > -Location of ganesha-ha.conf, ganesha.conf files ?<br>
> ><br>
> ><br>
> > And also can u cross check whether all the prerequisites<br>
before HA<br>
> > setup satisfied ?<br>
> ><br>
> > --<br>
> > With Regards,<br>
> > Jiffin<br>
> ><br>
> ><br>
> >> [2015-09-21 07:59:48.653912] E [MSGID: 106123]<br>
> >> [glusterd-syncop.c:1404:gd_commit_op_phase] 0-management:<br>
Commit<br>
> >> of operation 'Volume (null)' failed on localhost : Failed<br>
to set<br>
> >> up HA config for NFS-Ganesha. Please check the log file<br>
for details<br>
> >> [2015-09-21 07:59:45.402458] I [MSGID: 106006]<br>
> >> [glusterd-svc-mgmt.c:323:glusterd_svc_common_rpc_notify]<br>
> >> 0-management: nfs has disconnected from glusterd.<br>
> >> [2015-09-21 07:59:48.071578] I [MSGID: 106474]<br>
> >> [glusterd-ganesha.c:403:check_host_list] 0-management: ganesha<br>
> >> host found Hostname is cobalt<br>
> >><br>
> >> from etc-glusterfs-glusterd.vol.log on the other node:<br>
> >><br>
> >> [2015-09-21 08:12:50.111877] E [MSGID: 106062]<br>
> >> [glusterd-op-sm.c:3698:glusterd_op_ac_unlock] 0-management:<br>
> >> Unable to acquire volname<br>
> >> [2015-09-21 08:14:50.548087] E [MSGID: 106062]<br>
> >> [glusterd-op-sm.c:3635:glusterd_op_ac_lock] 0-management:<br>
Unable<br>
> >> to acquire volname<br>
> >> [2015-09-21 08:14:50.654746] I [MSGID: 106132]<br>
> >> [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs<br>
> >> already stopped<br>
> >> [2015-09-21 08:14:50.655095] I [MSGID: 106474]<br>
> >> [glusterd-ganesha.c:403:check_host_list] 0-management: ganesha<br>
> >> host found Hostname is cobalt<br>
> >> [2015-09-21 08:14:51.287156] E [MSGID: 106062]<br>
> >> [glusterd-op-sm.c:3698:glusterd_op_ac_unlock] 0-management:<br>
> >> Unable to acquire volname<br>
> >><br>
> >><br>
> >> from etc-glusterfs-glusterd.vol.log on the arbiter node:<br>
> >><br>
> >> [2015-09-21 08:18:50.934713] E [MSGID: 101075]<br>
> >> [common-utils.c:3127:gf_is_local_addr] 0-management: error in<br>
> >> getaddrinfo: Name or service not known<br>
> >> [2015-09-21 08:18:51.504694] E [MSGID: 106062]<br>
> >> [glusterd-op-sm.c:3698:glusterd_op_ac_unlock] 0-management:<br>
> >> Unable to acquire volname<br>
> >><br>
> >> I have put the hostnames of all servers in my /etc/hosts file,<br>
> >> including the arbiter node.<br>
> >><br>
> >><br>
> >> On 18 September 2015 at 16:52, Soumya Koduri<br>
<<a href="mailto:skoduri@redhat.com" target="_blank">skoduri@redhat.com</a> <mailto:<a href="mailto:skoduri@redhat.com" target="_blank">skoduri@redhat.com</a>><br></div></div>
> >> <mailto:<a href="mailto:skoduri@redhat.com" target="_blank">skoduri@redhat.com</a> <mailto:<a href="mailto:skoduri@redhat.com" target="_blank">skoduri@redhat.com</a>>>><div><div><br>
wrote:<br>
> >><br>
> >> Hi Tiemen,<br>
> >><br>
> >> One of the pre-requisites before setting up<br>
nfs-ganesha HA is<br>
> >> to create and mount shared_storage volume. Use below<br>
CLI for that<br>
> >><br>
> >> "gluster volume set all cluster.enable-shared-storage<br>
enable"<br>
> >><br>
> >> It shall create the volume and mount in all the nodes<br>
> >> (including the arbiter node). Note this volume shall be<br>
> >> mounted on all the nodes of the gluster storage pool<br>
(though<br>
> >> in this case it may not be part of nfs-ganesha cluster).<br>
> >><br>
> >> So instead of manually creating those directory paths,<br>
please<br>
> >> use above CLI and try re-configuring the setup.<br>
> >><br>
> >> Thanks,<br>
> >> Soumya<br>
> >><br>
> >> On 09/18/2015 07:29 PM, Tiemen Ruiten wrote:<br>
> >><br>
> >> Hello Kaleb,<br>
> >><br>
> >> I don't:<br>
> >><br>
> >> # Name of the HA cluster created.<br>
> >> # must be unique within the subnet<br>
> >> HA_NAME="rd-ganesha-ha"<br>
> >> #<br>
> >> # The gluster server from which to mount the<br>
shared data<br>
> >> volume.<br>
> >> HA_VOL_SERVER="iron"<br>
> >> #<br>
> >> # N.B. you may use short names or long names; you<br>
may not<br>
> >> use IP addrs.<br>
> >> # Once you select one, stay with it as it will be<br>
mildly<br>
> >> unpleasant to<br>
> >> # clean up if you switch later on. Ensure that all<br>
names<br>
> >> - short and/or<br>
> >> # long - are in DNS or /etc/hosts on all machines<br>
in the<br>
> >> cluster.<br>
> >> #<br>
> >> # The subset of nodes of the Gluster Trusted Pool that<br>
> >> form the ganesha<br>
> >> # HA cluster. Hostname is specified.<br>
> >> HA_CLUSTER_NODES="cobalt,iron"<br>
> >> #HA_CLUSTER_NODES="<a href="http://server1.lab.redhat.com" rel="noreferrer" target="_blank">server1.lab.redhat.com</a><br>
<<a href="http://server1.lab.redhat.com" rel="noreferrer" target="_blank">http://server1.lab.redhat.com</a>><br>
> >> <<a href="http://server1.lab.redhat.com" rel="noreferrer" target="_blank">http://server1.lab.redhat.com</a>><br>
> >><br>
<<a href="http://server1.lab.redhat.com" rel="noreferrer" target="_blank">http://server1.lab.redhat.com</a>>,<a href="http://server2.lab.redhat.com" rel="noreferrer" target="_blank">server2.lab.redhat.com</a><br>
<<a href="http://server2.lab.redhat.com" rel="noreferrer" target="_blank">http://server2.lab.redhat.com</a>><br>
> >> <<a href="http://server2.lab.redhat.com" rel="noreferrer" target="_blank">http://server2.lab.redhat.com</a>><br>
> >> <<a href="http://server2.lab.redhat.com" rel="noreferrer" target="_blank">http://server2.lab.redhat.com</a>>,..."<br>
> >> #<br>
> >> # Virtual IPs for each of the nodes specified above.<br>
> >> VIP_server1="10.100.30.101"<br>
> >> VIP_server2="10.100.30.102"<br>
> >> #VIP_server1_lab_redhat_com="10.0.2.1"<br>
> >> #VIP_server2_lab_redhat_com="10.0.2.2"<br>
> >><br>
> >> hosts cobalt & iron are the data nodes, the arbiter<br>
> >> ip/hostname (neon)<br>
> >> isn't mentioned anywhere in this config file.<br>
> >><br>
> >><br>
> >> On 18 September 2015 at 15:56, Kaleb S. KEITHLEY<br>
> >> <<a href="mailto:kkeithle@redhat.com" target="_blank">kkeithle@redhat.com</a> <mailto:<a href="mailto:kkeithle@redhat.com" target="_blank">kkeithle@redhat.com</a>><br>
<mailto:<a href="mailto:kkeithle@redhat.com" target="_blank">kkeithle@redhat.com</a> <mailto:<a href="mailto:kkeithle@redhat.com" target="_blank">kkeithle@redhat.com</a>>><br>
> >> <mailto:<a href="mailto:kkeithle@redhat.com" target="_blank">kkeithle@redhat.com</a><br>
<mailto:<a href="mailto:kkeithle@redhat.com" target="_blank">kkeithle@redhat.com</a>><br>
> >> <mailto:<a href="mailto:kkeithle@redhat.com" target="_blank">kkeithle@redhat.com</a><br>
<mailto:<a href="mailto:kkeithle@redhat.com" target="_blank">kkeithle@redhat.com</a>>>>> wrote:<br>
> >><br>
> >> On 09/18/2015 09:46 AM, Tiemen Ruiten wrote:<br>
> >> > Hello,<br>
> >> ><br>
> >> > I have a Gluster cluster with a single<br>
replica 3,<br>
> >> arbiter 1 volume (so<br>
> >> > two nodes with actual data, one arbiter node). I<br>
> >> would like to setup<br>
> >> > NFS-Ganesha HA for this volume but I'm<br>
having some<br>
> >> difficulties.<br>
> >> ><br>
> >> > - I needed to create a directory<br>
> >> /var/run/gluster/shared_storage<br>
> >> > manually on all nodes, or the command 'gluster<br>
> >> nfs-ganesha enable would<br>
> >> > fail with the following error:<br>
> >> > [2015-09-18 13:13:34.690416] E [MSGID: 106032]<br>
> >> > [glusterd-ganesha.c:708:pre_setup] 0-THIS->name:<br>
> >> mkdir() failed on path<br>
> >> > /var/run/gluster/shared_storage/nfs-ganesha, [No<br>
> >> such file or directory]<br>
> >> ><br>
> >> > - Then I found out that the command connects<br>
to the<br>
> >> arbiter node as<br>
> >> > well, but obviously I don't want to set up<br>
> >> NFS-Ganesha there. Is it<br>
> >> > actually possible to setup NFS-Ganesha HA<br>
with an<br>
> >> arbiter node? If it's<br>
> >> > possible, is there any documentation on how<br>
to do that?<br>
> >> ><br>
> >><br>
> >> Please send the /etc/ganesha/ganesha-ha.conf file<br>
> >> you're using.<br>
> >><br>
> >> Probably you have included the arbiter in your HA<br>
> >> config; that would be<br>
> >> a mistake.<br>
> >><br>
> >> --<br>
> >><br>
> >> Kaleb<br>
> >><br>
> >><br>
> >><br>
> >><br>
> >> --<br>
> >> Tiemen Ruiten<br>
> >> Systems Engineer<br>
> >> R&D Media<br>
> >><br>
> >><br>
> >> _______________________________________________<br>
> >> Gluster-users mailing list<br>
> >> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>><br></div></div>
<mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>>><span><br>
> >><a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> >><br>
> >><br>
> >><br>
> >><br>
> >> --<br>
> >> Tiemen Ruiten<br>
> >> Systems Engineer<br>
> >> R&D Media<br>
> >><br>
> >><br>
> >> _______________________________________________<br>
> >> Gluster-users mailing list<br>
> >> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>><br></span>
<mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>>><span><br>
> >><a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Gluster-users mailing list<br>
> > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>><br></span>
<mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>>><div><div><br>
> ><a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Tiemen Ruiten<br>
> > Systems Engineer<br>
> > R&D Media<br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Tiemen Ruiten<br>
> > Systems Engineer<br>
> > R&D Media<br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Gluster-users mailing list<br>
> ><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>><br>
> ><a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
><br>
><br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
<br>
<br>
<br>
<br>
--<br>
Tiemen Ruiten<br>
Systems Engineer<br>
R&D Media<br>
<br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
<br>
</div></div></blockquote>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr">Tiemen Ruiten<br>Systems Engineer<br>R&D Media<br></div></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">Tiemen Ruiten<br>Systems Engineer<br>R&D Media<br></div></div>
</div>