<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Raghavendra,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">This is exactly what I was missing, thanks! Based on the information you gave I was I was able to find and adapt the script from
<a href="http://xrsa.net/2015/04/25/ctdb-glusterfs-nfs-event-monitor-script/"><span style="color:windowtext">http://xrsa.net/2015/04/25/ctdb-glusterfs-nfs-event-monitor-script/</span></a> to work with CentOS 6.7 and stopping the NFS PID now results in CTDB
failover in 20 seconds or less.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I didn’t actually need to set
</span>CTDB_MANAGES_NFS, just adding the new event monitor in /etc/ctdb/events.d did the trick.<span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thanks!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Raghavendra Talur [mailto:rtalur@redhat.com]
<br>
<b>Sent:</b> Wednesday, January 27, 2016 10:22 PM<br>
<b>To:</b> Kris Laib <Kris.Laib@nwea.org><br>
<b>Cc:</b> Soumya Koduri <skoduri@redhat.com>; gluster-users@gluster.org<br>
<b>Subject:</b> Re: [Gluster-users] How to maintain HA using NFS clients if the NFS daemon process gets killed on a gluster node?<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Thu, Jan 28, 2016 at 11:08 AM, Kris Laib <<a href="mailto:Kris.Laib@nwea.org" target="_blank">Kris.Laib@nwea.org</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal" style="margin-bottom:12.0pt">Soumya,<br>
<br>
CTDB failover works great if the server crashes or the NIC is pulled, but I don't believe there's anything in the CTDB setup that would cause it to realize there is a problem if only the glusterfs process responsible for serving NFS is killed but network connectivity
with other CTDB nodes remains intact. If others are able to kill just the PID for the associated "NFS Server on localhost" process and have CTDB issue a failover, I'd be very interested to know how their setup differs from mine.<o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I think you can achieve that by CTDB_MANAGES_NFS option. Refer to last four sections on this link <a href="https://ctdb.samba.org/nfs.html">https://ctdb.samba.org/nfs.html</a> . I have not personally used this option and because this is
gluster-NFS and not kernel NFS, you might have to edit the scripts like /etc/ctdb/events.d/60.nfs.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Thanks for the nfs-ganesha suggestion, I'm not very familiar with that option and don't have enough time in my timeline to properly test it before moving to production, but I will look into it further for a possible solution down the road
or if my deadline gets extended. The FUSE client may be a good option for us as well, but I can't seem to get speeds higher than 30 MB/s using the Gluster FUSE client (I posted more details on that earlier today to this group as well, looking for advice
there).<br>
<br>
-Kris<br>
<br>
________________________________________<br>
From: Soumya Koduri <<a href="mailto:skoduri@redhat.com">skoduri@redhat.com</a>><br>
Sent: Wednesday, January 27, 2016 8:15 PM<br>
To: Kris Laib; <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
Subject: Re: [Gluster-users] How to maintain HA using NFS clients if the NFS daemon process gets killed on a gluster node?<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><br>
On 01/27/2016 09:39 PM, Kris Laib wrote:<br>
> Hi all,<br>
><br>
> We're getting ready to roll out Gluster using standard NFS from the<br>
> clients, and CTDB and RRDNS to help facilitate HA. I thought we were<br>
> good to know, but recently had an issue where there wasn't enough memory<br>
> on one of the gluster nodes in a test cluster, and OOM killer took out<br>
> the NFS daemon process. Since there was still IP traffic between nodes<br>
> and the gluster-based local CTDB mount for the lock file was intact,<br>
> CTDB didn't kick in an initiate failover, and all clients connected to<br>
<br>
For gluster-NFS, CTDB is typically configured to maintain high<br>
availability and I guess you have done the same. Could you check why<br>
CTDB hasn't initiated IP failover?<br>
<br>
An alternative solution is to use nfs-ganesha [1][2] to provide NFS<br>
support for gluster volumes and can be configured to maintain HA using<br>
gluster CLI.<br>
<br>
Thanks,<br>
Soumya<br>
<br>
[1]<br>
<a href="http://blog.gluster.org/2015/10/linux-scale-out-nfsv4-using-nfs-ganesha-and-glusterfs-one-step-at-a-time/" target="_blank">http://blog.gluster.org/2015/10/linux-scale-out-nfsv4-using-nfs-ganesha-and-glusterfs-one-step-at-a-time/</a><br>
<br>
[2]<br>
<a href="http://gluster.readthedocs.org/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Intergration/" target="_blank">http://gluster.readthedocs.org/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Intergration/</a><br>
(section# Using Highly Available Active-Active NFS-Ganesha And GlusterFS<br>
cli)<br>
<br>
> the node where NFS was killed lost their connections. We'll obviously<br>
> fix the lack of memory, but going forward how can we protect against<br>
> clients getting disconnected if the NFS daemon should be stopped for any<br>
> reason?<br>
><br>
> Our cluster is 3 nodes, 1 is a silent witness node to help with split<br>
> brain, and the other 2 host the volumes with one brick per node, and 1x2<br>
> replication.<br>
><br>
> Is there something incorrect about my setup, or is this a known downfall<br>
> to using standard NFS mounts with gluster?<br>
><br>
> Thanks,<br>
><br>
> Kris<br>
><br>
><br>
><br>
> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">
http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
><br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>