<div dir="ltr">Thanks for the info Joe! responded in line<div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 31, 2015 at 3:00 PM, Joe Julian <span dir="ltr"><<a href="mailto:joe@julianfamily.org" target="_blank">joe@julianfamily.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class=""><div class="h5">On 08/31/2015 12:35 PM, Grant Ridder wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Hi,<br>
<br>
I am testing out several failure scenarios with GlusterFS. I have a 3 node replicated gluster that i am testing with.<br>
<br>
One test i am having trouble solving is when a host dies. (i.e. as if someone pulled the power cord out).<br>
- Firewall off a host from the rest of the cluster<br>
- Test time it takes for a fuse mount to respond once the iptables rule is added<br>
<br>
With the default settings, the mount hangs for 39 seconds. If i change ping-timeout to 5 then the mount only hangs for 9.3 seconds. Is there anyway to eliminate or get the hang time to a negligible value (less than 1 second)?<br>
<br>
I have not seem much about handling GlusterFS failure scenarios with my Googling around.<br>
<br>
Several blog posts i have looked at:<br>
<a href="http://thornelabs.net/2015/02/24/change-gluster-volume-connection-timeout-for-glusterfs-native-client.html" rel="noreferrer" target="_blank">http://thornelabs.net/2015/02/24/change-gluster-volume-connection-timeout-for-glusterfs-native-client.html</a><br>
<a href="https://joejulian.name/blog/keeping-your-vms-from-going-read-only-when-encountering-a-ping-timeout-in-glusterfs/" rel="noreferrer" target="_blank">https://joejulian.name/blog/keeping-your-vms-from-going-read-only-when-encountering-a-ping-timeout-in-glusterfs/</a><br>
<br>
Thanks,<br>
Grant<br>
<br>
</blockquote></div></div>
If a client disconnects from a server, you have to reestablish all the file descriptors and synchronize the locks when the client reconnects. This can be pretty expensive and there's no way to avoid it. To balance that, you don't want your clients to disconnect from the servers if a packet is lost or takes too long to get a response. That's why the connections are TCP, to help mitigate that, and why the client waits for some ping-timeout. If it was too short, even server load could trigger a disconnection which would be followed by high server load as the connection was reestablished, potentially causing a disconnection again.<br>
<br>
Pulled power cords or complete system failures, should be a very rare occurrence.</blockquote><div><br></div><div>This unfortunately isn't the case with AWS EC2 instances.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> Typically this is much rarer than a temporary network issue which is much more likely to be mitigated in the network fabric and is transient enough to allow the ping-timeout to hold the connection long enough to avoid the reestablishment of FDs and locks. It's also only an issue if your file hash hits that specific replica out of the dht set (or the file doesn't exist). If you're using a cluster where server failure is frequent enough to be an issue, your dht distribution lowers the likelihood of the file being hit to an insignificant statistic.<br>
<br>
If you're working with reasonably resilient hardware, you should easily be able to engineer for 5 or 6 nines even with a 42 second ping-timeout.<br></blockquote><div><br></div><div>What settings do people normally use for EC2? I have to account for instances dying and not having the clients have a significant impact. 42 seconds is a VERY long time for a data store to be unavailable.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br></div></div>