<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Feb 13, 2016 at 12:38 AM, Richard Wareing <span dir="ltr"><<a href="mailto:rwareing@fb.com" target="_blank">rwareing@fb.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">Hey,
<div><br>
</div>
<div>Sorry for the late reply but I missed this e-mail. With respect to identifying locking domains,<span style="font-size:13.3333px"> we use the identical logic that GlusterFS itself uses to identify the domains; which is just a simple string comparison
if I'm not mistaken. System processes (SHD/Rebalance) locking domains are treated identical to any other, this is specifically critical to things like DHT healing as this locking domain is used both in userland and by SHDs (you cannot disable DHT healing).
</span></div></div></div></blockquote><div><br></div><div style="">We cannot disable DHT healing altogether. But we _can_ identify whether healing is done by a mount process (on behalf of application) or a rebalance process. All internal processes (rebalance, shd, quotad etc) have a negative value in frame->root->pid (as opposed to a positive value for a fop request from a mount process). I agree with you that just by looking at domain, we cannot figure out whether lock request is from internal process or a mount process. But, with the help of frame->root->pid, we can. By choosing to flush locks from rebalance process (instead of locks from mount process), I thought we can reduce the scenarios where application sees errors. Of course we'll see more of rebalance failures, but that is a trade-off we perhaps have to live with. Just a thought :).</div><div style=""><br></div><div style=""> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt"><div><span style="font-size:13.3333px"> To illustrate this, consider the case where a SHD holds a lock to do a DHT heal but can't because of GFID split-brain....a user comes along a hammers that directory attempting to get a lock....you can pretty much kiss your cluster good-bye after that :).</span></div>
<div><span style="font-size:13.3333px"><br>
</span></div>
<div>With this in mind, we explicitly choose not to respect system process (SHD/rebalance) locks any more than a user lock request as they can be just as likely (if not more so) to cause a system to fall over vs. a user (see example above). Although this might
seem unwise at first, I'd put forth that having clusters fall over catastrophically pushes far worse decisions on operators such as re-kicking random bricks or entire clusters in desperate attempts at freeing locks (the CLI is often unable to free the locks
in our experience) or stopping run away memory consumption due to frames piling up on the bricks. To date, we haven't even observed a single instance of data corruption (and we've been looking for it!) due to this feature.<br>
<br>
We've even used it on clusters where they were on the verge of falling over and we enable revocation and the entire system stabilizes almost instantly (it's really like magic when you see it :) ).<br>
<br>
</div>
<div>Hope this helps!</div>
<div><br>
</div>
<div>Richard</div>
<div><br>
</div>
<div><br>
<div style="font-family:Times New Roman;color:#000000;font-size:16px">
<hr>
<div style="direction:ltr"><font face="Tahoma" size="2" color="#000000"><b>From:</b> <a href="mailto:raghavendra.hg@gmail.com" target="_blank">raghavendra.hg@gmail.com</a> [<a href="mailto:raghavendra.hg@gmail.com" target="_blank">raghavendra.hg@gmail.com</a>] on behalf of Raghavendra G [<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>]<br>
<b>Sent:</b> Tuesday, January 26, 2016 9:49 PM<br>
<b>To:</b> Raghavendra Gowdappa<span class=""><br>
<b>Cc:</b> Richard Wareing; Gluster Devel<br>
</span><div><div class="h5"><b>Subject:</b> Re: [Gluster-devel] Feature: Automagic lock-revocation for features/locks xlator (v3.7.x)<br>
</div></div></font><br>
</div><div><div class="h5">
<div></div>
<div>
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Jan 25, 2016 at 10:39 AM, Raghavendra Gowdappa <span dir="ltr">
<<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span><br>
<br>
----- Original Message -----<br>
> From: "Richard Wareing" <<a href="mailto:rwareing@fb.com" target="_blank">rwareing@fb.com</a>><br>
> To: "Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>><br>
> Cc: <a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a><br>
> Sent: Monday, January 25, 2016 8:17:11 AM<br>
> Subject: Re: [Gluster-devel] Feature: Automagic lock-revocation for features/locks xlator (v3.7.x)<br>
><br>
> Yup per domain would be useful, the patch itself currently honors domains as<br>
> well. So locks in a different domains will not be touched during revocation.<br>
><br>
> In our cases we actually prefer to pull the plug on SHD/DHT domains to ensure<br>
> clients do not hang, this is important for DHT self heals which cannot be<br>
> disabled via any option, we've found in most cases once we reap the lock<br>
> another properly behaving client comes along and completes the DHT heal<br>
> properly.<br>
<br>
</span>Flushing waiting locks of DHT can affect application continuity too. Though locks requested by rebalance process can be flushed to certain extent without applications noticing any failures, there is no guarantee that locks requested in DHT_LAYOUT_HEAL_DOMAIN
and DHT_FILE_MIGRATE_DOMAIN, are issued by only rebalance process.</blockquote>
<div><br>
</div>
<div>I missed this point in my previous mail. Now I remember that we can use frame->root->pid (being negative) to identify internal processes. Was this the approach you followed to identify locks from rebalance process?</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
These two domains are used for locks to synchronize among and between rebalance process(es) and client(s). So, there is equal probability that these locks might be requests from clients and hence application can see some file operations failing.<br>
<br>
In case of pulling plug on DHT_LAYOUT_HEAL_DOMAIN, dentry operations that depend on layout can fail. These operations can include create, link, unlink, symlink, mknod, mkdir, rename for files/directory within the directory on which lock request is failed.<br>
<br>
In case of pulling plug on DHT_FILE_MIGRATE_DOMAIN, rename of immediate subdirectories/files can fail.<br>
<div>
<div><br>
<br>
><br>
> Richard<br>
><br>
><br>
> Sent from my iPhone<br>
><br>
> On Jan 24, 2016, at 6:42 PM, Pranith Kumar Karampuri < <a href="mailto:pkarampu@redhat.com" target="_blank">
pkarampu@redhat.com</a> ><br>
> wrote:<br>
><br>
><br>
><br>
><br>
><br>
><br>
> On 01/25/2016 02:17 AM, Richard Wareing wrote:<br>
><br>
><br>
><br>
> Hello all,<br>
><br>
> Just gave a talk at SCaLE 14x today and I mentioned our new locks revocation<br>
> feature which has had a significant impact on our GFS cluster reliability.<br>
> As such I wanted to share the patch with the community, so here's the<br>
> bugzilla report:<br>
><br>
> <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1301401" rel="noreferrer" target="_blank">
https://bugzilla.redhat.com/show_bug.cgi?id=1301401</a><br>
><br>
> =====<br>
> Summary:<br>
> Mis-behaving brick clients (gNFSd, FUSE, gfAPI) can cause cluster instability<br>
> and eventual complete unavailability due to failures in releasing<br>
> entry/inode locks in a timely manner.<br>
><br>
> Classic symptoms on this are increased brick (and/or gNFSd) memory usage due<br>
> the high number of (lock request) frames piling up in the processes. The<br>
> failure-mode results in bricks eventually slowing down to a crawl due to<br>
> swapping, or OOMing due to complete memory exhaustion; during this period<br>
> the entire cluster can begin to fail. End-users will experience this as<br>
> hangs on the filesystem, first in a specific region of the file-system and<br>
> ultimately the entire filesystem as the offending brick begins to turn into<br>
> a zombie (i.e. not quite dead, but not quite alive either).<br>
><br>
> Currently, these situations must be handled by an administrator detecting &<br>
> intervening via the "clear-locks" CLI command. Unfortunately this doesn't<br>
> scale for large numbers of clusters, and it depends on the correct<br>
> (external) detection of the locks piling up (for which there is little<br>
> signal other than state dumps).<br>
><br>
> This patch introduces two features to remedy this situation:<br>
><br>
> 1. Monkey-unlocking - This is a feature targeted at developers (only!) to<br>
> help track down crashes due to stale locks, and prove the utility of he lock<br>
> revocation feature. It does this by silently dropping 1% of unlock requests;<br>
> simulating bugs or mis-behaving clients.<br>
><br>
> The feature is activated via:<br>
> features.locks-monkey-unlocking <on/off><br>
><br>
> You'll see the message<br>
> "[<timestamp>] W [inodelk.c:653:pl_inode_setlk] 0-groot-locks: MONKEY LOCKING<br>
> (forcing stuck lock)!" ... in the logs indicating a request has been<br>
> dropped.<br>
><br>
> 2. Lock revocation - Once enabled, this feature will revoke a *contended*lock<br>
> (i.e. if nobody else asks for the lock, we will not revoke it) either by the<br>
> amount of time the lock has been held, how many other lock requests are<br>
> waiting on the lock to be freed, or some combination of both. Clients which<br>
> are losing their locks will be notified by receiving EAGAIN (send back to<br>
> their callback function).<br>
><br>
> The feature is activated via these options:<br>
> features.locks-revocation-secs <integer; 0 to disable><br>
> features.locks-revocation-clear-all [on/off]<br>
> features.locks-revocation-max-blocked <integer><br>
><br>
> Recommended settings are: 1800 seconds for a time based timeout (give clients<br>
> the benefit of the doubt, or chose a max-blocked requires some<br>
> experimentation depending on your workload, but generally values of hundreds<br>
> to low thousands (it's normal for many ten's of locks to be taken out when<br>
> files are being written @ high throughput).<br>
><br>
> I really like this feature. One question though, self-heal, rebalance domain<br>
> locks are active until self-heal/rebalance is complete which can take more<br>
> than 30 minutes if the files are in TBs. I will try to see what we can do to<br>
> handle these without increasing the revocation-secs too much. May be we can<br>
> come up with per domain revocation timeouts. Comments are welcome.<br>
><br>
> Pranith<br>
><br>
><br>
><br>
><br>
> =====<br>
><br>
> The patch supplied will patch clean the the v3.7.6 release tag, and probably<br>
> to any 3.7.x release & master (posix locks xlator is rarely touched).<br>
><br>
> Richard<br>
><br>
><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Gluster-devel mailing list <a href="mailto:Gluster-devel@gluster.org" target="_blank">
Gluster-devel@gluster.org</a><br>
> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.gluster.org_mailman_listinfo_gluster-2Ddevel&d=CwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=qJ8Lp7ySfpQklq3QZr44Iw&m=PZGWfy5Y16_RQ1y2K73ShgTiHmTdeo4ZTHlPNp5ENd8&s=hgqwXn2r8O02F2c4nk3ssRN_DNRBtaa7vsBxJbRwi1g&e=" rel="noreferrer" target="_blank">
http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
><br>
><br>
> _______________________________________________<br>
> Gluster-devel mailing list<br>
> <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.gluster.org_mailman_listinfo_gluster-2Ddevel&d=CwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=qJ8Lp7ySfpQklq3QZr44Iw&m=PZGWfy5Y16_RQ1y2K73ShgTiHmTdeo4ZTHlPNp5ENd8&s=hgqwXn2r8O02F2c4nk3ssRN_DNRBtaa7vsBxJbRwi1g&e=" rel="noreferrer" target="_blank">
http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.gluster.org_mailman_listinfo_gluster-2Ddevel&d=CwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=qJ8Lp7ySfpQklq3QZr44Iw&m=PZGWfy5Y16_RQ1y2K73ShgTiHmTdeo4ZTHlPNp5ENd8&s=hgqwXn2r8O02F2c4nk3ssRN_DNRBtaa7vsBxJbRwi1g&e=" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div>Raghavendra G<br>
</div>
</div>
</div>
</div>
</div></div></div>
</div>
</div>
</div>
<br>_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Raghavendra G<br></div>
</div></div>