<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 17, 2016 at 2:52 PM, Raghavendra Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class=""><br>
<br>
----- Original Message -----<br>
> From: "Raghavendra Gowdappa" <<a href="mailto:rgowdapp@redhat.com">rgowdapp@redhat.com</a>><br>
> To: "Nithya Balachandran" <<a href="mailto:nbalacha@redhat.com">nbalacha@redhat.com</a>><br>
> Cc: <a href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a><br>
</span><div><div class="h5">> Sent: Tuesday, May 17, 2016 2:40:32 PM<br>
> Subject: Re: [Gluster-devel] 'mv' of ./tests/bugs/posix/bug-1113960.t causes 100% CPU<br>
><br>
><br>
><br>
> ----- Original Message -----<br>
> > From: "Nithya Balachandran" <<a href="mailto:nbalacha@redhat.com">nbalacha@redhat.com</a>><br>
> > To: "Niels de Vos" <<a href="mailto:ndevos@redhat.com">ndevos@redhat.com</a>><br>
> > Cc: <a href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a><br>
> > Sent: Tuesday, May 17, 2016 2:25:20 PM<br>
> > Subject: Re: [Gluster-devel] 'mv' of ./tests/bugs/posix/bug-1113960.t<br>
> > causes 100% CPU<br>
> ><br>
> > Hi,<br>
> ><br>
> > I have looked into this on another system earlier and this is what I have<br>
> > so<br>
> > far:<br>
> ><br>
> > 1. The test involves moving and renaming directories and files within those<br>
> > dirs.<br>
> > 2. A rename dir operation failed on one subvol. So we have 3 subvols where<br>
> > the directory has the new name and one where it has the old name.<br>
> > 3. Some operation - perhaps a revalidate - has added a dentry with the old<br>
> > name to the inode . So there are now 2 dentries for the same inode for a<br>
> > directory.<br>
><br>
> I think the stale dentry is caused by a racing lookup and rename.<br>
<br>
</div></div>Please refer to bz 1335373 [1] for more details.<br>
<br>
[1] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1335373" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1335373</a></blockquote><div><br></div><div><br></div><div>I have posted a "hacky" patch at [2]</div><div><br></div><div>[2] <a href="http://review.gluster.org/#/c/14373/">http://review.gluster.org/#/c/14373/</a></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
<div class=""><div class="h5"><br>
> Apart from<br>
> that, I don't know of any other reasons for stale dentries in inode table.<br>
> "Dentry fop serializer" (DFS) [1], aims to solve these kind of races.<br>
><br>
> [1] <a href="http://review.gluster.org/14286" rel="noreferrer" target="_blank">http://review.gluster.org/14286</a><br>
><br>
> > 4. Renaming a file inside that directory calls an inode_link which end up<br>
> > traversing the dentry list for each entry all the way up to the root in the<br>
> > __foreach_ancestor_dentry function. If there are multiple deep directories<br>
> > with the same problem in the path, this takes a very long time (hours)<br>
> > because of the number of times the function is called.<br>
> ><br>
> > I do not know why the rename dir failed. However, is the following a<br>
> > correct/acceptable fix for the traversal issue?<br>
> ><br>
> > 1. A directory should never have more than one dentry<br>
> > 2. __foreach_ancestor_dentry uses the dentry list of the parent inode.<br>
> > Parent<br>
> > inode will always be a directory.<br>
> > 3. Can we just take the first dentry in the list for the cycle check as we<br>
> > are really only comparing inodes? In the scenarios I have tried, all the<br>
> > dentries in the dentry_list always have the same inode. This would prevent<br>
> > the hang. If there is more than one dentry for a directory, flag an error<br>
> > somehow.<br>
> > 4. Is there any chance that a dentry in the list can have a different<br>
> > inode?<br>
> > If yes, that is a different problem and 3 does not apply.<br>
> ><br>
> > It would work like this:<br>
> ><br>
> ><br>
> > last_parent_inode = NULL;<br>
> ><br>
> > list_for_each_entry (each, &parent->dentry_list, inode_list) {<br>
> ><br>
> > //Since we are only using the each->parent to check, stop if we have<br>
> > already<br>
> > checked it<br>
> ><br>
> > if(each->parent != last_parent_inode) {<br>
> > ret = __foreach_ancestor_dentry (each, per_dentry_fn, data);<br>
> > if (ret)<br>
> > goto out;<br>
> > }<br>
> > last_parent_inode = each->parent;<br>
> > }<br>
> ><br>
> ><br>
> > This would prevent the hang but leads to other issues which would exist in<br>
> > the current code anyway - mainly, which dentry is the correct one and how<br>
> > do<br>
> > we recover?<br>
> ><br>
> ><br>
> > Regards,<br>
> > Nithya<br>
> ><br>
> > On Wed, May 11, 2016 at 7:09 PM, Niels de Vos < <a href="mailto:ndevos@redhat.com">ndevos@redhat.com</a> > wrote:<br>
> ><br>
> ><br>
> > Could someone look into this busy loop?<br>
> > <a href="https://paste.fedoraproject.org/365207/29732171/raw/" rel="noreferrer" target="_blank">https://paste.fedoraproject.org/365207/29732171/raw/</a><br>
> ><br>
> > This was happening in a regression-test burn-in run, occupying a Jenkins<br>
> > slave for 2+ days:<br>
> > <a href="https://build.gluster.org/job/regression-test-burn-in/936/" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-burn-in/936/</a><br>
> > (run with commit f0ade919006b2581ae192f997a8ae5bacc2892af from master)<br>
> ><br>
> > A coredump of the mount process is available from here:<br>
> > <a href="http://slave20.cloud.gluster.org/archived_builds/crash.tar.gz" rel="noreferrer" target="_blank">http://slave20.cloud.gluster.org/archived_builds/crash.tar.gz</a><br>
> ><br>
> > Thanks misc for reporting and gathering the debugging info.<br>
> > Niels<br>
> ><br>
> > _______________________________________________<br>
> > Gluster-devel mailing list<br>
> > <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Gluster-devel mailing list<br>
> > <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
> _______________________________________________<br>
> Gluster-devel mailing list<br>
> <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
><br>
</div></div></blockquote></div><br></div></div>