<div dir="ltr"><span style="font-size:12.8px">Hi,</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">I have looked into this on another system earlier and this is what I have so far:</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">1. The test involves moving and renaming directories and files within those dirs.</span><br style="font-size:12.8px"><span style="font-size:12.8px">2. A rename dir operation failed on one subvol. So we have 3 subvols where the directory has the new name and one where it has the old name.</span><br style="font-size:12.8px"><span style="font-size:12.8px">3. Some operation - perhaps a revalidate - has added a dentry with the old name to the inode . So there are now 2 dentries for the same inode for a directory.</span><br style="font-size:12.8px"><span style="font-size:12.8px">4. Renaming a file inside that directory calls an inode_link which end up traversing the dentry list for each entry all the way up to the root in the __foreach_ancestor_dentry function. If there are multiple deep directories with the same problem in the path, this takes a very long time (hours) because of the number of times the function is called.</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">I do not know why the rename dir failed. However, is the following a correct/acceptable fix for the traversal issue?</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">1. A directory should never have more than one dentry</span><br style="font-size:12.8px"><span style="font-size:12.8px">2. __foreach_ancestor_dentry uses the dentry list of the parent inode. Parent inode will always be a directory.</span><br style="font-size:12.8px"><span style="font-size:12.8px">3. Can we just take the first dentry in the list for the cycle check as we are really only comparing inodes? In the scenarios I have tried, all the dentries in the dentry_list always have the same inode. This would prevent the hang. If there is more than one dentry for a directory, flag an error somehow.</span><br style="font-size:12.8px"><span style="font-size:12.8px">4. Is there any chance that a dentry in the list can have a different inode? If yes, that is a different problem and 3 does not apply.</span><br style="font-size:12.8px"><br>It would work like this:<div><br></div><div><div><br></div><div>last_parent_inode = NULL;</div><div><br></div><div> list_for_each_entry (each, &parent->dentry_list, inode_list) {</div><div><br></div><div>//Since we are only using the each->parent to check, stop if we have already checked it</div><div><br></div><div> if(each->parent != last_parent_inode) {</div><div> ret = __foreach_ancestor_dentry (each, per_dentry_fn, data);</div><div> if (ret)</div><div> goto out;</div><div> }</div><div> last_parent_inode = each->parent;</div><div> }</div><br><br style="font-size:12.8px"><span style="font-size:12.8px">This would prevent the hang but leads to other issues which would exist in the current code anyway - mainly, which dentry is the correct one and how do we recover?</span><br style="font-size:12.8px"><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">Regards,</span><br style="font-size:12.8px"><span style="font-size:12.8px">Nithya</span><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 11, 2016 at 7:09 PM, Niels de Vos <span dir="ltr"><<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Could someone look into this busy loop?<br>
<a href="https://paste.fedoraproject.org/365207/29732171/raw/" rel="noreferrer" target="_blank">https://paste.fedoraproject.org/365207/29732171/raw/</a><br>
<br>
This was happening in a regression-test burn-in run, occupying a Jenkins<br>
slave for 2+ days:<br>
<a href="https://build.gluster.org/job/regression-test-burn-in/936/" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-burn-in/936/</a><br>
(run with commit f0ade919006b2581ae192f997a8ae5bacc2892af from master)<br>
<br>
A coredump of the mount process is available from here:<br>
<a href="http://slave20.cloud.gluster.org/archived_builds/crash.tar.gz" rel="noreferrer" target="_blank">http://slave20.cloud.gluster.org/archived_builds/crash.tar.gz</a><br>
<br>
Thanks misc for reporting and gathering the debugging info.<br>
<span class="HOEnZb"><font color="#888888">Niels<br>
</font></span><br>_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br></blockquote></div><br></div>