<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN">
<html><body>
<p> </p>
<p>On 02.01.2015 05:45, Raghavendra G wrote:</p>
<blockquote type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%">
<div dir="ltr"><br />
<div class="gmail_extra"><br />
<div class="gmail_quote">On Wed, Dec 31, 2014 at 11:25 PM, Xavier Hernandez <span><<a href="mailto:xhernandez@datalab.es">xhernandez@datalab.es</a>></span> wrote:<br />
<blockquote class="gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;">
<div>
<p>On 27.12.2014 13:43, <a href="mailto:lidi@perabytes.com">lidi@perabytes.com</a> wrote:</p>
<blockquote style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
<div>I tracked this problem, and found that the loc.parent and loc.pargfid are all null in the call <span>sequences below</span>:<br /><br />ec_manager_writev() -> ec_get_size_version() -> ec_lookup(). This can cause server_resolve() return an EINVAL.<br /><br />A replace-brick will cause all opened fd and inode table recreate, but ec_lookup() get the loc from fd->_ctx. <br /><br />So loc.parent and loc.pargfid are missing while fd changed. Other xlators always do a lookup from root <br /><br />directory, so never cause this problem. It seems that a recursive lookup from root directory may address this <br /><br />issue.</div>
</blockquote>
<div>
<pre>EINVAL error is returned by protocol/server when it tries to resolve an inode based on a loc. If loc's 'name' field is not NULL nor empty, it tries to resolve the inode based on /. The problem here is that pargfid is 00...00.<br /><br />To solve this issue I've modified ec_loc_setup_parent() so that it clears loc's 'name' if parent inode cannot be determined. This forces protocol/server to resolve the inode based on , which is valid and can be resolved successfully.<br /><br />However this doesn't fully solve the bug. After solving this issue, I get an EIO error. Further investigations seems to indicate that this is caused by a locking problem caused by an incorrect management of ESTALE when the brick is replaced.</pre>
</div>
</div>
</blockquote>
<div>ESTALE indicates either any of the following situations:<br /><br /></div>
<div>1. In the case of named-lookup (loc containing /), is not present. Which means parent is not present on the brick</div>
<div>2. In the case of nameless lookup (loc containing only of the file), file/directory represented by gfid is not present on brick.<br /><br /></div>
<div>Which among the above two scenarios is your case?</div>
</div>
</div>
</div>
</blockquote>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>In this particular case, the problem is with the second scenario, however there are other combinations that could lead to the first one. Basically the root cause is that after replacing a brick, the new brick is totally empty, so self heal needs to recover directory contents, but some running operations may try to use gfid already resolved that the new brick has never seen. In these cases the brick returns ESTALE, but ec incorrectly handled this as a fatal error while trying to acquire a lock, returning EIO for the full operation.</div>
</div>
<div>I'll upload a patch to solve this problem.</div>
<div>Xavi</div>
</div>
</div>
</body></html>