<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Dec 31, 2014 at 11:25 PM, Xavier Hernandez <span dir="ltr"><<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div><span class="">
<p>On 27.12.2014 13:43, <a href="mailto:lidi@perabytes.com" target="_blank">lidi@perabytes.com</a> wrote:</p>
<blockquote type="cite" style="padding-left:5px;border-left:#1010ff 2px solid;margin-left:5px;width:100%">
<p> </p>
<div>I tracked this problem, and found that the loc.parent and loc.pargfid are all null in the call <span>sequences below</span>:<br><br>ec_manager_writev() -> ec_get_size_version() -> ec_lookup(). This can cause server_resolve() return an EINVAL.<br><br>A replace-brick will cause all opened fd and inode table recreate, but ec_lookup() get the loc from fd->_ctx. <br><br>So loc.parent and loc.pargfid are missing while fd changed. Other xlators always do a lookup from root <br><br>directory, so never cause this problem. It seems that a recursive lookup from root directory may address this <br><br>issue.</div>
</blockquote>
</span><div>
<pre>EINVAL error is returned by protocol/server when it tries to resolve an inode based on a loc. If loc's 'name' field is not NULL nor empty, it tries to resolve the inode based on <pargfid>/<name>. The problem here is that pargfid is 00...00.<br><br>To solve this issue I've modified ec_loc_setup_parent() so that it clears loc's 'name' if parent inode cannot be determined. This forces protocol/server to resolve the inode based on <gfid>, which is valid and can be resolved successfully.<br><br>However this doesn't fully solve the bug. After solving this issue, I get an EIO error. Further investigations seems to indicate that this is caused by a locking problem caused by an incorrect management of ESTALE when the brick is replaced.</pre></div></div></blockquote><div><br></div><div>ESTALE indicates either any of the following situations:<br><br></div><div>1. In the case of named-lookup (loc containing <pargfid>/<name>), <pargfid> is not present. Which means parent is not present on the brick<br></div><div>2. In the case of nameless lookup (loc containing only <gfid> of the file), file/directory represented by gfid is not present on brick.<br><br></div><div>Which among the above two scenarios is your case?<br><br></div><div>> <br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><pre> I'll upload a patch shortly to solve these issues.</pre>
</div>
<p>Xavi</p>
<blockquote type="cite" style="padding-left:5px;border-left:#1010ff 2px solid;margin-left:5px;width:100%">
<div><span class=""><br><br>----- 原邮件信息 -----<br><b>发件人：</b>Raghavendra Gowdappa <br></span><span class=""><b>发送时间：</b>14-12-24 21:48:56<br><b>收件人：</b>Xavier Hernandez <br></span><b>抄送人：</b>Gluster Devel <span class=""><br><b>主题：</b>Re: [Gluster-devel] Problems with graph switch in disperse<br><br>Do you know the origins of EIO? fuse-bridge only fails a lookup fop with EIO (when NULL gfid is received in a successful lookup reply). So, there might be other xlator which is sending EIO.<br><br> ----- Original Message -----<br></span> > From: "Xavier Hernandez" <br> > To: "Gluster Devel" <div><div class="h5"><br> > Sent: Wednesday, December 24, 2014 6:25:17 PM<br> > Subject: [Gluster-devel] Problems with graph switch in disperse<br> > <br> > Hi,<br> > <br> > I'm experiencing a problem when gluster graph is changed as a result of<br> > a replace-brick operation (probably with any other operation that<br> > changes the graph) while the client is also doing other tasks, like<br> > writing a file.<br> > <br> > When operation starts, I see that the replaced brick is disconnected,<br> > but writes continue working normally with one brick less.<br> > <br> > At some point, another graph is created and comes online. Remaining<br> > bricks on the old graph are disconnected and the old graph is destroyed.<br> > I see how new write requests are sent to the new graph.<br> > <br> > This seems correct. However there's a point where I see this:<br> > <br> > [2014-12-24 11:29:58.541130] T [fuse-bridge.c:2305:fuse_write_resume]<br> > 0-glusterfs-fuse: 2234: WRITE (0x16dcf3c, size=131072, offset=255721472)<br> > [2014-12-24 11:29:58.541156] T [ec-helpers.c:101:ec_trace] 2-ec:<br> > WIND(INODELK) 0x7f8921b7a9a4(0x7f8921b78e14) [refs=5, winds=3, jobs=1]<br> > frame=0x7f8932e92c38/0x7f8932e9e6b0, min/exp=3/3, err=0 state=1<br> > {111:000:000} idx=0<br> > [2014-12-24 11:29:58.541292] T [rpc-clnt.c:1384:rpc_clnt_record]<br> > 2-patchy-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner:<br> > d025e932897f0000<br> > [2014-12-24 11:29:58.541296] T [io-cache.c:133:ioc_inode_flush]<br> > 2-patchy-io-cache: locked inode(0x16d2810)<br> > [2014-12-24 11:29:58.541354] T<br> > [rpc-clnt.c:1241:rpc_clnt_record_build_header] 2-rpc-clnt: Request<br> > fraglen 152, payload: 84, rpc hdr: 68<br> > [2014-12-24 11:29:58.541408] T [io-cache.c:137:ioc_inode_flush]<br> > 2-patchy-io-cache: unlocked inode(0x16d2810)<br> > [2014-12-24 11:29:58.541493] T [io-cache.c:133:ioc_inode_flush]<br> > 2-patchy-io-cache: locked inode(0x16d2810)<br> > [2014-12-24 11:29:58.541536] T [io-cache.c:137:ioc_inode_flush]<br> > 2-patchy-io-cache: unlocked inode(0x16d2810)<br> > [2014-12-24 11:29:58.541537] T [rpc-clnt.c:1577:rpc_clnt_submit]<br> > 2-rpc-clnt: submitted request (XID: 0x17 Program: GlusterFS 3.3,<br> > ProgVers: 330, Proc: 29) to rpc-transport (patchy-client-0)<br> > [2014-12-24 11:29:58.541646] W [fuse-bridge.c:2271:fuse_writev_cbk]<br> > 0-glusterfs-fuse: 2234: WRITE => -1 (Input/output error)<br> > <br> > It seems that fuse still has a write request pending for graph 0. It is<br> > resumed but it returns EIO without calling the xlator stack (operations<br> > seen between the two log messages are from other operations and they are<br> > sent to graph 2). I'm not sure why this happens and how I should aviod this.<br> > <br> > I tried the same scenario with replicate and it seems to work, so there<br> > must be something wrong in disperse, but I don't see where the problem<br> > could be.<br> > <br> > Any ideas ?<br> > <br> > Thanks,<br> > <br> > Xavi<br> > _______________________________________________<br> > Gluster-devel mailing list<br> > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br> > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br> > <br> _______________________________________________<br> Gluster-devel mailing list<br> <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br><a href="http://www.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a></div></div></div>
<div>
<p> </p>
</div>
</blockquote>
</div>
<br>_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature">Raghavendra G<br></div>
</div></div>