<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On 22 November 2016 at 13:09, Raghavendra Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail-HOEnZb"><div class="gmail-h5"><br>
<br>
----- Original Message -----<br>
> From: "Vijay Bellur" <<a href="mailto:vbellur@redhat.com">vbellur@redhat.com</a>><br>
> To: "Nithya Balachandran" <<a href="mailto:nbalacha@redhat.com">nbalacha@redhat.com</a>><br>
> Cc: "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a>><br>
> Sent: Wednesday, November 16, 2016 9:41:12 AM<br>
> Subject: Re: [Gluster-devel] Upstream smoke test failures<br>
><br>
> On Tue, Nov 15, 2016 at 8:40 AM, Nithya Balachandran<br>
> <<a href="mailto:nbalacha@redhat.com">nbalacha@redhat.com</a>> wrote:<br>
> ><br>
> ><br>
> > On 15 November 2016 at 18:55, Vijay Bellur <<a href="mailto:vbellur@redhat.com">vbellur@redhat.com</a>> wrote:<br>
> >><br>
> >> On Mon, Nov 14, 2016 at 10:34 PM, Nithya Balachandran<br>
> >> <<a href="mailto:nbalacha@redhat.com">nbalacha@redhat.com</a>> wrote:<br>
> >> ><br>
> >> ><br>
> >> > On 14 November 2016 at 21:38, Vijay Bellur <<a href="mailto:vbellur@redhat.com">vbellur@redhat.com</a>> wrote:<br>
> >> >><br>
> >> >> I would prefer that we disable dbench only if we have an owner for<br>
> >> >> fixing the problem and re-enabling it as part of smoke tests. Running<br>
> >> >> dbench seamlessly on gluster has worked for a long while and if it is<br>
> >> >> failing today, we need to address this regression asap.<br>
> >> >><br>
> >> >> Does anybody have more context or clues on why dbench is failing now?<br>
> >> >><br>
> >> > While I agree that it needs to be looked at asap, leaving it in until we<br>
> >> > get<br>
> >> > an owner seems rather pointless as all it does is hold up various<br>
> >> > patches<br>
> >> > and waste machine time. Re-triggering it multiple times so that it<br>
> >> > eventually passes does not add anything to the regression test processes<br>
> >> > or<br>
> >> > validate the patch as we know there is a problem.<br>
> >> ><br>
> >> > I would vote for removing it and assigning someone to look at it<br>
> >> > immediately.<br>
> >> ><br>
> >><br>
> >> From the debugging done so far can we identify an owner to whom this<br>
> >> can be assigned? I looked around for related discussions and could<br>
> >> figure out that we are looking to get statedumps. Do we have more<br>
> >> information/context beyond this?<br>
> >><br>
> > I have updated the BZ (<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1379228" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/<wbr>show_bug.cgi?id=1379228</a>)<br>
> > with info from the last failure - looks like hangs in write-behind and<br>
> > read-ahead.<br>
> ><br>
><br>
><br>
> I spent some time on this today and it does look like write-behind is<br>
> absorbing READs without performing any WIND/UNWIND actions. I have<br>
> attached a statedump from a slave that had the dbench problem (thanks,<br>
> Nigel!) to the above bug.<br>
><br>
> Snip from statedump:<br>
><br>
> [global.callpool.stack.2]<br>
> stack=0x7fd970002cdc<br>
> uid=0<br>
> gid=0<br>
> pid=31884<br>
> unique=37870<br>
> lk-owner=0000000000000000<br>
> op=READ<br>
> type=1<br>
> cnt=2<br>
><br>
> [global.callpool.stack.2.<wbr>frame.1]<br>
> frame=0x7fd9700036ac<br>
> ref_count=0<br>
> translator=patchy-read-ahead<br>
> complete=0<br>
> parent=patchy-readdir-ahead<br>
> wind_from=ra_page_fault<br>
> wind_to=FIRST_CHILD (fault_frame->this)->fops-><wbr>readv<br>
> unwind_to=ra_fault_cbk<br>
><br>
> [global.callpool.stack.2.<wbr>frame.2]<br>
> frame=0x7fd97000346c<br>
> ref_count=1<br>
> translator=patchy-readdir-<wbr>ahead<br>
> complete=0<br>
><br>
><br>
> Note that the frame which was wound from ra_page_fault() to<br>
> write-behind is not yet complete and write-behind has not progressed<br>
> the call. There are several callstacks with a similar signature in<br>
> statedump.<br>
<br>
</div></div>I think the culprit here is read-ahead, not write-behind. If read fop was dropped in write-behind, we should've seen a frame associated with write-behind (complete=0 for a frame associated with a xlator indicates frame was not unwound from _that_ xlator). But I didn't see any. Also empty request queues in wb_inode corroborate the hypothesis. K</blockquote><div><br></div><div> </div><div>We have seen both . See comment#17 in <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1379228" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/<wbr>show_bug.cgi?id=1379228</a> .</div><div><br></div><div><br></div><div>regards,</div><div>Nithya</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">arthick subrahmanya is working on a similar issue reported by a user. However, we've not made much of a progress till now.<br>
<span class="gmail-im gmail-HOEnZb"><br>
><br>
> In write-behind's readv implementation, we stub READ fops and enqueue<br>
> them in the relevant inode context. Once enqueued the stub resumes<br>
> when appropriate set of conditions happen in write-behind. This is not<br>
> happening now and I am not certain if:<br>
><br>
> - READ fops are languishing in a queue and not being resumed or<br>
> - READ fops are pre-maturely dropped from a queue without winding or<br>
> unwinding<br>
><br>
> When I gdb'd into the client process and examined the inode contexts<br>
> for write-behind, I found all queues to be empty. This seems to<br>
> indicate that the latter reason is more plausible but I have not yet<br>
> found a code path to account for this possibility.<br>
><br>
> One approach to proceed further is to add more logs in write-behind to<br>
> get a better understanding of the problem. I will try that out<br>
> sometime later this week. We are also considering disabling<br>
> write-behind for smoke tests in the interim after a trial run (with<br>
> write-behind disabled) later in the day.<br>
><br>
> Thanks,<br>
> Vijay<br>
</span><div class="gmail-HOEnZb"><div class="gmail-h5">> ______________________________<wbr>_________________<br>
> Gluster-devel mailing list<br>
> <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br>
><br>
</div></div></blockquote></div><br></div></div>