<html><body><div style="font-family: times new roman,new york,times,serif; font-size: 12pt; color: #000000"><div>Hi,<br></div><div><br></div><div>Have posted a fix for hang in read : <a href="http://review.gluster.org/15901" data-mce-href="http://review.gluster.org/15901">http://review.gluster.org/15901</a><br></div><div>I think, it will fix the issue reported here. Please check the commit message of the patch<br></div><div>for more details.<br></div><div><br></div><div>Regards,<br></div><div>Poornima<br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Nithya Balachandran" <nbalacha@redhat.com><br><b>To: </b>"Raghavendra Gowdappa" <rgowdapp@redhat.com><br><b>Cc: </b>"Gluster Devel" <gluster-devel@gluster.org><br><b>Sent: </b>Tuesday, November 22, 2016 3:23:59 AM<br><b>Subject: </b>Re: [Gluster-devel] Upstream smoke test failures<br><div><br></div><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On 22 November 2016 at 13:09, Raghavendra Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank" data-mce-href="mailto:rgowdapp@redhat.com">rgowdapp@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;"><div class="gmail-HOEnZb"><div class="gmail-h5"><br> <br> ----- Original Message -----<br> > From: "Vijay Bellur" <<a href="mailto:vbellur@redhat.com" target="_blank" data-mce-href="mailto:vbellur@redhat.com">vbellur@redhat.com</a>><br> > To: "Nithya Balachandran" <<a href="mailto:nbalacha@redhat.com" target="_blank" data-mce-href="mailto:nbalacha@redhat.com">nbalacha@redhat.com</a>><br> > Cc: "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org" target="_blank" data-mce-href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a>><br> > Sent: Wednesday, November 16, 2016 9:41:12 AM<br> > Subject: Re: [Gluster-devel] Upstream smoke test failures<br> ><br> > On Tue, Nov 15, 2016 at 8:40 AM, Nithya Balachandran<br> > <<a href="mailto:nbalacha@redhat.com" target="_blank" data-mce-href="mailto:nbalacha@redhat.com">nbalacha@redhat.com</a>> wrote:<br> > ><br> > ><br> > > On 15 November 2016 at 18:55, Vijay Bellur <<a href="mailto:vbellur@redhat.com" target="_blank" data-mce-href="mailto:vbellur@redhat.com">vbellur@redhat.com</a>> wrote:<br> > >><br> > >> On Mon, Nov 14, 2016 at 10:34 PM, Nithya Balachandran<br> > >> <<a href="mailto:nbalacha@redhat.com" target="_blank" data-mce-href="mailto:nbalacha@redhat.com">nbalacha@redhat.com</a>> wrote:<br> > >> ><br> > >> ><br> > >> > On 14 November 2016 at 21:38, Vijay Bellur <<a href="mailto:vbellur@redhat.com" target="_blank" data-mce-href="mailto:vbellur@redhat.com">vbellur@redhat.com</a>> wrote:<br> > >> >><br> > >> >> I would prefer that we disable dbench only if we have an owner for<br> > >> >> fixing the problem and re-enabling it as part of smoke tests. Running<br> > >> >> dbench seamlessly on gluster has worked for a long while and if it is<br> > >> >> failing today, we need to address this regression asap.<br> > >> >><br> > >> >> Does anybody have more context or clues on why dbench is failing now?<br> > >> >><br> > >> > While I agree that it needs to be looked at asap, leaving it in until we<br> > >> > get<br> > >> > an owner seems rather pointless as all it does is hold up various<br> > >> > patches<br> > >> > and waste machine time. Re-triggering it multiple times so that it<br> > >> > eventually passes does not add anything to the regression test processes<br> > >> > or<br> > >> > validate the patch as we know there is a problem.<br> > >> ><br> > >> > I would vote for removing it and assigning someone to look at it<br> > >> > immediately.<br> > >> ><br> > >><br> > >> From the debugging done so far can we identify an owner to whom this<br> > >> can be assigned? I looked around for related discussions and could<br> > >> figure out that we are looking to get statedumps. Do we have more<br> > >> information/context beyond this?<br> > >><br> > > I have updated the BZ (<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1379228" rel="noreferrer" target="_blank" data-mce-href="https://bugzilla.redhat.com/show_bug.cgi?id=1379228">https://bugzilla.redhat.com/show_bug.cgi?id=1379228</a>)<br> > > with info from the last failure - looks like hangs in write-behind and<br> > > read-ahead.<br> > ><br> ><br> ><br> > I spent some time on this today and it does look like write-behind is<br> > absorbing READs without performing any WIND/UNWIND actions. I have<br> > attached a statedump from a slave that had the dbench problem (thanks,<br> > Nigel!) to the above bug.<br> ><br> > Snip from statedump:<br> ><br> > [global.callpool.stack.2]<br> > stack=0x7fd970002cdc<br> > uid=0<br> > gid=0<br> > pid=31884<br> > unique=37870<br> > lk-owner=0000000000000000<br> > op=READ<br> > type=1<br> > cnt=2<br> ><br> > [global.callpool.stack.2.frame.1]<br> > frame=0x7fd9700036ac<br> > ref_count=0<br> > translator=patchy-read-ahead<br> > complete=0<br> > parent=patchy-readdir-ahead<br> > wind_from=ra_page_fault<br> > wind_to=FIRST_CHILD (fault_frame->this)->fops->readv<br> > unwind_to=ra_fault_cbk<br> ><br> > [global.callpool.stack.2.frame.2]<br> > frame=0x7fd97000346c<br> > ref_count=1<br> > translator=patchy-readdir-ahead<br> > complete=0<br> ><br> ><br> > Note that the frame which was wound from ra_page_fault() to<br> > write-behind is not yet complete and write-behind has not progressed<br> > the call. There are several callstacks with a similar signature in<br> > statedump.<br> <br></div></div>I think the culprit here is read-ahead, not write-behind. If read fop was dropped in write-behind, we should've seen a frame associated with write-behind (complete=0 for a frame associated with a xlator indicates frame was not unwound from _that_ xlator). But I didn't see any. Also empty request queues in wb_inode corroborate the hypothesis. K</blockquote><div><br></div><div> </div><div>We have seen both . See comment#17 in <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1379228" rel="noreferrer" target="_blank" data-mce-href="https://bugzilla.redhat.com/show_bug.cgi?id=1379228">https://bugzilla.redhat.com/show_bug.cgi?id=1379228</a> .</div><div><br></div><div><br></div><div>regards,</div><div>Nithya</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;">arthick subrahmanya is working on a similar issue reported by a user. However, we've not made much of a progress till now.<br> <span class="gmail-im gmail-HOEnZb"><br> ><br> > In write-behind's readv implementation, we stub READ fops and enqueue<br> > them in the relevant inode context. Once enqueued the stub resumes<br> > when appropriate set of conditions happen in write-behind. This is not<br> > happening now and I am not certain if:<br> ><br> > - READ fops are languishing in a queue and not being resumed or<br> > - READ fops are pre-maturely dropped from a queue without winding or<br> > unwinding<br> ><br> > When I gdb'd into the client process and examined the inode contexts<br> > for write-behind, I found all queues to be empty. This seems to<br> > indicate that the latter reason is more plausible but I have not yet<br> > found a code path to account for this possibility.<br> ><br> > One approach to proceed further is to add more logs in write-behind to<br> > get a better understanding of the problem. I will try that out<br> > sometime later this week. We are also considering disabling<br> > write-behind for smoke tests in the interim after a trial run (with<br> > write-behind disabled) later in the day.<br> ><br> > Thanks,<br> > Vijay<br> </span><div class="gmail-HOEnZb"><div class="gmail-h5">> _______________________________________________<br> > Gluster-devel mailing list<br> > <a href="mailto:Gluster-devel@gluster.org" target="_blank" data-mce-href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br> > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank" data-mce-href="http://www.gluster.org/mailman/listinfo/gluster-devel">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br> ><br></div></div></blockquote></div><br></div></div><br>_______________________________________________<br>Gluster-devel mailing list<br>Gluster-devel@gluster.org<br>http://www.gluster.org/mailman/listinfo/gluster-devel</blockquote><div><br></div></div></body></html>