<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 19, 2016 at 5:21 PM, Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><br>
<br>
On 01/19/2016 10:45 AM, Emmanuel Dreyfus wrote:<br>
> Hi<br>
><br>
> Spurious regression failures make developers frustrated. One submits a<br>
> change and gets completely unrelated failures. The only way out is to<br>
> retrigger regression until it passes, a boring and time-wasting task.<br>
> Sometimes after 4 or 5 failed runs, the submitter realize there is a<br>
> real issue and look at it, which is a waste of time and resources.<br>
><br>
> The fact that we run regression on multiple platforms makes the<br>
> situation worse. If you have 10% of chances to hit a spurious failure on<br>
> Linux and a 20% chances to hit a spurious failure on NetBSD (random<br>
> number chosen), that means you get roughtly a failure for four<br>
> submissions (random prediction, as I used random input numbers, but you<br>
> get the idea)<br>
><br>
> Two solutions are proposed:<br>
><br>
> 1) do not run unreliable tests, as proposed by Raghavendra Talur:<br>
> <a href="http://review.gluster.org/13173" rel="noreferrer" target="_blank">http://review.gluster.org/13173</a><br>
><br>
> I have nothing against the idea, but I voted down the change because it<br>
> fails to address the need for different test blacklists on different<br>
> platforms: we do not have the same unreliable tests on Linux and NetBSD.<br></div></div></blockquote><div><br></div><div>Why I prefer having this solution:</div><div>a. Allowing re-running to tests to make them pass leads to complacency with how tests are written.</div><div>b. A test is bad if it is not deterministic and running a bad test has *no* value. We are wasting time even if the test runs for a few seconds.</div><div>c. I propose another method to overcome the technical difficulty of having blacklists for different platforms. We could have "[K[a-z]*-]*" as prefix of tests where [a-z]* could be L or N signify that the test is bad on Linux and NetBSD respectively. The run-tests.sh script can be made intelligent enough to determine host OS and skip them.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">
><br>
> 2) add a regression option to retry a failed test once, and to validate<br>
> the regression if second attempt passes, as I proposed:<br>
> <a href="http://review.gluster.org/13245" rel="noreferrer" target="_blank">http://review.gluster.org/13245</a><br>
><br>
> The idea is basicaly to automatically do what every submitter has been<br>
> doing: retry without a thought when regression fails. The benefit of<br>
> this approach is also that it gives us a better view of what test failed<br>
> because of the change, and what test failed because it was unreliable.<br>
><br>
> The retry feature is optionnal and triggered by using the -r flag to<br>
> run-tests.sh. I intend to use it on NetBSD regression to reduce the<br>
> number of failures that annoy people. It could be used on Linux<br>
> regression too, though I do not plan to touch that on my own.<br>
</div></div>+1 to option 2<br>
<span class="">><br>
> Please people tell us what approach you prefer.<br>
><br>
</span>_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
</blockquote></div><br></div></div>