<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 4, 2015 at 12:56 PM, Raghavendra Talur <span dir="ltr"><<a href="mailto:rtalur@redhat.com" target="_blank">rtalur@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
Maintainers - can you please take stock of this and ensure sanity of your components before merging patches that do not fix a failing test?<br><br></blockquote><div><br></div></span><div>Here is my proposal to get this fixed.</div><div><br></div><div><br></div><div>This weekend, 5th September 0400 UTC, I will start a jenkins run on master and 3.7 branches.</div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div class="gmail_extra"><div class="gmail_quote"><ul><li>It will be re-based with code just before it is run, so all patches merged by 4th September would be tested.</li><li>It will run each test for 10 times in succession. Why 10?</li><ul><li>Hope to find tests that fail occasionally.</li><li>If the tests fails only for 1st run, it could very well be a cleanup issue with last run test.</li><li>Failures within the 10 runs in a pattern is again indicative of some cleanup/timeout error.</li></ul><li>It will run all tests and not stop at the first failure.</li><li>I will have scripts modified to get maximum data from logs. (It will still be INFO level logs)</li></ul></div></div></blockquote>After the test completes, I will file a bug against the component of the .t tests that fail in this run and immediately add the test to bad tests list.<div><br></div><div>What should the maintainers do after that?</div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><ul><li>If a bug is filed against your component, please spend some time on Monday and root cause the issue by Monday EOD.</li><li>If the root cause proves that the bug is in .t file</li><ul><li>It is would be mostly because</li><ul><li>The timeouts are not enough all the time. Change EXPECT_WITHIN values and check.</li><li>The test is not deterministic enough ; some of the assumptions that test makes might not always be true. For example, a SIGTERM followed by a TEST which assumes that process is definitely killed is a wrong assumption. Use SIGKILL in such cases. (I know SIGKILL may not work too if the process is in D state, but its a good enough example)</li></ul><li>It is easier to fix bugs in.t once the root cause is found. Please fix the issue and remove it from bad tests list. Use the bug filed against this .t file.</li></ul><li>If the root cause proves that the bug is in Gluster code:</li><ul><li>If the bug is in same component as the .t file:</li><ul><li>In this case, you are the component owner, change the description and summary of the bug filed to indicate the actual issue.<br></li><li>If the time required to fix the issue in Gluster code is non-minimal</li><ul><li>Put a workaround in .t file with a comment clearly stating the bug number which would later fix it and remove the test from bad test list.</li><li>If a workaround is not possible let the test remain in bad test list.</li></ul></ul><li>If the bug is not in same component as the .t file:</li><ul><li>Update the bug with details which prove that bug is not in the same component and change the component accordingly.</li><li>It is new owner's responsibility to provide a workaround for all .t files hit by the issue and fix the code.</li></ul></ul></ul></blockquote>Note to all maintainers:</div><div><ul><li>I would request everyone to resist merging patches this weekend unless critically required. It would help us in debugging on Monday.</li></ul></div></div></blockquote><div><br></div><div> I did try this over the weekend. Refer to the patch at <a href="http://review.gluster.org/#/c/12109/">http://review.gluster.org/#/c/12109/</a>.</div><div><br></div><div>However, I discovered that tests failed continuously after certain tests failed in a run thereby</div><div>indicating that our cleanup function is not sufficient/complete.</div><div><br></div><div>I will be working on fixing few functions in run-tests.sh and include.rc before coming back to this next weekend.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div><br></div><div>Lets hope that when we do a similar jenkins run on next weekend, September 12th, we don't find any failures.</div><div><div><div><br></div></div></div></div><div>Suggestions welcome for any changes in the above plan.</div><div><br></div><div>Thanks,</div><div>Raghavendra Talur</div></div>
</blockquote></div><br></div></div>