<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi All <br>
I have suggestion to improve regression test. <br>
Issue : We are calling cleanup function at the starting and ending
of each test (twice for each test ). <br>
#time prove -vf tests/bugs/glusterd/cleanup.t #(cleanup
function called 20 times in loop ) -- find attached file <br>
<b>real 0m29.140s</b><br>
time taken cleanup function = 30/20 = ~1.5 sec<br>
#grep -rn "cleanup" ./tests/ |wc -l<br>
808<br>
808 times cleanup function called during regression test <br>
Time taken by cleanup in regression test =1.5 * 808 =1212 sec =
20.2 min<br>
Total time taken by both regression =20.2 * 2 = 40.4 min <br>
<br>
My suggestion : Call cleanup only at the start of each test case ,
so it reduce the 40.4/2=<b><u> 20.2 min</u></b> in total regression
test(both regression ).<br>
<br>
Note: These analysis done in below configuration<br>
model name : Intel(R) Core(TM) i5-4300U CPU @ 1.90GHz<br>
RAM : 8GB <br>
<br>
<div class="moz-cite-prefix">On 06/17/2015 05:18 PM, Atin Mukherjee
wrote:<br>
</div>
<blockquote cite="mid:55815E91.8050609@redhat.com" type="cite">
<pre wrap="">
On 06/17/2015 04:26 PM, Raghavendra Talur wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi,
MSV Bhat and I had presented in Gluster Design Summit some ideas about
improving our testing infrastructure.
Here is the link to the slides: <a class="moz-txt-link-freetext" href="http://redhat.slides.com/rtalur/distaf#">http://redhat.slides.com/rtalur/distaf#</a>
Here are the same suggestions,
1. *A .t file for a bug*
When a community user discovers a bug in Gluster, they contact us over
irc or email and eventually end up filling a bug in bugzilla.
Many times it so happens that we find a bug which we don't know the
fix for OR not a bug in our module and also end up filling a bug in
bugzilla.
If we could rather write a .t test to reproduce the bug and add it to
say /tests/bug/yet-to-be-fixed/ folder in gluster repo it would be
more helpful. As part of bug-triage we could try doing the same for bugs
filed by community users.
*What do we get?*
a. very easy for a new developer to pick up that bug and fix it.
If .t passes then the bug is fixed.
b. The regression on daily patch sets would skip this folder; but on a
nightly basis we could run a test on this folder to see if any of these
tests got fixed while we were fixing some other tests. Yay!
</pre>
</blockquote>
<pre wrap="">Attaching a reproducer in the form of .t might be difficult, specially
for the race conditions. It might pass pre and post fix as well. So it
*should not* be a must criteria to have .t file.
</pre>
<blockquote type="cite">
<pre wrap="">
2. *New gerrit/review work flow*
Our gerrit setup currently has a 2 hour average for regression run.
Due to long queue of commits the round about time is around 4-6 hours.
Kaushal has proposed on how to reduce round about time more in this
thread <a class="moz-txt-link-freetext" href="http://www.spinics.net/lists/gluster-devel/msg15798.html">http://www.spinics.net/lists/gluster-devel/msg15798.html</a>.
3. *Make sure tests can be done in docker and run in parallel*
To reduce time for one test run from 2 hours we can look at running
tests in parallel. I did a prototype and got test time down to 40 mins
on a 16 GB RAM and 4 core VM.
Current blocked at :
Some of the tests fail in docker while they pass in a VM.
Note that it is .t failing, Gluster works fine in docker.
Need some help on this. More on this in a mail I will be sending later
today at gluster-devel.
*what do we get?*
Running 4 docker containers on our Laptops itself can reduce time
taken by test runs down to 90 mins. Running them on powerful machines,
it is down to 40 mins as seen in the prototype.
</pre>
</blockquote>
<pre wrap="">How about NetBSD, yesterday Niels point out to me that there is no
docker service for NetBSD.
</pre>
<blockquote type="cite">
<pre wrap="">
4. *Test definitions for every .t*
May be the time has come to upgrade our test infra to have tests with
test definitions. Every .t file could have a corresponding .def file
which is
A JSON/YAML/XML config
Defines the requirements of test
Type of volume
Special knowledge of brick size required?
Which repo source folders should trigger this test
Running time
Test RUN level
*what do we get?*
a. Run a partial set of tests on a commit based on git log and test
definitions and run complete regression as nightly.
b. Order test run based on run times. This combined with fail on first
test setting we have, we will fail as early as possible.
c. Order tests based on functionality level, which means a mount.t basic
test should run before a complex DHT test that makes use of FUSE mount.
Again, this will help us to fail as early as possible in failure scenarios.
d. With knowledge of type of volume required and number of bricks
required, we can re-use volumes that are created for subsequent tests.
Even the cleanup() function we have takes time. DiSTAF already has a
function equivalent to use_existing_else_create_new.
5. *Testing GFAPI*
We don't have a good test framework for gfapi as of today.
However, with the recent design proposal at
<a class="moz-txt-link-freetext" href="https://docs.google.com/document/d/1yuRLRbdccx_0V0UDAxqWbz4g983q5inuINHgM1YO040/edit?usp=sharing">https://docs.google.com/document/d/1yuRLRbdccx_0V0UDAxqWbz4g983q5inuINHgM1YO040/edit?usp=sharing</a>
and
Craig Cabrey from Facebook developing a set of coreutils using
GFAPI as mentioned here
<a class="moz-txt-link-freetext" href="http://www.spinics.net/lists/gluster-devel/msg15753.html">http://www.spinics.net/lists/gluster-devel/msg15753.html</a>
I guess we have it well covered :)
Reviews and suggestions welcome!
Thanks,
Raghavendra Talur
_______________________________________________
Gluster-devel mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-devel">http://www.gluster.org/mailman/listinfo/gluster-devel</a>
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<br>
</body>
</html>