[Gluster-infra] Progress report for regression tests in Rackspace

[Justin Clift]

Hi us,

Was trying out the GlusterFS regression tests in Rackspace VMs last
night for each of the release-3.4, release-3.5, and master branches.

The regression test is just a run of "run-tests.sh", from a git
checkout of the appropriate branch.

The good news is we're adding a lot of testing code with each release:

 * release-3.4 -  6303 lines  (~30 mins to run test)
 * release-3.5 -  9776 lines  (~85 mins to run test)
 * master      - 11660 lines  (~90 mins to run test)

(lines counted using:
 $ find tests -type f -iname "*.t" -exec cat {} >> a \;; wc -l a; rm -f a)

The bad news is the tests only "kind of" pass now.  I say kind of because
although the regression run *can* pass for each of these branch's, it's
inconsistent. :(

Results from testing overnight:

 * release-3.4 - 20 runs - 17 PASS, 3 FAIL. 85% success.
   * bug-857330/normal.t failed in one run
   * bug-887098-gmount-crash.t failed in one run
   * bug-857330/normal.t failed in one run

 * release-3.5 - 20 runs, 18 PASS, 2 FAIL. 90% success.
   * bug-857330/xml.t failed in one run
   * bug-1004744.t failed in another run (same vm for both failures)

 * master - 20 runs, 6 PASS, 14 FAIL. 30% success.
   * bug-1070734.t failed in one run
   * bug-1087198.t & bug-860663.t failed in one run (same vm as bug-1070734.t failure above)
   * bug-1087198.t & bug-857330/normal.t failed in one run (new vm, a subsequent run on same vm passed)
   * bug-1087198.t & bug-948686.t failed in one run (new vm)
   * bug-1070734.t & bug-1087198.t failed in one run (new vm) 
   * bug-860663.t failed in one run
   * bug-1023974.t & bug-1087198.t & bug-948686.t failed in one run (new vm)
   * bug-1004744.t & bug-1023974.t & bug-1087198.t & bug-948686.t failed in one run (new vm)
   * bug-948686.t failed in one run (new vm)
   * bug-1070734.t failed in one run (new vm)
   * bug-1023974.t failed in one run (new vm)
   * bug-1087198.t & bug-948686.t failed in one run (new vm)
   * bug-1070734.t failed in one run (new vm)
   * bug-1087198.t failed in one run (new vm)

The occasional failing tests aren't completely random, suggesting
something is going on.  Possible race conditions maybe? (no idea).

 * 8 failures - bug-1087198.t
 * 5 failures - bug-948686.t
 * 4 failures - bug-1070734.t
 * 3 failures - bug-1023974.t
 * 3 failures - bug-857330/normal.t
 * 2 failures - bug-860663.t
 * 2 failures - bug-1004744.t
 * 1 failures - bug-857330/xml.t
 * 1 failures - bug-887098-gmount-crash.t

Anyone have suggestions on how to make this work reliably?

Regards and best wishes,

Justin Clift

--
Open Source and Standards @ Red Hat

twitter.com/realjustinclift