This fails only in slave28, in other places the test goes <span></span>through. I will run this test on a fresh node to see if any installation dependency is causing the failure or not.<br><br>On Thursday 1 September 2016, Atin Mukherjee <<a href="mailto:amukherj@redhat.com">amukherj@redhat.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>This is strange, Ashish has already fixed the issue of the missing SSL certs through fad93c1<br><br></div>From the logs it definitely looks like glusterd failed to come up:<br><br>[2016-09-01 13:30:28.909555] E [socket.c:4122:socket_init] 0-socket.management: could not load our cert<br><br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 1, 2016 at 7:38 PM, Vijay Bellur <span dir="ltr"><<a href="javascript:_e(%7B%7D,'cvml','vbellur@redhat.com');" target="_blank">vbellur@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">tests/bugs/cli/bug-1320388.t is failing quite frequently in<br>
regression-test burn in. Can you please take a look in?<br>
<br>
Thx,<br>
Vijay<br>
<div><div><br>
On Thu, Sep 1, 2016 at 9:33 AM, <<a href="javascript:_e(%7B%7D,'cvml','jenkins@build.gluster.org');" target="_blank">jenkins@build.gluster.org</a>> wrote:<br>
> See <<a href="http://build.gluster.org/job/regression-test-burn-in/1623/" rel="noreferrer" target="_blank">http://build.gluster.org/job/<wbr>regression-test-burn-in/1623/</a>><br>
><br>
> ------------------------------<wbr>------------<br>
> [...truncated 10509 lines...]<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> touch: cannot touch `/mnt/glusterfs/0/a': Transport endpoint is not connected<br>
> cat: /var/lib/glusterd/vols/patchy/<wbr>run/slave28.cloud.gluster.org-<wbr>d-backends-patchy5.pid: No such file or directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> ./tests/bugs/cli/bug-1320388.t<wbr>: line 19: /mnt/glusterfs/0/a: Transport endpoint is not connected<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> ./tests/bugs/cli/bug-1320388.t ..<br>
> 1..11<br>
> not ok 1 , LINENUM:11<br>
> FAILED COMMAND: glusterd<br>
> not ok 2 , LINENUM:12<br>
> FAILED COMMAND: pidof glusterd<br>
> ok 3, LINENUM:13<br>
> ok 4, LINENUM:14<br>
> not ok 5 , LINENUM:15<br>
> FAILED COMMAND: glusterfs --entry-timeout=0 --attribute-timeout=0 -s <a href="http://slave28.cloud.gluster.org" rel="noreferrer" target="_blank">slave28.cloud.gluster.org</a> --volfile-id patchy /mnt/glusterfs/0<br>
> not ok 6 Got "" instead of "^6$", LINENUM:16<br>
> FAILED COMMAND: ^6$ ec_child_up_count patchy 0<br>
> not ok 7 , LINENUM:18<br>
> FAILED COMMAND: kill_brick patchy <a href="http://slave28.cloud.gluster.org" rel="noreferrer" target="_blank">slave28.cloud.gluster.org</a> /d/backends/patchy5<br>
> not ok 8 Got "" instead of "^5$", LINENUM:20<br>
> FAILED COMMAND: ^5$ get_pending_heal_count patchy<br>
> not ok 9 Got "" instead of "^6$", LINENUM:22<br>
> FAILED COMMAND: ^6$ ec_child_up_count patchy 0<br>
> ok 10, LINENUM:23<br>
> not ok 11 Got "" instead of "^0$", LINENUM:24<br>
> FAILED COMMAND: ^0$ get_pending_heal_count patchy<br>
> Failed 8/11 subtests<br>
><br>
> Test Summary Report<br>
> -------------------<br>
> ./tests/bugs/cli/bug-1320388.t (Wstat: 0 Tests: 11 Failed: 8)<br>
> Failed tests: 1-2, 5-9, 11<br>
> Files=1, Tests=11, 202 wallclock secs ( 0.02 usr 0.01 sys + 12.55 cusr 3.28 csys = 15.86 CPU)<br>
> Result: FAIL<br>
> End of test ./tests/bugs/cli/bug-1320388.t<br>
> ==============================<wbr>==============================<wbr>====================<br>
><br>
><br>
> Run complete<br>
> ==============================<wbr>==============================<wbr>====================<br>
> Number of tests found: 159<br>
> Number of tests selected for run based on pattern: 159<br>
> Number of tests skipped as they were marked bad: 6<br>
> Number of tests skipped because of known_issues: 1<br>
> Number of tests that were run: 152<br>
><br>
> 1 test(s) failed<br>
> ./tests/bugs/cli/bug-1320388.t<br>
><br>
> 0 test(s) generated core<br>
><br>
><br>
> Tests ordered by time taken, slowest to fastest:<br>
> ==============================<wbr>==============================<wbr>====================<br>
> ./tests/basic/afr/split-brain-<wbr>favorite-child-policy.t - 578 second<br>
> ./tests/basic/ec/ec-12-4.t - 345 second<br>
> ./tests/basic/ec/ec-background<wbr>-heals.t - 302 second<br>
> ./tests/bugs/cli/bug-1320388.<wbr>t - 202 second<br>
> ./tests/basic/ec/ec-7-3.t - 191 second<br>
> ./tests/basic/ec/ec-6-2.t - 178 second<br>
> ./tests/basic/afr/entry-self-h<wbr>eal.t - 174 second<br>
> ./tests/basic/tier/tier-heald.<wbr>t - 166 second<br>
> ./tests/basic/ec/ec-5-2.t - 152 second<br>
> ./tests/basic/ec/ec-5-1.t - 151 second<br>
> ./tests/basic/afr/self-heal.t - 151 second<br>
> ./tests/basic/glusterd/heald.<wbr>t - 139 second<br>
> ./tests/basic/tier/tier.t - 132 second<br>
> ./tests/basic/ec/ec-4-1.t - 119 second<br>
> ./tests/basic/tier/legacy-many<wbr>.t - 115 second<br>
> ./tests/basic/afr/granular-esh<wbr>/conservative-merge.t - 114 second<br>
> ./tests/basic/ec/ec-root-heal.<wbr>t - 111 second<br>
> ./tests/basic/afr/granular-esh<wbr>/granular-esh.t - 100 second<br>
> ./tests/basic/afr/add-brick-se<wbr>lf-heal.t - 100 second<br>
> ./tests/basic/afr/granular-esh<wbr>/add-brick.t - 99 second<br>
> ./tests/basic/ec/ec-new-entry.<wbr>t - 92 second<br>
> ./tests/basic/ec/ec-3-1.t - 92 second<br>
> ./tests/basic/afr/split-brain-<wbr>heal-info.t - 88 second<br>
> ./tests/basic/afr/self-heald.<wbr>t - 86 second<br>
> ./tests/basic/afr/split-brain-<wbr>healing.t - 81 second<br>
> ./tests/basic/afr/metadata-sel<wbr>f-heal.t - 76 second<br>
> ./tests/basic/quota.t - 74 second<br>
> ./tests/basic/ec/self-heal.t - 68 second<br>
> ./tests/basic/tier/new-tier-cm<wbr>ds.t - 65 second<br>
> ./tests/basic/tier/tierd_check<wbr>.t - 64 second<br>
> ./tests/basic/afr/sparse-file-<wbr>self-heal.t - 62 second<br>
> ./tests/basic/tier/frequency-c<wbr>ounters.t - 61 second<br>
> ./tests/basic/volume-snapshot-<wbr>clone.t - 55 second<br>
> ./tests/basic/uss.t - 52 second<br>
> ./tests/basic/ec/ec-notify.t - 47 second<br>
> ./tests/basic/tier/fops-during<wbr>-migration-pause.t - 46 second<br>
> ./tests/basic/ec/ec-readdir.t - 44 second<br>
> ./tests/basic/volume-snapshot.<wbr>t - 42 second<br>
> ./tests/basic/mount-nfs-auth.<wbr>t - 40 second<br>
> ./tests/basic/tier/locked_file<wbr>_migration.t - 38 second<br>
> ./tests/basic/ec/ec-anonymous-<wbr>fd.t - 38 second<br>
> ./tests/basic/afr/arbiter.t - 38 second<br>
> ./tests/basic/tier/unlink-duri<wbr>ng-migration.t - 37 second<br>
> ./tests/basic/jbr/jbr.t - 37 second<br>
> ./tests/basic/ec/ec.t - 35 second<br>
> ./tests/basic/afr/data-self-he<wbr>al.t - 35 second<br>
> ./tests/basic/mgmt_v3-locks.t - 34 second<br>
> ./tests/basic/afr/quorum.t - 31 second<br>
> ./tests/basic/quota-ancestry-b<wbr>uilding.t - 29 second<br>
> ./tests/basic/afr/arbiter-add-<wbr>brick.t - 27 second<br>
> ./tests/basic/afr/durability-o<wbr>ff.t - 26 second<br>
> ./tests/bitrot/bug-1294786.t - 24 second<br>
> ./tests/basic/tier/file_with_s<wbr>paces.t - 24 second<br>
> ./tests/basic/afr/heal-quota.<wbr>t - 24 second<br>
> ./tests/basic/afr/gfid-self-he<wbr>al.t - 23 second<br>
> ./tests/basic/geo-replication/<wbr>marker-xattrs.t - 22 second<br>
> ./tests/basic/ec/quota.t - 22 second<br>
> ./tests/bugs/bitrot/bug-122799<wbr>6.t - 21 second<br>
> ./tests/basic/tier/readdir-dur<wbr>ing-migration.t - 21 second<br>
> ./tests/basic/op_errnos.t - 21 second<br>
> ./tests/basic/ec/statedump.t - 21 second<br>
> ./tests/basic/afr/replace-bric<wbr>k-self-heal.t - 21 second<br>
> ./tests/basic/glusterd/volfile<wbr>_server_switch.t - 20 second<br>
> ./tests/basic/afr/granular-esh<wbr>/replace-brick.t - 20 second<br>
> ./tests/basic/0symbol-check.t - 20 second<br>
> ./tests/bugs/cli/bug-1113476.<wbr>t - 19 second<br>
> ./tests/bugs/changelog/bug-122<wbr>5542.t - 19 second<br>
> ./tests/bugs/bitrot/bug-124598<wbr>1.t - 18 second<br>
> ./tests/basic/afr/split-brain-<wbr>resolution.t - 18 second<br>
> ./tests/bitrot/br-state-check.<wbr>t - 17 second<br>
> ./tests/bugs/cli/bug-1077682.<wbr>t - 16 second<br>
> ./tests/bugs/changelog/bug-132<wbr>1955.t - 16 second<br>
> ./tests/basic/afr/client-side-<wbr>heal.t - 16 second<br>
> ./tests/bugs/changelog/bug-121<wbr>1327.t - 15 second<br>
> ./tests/bugs/bitrot/bug-128849<wbr>0.t - 15 second<br>
> ./tests/bugs/bitrot/bug-122868<wbr>0.t - 15 second<br>
> ./tests/basic/glusterd/arbiter<wbr>-volume-probe.t - 15 second<br>
> ./tests/basic/bd.t - 15 second<br>
> ./tests/basic/afr/resolve.t - 15 second<br>
> ./tests/bugs/cli/bug-1047416.<wbr>t - 14 second<br>
> ./tests/bugs/bitrot/1209751-bi<wbr>trot-scrub-tunable-reset.t - 14 second<br>
> ./tests/basic/volume-status.t - 14 second<br>
> ./tests/basic/glusterd/dispers<wbr>e-create.t - 14 second<br>
> ./tests/bugs/bug-1110262.t - 13 second<br>
> ./tests/basic/tier/ctr-rename-<wbr>overwrite.t - 13 second<br>
> ./tests/basic/rpc-coverage.t - 13 second<br>
> ./tests/basic/pump.t - 13 second<br>
> ./tests/basic/cdc.t - 13 second<br>
> ./tests/bugs/bitrot/1209752-vo<wbr>lume-status-should-show-bitrot<wbr>-scrub-info.t - 12 second<br>
> ./tests/basic/volume.t - 12 second<br>
> ./tests/basic/quota-nfs.t - 12 second<br>
> ./tests/basic/quota-anon-fd-nf<wbr>s.t - 12 second<br>
> ./tests/basic/nufa.t - 12 second<br>
> ./tests/basic/inode-quota-enfo<wbr>rcing.t - 12 second<br>
> ./tests/basic/afr/stale-file-l<wbr>ookup.t - 12 second<br>
> ./tests/basic/afr/root-squash-<wbr>self-heal.t - 12 second<br>
> ./tests/basic/afr/read-subvol-<wbr>data.t - 12 second<br>
> ./tests/bugs/cli/bug-1087487.<wbr>t - 11 second<br>
> ./tests/bugs/cli/bug-1030580.<wbr>t - 11 second<br>
> ./tests/bugs/changelog/bug-120<wbr>8470.t - 11 second<br>
> ./tests/bugs/access-control/bu<wbr>g-958691.t - 11 second<br>
> ./tests/bugs/access-control/bu<wbr>g-887098-gmount-crash.t - 11 second<br>
> ./tests/bitrot/bug-1207627-bit<wbr>rot-scrub-status.t - 11 second<br>
> ./tests/basic/stats-dump.t - 11 second<br>
> ./tests/basic/mount.t - 11 second<br>
> ./tests/basic/glusterd/arbiter<wbr>-volume.t - 11 second<br>
> ./tests/basic/ec/ec-read-polic<wbr>y.t - 11 second<br>
> ./tests/basic/afr/arbiter-moun<wbr>t.t - 11 second<br>
> ./tests/bugs/bitrot/1207029-bi<wbr>trot-daemon-should-start-on-va<wbr>lid-node.t - 10 second<br>
> ./tests/basic/gfapi/bug1291259<wbr>.t - 10 second<br>
> ./tests/basic/fop-sampling.t - 10 second<br>
> ./tests/basic/afr/heal-info.t - 10 second<br>
> ./tests/bugs/cli/bug-1022905.<wbr>t - 9 second<br>
> ./tests/bugs/bitrot/1209818-vo<wbr>l-info-show-scrub-process-prop<wbr>erly.t - 9 second<br>
> ./tests/bitrot/bug-1244613.t - 9 second<br>
> ./tests/basic/meta.t - 9 second<br>
> ./tests/basic/afr/read-subvol-<wbr>entry.t - 9 second<br>
> ./tests/bugs/bitrot/bug-122913<wbr>4-bitd-not-support-vol-set.t - 8 second<br>
> ./tests/basic/pgfid-feat.t - 8 second<br>
> ./tests/basic/gfapi/gfapi-trun<wbr>c.t - 8 second<br>
> ./tests/basic/gfapi/bug-124110<wbr>4.t - 8 second<br>
> ./tests/basic/ec/ec-internal-x<wbr>attrs.t - 8 second<br>
> ./tests/basic/distribute/bug-1<wbr>265677-use-readdirp.t - 8 second<br>
> ./tests/basic/afr/gfid-mismatc<wbr>h.t - 8 second<br>
> ./tests/basic/afr/gfid-heal.t - 8 second<br>
> ./tests/basic/afr/arbiter-stat<wbr>fs.t - 8 second<br>
> ./tests/basic/afr/arbiter-remo<wbr>ve-brick.t - 8 second<br>
> ./tests/bugs/bug-1258069.t - 7 second<br>
> ./tests/bugs/bitrot/bug-121068<wbr>4-scrub-pause-resume-error-<wbr>handling.t - 7 second<br>
> ./tests/bitrot/bug-internal-xa<wbr>ttrs-check-1243391.t - 7 second<br>
> ./tests/bitrot/bug-1221914.t - 7 second<br>
> ./tests/bitrot/br-stub.t - 7 second<br>
> ./tests/basic/quota-rename.t - 7 second<br>
> ./tests/basic/gfapi/gfapi-dup.<wbr>t - 7 second<br>
> ./tests/basic/gfapi/anonymous_<wbr>fd.t - 7 second<br>
> ./tests/basic/ec/dht-rename.t - 7 second<br>
> ./tests/basic/distribute/throt<wbr>tle-rebal.t - 7 second<br>
> ./tests/bugs/cli/bug-1004218.<wbr>t - 6 second<br>
> ./tests/bugs/access-control/bu<wbr>g-1051896.t - 6 second<br>
> ./tests/basic/jbr/jbr-volgen.<wbr>t - 6 second<br>
> ./tests/basic/gfid-access.t - 6 second<br>
> ./tests/basic/gfapi/libgfapi-f<wbr>ini-hang.t - 6 second<br>
> ./tests/basic/fops-sanity.t - 6 second<br>
> ./tests/basic/ec/nfs.t - 6 second<br>
> ./tests/bugs/cli/bug-1047378.<wbr>t - 5 second<br>
> ./tests/basic/afr/arbiter-cli.<wbr>t - 5 second<br>
> ./tests/basic/rpm.t - 2 second<br>
> ./tests/basic/posixonly.t - 1 second<br>
> ./tests/basic/netgroup_parsing<wbr>.t - 1 second<br>
> ./tests/basic/gfapi/upcall-cac<wbr>he-invalidate.t - 1 second<br>
> ./tests/basic/exports_parsing.<wbr>t - 1 second<br>
> ./tests/basic/first-test.t - 0 second<br>
><br>
> Result is 1<br>
><br>
> tar: Removing leading `/' from member names<br>
> Logs archived in <a href="http://slave28.cloud.gluster.org/logs/glusterfs-logs-20160901:11:46:33.tgz" rel="noreferrer" target="_blank">http://slave28.cloud.gluster.o<wbr>rg/logs/glusterfs-logs-2016090<wbr>1:11:46:33.tgz</a><br>
> kernel.core_pattern = /%e-%p.core<br>
> Build step 'Execute shell' marked build as failure<br>
> ______________________________<wbr>_________________<br>
> maintainers mailing list<br>
> <a href="javascript:_e(%7B%7D,'cvml','maintainers@gluster.org');" target="_blank">maintainers@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/maintainers" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/maintainers</a><br>
______________________________<wbr>_________________<br>
maintainers mailing list<br>
<a href="javascript:_e(%7B%7D,'cvml','maintainers@gluster.org');" target="_blank">maintainers@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/maintainers" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/maintainers</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div><br></div>--Atin<br></div></div>
</div>
</blockquote><br><br>-- <br>--Atin<br>