<div dir="ltr"><div><div>This is strange, Ashish has already fixed the issue of the missing SSL certs through fad93c1<br><br></div>From the logs it definitely looks like glusterd failed to come up:<br><br>[2016-09-01 13:30:28.909555] E [socket.c:4122:socket_init] 0-socket.management: could not load our cert<br><br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 1, 2016 at 7:38 PM, Vijay Bellur <span dir="ltr"><<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">tests/bugs/cli/bug-1320388.t is failing quite frequently in<br>
regression-test burn in. Can you please take a look in?<br>
<br>
Thx,<br>
Vijay<br>
<div class="HOEnZb"><div class="h5"><br>
On Thu, Sep 1, 2016 at 9:33 AM, <<a href="mailto:jenkins@build.gluster.org">jenkins@build.gluster.org</a>> wrote:<br>
> See <<a href="http://build.gluster.org/job/regression-test-burn-in/1623/" rel="noreferrer" target="_blank">http://build.gluster.org/job/<wbr>regression-test-burn-in/1623/</a>><br>
><br>
> ------------------------------<wbr>------------<br>
> [...truncated 10509 lines...]<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> touch: cannot touch `/mnt/glusterfs/0/a': Transport endpoint is not connected<br>
> cat: /var/lib/glusterd/vols/patchy/<wbr>run/slave28.cloud.gluster.org-<wbr>d-backends-patchy5.pid: No such file or directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> ./tests/bugs/cli/bug-1320388.<wbr>t: line 19: /mnt/glusterfs/0/a: Transport endpoint is not connected<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]<br>
> sed: read error on /var/run/gluster/: Is a directory<br>
> rm: cannot remove `/var/run/gluster/': Is a directory<br>
> ./tests/bugs/cli/bug-1320388.t ..<br>
> 1..11<br>
> not ok 1 , LINENUM:11<br>
> FAILED COMMAND: glusterd<br>
> not ok 2 , LINENUM:12<br>
> FAILED COMMAND: pidof glusterd<br>
> ok 3, LINENUM:13<br>
> ok 4, LINENUM:14<br>
> not ok 5 , LINENUM:15<br>
> FAILED COMMAND: glusterfs --entry-timeout=0 --attribute-timeout=0 -s <a href="http://slave28.cloud.gluster.org" rel="noreferrer" target="_blank">slave28.cloud.gluster.org</a> --volfile-id patchy /mnt/glusterfs/0<br>
> not ok 6 Got "" instead of "^6$", LINENUM:16<br>
> FAILED COMMAND: ^6$ ec_child_up_count patchy 0<br>
> not ok 7 , LINENUM:18<br>
> FAILED COMMAND: kill_brick patchy <a href="http://slave28.cloud.gluster.org" rel="noreferrer" target="_blank">slave28.cloud.gluster.org</a> /d/backends/patchy5<br>
> not ok 8 Got "" instead of "^5$", LINENUM:20<br>
> FAILED COMMAND: ^5$ get_pending_heal_count patchy<br>
> not ok 9 Got "" instead of "^6$", LINENUM:22<br>
> FAILED COMMAND: ^6$ ec_child_up_count patchy 0<br>
> ok 10, LINENUM:23<br>
> not ok 11 Got "" instead of "^0$", LINENUM:24<br>
> FAILED COMMAND: ^0$ get_pending_heal_count patchy<br>
> Failed 8/11 subtests<br>
><br>
> Test Summary Report<br>
> -------------------<br>
> ./tests/bugs/cli/bug-1320388.t (Wstat: 0 Tests: 11 Failed: 8)<br>
> Failed tests: 1-2, 5-9, 11<br>
> Files=1, Tests=11, 202 wallclock secs ( 0.02 usr 0.01 sys + 12.55 cusr 3.28 csys = 15.86 CPU)<br>
> Result: FAIL<br>
> End of test ./tests/bugs/cli/bug-1320388.t<br>
> ==============================<wbr>==============================<wbr>====================<br>
><br>
><br>
> Run complete<br>
> ==============================<wbr>==============================<wbr>====================<br>
> Number of tests found: 159<br>
> Number of tests selected for run based on pattern: 159<br>
> Number of tests skipped as they were marked bad: 6<br>
> Number of tests skipped because of known_issues: 1<br>
> Number of tests that were run: 152<br>
><br>
> 1 test(s) failed<br>
> ./tests/bugs/cli/bug-1320388.t<br>
><br>
> 0 test(s) generated core<br>
><br>
><br>
> Tests ordered by time taken, slowest to fastest:<br>
> ==============================<wbr>==============================<wbr>====================<br>
> ./tests/basic/afr/split-brain-<wbr>favorite-child-policy.t - 578 second<br>
> ./tests/basic/ec/ec-12-4.t - 345 second<br>
> ./tests/basic/ec/ec-<wbr>background-heals.t - 302 second<br>
> ./tests/bugs/cli/bug-1320388.t - 202 second<br>
> ./tests/basic/ec/ec-7-3.t - 191 second<br>
> ./tests/basic/ec/ec-6-2.t - 178 second<br>
> ./tests/basic/afr/entry-self-<wbr>heal.t - 174 second<br>
> ./tests/basic/tier/tier-heald.<wbr>t - 166 second<br>
> ./tests/basic/ec/ec-5-2.t - 152 second<br>
> ./tests/basic/ec/ec-5-1.t - 151 second<br>
> ./tests/basic/afr/self-heal.t - 151 second<br>
> ./tests/basic/glusterd/heald.t - 139 second<br>
> ./tests/basic/tier/tier.t - 132 second<br>
> ./tests/basic/ec/ec-4-1.t - 119 second<br>
> ./tests/basic/tier/legacy-<wbr>many.t - 115 second<br>
> ./tests/basic/afr/granular-<wbr>esh/conservative-merge.t - 114 second<br>
> ./tests/basic/ec/ec-root-heal.<wbr>t - 111 second<br>
> ./tests/basic/afr/granular-<wbr>esh/granular-esh.t - 100 second<br>
> ./tests/basic/afr/add-brick-<wbr>self-heal.t - 100 second<br>
> ./tests/basic/afr/granular-<wbr>esh/add-brick.t - 99 second<br>
> ./tests/basic/ec/ec-new-entry.<wbr>t - 92 second<br>
> ./tests/basic/ec/ec-3-1.t - 92 second<br>
> ./tests/basic/afr/split-brain-<wbr>heal-info.t - 88 second<br>
> ./tests/basic/afr/self-heald.t - 86 second<br>
> ./tests/basic/afr/split-brain-<wbr>healing.t - 81 second<br>
> ./tests/basic/afr/metadata-<wbr>self-heal.t - 76 second<br>
> ./tests/basic/quota.t - 74 second<br>
> ./tests/basic/ec/self-heal.t - 68 second<br>
> ./tests/basic/tier/new-tier-<wbr>cmds.t - 65 second<br>
> ./tests/basic/tier/tierd_<wbr>check.t - 64 second<br>
> ./tests/basic/afr/sparse-file-<wbr>self-heal.t - 62 second<br>
> ./tests/basic/tier/frequency-<wbr>counters.t - 61 second<br>
> ./tests/basic/volume-snapshot-<wbr>clone.t - 55 second<br>
> ./tests/basic/uss.t - 52 second<br>
> ./tests/basic/ec/ec-notify.t - 47 second<br>
> ./tests/basic/tier/fops-<wbr>during-migration-pause.t - 46 second<br>
> ./tests/basic/ec/ec-readdir.t - 44 second<br>
> ./tests/basic/volume-snapshot.<wbr>t - 42 second<br>
> ./tests/basic/mount-nfs-auth.t - 40 second<br>
> ./tests/basic/tier/locked_<wbr>file_migration.t - 38 second<br>
> ./tests/basic/ec/ec-anonymous-<wbr>fd.t - 38 second<br>
> ./tests/basic/afr/arbiter.t - 38 second<br>
> ./tests/basic/tier/unlink-<wbr>during-migration.t - 37 second<br>
> ./tests/basic/jbr/jbr.t - 37 second<br>
> ./tests/basic/ec/ec.t - 35 second<br>
> ./tests/basic/afr/data-self-<wbr>heal.t - 35 second<br>
> ./tests/basic/mgmt_v3-locks.t - 34 second<br>
> ./tests/basic/afr/quorum.t - 31 second<br>
> ./tests/basic/quota-ancestry-<wbr>building.t - 29 second<br>
> ./tests/basic/afr/arbiter-add-<wbr>brick.t - 27 second<br>
> ./tests/basic/afr/durability-<wbr>off.t - 26 second<br>
> ./tests/bitrot/bug-1294786.t - 24 second<br>
> ./tests/basic/tier/file_with_<wbr>spaces.t - 24 second<br>
> ./tests/basic/afr/heal-quota.t - 24 second<br>
> ./tests/basic/afr/gfid-self-<wbr>heal.t - 23 second<br>
> ./tests/basic/geo-replication/<wbr>marker-xattrs.t - 22 second<br>
> ./tests/basic/ec/quota.t - 22 second<br>
> ./tests/bugs/bitrot/bug-<wbr>1227996.t - 21 second<br>
> ./tests/basic/tier/readdir-<wbr>during-migration.t - 21 second<br>
> ./tests/basic/op_errnos.t - 21 second<br>
> ./tests/basic/ec/statedump.t - 21 second<br>
> ./tests/basic/afr/replace-<wbr>brick-self-heal.t - 21 second<br>
> ./tests/basic/glusterd/<wbr>volfile_server_switch.t - 20 second<br>
> ./tests/basic/afr/granular-<wbr>esh/replace-brick.t - 20 second<br>
> ./tests/basic/0symbol-check.t - 20 second<br>
> ./tests/bugs/cli/bug-1113476.t - 19 second<br>
> ./tests/bugs/changelog/bug-<wbr>1225542.t - 19 second<br>
> ./tests/bugs/bitrot/bug-<wbr>1245981.t - 18 second<br>
> ./tests/basic/afr/split-brain-<wbr>resolution.t - 18 second<br>
> ./tests/bitrot/br-state-check.<wbr>t - 17 second<br>
> ./tests/bugs/cli/bug-1077682.t - 16 second<br>
> ./tests/bugs/changelog/bug-<wbr>1321955.t - 16 second<br>
> ./tests/basic/afr/client-side-<wbr>heal.t - 16 second<br>
> ./tests/bugs/changelog/bug-<wbr>1211327.t - 15 second<br>
> ./tests/bugs/bitrot/bug-<wbr>1288490.t - 15 second<br>
> ./tests/bugs/bitrot/bug-<wbr>1228680.t - 15 second<br>
> ./tests/basic/glusterd/<wbr>arbiter-volume-probe.t - 15 second<br>
> ./tests/basic/bd.t - 15 second<br>
> ./tests/basic/afr/resolve.t - 15 second<br>
> ./tests/bugs/cli/bug-1047416.t - 14 second<br>
> ./tests/bugs/bitrot/1209751-<wbr>bitrot-scrub-tunable-reset.t - 14 second<br>
> ./tests/basic/volume-status.t - 14 second<br>
> ./tests/basic/glusterd/<wbr>disperse-create.t - 14 second<br>
> ./tests/bugs/bug-1110262.t - 13 second<br>
> ./tests/basic/tier/ctr-rename-<wbr>overwrite.t - 13 second<br>
> ./tests/basic/rpc-coverage.t - 13 second<br>
> ./tests/basic/pump.t - 13 second<br>
> ./tests/basic/cdc.t - 13 second<br>
> ./tests/bugs/bitrot/1209752-<wbr>volume-status-should-show-<wbr>bitrot-scrub-info.t - 12 second<br>
> ./tests/basic/volume.t - 12 second<br>
> ./tests/basic/quota-nfs.t - 12 second<br>
> ./tests/basic/quota-anon-fd-<wbr>nfs.t - 12 second<br>
> ./tests/basic/nufa.t - 12 second<br>
> ./tests/basic/inode-quota-<wbr>enforcing.t - 12 second<br>
> ./tests/basic/afr/stale-file-<wbr>lookup.t - 12 second<br>
> ./tests/basic/afr/root-squash-<wbr>self-heal.t - 12 second<br>
> ./tests/basic/afr/read-subvol-<wbr>data.t - 12 second<br>
> ./tests/bugs/cli/bug-1087487.t - 11 second<br>
> ./tests/bugs/cli/bug-1030580.t - 11 second<br>
> ./tests/bugs/changelog/bug-<wbr>1208470.t - 11 second<br>
> ./tests/bugs/access-control/<wbr>bug-958691.t - 11 second<br>
> ./tests/bugs/access-control/<wbr>bug-887098-gmount-crash.t - 11 second<br>
> ./tests/bitrot/bug-1207627-<wbr>bitrot-scrub-status.t - 11 second<br>
> ./tests/basic/stats-dump.t - 11 second<br>
> ./tests/basic/mount.t - 11 second<br>
> ./tests/basic/glusterd/<wbr>arbiter-volume.t - 11 second<br>
> ./tests/basic/ec/ec-read-<wbr>policy.t - 11 second<br>
> ./tests/basic/afr/arbiter-<wbr>mount.t - 11 second<br>
> ./tests/bugs/bitrot/1207029-<wbr>bitrot-daemon-should-start-on-<wbr>valid-node.t - 10 second<br>
> ./tests/basic/gfapi/<wbr>bug1291259.t - 10 second<br>
> ./tests/basic/fop-sampling.t - 10 second<br>
> ./tests/basic/afr/heal-info.t - 10 second<br>
> ./tests/bugs/cli/bug-1022905.t - 9 second<br>
> ./tests/bugs/bitrot/1209818-<wbr>vol-info-show-scrub-process-<wbr>properly.t - 9 second<br>
> ./tests/bitrot/bug-1244613.t - 9 second<br>
> ./tests/basic/meta.t - 9 second<br>
> ./tests/basic/afr/read-subvol-<wbr>entry.t - 9 second<br>
> ./tests/bugs/bitrot/bug-<wbr>1229134-bitd-not-support-vol-<wbr>set.t - 8 second<br>
> ./tests/basic/pgfid-feat.t - 8 second<br>
> ./tests/basic/gfapi/gfapi-<wbr>trunc.t - 8 second<br>
> ./tests/basic/gfapi/bug-<wbr>1241104.t - 8 second<br>
> ./tests/basic/ec/ec-internal-<wbr>xattrs.t - 8 second<br>
> ./tests/basic/distribute/bug-<wbr>1265677-use-readdirp.t - 8 second<br>
> ./tests/basic/afr/gfid-<wbr>mismatch.t - 8 second<br>
> ./tests/basic/afr/gfid-heal.t - 8 second<br>
> ./tests/basic/afr/arbiter-<wbr>statfs.t - 8 second<br>
> ./tests/basic/afr/arbiter-<wbr>remove-brick.t - 8 second<br>
> ./tests/bugs/bug-1258069.t - 7 second<br>
> ./tests/bugs/bitrot/bug-<wbr>1210684-scrub-pause-resume-<wbr>error-handling.t - 7 second<br>
> ./tests/bitrot/bug-internal-<wbr>xattrs-check-1243391.t - 7 second<br>
> ./tests/bitrot/bug-1221914.t - 7 second<br>
> ./tests/bitrot/br-stub.t - 7 second<br>
> ./tests/basic/quota-rename.t - 7 second<br>
> ./tests/basic/gfapi/gfapi-dup.<wbr>t - 7 second<br>
> ./tests/basic/gfapi/anonymous_<wbr>fd.t - 7 second<br>
> ./tests/basic/ec/dht-rename.t - 7 second<br>
> ./tests/basic/distribute/<wbr>throttle-rebal.t - 7 second<br>
> ./tests/bugs/cli/bug-1004218.t - 6 second<br>
> ./tests/bugs/access-control/<wbr>bug-1051896.t - 6 second<br>
> ./tests/basic/jbr/jbr-volgen.t - 6 second<br>
> ./tests/basic/gfid-access.t - 6 second<br>
> ./tests/basic/gfapi/libgfapi-<wbr>fini-hang.t - 6 second<br>
> ./tests/basic/fops-sanity.t - 6 second<br>
> ./tests/basic/ec/nfs.t - 6 second<br>
> ./tests/bugs/cli/bug-1047378.t - 5 second<br>
> ./tests/basic/afr/arbiter-cli.<wbr>t - 5 second<br>
> ./tests/basic/rpm.t - 2 second<br>
> ./tests/basic/posixonly.t - 1 second<br>
> ./tests/basic/netgroup_<wbr>parsing.t - 1 second<br>
> ./tests/basic/gfapi/upcall-<wbr>cache-invalidate.t - 1 second<br>
> ./tests/basic/exports_parsing.<wbr>t - 1 second<br>
> ./tests/basic/first-test.t - 0 second<br>
><br>
> Result is 1<br>
><br>
> tar: Removing leading `/' from member names<br>
> Logs archived in <a href="http://slave28.cloud.gluster.org/logs/glusterfs-logs-20160901:11:46:33.tgz" rel="noreferrer" target="_blank">http://slave28.cloud.gluster.<wbr>org/logs/glusterfs-logs-<wbr>20160901:11:46:33.tgz</a><br>
> kernel.core_pattern = /%e-%p.core<br>
> Build step 'Execute shell' marked build as failure<br>
> ______________________________<wbr>_________________<br>
> maintainers mailing list<br>
> <a href="mailto:maintainers@gluster.org">maintainers@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/maintainers" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/maintainers</a><br>
______________________________<wbr>_________________<br>
maintainers mailing list<br>
<a href="mailto:maintainers@gluster.org">maintainers@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/maintainers" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/maintainers</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><br></div>--Atin<br></div></div>
</div>