<div dir="ltr"><div>Hi,</div><div>The primary cause of the failures are provided below. We have come up with the following action plan to resolve these failures.</div><div><br></div><div>1. Array virtual disks/volumes were not getting cleaned up after a CI job run, and were piling up causing some tests to fail in the next CI job run. We have written scripts for automated array cleanup at the beginning of a CI job.</div><div><br>2. We were also having intermittent network issues and this was causing failures while the Jenkins slave node VM was connecting to the git repository. </div><div>===============<br>20:13:36 [vedams-lenovo-dsvm-tempest-full-iscsi] $ /bin/bash -xe /tmp/hudson5584133818276563996.sh<br>20:13:36 + [[ ! -e devstack-gate ]]<br>20:13:36 + git clone git://<a href="http://git.openstack.org/openstack-infra/devstack-gate">git.openstack.org/openstack-infra/devstack-gate</a><br>20:13:36 Cloning into 'devstack-gate'...<br>20:13:37 fatal: unable to connect to <a href="http://git.openstack.org">git.openstack.org</a>:<br>20:13:37 <a href="http://git.openstack.org">git.openstack.org</a>: Name or service not known<br>20:13:37<br>20:13:37 Build step 'Execute shell' marked build as failure<br>===============</div><div><br>We are enhancing the network infrastructure and also planning moving the servers from our lab to DotHill datacenter with better network connectivity, so we don't have these failures in the future.</div><div><br></div><div>Currently, we are testing with our modifications, and we have stopped posting CI results to the patch. We will be completing the testing/sandbox testing within a few days and will start posting the CI results to the patch.</div><div>We are extremely sorry for the inconvenience and hope to have a stable CI very soon.</div><div><br></div><div>Regards</div><div>-Nikesh</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 24, 2015 at 5:13 AM, Mike Perez <span dir="ltr"><<a href="mailto:thingee@gmail.com" target="_blank">thingee@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On Thu, Jul 16, 2015 at 7:24 AM, Anita Kuno <<a href="mailto:anteaya@anteaya.info">anteaya@anteaya.info</a>> wrote:<br>
> On 07/16/2015 09:06 AM, Nikesh Kumar Mahalka wrote:<br>
>> Hi Anita,<br>
>> We debugged the system logs when the incident occurred and found that<br>
>> there was an I/O error that occurred on the operating system disk of the<br>
>> Jenkins Master node.<br>
>> We have setup a new Jenkins master node that resolves the I/O error issue.<br>
>> I am also providing below a snippet of the system log file that indicates<br>
>> the failure that occurred.<br>
>> Could you please enable the HPMSA CI account, so that we can complete the<br>
>> sandbox testing and have the HPMSA CI report on the cinder patches after<br>
>> discussing with cinder developers.<br>
>><br>
>> To prevent this issue from re-occurring, we will pro-actively monitor the<br>
>> operating system logs for any warnings from the disk subsystem. Also, we<br>
>> will pro-actively run disk checks to make sure the disk is healthy.<br>
><br>
> Thank you for clearly stating your commitment to taking responsibility<br>
> for your system and paying attention to how it is affecting the<br>
> community. Thank you.<br>
><br>
> This CI gerrit account is now re-enabled.<br>
<br>
</span>I would like to please re-request the CI's to be disabled again. There<br>
is just way too many false failures in all three in the last 60 runs:<br>
<br>
Vedams-HPMSA FCISCSIDriver CI<br>
Vedams DotHillDriver CI<br>
Vedams-LenovoStorage FCISCSI CI<br>
<br>
<a href="http://paste.openstack.org/show/404811/" target="_blank" rel="noreferrer">http://paste.openstack.org/show/404811/</a><br>
<a href="http://paste.openstack.org/show/404812/" target="_blank" rel="noreferrer">http://paste.openstack.org/show/404812/</a><br>
<a href="http://paste.openstack.org/show/404813/" target="_blank" rel="noreferrer">http://paste.openstack.org/show/404813/</a><br>
<br>
--<br>
Mike Perez<br>
</blockquote></div><br></div>