<div dir="ltr">Thanks!</div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 17, 2015 at 1:32 PM, Anita Kuno <span dir="ltr"><<a href="mailto:anteaya@anteaya.info" target="_blank">anteaya@anteaya.info</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 07/14/2015 02:16 PM, Erlon Cruz wrote:<br>
<span class="">> On Tue, Jul 14, 2015 at 1:06 PM, Anita Kuno <<a href="mailto:anteaya@anteaya.info">anteaya@anteaya.info</a>> wrote:<br>
><br>
>> On 07/14/2015 11:39 AM, Erlon Cruz wrote:<br>
>>> Hi Anita,<br>
>>><br>
>>> I think I knew what happened, we had a power failure this weekend, and<br>
>> the<br>
>>> VM that was running the Jenkins master for this CI was automatically<br>
>>> booted, if you see on the CI reports all of then where sent on 13/07.<br>
>><br>
>> Great are you able to take any steps so that if you have a power failure<br>
>> in future that a human needs to evaluate the system prior to it<br>
>> reporting on patches.<br>
>><br>
>><br>
> Sure, we have done it!<br>
><br>
><br>
>>><br>
>>> Whats is there any criteria for removing CIs?<br>
>><br>
>> This gerrit account has been disabled. The CI has not be removed. There<br>
>> is a difference.<br>
>><br>
>> That was what I meant. Disable.<br>
><br>
><br>
>> The criteria for disabling CIs is meeting infra requirements as outlined<br>
>> here:<br>
>> <a href="http://docs.openstack.org/infra/system-config/third_party.html#requirements" rel="noreferrer" target="_blank">http://docs.openstack.org/infra/system-config/third_party.html#requirements</a><br>
</span><span class="">>> as well as infra having discretion at disabling a system at any time:<br>
>> "The OpenStack Infrastructure team disables mis-behaving third-party ci<br>
>> accounts at its discretion." from<br>
>><br>
>> <a href="http://docs.openstack.org/infra/system-config/third_party.html#permissions-on-your-third-party-system" rel="noreferrer" target="_blank">http://docs.openstack.org/infra/system-config/third_party.html#permissions-on-your-third-party-system</a><br>
>><br>
</span><span class="">>>> I fell this removal was very<br>
>>> unfair.<br>
>><br>
>> That's fine and I appreciate you sharing your thoughts with me.<br>
>><br>
>><br>
>>> I have seen CIs reporting errors for weeks<br>
>><br>
>> Have you communicated those errors to anyone? The core reviewers of the<br>
>> project? The operators of the CI in question? Shared them at a third<br>
>> party meeting?<br>
>><br>
>><br>
> Yes, we talked[1] to cinder guys and informed about the problem when it<br>
> happened, also informed about the patch that was ready to fix the<br>
> problem[2], that was only merged after I asked again.<br>
><br>
><br>
>> Part of the reason why my actions may seem so heavy handed is that my<br>
>> vision of CI operators working collaboratively with each other to ensure<br>
>> high quality of all systems has not come to pass. Quality as a whole is<br>
>> low and folks tend to ignore CI reports as noise, even the useful ones.<br>
>> As a survival technique I disable systems when I find out about them.<br>
>> This system came to my attention and I diabled it.<br>
>><br>
><br>
> We worked in a collaboration scheme since the problem came up. Since we<br>
> detected the problem, that was not our fault BTW, we are working with<br>
> Cinder team to get it fixed. Collaboration is opposed to the action that<br>
> was taken here. You see a problem and before informing the people that<br>
> could solve the problem or trying to understand the status of the problem<br>
> just disable the CI.<br>
><br>
><br>
>><br>
>><br>
>> <a href="http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-07-13.log.html#t2015-07-13T21:36:48" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-07-13.log.html#t2015-07-13T21:36:48</a><br>
>><br>
>><br>
>><br>
</span><span class="">>>> and then, only after<br>
>>> warnings to operators in this mailing list, they are removed.<br>
>><br>
>> Nothing has been removed. Infra takes care of the gerrit accounts,<br>
>> nothing more.<br>
>><br>
>>> In our case,<br>
>>> it was less than 24h, we got no warnings and got this account disabled.<br>
>>> Today, we can't test the CI against the patch that entered (yesterday)<br>
>> and<br>
>>> will have to do all the process to get the account again. Is it possible<br>
>> to<br>
>>> re-enable the account so we can continue the process to test the patch<br>
>> that<br>
>>> should fix the problem?<br>
>><br>
>> Let's get some confirmation a power outage in future will have a<br>
>> different outcome.<br>
>><br>
><br>
> I have done this. If you re-enable it, it will run in silent mode until we<br>
> assure that it is stable enough to post to gerrit.<br>
><br>
><br>
>><br>
>> Thanks Erlon,<br>
>> Anita.<br>
>><br>
>>><br>
>>> Thanks,<br>
>>> Erlon<br>
>>><br>
</span>>>> On Tue, Jul 14, 2015 at 12:06 PM, Anita Kuno <<a href="mailto:anteaya@anteaya.info">anteaya@anteaya.info</a>><br>
<span class="">>> wrote:<br>
>>><br>
>>>> On 07/14/2015 10:18 AM, Erlon Cruz wrote:<br>
>>>>> Hi Anita,<br>
>>>>><br>
>>>>> There was a change in an external library (lockutils) that was causing<br>
>>>> the<br>
>>>>> patches run in our CI to fail. We have reported the problem[1],<br>
>>>> submitted a<br>
>>>>> fix[2], and shut down the CI while the fix wasn't merged.<br>
>>>><br>
>>>> Okay well somehow you shut down your system but it was still leaving<br>
>>>> comments on gerrit patches. How did that happen?<br>
>>>><br>
>>>> Thanks Erlon,<br>
>>>> Anita.<br>
>>>><br>
>>>><br>
>>>>> That was about 10<br>
>>>>> days ago, and the patch only landed yesterday. We will re-run the above<br>
>>>>> tests and submit the successful results.<br>
>>>>><br>
>>>>> [1] <a href="https://bugs.launchpad.net/cinder/+bug/1470112" rel="noreferrer" target="_blank">https://bugs.launchpad.net/cinder/+bug/1470112</a><br>
</span>>>>>> [2] <a href="https://review.openstack.org/#/c/197686/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/197686/</a><br>
>>>>><br>
>>>>> On Mon, Jul 13, 2015 at 7:06 PM, Anita Kuno <<a href="mailto:anteaya@anteaya.info">anteaya@anteaya.info</a>><br>
>>>> wrote:<br>
>>>>><br>
>>>>>> <a href="https://review.openstack.org/#/c/199537/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/199537/</a><br>
>>>>>><br>
>>>>>> <a href="https://review.openstack.org/#/c/198853/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/198853/</a><br>
>>>>>><br>
>>>>>> <a href="https://review.openstack.org/#/c/193937/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/193937/</a><br>
<span class="">>>>>>><br>
>>>>>> <a href="https://review.openstack.org/#/c/201241/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/201241/</a><br>
>>>>>><br>
>>>>>> This account is posting NOT_REGISTERED on patches.<br>
>>>>>><br>
>>>>>> This is an indication that zuul (if you are using zuul) is unable to<br>
>>>>>> clearly communicate with Jenkins nodes. This is an internal error and<br>
>>>>>> has nothing to do with the patch.<br>
>>>>>><br>
>>>>>> Please fix your system and post your incident report with details as a<br>
>>>>>> response.<br>
>>>>>><br>
>>>>>> Please test your system on ci-sandbox after it is working again.<br>
>>>>>><br>
>>>>>> Thank you,<br>
>>>>>> Anita.<br>
>>>>>><br>
>>>>>> _______________________________________________<br>
>>>>>> Third-party-announce mailing list<br>
</span>>>>>>> <a href="mailto:Third-party-announce@lists.openstack.org">Third-party-announce@lists.openstack.org</a><br>
<span class="">>>>>>><br>
>>>><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce</a><br>
>>>>>> Please attend the third party meetings:<br>
>>>>>> <a href="http://eavesdrop.openstack.org/#Third_Party_Meeting" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/#Third_Party_Meeting</a><br>
>>>>>><br>
>>>>><br>
>>>>><br>
>>>>><br>
</span><span class="">>>>>> _______________________________________________<br>
>>>>> Third-party-announce mailing list<br>
>>>>> <a href="mailto:Third-party-announce@lists.openstack.org">Third-party-announce@lists.openstack.org</a><br>
>>>>><br>
</span><span class="">>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce</a><br>
>>>>> Please attend the third party meetings:<br>
>>>> <a href="http://eavesdrop.openstack.org/#Third_Party_Meeting" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/#Third_Party_Meeting</a><br>
>>>>><br>
>>>><br>
>>>><br>
</span><span class="">>>>> _______________________________________________<br>
>>>> Third-party-announce mailing list<br>
>>>> <a href="mailto:Third-party-announce@lists.openstack.org">Third-party-announce@lists.openstack.org</a><br>
>>>><br>
</span><span class="">>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce</a><br>
>>>> Please attend the third party meetings:<br>
>>>> <a href="http://eavesdrop.openstack.org/#Third_Party_Meeting" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/#Third_Party_Meeting</a><br>
>>>><br>
>>><br>
>>><br>
>>><br>
</span><span class="">>>> _______________________________________________<br>
>>> Third-party-announce mailing list<br>
>>> <a href="mailto:Third-party-announce@lists.openstack.org">Third-party-announce@lists.openstack.org</a><br>
</span><span class="">>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce</a><br>
>>> Please attend the third party meetings:<br>
>> <a href="http://eavesdrop.openstack.org/#Third_Party_Meeting" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/#Third_Party_Meeting</a><br>
>>><br>
>><br>
>><br>
</span><span class="">>> _______________________________________________<br>
>> Third-party-announce mailing list<br>
>> <a href="mailto:Third-party-announce@lists.openstack.org">Third-party-announce@lists.openstack.org</a><br>
</span><span class="">>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce</a><br>
>> Please attend the third party meetings:<br>
>> <a href="http://eavesdrop.openstack.org/#Third_Party_Meeting" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/#Third_Party_Meeting</a><br>
>><br>
><br>
><br>
</span>> [1]<br>
<span class="">> <a href="http://eavesdrop.openstack.org/irclogs/%23openstack-cinder/%23openstack-cinder.2015-07-02.log.html" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/irclogs/%23openstack-cinder/%23openstack-cinder.2015-07-02.log.html</a><br>
> [2<br>
> <a href="http://eavesdrop.openstack.org/irclogs/%23openstack-cinder/%23openstack-cinder.2015-07-08.log.html" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/irclogs/%23openstack-cinder/%23openstack-cinder.2015-07-08.log.html</a><br>
><br>
><br>
><br>
</span><span class="">> _______________________________________________<br>
> Third-party-announce mailing list<br>
> <a href="mailto:Third-party-announce@lists.openstack.org">Third-party-announce@lists.openstack.org</a><br>
</span><span class="">> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce</a><br>
> Please attend the third party meetings: <a href="http://eavesdrop.openstack.org/#Third_Party_Meeting" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/#Third_Party_Meeting</a><br>
><br>
<br>
</span>This account is now re-enabled.<br>
<br>
In future please keep a conversation in the same thread, starting a new<br>
thread which is a part of the conversation is confusing when I go to<br>
look for information.<br>
<br>
Reference:<br>
<a href="http://lists.openstack.org/pipermail/third-party-announce/2015-July/000245.htmls" rel="noreferrer" target="_blank">lists.openstack.org/pipermail/third-party-announce/2015-July/000245.htmls</a><br>
<br>
Thank you Erlon,<br>
<span class="">Anita.<br>
<br>
_______________________________________________<br>
Third-party-announce mailing list<br>
</span><a href="mailto:Third-party-announce@lists.openstack.org">Third-party-announce@lists.openstack.org</a><br>
<div class="HOEnZb"><div class="h5"><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce</a><br>
Please attend the third party meetings: <a href="http://eavesdrop.openstack.org/#Third_Party_Meeting" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/#Third_Party_Meeting</a><br>
</div></div></blockquote></div><br></div>