<div dir="ltr">Hi Mike!<br>While we were busy on stabilizing CI, we have temporary disabled the runs not to give false negatives.<div>I think we are ready and want to re-enable our CI. However I want to finish running multiple internal tests to make sure the infrastructure is stable first.</div><div>Best Regards,</div><div>Ilya</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 16, 2015 at 10:26 AM, Mike Perez <span dir="ltr"><<a href="mailto:thingee@gmail.com" target="_blank">thingee@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">There has at least been 34 patches proposed just today in Cinder,<br>
however, the Nexenta CI has only commented once.<br>
<br>
There was only 6 runs from the Nexenta CI yesterday, and I'm very sure<br>
there was more than that in proposed patches for Cinder. However,<br>
those six at least succeeded.<br>
<br>
I think it's out of the question for the community to accept Nexenta<br>
Edge and the Nexenta ns5 NFS and ISCSI drivers at this point given how<br>
unstable things are.<br>
<br>
Is anyone fixing the CI to actually to do runs today?<br>
<br>
--<br>
Mike Perez<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
On Fri, Jun 12, 2015 at 2:07 PM, Zohar Mamedov<br>
<<a href="mailto:zohar.mamedov@nexenta.com">zohar.mamedov@nexenta.com</a>> wrote:<br>
> Hi Mike,<br>
><br>
> Thank you very much for this update!<br>
><br>
> I reviewed these failures, there are a few things going on:<br>
><br>
> 1. A chunk of those items are starting from yesterday morning (June 11, PST)<br>
> when we were doing some migration of CI environments during which we had<br>
> some issues, hence a few quick failures. In the future we will be smarter<br>
> about minimizing failures when changing CI infrastructure.<br>
><br>
> 2. We were seeing many ssh timeouts in the days before that, and after the<br>
> migration (hardware reorganization, which we are still scrambling over) we<br>
> ended up having that timeout hit again, until we figured out some virtual<br>
> network interface driver optimization for our hypervisor and it seems more<br>
> stable now. We are also adding in SSDs as we speak, and tweaking the<br>
> environment to further optimize. It seems to be running a lot better<br>
> already, and there were a few recent successful runs.<br>
><br>
> 3. During our migration, a certain tweak to the NEDGE backend was lost<br>
> (increasing of internal timeout) which is a tweak that highly improved<br>
> success rate during a working test run (as opposed to the many 6min failures<br>
> caused by our CI infrastructure rather than the backend.) We are currently<br>
> also scrambling over backend hardware too, and may not need that tweak in<br>
> the future, but I just recently put it back in and it looks like those<br>
> internal timeouts are gone again.<br>
><br>
> So to summarize, a lot of these failures are due to our CI infrastructure<br>
> issues and non-backend related, many of the failures before today's morning<br>
> are <6min failures caused by ssh timeout to the instance where the backend<br>
> was untouched. Now there do exist some issues with the backend itself, but<br>
> they are minor, and we are working through them by improving backend<br>
> hardware as well as continued effort on NexentaEdge source code.<br>
><br>
> Finally we are doing our best to give timely responses to comments and<br>
> questions. The Nexenta Edge review was updated with new driver changes based<br>
> on comments from today, and all other questions, comments and suggestions<br>
> are very welcomed and highly appreciated!<br>
><br>
> Thanks again<br>
><br>
> --<br>
> Zohar<br>
><br>
> On Fri, Jun 12, 2015 at 9:27 AM, Mike Perez <<a href="mailto:thingee@gmail.com">thingee@gmail.com</a>> wrote:<br>
>><br>
>> This just today:<br>
>><br>
>> <a href="https://review.openstack.org/#/c/190677/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/190677/</a> failed<br>
>> <a href="https://review.openstack.org/#/c/190725/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/190725/</a> failed<br>
>> <a href="https://review.openstack.org/#/c/187624/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/187624/</a> failed<br>
>> <a href="https://review.openstack.org/#/c/190169/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/190169/</a> failed<br>
>> <a href="https://review.openstack.org/#/c/189135/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/189135/</a> failed<br>
>> <a href="https://review.openstack.org/#/c/185906/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/185906/</a> failed<br>
>> <a href="https://review.openstack.org/#/c/190173/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/190173/</a> failed<br>
>> <a href="https://review.openstack.org/#/c/186580/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/186580/</a> failed<br>
>> <a href="https://review.openstack.org/#/c/189517/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/189517/</a> failed<br>
>> <a href="https://review.openstack.org/#/c/185545/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/185545/</a> failed<br>
>> <a href="https://review.openstack.org/#/c/184596/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/184596/</a> failed<br>
>> <a href="https://review.openstack.org/#/c/182985/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/182985/</a> failed<br>
>> <a href="https://review.openstack.org/#/c/189720/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/189720/</a> failed<br>
>> <a href="https://review.openstack.org/#/c/188649/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/188649/</a> failed<br>
>> <a href="https://review.openstack.org/#/c/178573/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/178573/</a> failed<br>
>><br>
>> What's going on here, and when will this be corrected. This is<br>
>> publishing false failures to Cinder patches.<br>
>><br>
>><br>
>> --<br>
>> Mike Perez<br>
><br>
><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><p><a></a></p><br style="font-size:small"><div style="font-size:small"><div dir="ltr"><div style="font-size:12.8000001907349px;max-width:470px;margin:8px 8px 0px 0px;background:url(https://s3.amazonaws.com/images.wisestamp.com/symbols/bg_colors/ff6600.png) 0% 0% repeat-x"><table border="0" cellspacing="0" cellpadding="0" style="margin-bottom:3px;width:469.600006103516px"><tbody><tr><td style="width:141.600006103516px;vertical-align:top"><table border="0" cellspacing="0" cellpadding="0" style="width:141.600006103516px"><tbody><tr><td><div style="width:141.600006103516px;font-family:Helvetica,Arial,sans-serif;color:rgb(255,255,255);vertical-align:top;text-align:center;padding-top:14px;padding-right:5px;font-size:16px">Ilya Grafutko</div></td></tr><tr><td style="width:141.600006103516px;vertical-align:top;padding-top:20px;text-align:center"><img src="cid:ii_14d12b92676265ae" alt="Inline image 1" width="84" height="96" style="font-size:12.8000001907349px;text-align:start"><div style="font-size:12.8000001907349px;text-align:start"></div></td></tr></tbody></table></td><td style="width:8.80000019073486px;vertical-align:top"><div style="margin-top:10px;border-left-width:3px;border-left-style:solid;border-left-color:rgb(255,255,255);width:1px;min-height:80px;display:inline-block"></div></td><td style="width:319.200012207031px;vertical-align:top"><table border="0" cellspacing="0" cellpadding="0" style="width:319.200012207031px"><tbody><tr><td style="vertical-align:top;width:319.200012207031px;height:100px;padding-top:10px"><table border="0" cellspacing="0" cellpadding="0" style="font-family:Tahoma,sans-serif,arial;color:rgb(255,255,255)"><tbody><tr><td valign="middle" style="padding-left:3px;padding-top:7px;white-space:nowrap;font-size:13px">Director of Solutions Engineering, Nexenta Systems</td></tr><tr><td valign="middle" style="padding-left:5px;padding-top:5px">P 614-636-3895 //<span style="font-size:12px;white-space:nowrap"> </span>M 408-887-5727</td></tr><tr><td valign="middle" style="padding-left:5px;padding-top:3px;white-space:nowrap;font-size:13px"><a href="http://www.nexenta.com/" style="color:rgb(255,255,255);text-decoration:none" target="_blank">http://www.nexenta.com</a></td></tr></tbody></table></td></tr><tr><td valign="middle" style="white-space:nowrap;text-align:right"><a href="https://www.linkedin.com/in/ilyagrafutko" style="color:rgb(17,85,204)" target="_blank"><img src="https://s3.amazonaws.com/images.wisestamp.com/icons/linkedin.png" width="16" height="16"></a> <a href="https://twitter.com/IlyaGraf" style="color:rgb(17,85,204)" target="_blank"><img src="https://s3.amazonaws.com/images.wisestamp.com/icons/twitter.png" width="16" height="16" alt=""></a> <a href="http://skype:n3m8tz/?call" style="color:rgb(17,85,204)" target="_blank"><img src="https://s3.amazonaws.com/webapp.wisestamp.com/uSh6xg7YT6qErsEpJdSd_skype_circle_color-512.png" width="16" height="16" alt=""></a> <a href="http://youtube.com/user/NexentaVideos" style="color:rgb(17,85,204)" target="_blank"><img src="https://s3.amazonaws.com/images.wisestamp.com/icons/youtube.png" width="16" height="16"></a></td></tr></tbody></table></td></tr></tbody></table></div><div style="font-size:12.8000001907349px;clear:both"></div><div style="max-width:469px;margin:8px 8px 0px 0px;font-size:12px"><table><tbody><tr><td style="vertical-align:middle"><a href="http://www.nexenta.com/openstack-vancouver" style="color:rgb(17,85,204);vertical-align:middle" target="_blank"><img src="http://images.wisestamp.com.s3.amazonaws.com/email-apps/event_button/event_button_1.png" style="display:block;margin-bottom:auto;margin-right:4px"></a></td><td style="margin-right:4px;margin-left:4px;vertical-align:middle">Join Nexenta at OpenStack Vancouver<a href="http://www.nexenta.com/openstack-vancouver" style="color:rgb(17,85,204);margin:0px 4px" target="_blank">Nexenta OpenSDx VIP Reception</a></td></tr></tbody></table></div><div style="font-size:13px;max-width:600px;margin:8px 8px 0px 0px;color:gray;border-top-width:1px;border-top-style:solid;border-top-color:gray">IMPORTANT: The contents of this email and any attachments are confidential. <br>They are intended for the named recipient(s) only. If you have received this email <br>by mistake, please notify the sender immediately and do not disclose the contents <br>to anyone or make copies thereof.</div></div></div><table border="1" cellpadding="0" width="100%" style="width:607px;background-color:rgb(230,230,230);border:1pt solid rgb(240,138,44)"><tbody></tbody></table></div></div></div></div></div></div></div></div></div></div>
</div>