<div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Nov 5, 2016 at 3:32 PM, Michael Scherer <span dir="ltr"><<a href="mailto:mscherer@redhat.com" target="_blank">mscherer@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
so people might have seen that last week, a rather severe vuln was<br>
found: <a href="https://dirtycow.ninja/" rel="noreferrer" target="_blank">https://dirtycow.ninja/</a><br>
<br>
I was at Openstack summit when it was found, and the updated kernel<br>
package wasn't on the CDN until I was out for holiday[1]. The main<br>
reason is that RH test kernel patchs a bit more than others, especially<br>
for something as critical. And Centos wait on RH to push update<br>
<br>
So while this was not uber urgent as shellshock or heartbleed, it was<br>
still rather critical to fix as I have a rather minimal trust in Jenkins<br>
and Gerrit to be secure.<br>
<br>
So once I was back on friday, and after dealing with others fires and<br>
infra, I did reboot stuff that wouldn't impact too much production (like<br>
rsyslog, freeipa servers, the salt server, the virt hosts with builders)<br>
and decided to push for a reboot of jenkins and gerrit for the weekend.<br>
<br>
In retrospect, I tought I did discuss on irc, but I forgot, sorry about<br>
that.<br>
<br>
Of course, because I like to live dangerously, I did that in the<br>
saturday morning, on a travel day. It should have been fast[2].<br>
<br>
However, things never go as expected and we did face a few issues:<br>
<br>
- <a href="http://myrmicinae.rht.gluster.org" rel="noreferrer" target="_blank">myrmicinae.rht.gluster.org</a>, the host running our VM decided to take 1h<br>
to boot. At the firmware/BIOS level. That's slightly inacceptable, but I<br>
have also a limited capacity to fix, since this would requires 1) to<br>
test reboot (so lose 1h) 2) to fiddle in the Bios (and so reboot).<br>
<br>
So that's why jenkins/gerrit were down around 10h CET until 11h.<br>
<br>
- jenkins didn't (as usual) restart. I found the root cause, this was<br>
due to NetworkManager and network init script kinda doing the same<br>
stuff, but in different way. This is now fixed, and jenkins VM should<br>
reboot without a human to fix stuff around.<br>
<br>
- gerrit for some reason do not start at boot. I am not sure what was<br>
the way it was done before, but I suspect something related<br>
to /etc/init.d that got wiped after a upgrade or something, because<br>
gerrit initscript is not a real initscript. So I did some hack<br>
in /etc/rc.local, since the upgrade to EL7 is around the corner, and I<br>
had better things to do in the weekend that fixing some bash stuff (like<br>
fixing python stuff).<br>
<br>
- gerrit VM DNS was incorrect, and no one told me until 6h after the<br>
reboot (why no one told on irc and or on the list and or bugzilla is a<br>
issue that I will surely have to investigate). Why did the DNS got<br>
changed (or if it didn't changed, how did it worked before ?) is the<br>
part that I still cannot explain. But it got for some reason reverted to<br>
the old setting, using the libvirt gateway as dns, which wasn't working<br>
with the current setup. So this was fixed after Nigel pinged me on my<br>
phone, and I managed to connect from the train to fix it.<br>
<br>
So I suspect that's all for today, I will try to schedule my next<br>
vacation outside of the unexpected release of a critical kernel patch.<br>
<br>
[1] yes, it was nice, thanks for asking.<br>
<br>
[2] famous last word<br>
<span class="HOEnZb"><font color="#888888">--<br>
Michael Scherer<br>
Sysadmin, Community Infrastructure and Platform, OSAS<br>
<br>
<br>
</font></span><br>______________________________<wbr>_________________<br>
Gluster-infra mailing list<br>
<a href="mailto:Gluster-infra@gluster.org">Gluster-infra@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-infra" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-infra</a><br></blockquote></div><br>Hey, thanks for the update! We'll work on moving those kernel patches timelines around.</div><div class="gmail_extra">- amye </div><div class="gmail_extra"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Amye Scavarda | <a href="mailto:amye@redhat.com" target="_blank">amye@redhat.com</a> | Gluster Community Lead</div></div>
</div></div>