<div dir="ltr">Excellent post-mortem!<div><br></div><div>Do you think its worth adding mirrors to gluster repos like oVirt is doing? [1]</div><div><br></div><div>[1] <a href="http://ovirt-infra-docs.readthedocs.org/en/latest/General/Mirror.html">http://ovirt-infra-docs.readthedocs.org/en/latest/General/Mirror.html</a></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 27, 2016 at 1:56 PM, Michael Scherer <span dir="ltr"><<a href="mailto:mscherer@redhat.com" target="_blank">mscherer@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
as promised, here is the post-mortem of the incident, if you would like<br>
to see more information, or any remarks, please do not hesitate, since<br>
that's the first attempt at it we do.<br>
<br>
I modelled it based on the example of<br>
<a href="http://shop.oreilly.com/product/0636920041528.do" rel="noreferrer" target="_blank">http://shop.oreilly.com/product/0636920041528.do</a>, as that the book I am<br>
reading at the moment (Appendix D). We will formalize that later.<br>
<br>
<br>
<br>
<a href="http://Download.gluster.org" rel="noreferrer" target="_blank">Download.gluster.org</a> was not serving file<br>
Date: 2016-04-27<br>
Participating people:<br>
- misc<br>
<br>
Summary:<br>
<br>
<a href="http://Download.gluster.org" rel="noreferrer" target="_blank">Download.gluster.org</a> http server was showing error 403 for all url,<br>
which did impact ovirt jenkins jobs, and users using the repository,<br>
among others. The server is used to distribute gluster rpms.<br>
<br>
Impact:<br>
- ovirt CI jobs got blocked<br>
- user couldn't install gluster<br>
<br>
Root cause:<br>
the underlying block device on rackspace was down for a undiagnosed<br>
reason, triggering xfs error on the server and thus 403 on the http<br>
level.<br>
<br>
the root cause of the block device error is for still unknown, no error<br>
have been seen on the rackspace status page for this DC. A ticket was<br>
opened with rackspace to see what was going on (160427-iad-0000814), a<br>
follow up of this post-mortem will be done if the ticket say something<br>
more than "shit happens".<br>
<br>
Resolution:<br>
<br>
The whole server was rebooted, and upon reboot, the block device came<br>
back.<br>
<br>
Lessons learned:<br>
- what went well:<br>
- people notified the admin quickly on irc and on gluster-infra<br>
<br>
- when we were lucky<br>
- the server and block device came back immediately<br>
- it failed during business hours of EMEA with misc being on irc (just<br>
arrived at the office)<br>
<br>
<br>
- what went bad<br>
- we do not have proper HA for the service<br>
- we do not have automated monitoring for it<br>
- the setup is using 2 blocks device of 120G in lvm, thus making it<br>
twice as risky to fail<br>
<br>
Timeline (in UTC)<br>
- 05:39 first error message in the log about XFS error<br>
- 08:41 misc is pinged on irc<br>
- 08:56 misc ack and diagnose the issue<br>
- 09:00 the server and service is back to normal<br>
- 09:00 first mail about the problem hit gluster-infra<br>
<br>
<br>
Potential improvement to make:<br>
- add monitoring on gluster side<br>
- use the centos sig repo on ovirt side<br>
- add more sysadmin for gluster<br>
- add a redundant service for that<br>
- a 2nd download server with a shared gluster backend<br>
- migrate the storage to a proper setup with 1 single block device,<br>
rather than 2.<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
--<br>
Michael Scherer<br>
Sysadmin, Community Infrastructure and Platform, OSAS<br>
<br>
<br>
</font></span><br>_______________________________________________<br>
Infra mailing list<br>
<a href="mailto:Infra@ovirt.org">Infra@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/infra" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/infra</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Eyal Edri<br>Associate Manager</div><div>RHEV DevOps<br>EMEA ENG Virtualization R&D<br>Red Hat Israel<br><br>phone: +972-9-7692018<br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div>
</div>