<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">FYI, it’s my experience that “yum upgrade” will stop the running glistered (and possibly the running glusterfsds) during it’s installation of new gluster components. I’ve also noticed it starts them back up again during the process.<div class=""><br class=""></div><div class="">Ie, yesterday I upgraded a system to 3.7.13:</div><div class=""><br class=""></div><div class="">systemctl stop glusterd</div><div class=""><manually kill all glusterfsds></div><div class="">yum upgrade</div><div class=""><br class=""></div><div class="">and discovered that glusterd was running again, and had started during the yum upgrade processing. All it’s glusterfsds had also started. Somewhat annoying, actually, because I had been planning to reboot the server to switch to the latest kernel as part of the process, but really didn’t feel like interrupting the heals at that time.</div><div class=""><br class=""></div><div class="">This probably didn’t have much impact on you, but it would have restarted any healing that was a result of the upgrade downtime twice. You may have caused yourself some extra wait for the first round of healing to conclude if stuff was using those volumes at the time. If you didn’t wait for those to conclude before starting your next upgrade, you could have caused a split brain on affected active files.</div><div class=""><br class=""></div><div class=""> -Darrell</div><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On Jul 12, 2016, at 10:57 AM, Dmitry Melekhov <<a href="mailto:dm@belkam.com" class="">dm@belkam.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type" class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">12.07.2016 17:38, Pranith Kumar
Karampuri пишет:<br class="">
</div>
<blockquote cite="mid:CAOgeEnbFyZiUdwN-SEN+uyQBUxcKAK55TFVQMnmcxS7hF3pyMw@mail.gmail.com" type="cite" class="">
<div dir="ltr" class="">Did you wait for heals to complete before
upgrading second node?<br class="">
</div>
</blockquote>
<br class="">
no...<br class="">
<br class="">
<blockquote cite="mid:CAOgeEnbFyZiUdwN-SEN+uyQBUxcKAK55TFVQMnmcxS7hF3pyMw@mail.gmail.com" type="cite" class="">
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Tue, Jul 12, 2016 at 3:08 PM, Dmitry
Melekhov <span dir="ltr" class=""><<a moz-do-not-send="true" href="mailto:dm@belkam.com" target="_blank" class="">dm@belkam.com</a>></span>
wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000" class="">
<div class="">12.07.2016 13:31, Pranith Kumar Karampuri пишет:<br class="">
</div>
<span class="">
<blockquote type="cite" class="">
<div dir="ltr" class=""><br class="">
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Mon, Jul 11, 2016 at
2:26 PM, Dmitry Melekhov <span dir="ltr" class=""><<a moz-do-not-send="true" href="mailto:dm@belkam.com" target="_blank" class=""></a><a class="moz-txt-link-abbreviated" href="mailto:dm@belkam.com">dm@belkam.com</a>></span>
wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">11.07.2016 12:47,
Gandalf Corvotempesta пишет:<span class=""><br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex"> 2016-07-11
9:54 GMT+02:00 Dmitry Melekhov <<a moz-do-not-send="true" href="mailto:dm@belkam.com" target="_blank" class=""></a><a class="moz-txt-link-abbreviated" href="mailto:dm@belkam.com">dm@belkam.com</a>>:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex"> We just
got split-brain during update to 3.7.13
;-)<br class="">
</blockquote>
This is an interesting point.<br class="">
Could you please tell me which replica
count did you set ?<br class="">
</blockquote>
<br class="">
</span> 3<span class=""><br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex"> <br class="">
With replica "3" split brain should not
occurs, right ?<br class="">
</blockquote>
<br class="">
</span> I guess we did something wrong :-)</blockquote>
<div class=""><br class="">
</div>
<div class="">Or there is a bug we never found? Could you
please share details about what you did?<br class="">
</div>
</div>
</div>
</div>
</blockquote>
<br class="">
</span> upgraded to 3.7.13 from 3.7.11 using yum, while at
least one VM is running :-)<br class="">
on all 3 servers, one by one:<br class="">
<br class="">
yum upgrade<br class="">
systemctl stop glusterd <br class="">
than killed glusterfsd processes using kill <br class="">
and systemctl start glusterd<br class="">
<br class="">
then next server....<br class="">
<br class="">
after this we tried to restart VM, but it failed, because
we forget to restart libvirtd, and it used old libraries,<br class="">
I guess this is point where we got this problem.<span class=""><br class="">
<br class="">
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div class="gmail_extra">
<div class="gmail_quote">
<div class=""> </div>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div class="">
<div class=""><br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex"> I'm
planning a new cluster and I would like
to be protected against<br class="">
split brains.<br class="">
</blockquote>
<br class="">
_______________________________________________<br class="">
Gluster-users mailing list<br class="">
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank" class="">Gluster-users@gluster.org</a><br class="">
<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank" class="">http://www.gluster.org/mailman/listinfo/gluster-users</a></div>
</div>
</blockquote>
</div>
<br class="">
<br clear="all" class="">
<br class="">
-- <br class="">
<div data-smartmail="gmail_signature" class="">
<div dir="ltr" class="">Pranith<br class="">
</div>
</div>
</div>
</div>
</blockquote>
<br class="">
</span></div>
</blockquote>
</div>
<br class="">
<br clear="all" class="">
<br class="">
-- <br class="">
<div class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr" class="">Pranith<br class="">
</div>
</div>
</div>
</blockquote>
<br class="">
</div>
_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">http://www.gluster.org/mailman/listinfo/gluster-users</div></blockquote></div><br class=""></div></body></html>