<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 12, 2016 at 9:27 PM, Dmitry Melekhov <span dir="ltr"><<a href="mailto:dm@belkam.com" target="_blank">dm@belkam.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<br>
<br>
<div>12.07.2016 17:38, Pranith Kumar
Karampuri пишет:<br>
</div><span class="">
<blockquote type="cite">
<div dir="ltr">Did you wait for heals to complete before
upgrading second node?<br>
</div>
</blockquote>
<br></span>
no...</div></blockquote><div><br></div><div>So basically if you have operations in progress on the mount, you should wait for heals to complete before you upgrade second node. If you have all the operations on all the mounts stopped or you unmounted all the mounts for the volume, then you can upgrade all the servers one by one then clients. Otherwise it will lead to problems. That said in 3 way replica it shouldn't cause split-brains. So I would like to know exact steps that lead to this problem. We know of one issue which leads to split-brains in case of VM workloads where we take down bricks in cyclic manner without waiting for heals to complete. I wonder if the steps that lead to split-brain on your setup are similar. We are targetting this for future releases...<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><div><div class="h5"><br>
<br>
<blockquote type="cite">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Jul 12, 2016 at 3:08 PM, Dmitry
Melekhov <span dir="ltr"><<a href="mailto:dm@belkam.com" target="_blank">dm@belkam.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>12.07.2016 13:31, Pranith Kumar Karampuri пишет:<br>
</div>
<span>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Jul 11, 2016 at
2:26 PM, Dmitry Melekhov <span dir="ltr"><<a href="mailto:dm@belkam.com" target="_blank"></a><a href="mailto:dm@belkam.com" target="_blank">dm@belkam.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">11.07.2016 12:47,
Gandalf Corvotempesta пишет:<span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> 2016-07-11
9:54 GMT+02:00 Dmitry Melekhov <<a href="mailto:dm@belkam.com" target="_blank"></a><a href="mailto:dm@belkam.com" target="_blank">dm@belkam.com</a>>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> We just
got split-brain during update to 3.7.13
;-)<br>
</blockquote>
This is an interesting point.<br>
Could you please tell me which replica
count did you set ?<br>
</blockquote>
<br>
</span> 3<span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <br>
With replica "3" split brain should not
occurs, right ?<br>
</blockquote>
<br>
</span> I guess we did something wrong :-)</blockquote>
<div><br>
</div>
<div>Or there is a bug we never found? Could you
please share details about what you did?<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</span> upgraded to 3.7.13 from 3.7.11 using yum, while at
least one VM is running :-)<br>
on all 3 servers, one by one:<br>
<br>
yum upgrade<br>
systemctl stop glusterd <br>
than killed glusterfsd processes using kill <br>
and systemctl start glusterd<br>
<br>
then next server....<br>
<br>
after this we tried to restart VM, but it failed, because
we forget to restart libvirtd, and it used old libraries,<br>
I guess this is point where we got this problem.<span><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> I'm
planning a new cluster and I would like
to be protected against<br>
split brains.<br>
</blockquote>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div data-smartmail="gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</span></div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div data-smartmail="gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>