<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">13.07.2016 07:44, Pranith Kumar
Karampuri пишет:<br>
</div>
<blockquote
cite="mid:CAOgeEnZe8wuyNyv_fdH3NxPvy91bPHgJriwMJN0LwF7rOJRwEw@mail.gmail.com"
type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Jul 12, 2016 at 9:27 PM,
Dmitry Melekhov <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:dm@belkam.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:dm@belkam.com">dm@belkam.com</a></a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
<br>
<div>12.07.2016 17:38, Pranith Kumar Karampuri пишет:<br>
</div>
<span class="">
<blockquote type="cite">
<div dir="ltr">Did you wait for heals to complete
before upgrading second node?<br>
</div>
</blockquote>
<br>
</span> no...</div>
</blockquote>
<div><br>
</div>
<div>So basically if you have operations in progress on the
mount, you should wait for heals to complete before you
upgrade second node. If you have all the operations on all
the mounts stopped or you unmounted all the mounts for the
volume, then you can upgrade all the servers one by one
then clients. Otherwise it will lead to problems. That
said in 3 way replica it shouldn't cause split-brains. So
I would like to know exact steps that lead to this
problem. </div>
</div>
</div>
</div>
</blockquote>
<br>
Thank you, this is all I can remember :-(<br>
<br>
<blockquote
cite="mid:CAOgeEnZe8wuyNyv_fdH3NxPvy91bPHgJriwMJN0LwF7rOJRwEw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>We know of one issue which leads to split-brains in
case of VM workloads where we take down bricks in cyclic
manner without waiting for heals to complete. I wonder if
the steps that lead to split-brain on your setup are
similar. We are targetting this for future releases...<br>
</div>
</div>
</div>
</div>
</blockquote>
I guess we hit this...<br>
<br>
<blockquote
cite="mid:CAOgeEnZe8wuyNyv_fdH3NxPvy91bPHgJriwMJN0LwF7rOJRwEw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>
<div class="h5"><br>
<br>
<blockquote type="cite">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Jul 12, 2016 at
3:08 PM, Dmitry Melekhov <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:dm@belkam.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:dm@belkam.com">dm@belkam.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>12.07.2016 13:31, Pranith Kumar
Karampuri пишет:<br>
</div>
<span>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon,
Jul 11, 2016 at 2:26 PM, Dmitry
Melekhov <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:dm@belkam.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:dm@belkam.com">dm@belkam.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">11.07.2016
12:47, Gandalf Corvotempesta
пишет:<span><br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
2016-07-11 9:54 GMT+02:00
Dmitry Melekhov <<a
moz-do-not-send="true"
href="mailto:dm@belkam.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:dm@belkam.com">dm@belkam.com</a></a>>:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
We just got split-brain
during update to 3.7.13
;-)<br>
</blockquote>
This is an interesting
point.<br>
Could you please tell me
which replica count did
you set ?<br>
</blockquote>
<br>
</span> 3<span><br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex"> <br>
With replica "3" split
brain should not occurs,
right ?<br>
</blockquote>
<br>
</span> I guess we did
something wrong :-)</blockquote>
<div><br>
</div>
<div>Or there is a bug we never
found? Could you please share
details about what you did?<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</span> upgraded to 3.7.13 from 3.7.11
using yum, while at least one VM is
running :-)<br>
on all 3 servers, one by one:<br>
<br>
yum upgrade<br>
systemctl stop glusterd <br>
than killed glusterfsd processes using
kill <br>
and systemctl start glusterd<br>
<br>
then next server....<br>
<br>
after this we tried to restart VM, but it
failed, because we forget to restart
libvirtd, and it used old libraries,<br>
I guess this is point where we got this
problem.<span><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div>
<div><br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
I'm planning a new
cluster and I would like
to be protected against<br>
split brains.<br>
</blockquote>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a moz-do-not-send="true"
href="http://www.gluster.org/mailman/listinfo/gluster-users"
rel="noreferrer"
target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div
data-smartmail="gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</span></div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div data-smartmail="gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>