<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 15, 2016 at 4:33 AM, Micha Ober <span dir="ltr"><<a href="mailto:micha2k@gmail.com" target="_blank">micha2k@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div style="font-family:monospace,monospace">Hi,</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">I upgraded an installation of GlusterFS on Ubuntu 14.04.3 from version 3.4.2 to 3.8.5.</div><div style="font-family:monospace,monospace">Few hours after the upgrade, I noticed files in "split-brain" state. I never had split-brain files in months of operation before, with the old version.</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">Using htop, I observed the "glusterfs" process jumping from 0% to 100+% CPU usage every now and then.</div><div style="font-family:monospace,monospace">Using iostat, I confirmed there is no bottleneck on the local disks (util is well below 10%)</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">Inspecting the logfiles, it looks like clients are losing connection quite often:</div><div style="font-family:monospace,monospace"><br></div><div><div style="font-family:monospace,monospace">[2016-11-14 16:34:56.685349] C [rpc-clnt-ping.c:160:rpc_clnt_<wbr>ping_timer_expired] 0-gv0-client-1: server X.X.X.62:49152 has not responded in the last 42 seconds, disconnecting.</div><div style="font-family:monospace,monospace">[2016-11-14 16:35:47.690348] C [rpc-clnt-ping.c:160:rpc_clnt_<wbr>ping_timer_expired] 0-gv0-client-8: server X.X.X.219:49153 has not responded in the last 42 seconds, disconnecting.</div><div style="font-family:monospace,monospace">[2016-11-14 17:09:33.903096] C [rpc-clnt-ping.c:160:rpc_clnt_<wbr>ping_timer_expired] 0-gv0-client-7: server X.X.X.62:49153 has not responded in the last 42 seconds, disconnecting.</div></div></div></blockquote><div><br></div><div>You need to find out what is leading to the disconnects above. These could be the reason for split-brain.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">There are a total of 6 servers with 2 bricks each (Distribute-Replicate)</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">The result of a 60 second gluster volume profile can be seen here: <a href="http://pastebin.com/5WN5S63B" target="_blank">http://pastebin.com/<wbr>5WN5S63B</a></div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">After upgrading, I set:</div><div style="font-family:monospace,monospace"><br></div><div><div><font face="monospace, monospace">cluster.granular-entry-heal yes</font></div><div><font face="monospace, monospace">cluster.locking-scheme granular</font></div></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">I now reverted to no/full to see if files are still going "split-brain".</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">Best regards</font></div></div></div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>