<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hi Micha,</p>
<p>I have changed the thread and subject so that your original
thread remain same for your query. Let's try to fix the problem
what you observed with 3.8.4, So I have started a new thread to
discuss the frequent disconnect problem.</p>
<p><b>If any one else has experienced the same problem, please
respond to the mail.</b><br>
</p>
<p>It would be very helpful if you could give us some more logs from
clients and bricks. Also any reproducible steps will surely help
to chase the problem further.</p>
<p>Regards</p>
<p>Rafi KC<br>
</p>
<div class="moz-cite-prefix">On 11/30/2016 04:44 AM, Micha Ober
wrote:<br>
</div>
<blockquote
cite="mid:CAK9oAHZtNWdXuVCPHPF56+ZuiqAAkE-=YsWvNiEkPn2hGy6WKQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_default">
<div class="gmail_default"><font face="monospace, monospace">I
had opened another thread on this mailing list (Subject:
"After upgrade from 3.4.2 to 3.8.5 - High CPU usage
resulting in disconnects and split-brain").</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">The
title may be a bit misleading now, as I am no longer
observing high CPU usage after upgrading to 3.8.6, but the
disconnects are still happening and the number of files in
split-brain is growing.</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">Setup:
6 compute nodes, each serving as a glusterfs server and
client, Ubuntu 14.04, two bricks per node,
distribute-replicate</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">I
have two gluster volumes set up (one for scratch data, one
for the slurm scheduler). Only the scratch data volume
shows critical errors "[...] has not responded in the last
42 seconds, disconnecting.". So I can rule out network
problems, the gigabit link between the nodes is not
saturated at all. The disks are almost idle (<10%).</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">I
have glusterfs 3.4.2 on Ubuntu 12.04 on a another compute
cluster, running fine since it was deployed.</font></div>
<div class="gmail_default"><font face="monospace, monospace">I
had glusterfs 3.4.2 on Ubuntu 14.04 on this cluster,
running fine for almost a year.</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">After
upgrading to 3.8.5, the problems (as described) started. I
would like to use some of the new features of the newer
versions (like bitrot), but the users can't run their
compute jobs right now because the result files are
garbled.</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">There
also seems to be a bug report with a smiliar problem: (but
no progress)</font></div>
<div class="gmail_default"><font face="monospace, monospace"><a
moz-do-not-send="true"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1370683"><a class="moz-txt-link-freetext" href="https://bugzilla.redhat.com/show_bug.cgi?id=1370683">https://bugzilla.redhat.com/show_bug.cgi?id=1370683</a></a></font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">For
me, ALL servers are affected (not isolated to one or two
servers)</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">I
also see messages like <a class="moz-txt-link-rfc2396E" href="INFO:taskgpu_graphene_bv:4476blockedformorethan120seconds.">"INFO: task gpu_graphene_bv:4476
blocked for more than 120 seconds."</a> in the syslog.</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">For
completeness (gv0 is the scratch volume, gv2 the slurm
volume):</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">[root@giant2:
~]# gluster v info</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">Volume
Name: gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Type:
Distributed-Replicate</font></div>
<div class="gmail_default"><font face="monospace, monospace">Volume
ID: 993ec7c9-e4bc-44d0-b7c4-2d977e622e86</font></div>
<div class="gmail_default"><font face="monospace, monospace">Status:
Started</font></div>
<div class="gmail_default"><font face="monospace, monospace">Snapshot
Count: 0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Number
of Bricks: 6 x 2 = 12</font></div>
<div class="gmail_default"><font face="monospace, monospace">Transport-type:
tcp</font></div>
<div class="gmail_default"><font face="monospace, monospace">Bricks:</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick1:
giant1:/gluster/sdc/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick2:
giant2:/gluster/sdc/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick3:
giant3:/gluster/sdc/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick4:
giant4:/gluster/sdc/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick5:
giant5:/gluster/sdc/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick6:
giant6:/gluster/sdc/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick7:
giant1:/gluster/sdd/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick8:
giant2:/gluster/sdd/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick9:
giant3:/gluster/sdd/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick10:
giant4:/gluster/sdd/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick11:
giant5:/gluster/sdd/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick12:
giant6:/gluster/sdd/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Options
Reconfigured:</font></div>
<div class="gmail_default"><font face="monospace, monospace">auth.allow:
X.X.X.*,127.0.0.1</font></div>
<div class="gmail_default"><font face="monospace, monospace">nfs.disable:
on</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">Volume
Name: gv2</font></div>
<div class="gmail_default"><font face="monospace, monospace">Type:
Replicate</font></div>
<div class="gmail_default"><font face="monospace, monospace">Volume
ID: 30c78928-5f2c-4671-becc-8deaee1a7a8d</font></div>
<div class="gmail_default"><font face="monospace, monospace">Status:
Started</font></div>
<div class="gmail_default"><font face="monospace, monospace">Snapshot
Count: 0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Number
of Bricks: 1 x 2 = 2</font></div>
<div class="gmail_default"><font face="monospace, monospace">Transport-type:
tcp</font></div>
<div class="gmail_default"><font face="monospace, monospace">Bricks:</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick1:
giant1:/gluster/sdd/gv2</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick2:
giant2:/gluster/sdd/gv2</font></div>
<div class="gmail_default"><font face="monospace, monospace">Options
Reconfigured:</font></div>
<div class="gmail_default"><font face="monospace, monospace">auth.allow:
X.X.X.*,127.0.0.1</font></div>
<div class="gmail_default"><font face="monospace, monospace">cluster.granular-entry-heal:
on</font></div>
<div class="gmail_default"><font face="monospace, monospace">cluster.locking-scheme:
granular</font></div>
<div class="gmail_default"><font face="monospace, monospace">nfs.disable:
on</font></div>
<div style="font-family:monospace,monospace"><br>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2016-11-30 0:10 GMT+01:00 Micha Ober <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:micha2k@gmail.com" target="_blank">micha2k@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_default"
style="font-family:monospace,monospace">There also seems
to be a bug report with a smiliar problem: (but no
progress)</div>
<div class="gmail_default"><font face="monospace,
monospace"><a moz-do-not-send="true"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1370683"
target="_blank">https://bugzilla.redhat.com/<wbr>show_bug.cgi?id=1370683</a></font><br>
</div>
<div class="gmail_default"><font face="monospace,
monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace,
monospace">For me, ALL servers are affected (not
isolated to one or two servers)</font></div>
<div class="gmail_default"><font face="monospace,
monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace,
monospace">I also see messages like <a class="moz-txt-link-rfc2396E" href="INFO:taskgpu_graphene_bv:4476blockedformorethan120seconds.">"INFO: task
gpu_graphene_bv:4476 blocked for more than 120
seconds."</a> in the syslog.</font></div>
<div class="gmail_default"><font face="monospace,
monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace,
monospace">For completeness (gv0 is the scratch
volume, gv2 the slurm volume):</font></div>
<div class="gmail_default"><font face="monospace,
monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace,
monospace">
<div class="gmail_default">[root@giant2: ~]# gluster v
info</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">Volume Name: gv0</div>
<div class="gmail_default">Type: Distributed-Replicate</div>
<div class="gmail_default">Volume ID:
993ec7c9-e4bc-44d0-b7c4-<wbr>2d977e622e86</div>
<div class="gmail_default">Status: Started</div>
<div class="gmail_default">Snapshot Count: 0</div>
<div class="gmail_default">Number of Bricks: 6 x 2 =
12</div>
<div class="gmail_default">Transport-type: tcp</div>
<div class="gmail_default">Bricks:</div>
<div class="gmail_default">Brick1:
giant1:/gluster/sdc/gv0</div>
<div class="gmail_default">Brick2:
giant2:/gluster/sdc/gv0</div>
<div class="gmail_default">Brick3:
giant3:/gluster/sdc/gv0</div>
<div class="gmail_default">Brick4:
giant4:/gluster/sdc/gv0</div>
<div class="gmail_default">Brick5:
giant5:/gluster/sdc/gv0</div>
<div class="gmail_default">Brick6:
giant6:/gluster/sdc/gv0</div>
<div class="gmail_default">Brick7:
giant1:/gluster/sdd/gv0</div>
<div class="gmail_default">Brick8:
giant2:/gluster/sdd/gv0</div>
<div class="gmail_default">Brick9:
giant3:/gluster/sdd/gv0</div>
<div class="gmail_default">Brick10:
giant4:/gluster/sdd/gv0</div>
<div class="gmail_default">Brick11:
giant5:/gluster/sdd/gv0</div>
<div class="gmail_default">Brick12:
giant6:/gluster/sdd/gv0</div>
<div class="gmail_default">Options Reconfigured:</div>
<div class="gmail_default">auth.allow:
X.X.X.*,127.0.0.1</div>
<div class="gmail_default">nfs.disable: on</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">Volume Name: gv2</div>
<div class="gmail_default">Type: Replicate</div>
<div class="gmail_default">Volume ID:
30c78928-5f2c-4671-becc-<wbr>8deaee1a7a8d</div>
<div class="gmail_default">Status: Started</div>
<div class="gmail_default">Snapshot Count: 0</div>
<div class="gmail_default">Number of Bricks: 1 x 2 = 2</div>
<div class="gmail_default">Transport-type: tcp</div>
<div class="gmail_default">Bricks:</div>
<div class="gmail_default">Brick1:
giant1:/gluster/sdd/gv2</div>
<div class="gmail_default">Brick2:
giant2:/gluster/sdd/gv2</div>
<div class="gmail_default">Options Reconfigured:</div>
<div class="gmail_default">auth.allow:
X.X.X.*,127.0.0.1</div>
<div class="gmail_default">cluster.granular-entry-heal:
on</div>
<div class="gmail_default">cluster.locking-scheme:
granular</div>
<div class="gmail_default">nfs.disable: on</div>
<div><br>
</div>
</font></div>
</div>
<div class="HOEnZb">
<div class="h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">2016-11-29 19:21 GMT+01:00
Micha Ober <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:micha2k@gmail.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:micha2k@gmail.com">micha2k@gmail.com</a></a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_default"
style="font-family:monospace,monospace">I had
opened another thread on this mailing list
(Subject: "After upgrade from 3.4.2 to 3.8.5 -
High CPU usage resulting in disconnects and
split-brain").</div>
<div class="gmail_default"
style="font-family:monospace,monospace"><br>
</div>
<div class="gmail_default"
style="font-family:monospace,monospace">The
title may be a bit misleading now, as I am no
longer observing high CPU usage after
upgrading to 3.8.6, but the disconnects are
still happening and the number of files in
split-brain is growing.<br>
</div>
<div class="gmail_default"
style="font-family:monospace,monospace"><br>
</div>
<div class="gmail_default"
style="font-family:monospace,monospace">Setup:
6 compute nodes, each serving as a glusterfs
server and client, Ubuntu 14.04, two bricks
per node, distribute-replicate</div>
<div class="gmail_default"
style="font-family:monospace,monospace"><br>
</div>
<div class="gmail_default"
style="font-family:monospace,monospace">I have
two gluster volumes set up (one for scratch
data, one for the slurm scheduler). Only the
scratch data volume shows critical errors
"[...] has not responded in the last 42
seconds, disconnecting.". So I can rule out
network problems, the gigabit link between the
nodes is not saturated at all. The disks are
almost idle (<10%).</div>
<div class="gmail_default"
style="font-family:monospace,monospace"><br>
</div>
<div class="gmail_default"
style="font-family:monospace,monospace">I have
glusterfs 3.4.2 on Ubuntu 12.04 on a another
compute cluster, running fine since it was
deployed.</div>
<div class="gmail_default"
style="font-family:monospace,monospace">I had
glusterfs 3.4.2 on Ubuntu 14.04 on this
cluster, running fine for almost a year.</div>
<div class="gmail_default"
style="font-family:monospace,monospace"><br>
</div>
<div class="gmail_default"
style="font-family:monospace,monospace">After
upgrading to 3.8.5, the problems (as
described) started. I would like to use some
of the new features of the newer versions
(like bitrot), but the users can't run their
compute jobs right now because the result
files are garbled.</div>
</div>
<div class="m_-1578094958703753071HOEnZb">
<div class="m_-1578094958703753071h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">2016-11-29 18:53
GMT+01:00 Atin Mukherjee <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:amukherj@redhat.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:amukherj@redhat.com">amukherj@redhat.com</a></a>></span>:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div style="white-space:pre-wrap">Would you be able to share what is not working for you in 3.8.x (mention the exact version). 3.4 is quite old and falling back to an unsupported version doesn't look a feasible option.</div>
<br>
<div class="gmail_quote">
<div>
<div
class="m_-1578094958703753071m_-2811647508981727209h5">
<div dir="ltr">On Tue, 29 Nov 2016
at 17:01, Micha Ober <<a
moz-do-not-send="true"
href="mailto:micha2k@gmail.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:micha2k@gmail.com">micha2k@gmail.com</a></a>>
wrote:<br>
</div>
</div>
</div>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div>
<div
class="m_-1578094958703753071m_-2811647508981727209h5">
<div dir="ltr"
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">Hi,</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace"><br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">I was using gluster 3.4 and
upgraded to 3.8, but that
version showed to be
unusable for me. I now need
to downgrade.</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace"><br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">I'm running Ubuntu 14.04. As
upgrades of the op version
are irreversible, I guess I
have to delete all gluster
volumes and re-create them
with the downgraded
version. </div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace"><br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">0. Backup data</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">1. Unmount all gluster volumes</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">2. apt-get purge
glusterfs-server
glusterfs-client</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">3. Remove PPA for 3.8</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">4. Add PPA for older version</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">5. apt-get install
glusterfs-server
glusterfs-client</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">6. Create volumes</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace"><br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">Is "purge" enough to delete all
configuration files of the
currently installed version
or do I need to manually
clear some residues before
installing an older version?</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace"><br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">Thanks.</div>
</div>
</div>
</div>
<span>
______________________________<wbr>_________________<br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
Gluster-users mailing list<br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
<a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org"
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
target="_blank">Gluster-users@gluster.org</a><br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
<a moz-do-not-send="true"
href="http://www.gluster.org/mailman/listinfo/gluster-users"
rel="noreferrer"
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a></span></blockquote>
</div>
<span
class="m_-1578094958703753071m_-2811647508981727209HOEnZb"><font
color="#888888">
<div dir="ltr">-- <br>
</div>
<div
data-smartmail="gmail_signature">-
Atin (atinm)</div>
</font></span></blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>