<div dir="ltr"><div class="gmail_default"><div class="gmail_default"><font face="monospace, monospace">I had opened another thread on this mailing list (Subject: "After upgrade from 3.4.2 to 3.8.5 - High CPU usage resulting in disconnects and split-brain").</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">The title may be a bit misleading now, as I am no longer observing high CPU usage after upgrading to 3.8.6, but the disconnects are still happening and the number of files in split-brain is growing.</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">Setup: 6 compute nodes, each serving as a glusterfs server and client, Ubuntu 14.04, two bricks per node, distribute-replicate</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">I have two gluster volumes set up (one for scratch data, one for the slurm scheduler). Only the scratch data volume shows critical errors "[...] has not responded in the last 42 seconds, disconnecting.". So I can rule out network problems, the gigabit link between the nodes is not saturated at all. The disks are almost idle (<10%).</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">I have glusterfs 3.4.2 on Ubuntu 12.04 on a another compute cluster, running fine since it was deployed.</font></div><div class="gmail_default"><font face="monospace, monospace">I had glusterfs 3.4.2 on Ubuntu 14.04 on this cluster, running fine for almost a year.</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">After upgrading to 3.8.5, the problems (as described) started. I would like to use some of the new features of the newer versions (like bitrot), but the users can't run their compute jobs right now because the result files are garbled.</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">There also seems to be a bug report with a smiliar problem: (but no progress)</font></div><div class="gmail_default"><font face="monospace, monospace"><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1370683">https://bugzilla.redhat.com/show_bug.cgi?id=1370683</a></font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">For me, ALL servers are affected (not isolated to one or two servers)</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">I also see messages like "INFO: task gpu_graphene_bv:4476 blocked for more than 120 seconds." in the syslog.</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">For completeness (gv0 is the scratch volume, gv2 the slurm volume):</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">[root@giant2: ~]# gluster v info</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">Volume Name: gv0</font></div><div class="gmail_default"><font face="monospace, monospace">Type: Distributed-Replicate</font></div><div class="gmail_default"><font face="monospace, monospace">Volume ID: 993ec7c9-e4bc-44d0-b7c4-2d977e622e86</font></div><div class="gmail_default"><font face="monospace, monospace">Status: Started</font></div><div class="gmail_default"><font face="monospace, monospace">Snapshot Count: 0</font></div><div class="gmail_default"><font face="monospace, monospace">Number of Bricks: 6 x 2 = 12</font></div><div class="gmail_default"><font face="monospace, monospace">Transport-type: tcp</font></div><div class="gmail_default"><font face="monospace, monospace">Bricks:</font></div><div class="gmail_default"><font face="monospace, monospace">Brick1: giant1:/gluster/sdc/gv0</font></div><div class="gmail_default"><font face="monospace, monospace">Brick2: giant2:/gluster/sdc/gv0</font></div><div class="gmail_default"><font face="monospace, monospace">Brick3: giant3:/gluster/sdc/gv0</font></div><div class="gmail_default"><font face="monospace, monospace">Brick4: giant4:/gluster/sdc/gv0</font></div><div class="gmail_default"><font face="monospace, monospace">Brick5: giant5:/gluster/sdc/gv0</font></div><div class="gmail_default"><font face="monospace, monospace">Brick6: giant6:/gluster/sdc/gv0</font></div><div class="gmail_default"><font face="monospace, monospace">Brick7: giant1:/gluster/sdd/gv0</font></div><div class="gmail_default"><font face="monospace, monospace">Brick8: giant2:/gluster/sdd/gv0</font></div><div class="gmail_default"><font face="monospace, monospace">Brick9: giant3:/gluster/sdd/gv0</font></div><div class="gmail_default"><font face="monospace, monospace">Brick10: giant4:/gluster/sdd/gv0</font></div><div class="gmail_default"><font face="monospace, monospace">Brick11: giant5:/gluster/sdd/gv0</font></div><div class="gmail_default"><font face="monospace, monospace">Brick12: giant6:/gluster/sdd/gv0</font></div><div class="gmail_default"><font face="monospace, monospace">Options Reconfigured:</font></div><div class="gmail_default"><font face="monospace, monospace">auth.allow: X.X.X.*,127.0.0.1</font></div><div class="gmail_default"><font face="monospace, monospace">nfs.disable: on</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">Volume Name: gv2</font></div><div class="gmail_default"><font face="monospace, monospace">Type: Replicate</font></div><div class="gmail_default"><font face="monospace, monospace">Volume ID: 30c78928-5f2c-4671-becc-8deaee1a7a8d</font></div><div class="gmail_default"><font face="monospace, monospace">Status: Started</font></div><div class="gmail_default"><font face="monospace, monospace">Snapshot Count: 0</font></div><div class="gmail_default"><font face="monospace, monospace">Number of Bricks: 1 x 2 = 2</font></div><div class="gmail_default"><font face="monospace, monospace">Transport-type: tcp</font></div><div class="gmail_default"><font face="monospace, monospace">Bricks:</font></div><div class="gmail_default"><font face="monospace, monospace">Brick1: giant1:/gluster/sdd/gv2</font></div><div class="gmail_default"><font face="monospace, monospace">Brick2: giant2:/gluster/sdd/gv2</font></div><div class="gmail_default"><font face="monospace, monospace">Options Reconfigured:</font></div><div class="gmail_default"><font face="monospace, monospace">auth.allow: X.X.X.*,127.0.0.1</font></div><div class="gmail_default"><font face="monospace, monospace">cluster.granular-entry-heal: on</font></div><div class="gmail_default"><font face="monospace, monospace">cluster.locking-scheme: granular</font></div><div class="gmail_default"><font face="monospace, monospace">nfs.disable: on</font></div><div style="font-family:monospace,monospace"><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-11-30 0:10 GMT+01:00 Micha Ober <span dir="ltr"><<a href="mailto:micha2k@gmail.com" target="_blank">micha2k@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">There also seems to be a bug report with a smiliar problem: (but no progress)</div><div class="gmail_default"><font face="monospace, monospace"><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1370683" target="_blank">https://bugzilla.redhat.com/<wbr>show_bug.cgi?id=1370683</a></font><br></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">For me, ALL servers are affected (not isolated to one or two servers)</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">I also see messages like "INFO: task gpu_graphene_bv:4476 blocked for more than 120 seconds." in the syslog.</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">For completeness (gv0 is the scratch volume, gv2 the slurm volume):</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace"><div class="gmail_default">[root@giant2: ~]# gluster v info</div><div class="gmail_default"><br></div><div class="gmail_default">Volume Name: gv0</div><div class="gmail_default">Type: Distributed-Replicate</div><div class="gmail_default">Volume ID: 993ec7c9-e4bc-44d0-b7c4-<wbr>2d977e622e86</div><div class="gmail_default">Status: Started</div><div class="gmail_default">Snapshot Count: 0</div><div class="gmail_default">Number of Bricks: 6 x 2 = 12</div><div class="gmail_default">Transport-type: tcp</div><div class="gmail_default">Bricks:</div><div class="gmail_default">Brick1: giant1:/gluster/sdc/gv0</div><div class="gmail_default">Brick2: giant2:/gluster/sdc/gv0</div><div class="gmail_default">Brick3: giant3:/gluster/sdc/gv0</div><div class="gmail_default">Brick4: giant4:/gluster/sdc/gv0</div><div class="gmail_default">Brick5: giant5:/gluster/sdc/gv0</div><div class="gmail_default">Brick6: giant6:/gluster/sdc/gv0</div><div class="gmail_default">Brick7: giant1:/gluster/sdd/gv0</div><div class="gmail_default">Brick8: giant2:/gluster/sdd/gv0</div><div class="gmail_default">Brick9: giant3:/gluster/sdd/gv0</div><div class="gmail_default">Brick10: giant4:/gluster/sdd/gv0</div><div class="gmail_default">Brick11: giant5:/gluster/sdd/gv0</div><div class="gmail_default">Brick12: giant6:/gluster/sdd/gv0</div><div class="gmail_default">Options Reconfigured:</div><div class="gmail_default">auth.allow: X.X.X.*,127.0.0.1</div><div class="gmail_default">nfs.disable: on</div><div class="gmail_default"><br></div><div class="gmail_default">Volume Name: gv2</div><div class="gmail_default">Type: Replicate</div><div class="gmail_default">Volume ID: 30c78928-5f2c-4671-becc-<wbr>8deaee1a7a8d</div><div class="gmail_default">Status: Started</div><div class="gmail_default">Snapshot Count: 0</div><div class="gmail_default">Number of Bricks: 1 x 2 = 2</div><div class="gmail_default">Transport-type: tcp</div><div class="gmail_default">Bricks:</div><div class="gmail_default">Brick1: giant1:/gluster/sdd/gv2</div><div class="gmail_default">Brick2: giant2:/gluster/sdd/gv2</div><div class="gmail_default">Options Reconfigured:</div><div class="gmail_default">auth.allow: X.X.X.*,127.0.0.1</div><div class="gmail_default">cluster.granular-entry-heal: on</div><div class="gmail_default">cluster.locking-scheme: granular</div><div class="gmail_default">nfs.disable: on</div><div><br></div></font></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">2016-11-29 19:21 GMT+01:00 Micha Ober <span dir="ltr"><<a href="mailto:micha2k@gmail.com" target="_blank">micha2k@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">I had opened another thread on this mailing list (Subject: "After upgrade from 3.4.2 to 3.8.5 - High CPU usage resulting in disconnects and split-brain").</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">The title may be a bit misleading now, as I am no longer observing high CPU usage after upgrading to 3.8.6, but the disconnects are still happening and the number of files in split-brain is growing.<br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">Setup: 6 compute nodes, each serving as a glusterfs server and client, Ubuntu 14.04, two bricks per node, distribute-replicate</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">I have two gluster volumes set up (one for scratch data, one for the slurm scheduler). Only the scratch data volume shows critical errors "[...] has not responded in the last 42 seconds, disconnecting.". So I can rule out network problems, the gigabit link between the nodes is not saturated at all. The disks are almost idle (<10%).</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">I have glusterfs 3.4.2 on Ubuntu 12.04 on a another compute cluster, running fine since it was deployed.</div><div class="gmail_default" style="font-family:monospace,monospace">I had glusterfs 3.4.2 on Ubuntu 14.04 on this cluster, running fine for almost a year.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">After upgrading to 3.8.5, the problems (as described) started. I would like to use some of the new features of the newer versions (like bitrot), but the users can't run their compute jobs right now because the result files are garbled.</div></div><div class="m_-1578094958703753071HOEnZb"><div class="m_-1578094958703753071h5"><div class="gmail_extra"><br><div class="gmail_quote">2016-11-29 18:53 GMT+01:00 Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="white-space:pre-wrap">Would you be able to share what is not working for you in 3.8.x (mention the exact version). 3.4 is quite old and falling back to an unsupported version doesn't look a feasible option.</div><br><div class="gmail_quote"><div><div class="m_-1578094958703753071m_-2811647508981727209h5"><div dir="ltr">On Tue, 29 Nov 2016 at 17:01, Micha Ober <<a href="mailto:micha2k@gmail.com" target="_blank">micha2k@gmail.com</a>> wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_-1578094958703753071m_-2811647508981727209h5"><div dir="ltr" class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"><div class="gmail_default m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" style="font-family:monospace,monospace">Hi,</div><div class="gmail_default m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" style="font-family:monospace,monospace"><br class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"></div><div class="gmail_default m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" style="font-family:monospace,monospace">I was using gluster 3.4 and upgraded to 3.8, but that version showed to be unusable for me. I now need to downgrade.</div><div class="gmail_default m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" style="font-family:monospace,monospace"><br class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"></div><div class="gmail_default m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" style="font-family:monospace,monospace">I'm running Ubuntu 14.04. As upgrades of the op version are irreversible, I guess I have to delete all gluster volumes and re-create them with the downgraded version. </div><div class="gmail_default m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" style="font-family:monospace,monospace"><br class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"></div><div class="gmail_default m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" style="font-family:monospace,monospace">0. Backup data</div><div class="gmail_default m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" style="font-family:monospace,monospace">1. Unmount all gluster volumes</div><div class="gmail_default m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" style="font-family:monospace,monospace">2. apt-get purge glusterfs-server glusterfs-client</div><div class="gmail_default m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" style="font-family:monospace,monospace">3. Remove PPA for 3.8</div><div class="gmail_default m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" style="font-family:monospace,monospace">4. Add PPA for older version</div><div class="gmail_default m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" style="font-family:monospace,monospace">5. apt-get install glusterfs-server glusterfs-client</div><div class="gmail_default m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" style="font-family:monospace,monospace">6. Create volumes</div><div class="gmail_default m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" style="font-family:monospace,monospace"><br class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"></div><div class="gmail_default m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" style="font-family:monospace,monospace">Is "purge" enough to delete all configuration files of the currently installed version or do I need to manually clear some residues before installing an older version?</div><div class="gmail_default m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" style="font-family:monospace,monospace"><br class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"></div><div class="gmail_default m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" style="font-family:monospace,monospace">Thanks.</div></div></div></div><span>
______________________________<wbr>_________________<br class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
Gluster-users mailing list<br class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
<a href="mailto:Gluster-users@gluster.org" class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" target="_blank">Gluster-users@gluster.org</a><br class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a></span></blockquote></div><span class="m_-1578094958703753071m_-2811647508981727209HOEnZb"><font color="#888888"><div dir="ltr">-- <br></div><div data-smartmail="gmail_signature">- Atin (atinm)</div>
</font></span></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>