<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi,<br>
<br>
as 6 servers with 12 bricks produce a lot of log files, I have
uploaded the last 200 lines of a client on one server here:<br>
<a class="moz-txt-link-freetext" href="http://paste.ubuntu.com/23558816/">http://paste.ubuntu.com/23558816/</a><br>
<br>
When greping the C(ritical) messages, there is for example this
one:<br>
[2016-11-30 12:01:06.813333] C
[rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-gv0-client-3:
server X.X.X.107:49154 has not responded in the last 42 seconds,
disconnecting.<br>
<br>
For client-3, which is giant4:/gluster/sdc/gv0, I have uploaded
the log for this brick here:<br>
<a class="moz-txt-link-freetext" href="http://paste.ubuntu.com/23558818/">http://paste.ubuntu.com/23558818/</a><br>
<br>
It's hard to tell how to reproduce this issue other than "put load
on the servers/clients".<br>
There are GPGPU compute jobs running on the nodes, but those only
consume 4 of the 6 CPU cores.<br>
The servers are not overloaded. All of them have 16 GB RAM and
most of it is empty.<br>
The load on the disks is also very small (<10% according to
iostat)<br>
<br>
What other information/logs can I provide?<br>
<br>
Thanks,<br>
Micha<br>
<br>
Am 30.11.2016 um 06:57 schrieb Mohammed Rafi K C:<br>
</div>
<blockquote
cite="mid:bd45cecb-374a-aebe-5f21-2704215df66c@redhat.com"
type="cite">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
<p>Hi Micha,</p>
<p>I have changed the thread and subject so that your original
thread remain same for your query. Let's try to fix the problem
what you observed with 3.8.4, So I have started a new thread to
discuss the frequent disconnect problem.</p>
<p><b>If any one else has experienced the same problem, please
respond to the mail.</b><br>
</p>
<p>It would be very helpful if you could give us some more logs
from clients and bricks. Also any reproducible steps will
surely help to chase the problem further.</p>
<p>Regards</p>
<p>Rafi KC<br>
</p>
<div class="moz-cite-prefix">On 11/30/2016 04:44 AM, Micha Ober
wrote:<br>
</div>
<blockquote
cite="mid:CAK9oAHZtNWdXuVCPHPF56+ZuiqAAkE-=YsWvNiEkPn2hGy6WKQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_default">
<div class="gmail_default"><font face="monospace, monospace">I
had opened another thread on this mailing list (Subject:
"After upgrade from 3.4.2 to 3.8.5 - High CPU usage
resulting in disconnects and split-brain").</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">The
title may be a bit misleading now, as I am no longer
observing high CPU usage after upgrading to 3.8.6, but
the disconnects are still happening and the number of
files in split-brain is growing.</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">Setup:
6 compute nodes, each serving as a glusterfs server and
client, Ubuntu 14.04, two bricks per node,
distribute-replicate</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">I
have two gluster volumes set up (one for scratch data,
one for the slurm scheduler). Only the scratch data
volume shows critical errors "[...] has not responded in
the last 42 seconds, disconnecting.". So I can rule out
network problems, the gigabit link between the nodes is
not saturated at all. The disks are almost idle
(<10%).</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">I
have glusterfs 3.4.2 on Ubuntu 12.04 on a another
compute cluster, running fine since it was deployed.</font></div>
<div class="gmail_default"><font face="monospace, monospace">I
had glusterfs 3.4.2 on Ubuntu 14.04 on this cluster,
running fine for almost a year.</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">After
upgrading to 3.8.5, the problems (as described) started.
I would like to use some of the new features of the
newer versions (like bitrot), but the users can't run
their compute jobs right now because the result files
are garbled.</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">There
also seems to be a bug report with a smiliar problem:
(but no progress)</font></div>
<div class="gmail_default"><font face="monospace, monospace"><a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1370683">https://bugzilla.redhat.com/show_bug.cgi?id=1370683</a></font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">For
me, ALL servers are affected (not isolated to one or two
servers)</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">I
also see messages like <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="INFO:taskgpu_graphene_bv:4476blockedformorethan120seconds.">"INFO:
task gpu_graphene_bv:4476 blocked for more than 120
seconds."</a> in the syslog.</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">For
completeness (gv0 is the scratch volume, gv2 the slurm
volume):</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">[root@giant2:
~]# gluster v info</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">Volume
Name: gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Type:
Distributed-Replicate</font></div>
<div class="gmail_default"><font face="monospace, monospace">Volume
ID: 993ec7c9-e4bc-44d0-b7c4-2d977e622e86</font></div>
<div class="gmail_default"><font face="monospace, monospace">Status:
Started</font></div>
<div class="gmail_default"><font face="monospace, monospace">Snapshot
Count: 0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Number
of Bricks: 6 x 2 = 12</font></div>
<div class="gmail_default"><font face="monospace, monospace">Transport-type:
tcp</font></div>
<div class="gmail_default"><font face="monospace, monospace">Bricks:</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick1:
giant1:/gluster/sdc/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick2:
giant2:/gluster/sdc/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick3:
giant3:/gluster/sdc/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick4:
giant4:/gluster/sdc/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick5:
giant5:/gluster/sdc/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick6:
giant6:/gluster/sdc/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick7:
giant1:/gluster/sdd/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick8:
giant2:/gluster/sdd/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick9:
giant3:/gluster/sdd/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick10:
giant4:/gluster/sdd/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick11:
giant5:/gluster/sdd/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick12:
giant6:/gluster/sdd/gv0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Options
Reconfigured:</font></div>
<div class="gmail_default"><font face="monospace, monospace">auth.allow:
X.X.X.*,127.0.0.1</font></div>
<div class="gmail_default"><font face="monospace, monospace">nfs.disable:
on</font></div>
<div class="gmail_default"><font face="monospace, monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace, monospace">Volume
Name: gv2</font></div>
<div class="gmail_default"><font face="monospace, monospace">Type:
Replicate</font></div>
<div class="gmail_default"><font face="monospace, monospace">Volume
ID: 30c78928-5f2c-4671-becc-8deaee1a7a8d</font></div>
<div class="gmail_default"><font face="monospace, monospace">Status:
Started</font></div>
<div class="gmail_default"><font face="monospace, monospace">Snapshot
Count: 0</font></div>
<div class="gmail_default"><font face="monospace, monospace">Number
of Bricks: 1 x 2 = 2</font></div>
<div class="gmail_default"><font face="monospace, monospace">Transport-type:
tcp</font></div>
<div class="gmail_default"><font face="monospace, monospace">Bricks:</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick1:
giant1:/gluster/sdd/gv2</font></div>
<div class="gmail_default"><font face="monospace, monospace">Brick2:
giant2:/gluster/sdd/gv2</font></div>
<div class="gmail_default"><font face="monospace, monospace">Options
Reconfigured:</font></div>
<div class="gmail_default"><font face="monospace, monospace">auth.allow:
X.X.X.*,127.0.0.1</font></div>
<div class="gmail_default"><font face="monospace, monospace">cluster.granular-entry-heal:
on</font></div>
<div class="gmail_default"><font face="monospace, monospace">cluster.locking-scheme:
granular</font></div>
<div class="gmail_default"><font face="monospace, monospace">nfs.disable:
on</font></div>
<div style="font-family:monospace,monospace"><br>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2016-11-30 0:10 GMT+01:00 Micha Ober
<span dir="ltr"><<a moz-do-not-send="true"
href="mailto:micha2k@gmail.com" target="_blank">micha2k@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_default"
style="font-family:monospace,monospace">There also
seems to be a bug report with a smiliar problem: (but
no progress)</div>
<div class="gmail_default"><font face="monospace,
monospace"><a moz-do-not-send="true"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1370683"
target="_blank">https://bugzilla.redhat.com/<wbr>show_bug.cgi?id=1370683</a></font><br>
</div>
<div class="gmail_default"><font face="monospace,
monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace,
monospace">For me, ALL servers are affected (not
isolated to one or two servers)</font></div>
<div class="gmail_default"><font face="monospace,
monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace,
monospace">I also see messages like <a
moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="INFO:taskgpu_graphene_bv:4476blockedformorethan120seconds.">"INFO:
task gpu_graphene_bv:4476 blocked for more than
120 seconds."</a> in the syslog.</font></div>
<div class="gmail_default"><font face="monospace,
monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace,
monospace">For completeness (gv0 is the scratch
volume, gv2 the slurm volume):</font></div>
<div class="gmail_default"><font face="monospace,
monospace"><br>
</font></div>
<div class="gmail_default"><font face="monospace,
monospace">
<div class="gmail_default">[root@giant2: ~]# gluster
v info</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">Volume Name: gv0</div>
<div class="gmail_default">Type:
Distributed-Replicate</div>
<div class="gmail_default">Volume ID:
993ec7c9-e4bc-44d0-b7c4-<wbr>2d977e622e86</div>
<div class="gmail_default">Status: Started</div>
<div class="gmail_default">Snapshot Count: 0</div>
<div class="gmail_default">Number of Bricks: 6 x 2 =
12</div>
<div class="gmail_default">Transport-type: tcp</div>
<div class="gmail_default">Bricks:</div>
<div class="gmail_default">Brick1:
giant1:/gluster/sdc/gv0</div>
<div class="gmail_default">Brick2:
giant2:/gluster/sdc/gv0</div>
<div class="gmail_default">Brick3:
giant3:/gluster/sdc/gv0</div>
<div class="gmail_default">Brick4:
giant4:/gluster/sdc/gv0</div>
<div class="gmail_default">Brick5:
giant5:/gluster/sdc/gv0</div>
<div class="gmail_default">Brick6:
giant6:/gluster/sdc/gv0</div>
<div class="gmail_default">Brick7:
giant1:/gluster/sdd/gv0</div>
<div class="gmail_default">Brick8:
giant2:/gluster/sdd/gv0</div>
<div class="gmail_default">Brick9:
giant3:/gluster/sdd/gv0</div>
<div class="gmail_default">Brick10:
giant4:/gluster/sdd/gv0</div>
<div class="gmail_default">Brick11:
giant5:/gluster/sdd/gv0</div>
<div class="gmail_default">Brick12:
giant6:/gluster/sdd/gv0</div>
<div class="gmail_default">Options Reconfigured:</div>
<div class="gmail_default">auth.allow:
X.X.X.*,127.0.0.1</div>
<div class="gmail_default">nfs.disable: on</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">Volume Name: gv2</div>
<div class="gmail_default">Type: Replicate</div>
<div class="gmail_default">Volume ID:
30c78928-5f2c-4671-becc-<wbr>8deaee1a7a8d</div>
<div class="gmail_default">Status: Started</div>
<div class="gmail_default">Snapshot Count: 0</div>
<div class="gmail_default">Number of Bricks: 1 x 2 =
2</div>
<div class="gmail_default">Transport-type: tcp</div>
<div class="gmail_default">Bricks:</div>
<div class="gmail_default">Brick1:
giant1:/gluster/sdd/gv2</div>
<div class="gmail_default">Brick2:
giant2:/gluster/sdd/gv2</div>
<div class="gmail_default">Options Reconfigured:</div>
<div class="gmail_default">auth.allow:
X.X.X.*,127.0.0.1</div>
<div class="gmail_default">cluster.granular-entry-heal:
on</div>
<div class="gmail_default">cluster.locking-scheme:
granular</div>
<div class="gmail_default">nfs.disable: on</div>
<div><br>
</div>
</font></div>
</div>
<div class="HOEnZb">
<div class="h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">2016-11-29 19:21 GMT+01:00
Micha Ober <span dir="ltr"><<a
moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:micha2k@gmail.com">micha2k@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_default"
style="font-family:monospace,monospace">I
had opened another thread on this mailing
list (Subject: "After upgrade from 3.4.2 to
3.8.5 - High CPU usage resulting in
disconnects and split-brain").</div>
<div class="gmail_default"
style="font-family:monospace,monospace"><br>
</div>
<div class="gmail_default"
style="font-family:monospace,monospace">The
title may be a bit misleading now, as I am
no longer observing high CPU usage after
upgrading to 3.8.6, but the disconnects are
still happening and the number of files in
split-brain is growing.<br>
</div>
<div class="gmail_default"
style="font-family:monospace,monospace"><br>
</div>
<div class="gmail_default"
style="font-family:monospace,monospace">Setup:
6 compute nodes, each serving as a glusterfs
server and client, Ubuntu 14.04, two bricks
per node, distribute-replicate</div>
<div class="gmail_default"
style="font-family:monospace,monospace"><br>
</div>
<div class="gmail_default"
style="font-family:monospace,monospace">I
have two gluster volumes set up (one for
scratch data, one for the slurm scheduler).
Only the scratch data volume shows critical
errors "[...] has not responded in the last
42 seconds, disconnecting.". So I can rule
out network problems, the gigabit link
between the nodes is not saturated at all.
The disks are almost idle (<10%).</div>
<div class="gmail_default"
style="font-family:monospace,monospace"><br>
</div>
<div class="gmail_default"
style="font-family:monospace,monospace">I
have glusterfs 3.4.2 on Ubuntu 12.04 on a
another compute cluster, running fine since
it was deployed.</div>
<div class="gmail_default"
style="font-family:monospace,monospace">I
had glusterfs 3.4.2 on Ubuntu 14.04 on this
cluster, running fine for almost a year.</div>
<div class="gmail_default"
style="font-family:monospace,monospace"><br>
</div>
<div class="gmail_default"
style="font-family:monospace,monospace">After
upgrading to 3.8.5, the problems (as
described) started. I would like to use some
of the new features of the newer versions
(like bitrot), but the users can't run their
compute jobs right now because the result
files are garbled.</div>
</div>
<div class="m_-1578094958703753071HOEnZb">
<div class="m_-1578094958703753071h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">2016-11-29 18:53
GMT+01:00 Atin Mukherjee <span
dir="ltr"><<a
moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:amukherj@redhat.com">amukherj@redhat.com</a>></span>:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div style="white-space:pre-wrap">Would you be able to share what is not working for you in 3.8.x (mention the exact version). 3.4 is quite old and falling back to an unsupported version doesn't look a feasible option.</div>
<br>
<div class="gmail_quote">
<div>
<div
class="m_-1578094958703753071m_-2811647508981727209h5">
<div dir="ltr">On Tue, 29 Nov
2016 at 17:01, Micha Ober <<a
moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:micha2k@gmail.com">micha2k@gmail.com</a>> wrote:<br>
</div>
</div>
</div>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div>
<div
class="m_-1578094958703753071m_-2811647508981727209h5">
<div dir="ltr"
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">Hi,</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace"><br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">I was using gluster 3.4 and
upgraded to 3.8, but that
version showed to be
unusable for me. I now
need to downgrade.</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace"><br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">I'm running Ubuntu 14.04. As
upgrades of the op version
are irreversible, I guess
I have to delete all
gluster volumes and
re-create them with the
downgraded version. </div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace"><br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">0. Backup data</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">1. Unmount all gluster volumes</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">2. apt-get purge
glusterfs-server
glusterfs-client</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">3. Remove PPA for 3.8</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">4. Add PPA for older version</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">5. apt-get install
glusterfs-server
glusterfs-client</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">6. Create volumes</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace"><br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">Is "purge" enough to delete all
configuration files of the
currently installed
version or do I need to
manually clear some
residues before installing
an older version?</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace"><br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
</div>
<div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">Thanks.</div>
</div>
</div>
</div>
<span>
______________________________<wbr>_________________<br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
Gluster-users mailing list<br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
<a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org"
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
target="_blank">Gluster-users@gluster.org</a><br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
<a moz-do-not-send="true"
href="http://www.gluster.org/mailman/listinfo/gluster-users"
rel="noreferrer"
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a></span></blockquote>
</div>
<span
class="m_-1578094958703753071m_-2811647508981727209HOEnZb"><font
color="#888888">
<div dir="ltr">-- <br>
</div>
<div
data-smartmail="gmail_signature">-
Atin (atinm)</div>
</font></span></blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</blockquote>
<p><br>
</p>
</body>
</html>