<html>
  <head>
    <meta content="text/html; charset=windows-1252"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <p>Hi Micha,</p>
    <p>I have changed the thread and subject so that your original
      thread remain same for your query. Let's try to fix the problem
      what you observed with 3.8.4, So I have started a new thread to
      discuss the frequent disconnect problem.</p>
    <p><b>If any one else has experienced the same problem, please
        respond to the mail.</b><br>
    </p>
    <p>It would be very helpful if you could give us some more logs from
      clients and bricks.  Also any reproducible steps will surely help
      to chase the problem further.</p>
    <p>Regards</p>
    <p>Rafi KC<br>
    </p>
    <div class="moz-cite-prefix">On 11/30/2016 04:44 AM, Micha Ober
      wrote:<br>
    </div>
    <blockquote
cite="mid:CAK9oAHZtNWdXuVCPHPF56+ZuiqAAkE-=YsWvNiEkPn2hGy6WKQ@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_default">
          <div class="gmail_default"><font face="monospace, monospace">I
              had opened another thread on this mailing list (Subject:
              "After upgrade from 3.4.2 to 3.8.5 - High CPU usage
              resulting in disconnects and split-brain").</font></div>
          <div class="gmail_default"><font face="monospace, monospace"><br>
            </font></div>
          <div class="gmail_default"><font face="monospace, monospace">The
              title may be a bit misleading now, as I am no longer
              observing high CPU usage after upgrading to 3.8.6, but the
              disconnects are still happening and the number of files in
              split-brain is growing.</font></div>
          <div class="gmail_default"><font face="monospace, monospace"><br>
            </font></div>
          <div class="gmail_default"><font face="monospace, monospace">Setup:
              6 compute nodes, each serving as a glusterfs server and
              client, Ubuntu 14.04, two bricks per node,
              distribute-replicate</font></div>
          <div class="gmail_default"><font face="monospace, monospace"><br>
            </font></div>
          <div class="gmail_default"><font face="monospace, monospace">I
              have two gluster volumes set up (one for scratch data, one
              for the slurm scheduler). Only the scratch data volume
              shows critical errors "[...] has not responded in the last
              42 seconds, disconnecting.". So I can rule out network
              problems, the gigabit link between the nodes is not
              saturated at all. The disks are almost idle (&lt;10%).</font></div>
          <div class="gmail_default"><font face="monospace, monospace"><br>
            </font></div>
          <div class="gmail_default"><font face="monospace, monospace">I
              have glusterfs 3.4.2 on Ubuntu 12.04 on a another compute
              cluster, running fine since it was deployed.</font></div>
          <div class="gmail_default"><font face="monospace, monospace">I
              had glusterfs 3.4.2 on Ubuntu 14.04 on this cluster,
              running fine for almost a year.</font></div>
          <div class="gmail_default"><font face="monospace, monospace"><br>
            </font></div>
          <div class="gmail_default"><font face="monospace, monospace">After
              upgrading to 3.8.5, the problems (as described) started. I
              would like to use some of the new features of the newer
              versions (like bitrot), but the users can't run their
              compute jobs right now because the result files are
              garbled.</font></div>
          <div class="gmail_default"><font face="monospace, monospace"><br>
            </font></div>
          <div class="gmail_default"><font face="monospace, monospace">There
              also seems to be a bug report with a smiliar problem: (but
              no progress)</font></div>
          <div class="gmail_default"><font face="monospace, monospace"><a
                moz-do-not-send="true"
                href="https://bugzilla.redhat.com/show_bug.cgi?id=1370683"><a class="moz-txt-link-freetext" href="https://bugzilla.redhat.com/show_bug.cgi?id=1370683">https://bugzilla.redhat.com/show_bug.cgi?id=1370683</a></a></font></div>
          <div class="gmail_default"><font face="monospace, monospace"><br>
            </font></div>
          <div class="gmail_default"><font face="monospace, monospace">For
              me, ALL servers are affected (not isolated to one or two
              servers)</font></div>
          <div class="gmail_default"><font face="monospace, monospace"><br>
            </font></div>
          <div class="gmail_default"><font face="monospace, monospace">I
              also see messages like <a class="moz-txt-link-rfc2396E" href="INFO:taskgpu_graphene_bv:4476blockedformorethan120seconds.">"INFO: task gpu_graphene_bv:4476
              blocked for more than 120 seconds."</a> in the syslog.</font></div>
          <div class="gmail_default"><font face="monospace, monospace"><br>
            </font></div>
          <div class="gmail_default"><font face="monospace, monospace">For
              completeness (gv0 is the scratch volume, gv2 the slurm
              volume):</font></div>
          <div class="gmail_default"><font face="monospace, monospace"><br>
            </font></div>
          <div class="gmail_default"><font face="monospace, monospace">[root@giant2:
              ~]# gluster v info</font></div>
          <div class="gmail_default"><font face="monospace, monospace"><br>
            </font></div>
          <div class="gmail_default"><font face="monospace, monospace">Volume
              Name: gv0</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Type:
              Distributed-Replicate</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Volume
              ID: 993ec7c9-e4bc-44d0-b7c4-2d977e622e86</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Status:
              Started</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Snapshot
              Count: 0</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Number
              of Bricks: 6 x 2 = 12</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Transport-type:
              tcp</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Bricks:</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Brick1:
              giant1:/gluster/sdc/gv0</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Brick2:
              giant2:/gluster/sdc/gv0</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Brick3:
              giant3:/gluster/sdc/gv0</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Brick4:
              giant4:/gluster/sdc/gv0</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Brick5:
              giant5:/gluster/sdc/gv0</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Brick6:
              giant6:/gluster/sdc/gv0</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Brick7:
              giant1:/gluster/sdd/gv0</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Brick8:
              giant2:/gluster/sdd/gv0</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Brick9:
              giant3:/gluster/sdd/gv0</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Brick10:
              giant4:/gluster/sdd/gv0</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Brick11:
              giant5:/gluster/sdd/gv0</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Brick12:
              giant6:/gluster/sdd/gv0</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Options
              Reconfigured:</font></div>
          <div class="gmail_default"><font face="monospace, monospace">auth.allow:
              X.X.X.*,127.0.0.1</font></div>
          <div class="gmail_default"><font face="monospace, monospace">nfs.disable:
              on</font></div>
          <div class="gmail_default"><font face="monospace, monospace"><br>
            </font></div>
          <div class="gmail_default"><font face="monospace, monospace">Volume
              Name: gv2</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Type:
              Replicate</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Volume
              ID: 30c78928-5f2c-4671-becc-8deaee1a7a8d</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Status:
              Started</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Snapshot
              Count: 0</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Number
              of Bricks: 1 x 2 = 2</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Transport-type:
              tcp</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Bricks:</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Brick1:
              giant1:/gluster/sdd/gv2</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Brick2:
              giant2:/gluster/sdd/gv2</font></div>
          <div class="gmail_default"><font face="monospace, monospace">Options
              Reconfigured:</font></div>
          <div class="gmail_default"><font face="monospace, monospace">auth.allow:
              X.X.X.*,127.0.0.1</font></div>
          <div class="gmail_default"><font face="monospace, monospace">cluster.granular-entry-heal:
              on</font></div>
          <div class="gmail_default"><font face="monospace, monospace">cluster.locking-scheme:
              granular</font></div>
          <div class="gmail_default"><font face="monospace, monospace">nfs.disable:
              on</font></div>
          <div style="font-family:monospace,monospace"><br>
          </div>
        </div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">2016-11-30 0:10 GMT+01:00 Micha Ober <span
            dir="ltr">&lt;<a moz-do-not-send="true"
              href="mailto:micha2k@gmail.com" target="_blank">micha2k@gmail.com</a>&gt;</span>:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div dir="ltr">
              <div class="gmail_default"
                style="font-family:monospace,monospace">There also seems
                to be a bug report with a smiliar problem: (but no
                progress)</div>
              <div class="gmail_default"><font face="monospace,
                  monospace"><a moz-do-not-send="true"
                    href="https://bugzilla.redhat.com/show_bug.cgi?id=1370683"
                    target="_blank">https://bugzilla.redhat.com/<wbr>show_bug.cgi?id=1370683</a></font><br>
              </div>
              <div class="gmail_default"><font face="monospace,
                  monospace"><br>
                </font></div>
              <div class="gmail_default"><font face="monospace,
                  monospace">For me, ALL servers are affected (not
                  isolated to one or two servers)</font></div>
              <div class="gmail_default"><font face="monospace,
                  monospace"><br>
                </font></div>
              <div class="gmail_default"><font face="monospace,
                  monospace">I also see messages like <a class="moz-txt-link-rfc2396E" href="INFO:taskgpu_graphene_bv:4476blockedformorethan120seconds.">"INFO: task
                  gpu_graphene_bv:4476 blocked for more than 120
                  seconds."</a> in the syslog.</font></div>
              <div class="gmail_default"><font face="monospace,
                  monospace"><br>
                </font></div>
              <div class="gmail_default"><font face="monospace,
                  monospace">For completeness (gv0 is the scratch
                  volume, gv2 the slurm volume):</font></div>
              <div class="gmail_default"><font face="monospace,
                  monospace"><br>
                </font></div>
              <div class="gmail_default"><font face="monospace,
                  monospace">
                  <div class="gmail_default">[root@giant2: ~]# gluster v
                    info</div>
                  <div class="gmail_default"><br>
                  </div>
                  <div class="gmail_default">Volume Name: gv0</div>
                  <div class="gmail_default">Type: Distributed-Replicate</div>
                  <div class="gmail_default">Volume ID:
                    993ec7c9-e4bc-44d0-b7c4-<wbr>2d977e622e86</div>
                  <div class="gmail_default">Status: Started</div>
                  <div class="gmail_default">Snapshot Count: 0</div>
                  <div class="gmail_default">Number of Bricks: 6 x 2 =
                    12</div>
                  <div class="gmail_default">Transport-type: tcp</div>
                  <div class="gmail_default">Bricks:</div>
                  <div class="gmail_default">Brick1:
                    giant1:/gluster/sdc/gv0</div>
                  <div class="gmail_default">Brick2:
                    giant2:/gluster/sdc/gv0</div>
                  <div class="gmail_default">Brick3:
                    giant3:/gluster/sdc/gv0</div>
                  <div class="gmail_default">Brick4:
                    giant4:/gluster/sdc/gv0</div>
                  <div class="gmail_default">Brick5:
                    giant5:/gluster/sdc/gv0</div>
                  <div class="gmail_default">Brick6:
                    giant6:/gluster/sdc/gv0</div>
                  <div class="gmail_default">Brick7:
                    giant1:/gluster/sdd/gv0</div>
                  <div class="gmail_default">Brick8:
                    giant2:/gluster/sdd/gv0</div>
                  <div class="gmail_default">Brick9:
                    giant3:/gluster/sdd/gv0</div>
                  <div class="gmail_default">Brick10:
                    giant4:/gluster/sdd/gv0</div>
                  <div class="gmail_default">Brick11:
                    giant5:/gluster/sdd/gv0</div>
                  <div class="gmail_default">Brick12:
                    giant6:/gluster/sdd/gv0</div>
                  <div class="gmail_default">Options Reconfigured:</div>
                  <div class="gmail_default">auth.allow:
                    X.X.X.*,127.0.0.1</div>
                  <div class="gmail_default">nfs.disable: on</div>
                  <div class="gmail_default"><br>
                  </div>
                  <div class="gmail_default">Volume Name: gv2</div>
                  <div class="gmail_default">Type: Replicate</div>
                  <div class="gmail_default">Volume ID:
                    30c78928-5f2c-4671-becc-<wbr>8deaee1a7a8d</div>
                  <div class="gmail_default">Status: Started</div>
                  <div class="gmail_default">Snapshot Count: 0</div>
                  <div class="gmail_default">Number of Bricks: 1 x 2 = 2</div>
                  <div class="gmail_default">Transport-type: tcp</div>
                  <div class="gmail_default">Bricks:</div>
                  <div class="gmail_default">Brick1:
                    giant1:/gluster/sdd/gv2</div>
                  <div class="gmail_default">Brick2:
                    giant2:/gluster/sdd/gv2</div>
                  <div class="gmail_default">Options Reconfigured:</div>
                  <div class="gmail_default">auth.allow:
                    X.X.X.*,127.0.0.1</div>
                  <div class="gmail_default">cluster.granular-entry-heal:
                    on</div>
                  <div class="gmail_default">cluster.locking-scheme:
                    granular</div>
                  <div class="gmail_default">nfs.disable: on</div>
                  <div><br>
                  </div>
                </font></div>
            </div>
            <div class="HOEnZb">
              <div class="h5">
                <div class="gmail_extra"><br>
                  <div class="gmail_quote">2016-11-29 19:21 GMT+01:00
                    Micha Ober <span dir="ltr">&lt;<a
                        moz-do-not-send="true"
                        href="mailto:micha2k@gmail.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:micha2k@gmail.com">micha2k@gmail.com</a></a>&gt;</span>:<br>
                    <blockquote class="gmail_quote" style="margin:0 0 0
                      .8ex;border-left:1px #ccc solid;padding-left:1ex">
                      <div dir="ltr">
                        <div class="gmail_default"
                          style="font-family:monospace,monospace">I had
                          opened another thread on this mailing list
                          (Subject: "After upgrade from 3.4.2 to 3.8.5 -
                          High CPU usage resulting in disconnects and
                          split-brain").</div>
                        <div class="gmail_default"
                          style="font-family:monospace,monospace"><br>
                        </div>
                        <div class="gmail_default"
                          style="font-family:monospace,monospace">The
                          title may be a bit misleading now, as I am no
                          longer observing high CPU usage after
                          upgrading to 3.8.6, but the disconnects are
                          still happening and the number of files in
                          split-brain is growing.<br>
                        </div>
                        <div class="gmail_default"
                          style="font-family:monospace,monospace"><br>
                        </div>
                        <div class="gmail_default"
                          style="font-family:monospace,monospace">Setup:
                          6 compute nodes, each serving as a glusterfs
                          server and client, Ubuntu 14.04, two bricks
                          per node, distribute-replicate</div>
                        <div class="gmail_default"
                          style="font-family:monospace,monospace"><br>
                        </div>
                        <div class="gmail_default"
                          style="font-family:monospace,monospace">I have
                          two gluster volumes set up (one for scratch
                          data, one for the slurm scheduler). Only the
                          scratch data volume shows critical errors
                          "[...] has not responded in the last 42
                          seconds, disconnecting.". So I can rule out
                          network problems, the gigabit link between the
                          nodes is not saturated at all. The disks are
                          almost idle (&lt;10%).</div>
                        <div class="gmail_default"
                          style="font-family:monospace,monospace"><br>
                        </div>
                        <div class="gmail_default"
                          style="font-family:monospace,monospace">I have
                          glusterfs 3.4.2 on Ubuntu 12.04 on a another
                          compute cluster, running fine since it was
                          deployed.</div>
                        <div class="gmail_default"
                          style="font-family:monospace,monospace">I had
                          glusterfs 3.4.2 on Ubuntu 14.04 on this
                          cluster, running fine for almost a year.</div>
                        <div class="gmail_default"
                          style="font-family:monospace,monospace"><br>
                        </div>
                        <div class="gmail_default"
                          style="font-family:monospace,monospace">After
                          upgrading to 3.8.5, the problems (as
                          described) started. I would like to use some
                          of the new features of the newer versions
                          (like bitrot), but the users can't run their
                          compute jobs right now because the result
                          files are garbled.</div>
                      </div>
                      <div class="m_-1578094958703753071HOEnZb">
                        <div class="m_-1578094958703753071h5">
                          <div class="gmail_extra"><br>
                            <div class="gmail_quote">2016-11-29 18:53
                              GMT+01:00 Atin Mukherjee <span dir="ltr">&lt;<a
                                  moz-do-not-send="true"
                                  href="mailto:amukherj@redhat.com"
                                  target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:amukherj@redhat.com">amukherj@redhat.com</a></a>&gt;</span>:<br>
                              <blockquote class="gmail_quote"
                                style="margin:0 0 0 .8ex;border-left:1px
                                #ccc solid;padding-left:1ex">
                                <div style="white-space:pre-wrap">Would you be able to share what is not working for you in 3.8.x (mention the exact version). 3.4 is quite old and falling back to an unsupported version doesn't look a feasible option.</div>
                                <br>
                                <div class="gmail_quote">
                                  <div>
                                    <div
                                      class="m_-1578094958703753071m_-2811647508981727209h5">
                                      <div dir="ltr">On Tue, 29 Nov 2016
                                        at 17:01, Micha Ober &lt;<a
                                          moz-do-not-send="true"
                                          href="mailto:micha2k@gmail.com"
                                          target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:micha2k@gmail.com">micha2k@gmail.com</a></a>&gt;
                                        wrote:<br>
                                      </div>
                                    </div>
                                  </div>
                                  <blockquote class="gmail_quote"
                                    style="margin:0 0 0
                                    .8ex;border-left:1px #ccc
                                    solid;padding-left:1ex">
                                    <div>
                                      <div
                                        class="m_-1578094958703753071m_-2811647508981727209h5">
                                        <div dir="ltr"
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
                                          <div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">Hi,</div>
                                          <div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace"><br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
                                          </div>
                                          <div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">I was using gluster 3.4 and
                                            upgraded to 3.8, but that
                                            version showed to be
                                            unusable for me. I now need
                                            to downgrade.</div>
                                          <div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace"><br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
                                          </div>
                                          <div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">I'm running Ubuntu 14.04. As
                                            upgrades of the op version
                                            are irreversible, I guess I
                                            have to delete all gluster
                                            volumes and re-create them
                                            with the downgraded
                                            version. </div>
                                          <div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace"><br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
                                          </div>
                                          <div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">0. Backup data</div>
                                          <div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">1. Unmount all gluster volumes</div>
                                          <div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">2. apt-get purge
                                            glusterfs-server
                                            glusterfs-client</div>
                                          <div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">3. Remove PPA for 3.8</div>
                                          <div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">4. Add PPA for older version</div>
                                          <div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">5. apt-get install
                                            glusterfs-server
                                            glusterfs-client</div>
                                          <div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">6. Create volumes</div>
                                          <div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace"><br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
                                          </div>
                                          <div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">Is "purge" enough to delete all
                                            configuration files of the
                                            currently installed version
                                            or do I need to  manually
                                            clear some residues before
                                            installing an older version?</div>
                                          <div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace"><br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
                                          </div>
                                          <div class="gmail_default
m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
style="font-family:monospace,monospace">Thanks.</div>
                                        </div>
                                      </div>
                                    </div>
                                    <span>
                                      ______________________________<wbr>_________________<br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
                                      Gluster-users mailing list<br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
                                      <a moz-do-not-send="true"
                                        href="mailto:Gluster-users@gluster.org"
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
                                        target="_blank">Gluster-users@gluster.org</a><br
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg">
                                      <a moz-do-not-send="true"
                                        href="http://www.gluster.org/mailman/listinfo/gluster-users"
                                        rel="noreferrer"
class="m_-1578094958703753071m_-2811647508981727209m_-2705140003504720857gmail_msg"
                                        target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a></span></blockquote>
                                </div>
                                <span
                                  class="m_-1578094958703753071m_-2811647508981727209HOEnZb"><font
                                    color="#888888">
                                    <div dir="ltr">-- <br>
                                    </div>
                                    <div
                                      data-smartmail="gmail_signature">-
                                      Atin (atinm)</div>
                                  </font></span></blockquote>
                            </div>
                            <br>
                          </div>
                        </div>
                      </div>
                    </blockquote>
                  </div>
                  <br>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
    </blockquote>
    <br>
  </body>
</html>