<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">Hi Kondo,<br>
      <br>
      Can u also mention the version of gluster you are using. <br>
      <br>
      +Adding gluster-users<br>
      <br>
      Regards,<br>
      Avra<br>
      On 04/21/2015 02:27 PM, Avra Sengupta wrote:<br>
    </div>
    <blockquote cite="mid:55361107.7090506@redhat.com" type="cite">
      <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
      <div class="moz-cite-prefix">Hi Kondo,<br>
        <br>
        I went through the gluster13 logs you had sent. Seems like
        something on that machine is holding the lock and is not
        releasing it. There are ways in which the system might end up in
        this scenario. I will try and explain the same with an example.<br>
        <br>
        Let's say I have gluster 11, gluster12, and gluster 13 in my
        cluster. I initiate a command from gluster11. Now the first
        thing that command does is, it holds a lock on all the nodes in
        the cluster on behalf of gluster11. Once the command does what's
        intended, it's last act before ending is to unlock all the nodes
        in the cluster. Now, only the node that has issued the lock, can
        issue the unlock.<br>
        <br>
        In your case what has happened is some command after
        successfully acquired the lock on gluster13. Now the node which
        initiated the command, went down or glusterd on that node went
        down before it could complete the command and it never got to
        send the unlock to gluster13. <br>
        <br>
        There's a workaround to it. You can restart glusterd on
        gluster13 and it should work fine.<br>
        <br>
        Regards,<br>
        Avra<br>
        <br>
        On 04/20/2015 06:55 PM, kenji kondo wrote:<br>
      </div>
      <blockquote
cite="mid:CAJDhDZa0CW5JjJsUy5MUFqmfKsJPhY7XNFWTZuNN08UZTGmXfw@mail.gmail.com"
        type="cite">
        <div dir="ltr">Hello Vijay,
          <div>Maybe this is very rare case. But is there any idea?</div>
          <div><br>
          </div>
          <div>Thanks,</div>
          <div>Kondo</div>
        </div>
        <div class="gmail_extra"><br>
          <div class="gmail_quote">2015-04-15 9:47 GMT+09:00 Vijaikumar
            M <span dir="ltr">&lt;<a moz-do-not-send="true"
                href="mailto:vmallika@redhat.com" target="_blank">vmallika@redhat.com</a>&gt;</span>:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div bgcolor="#FFFFFF" text="#000000"> <tt>Adding Avra</tt><tt>...</tt><tt><br>
                </tt><tt><br>
                </tt><tt>Thanks,</tt><tt><br>
                </tt><tt>Vijay</tt><tt><br>
                </tt>
                <div><br>
                  <br>
                  -------- Forwarded Message --------
                  <table border="0" cellpadding="0" cellspacing="0">
                    <tbody>
                      <tr>
                        <th align="RIGHT" nowrap="nowrap"
                          valign="BASELINE">Subject: </th>
                        <td>Re: [Gluster-users] Quota trouble</td>
                      </tr>
                      <tr>
                        <th align="RIGHT" nowrap="nowrap"
                          valign="BASELINE">Date: </th>
                        <td>Wed, 15 Apr 2015 00:27:26 +0900</td>
                      </tr>
                      <tr>
                        <th align="RIGHT" nowrap="nowrap"
                          valign="BASELINE">From: </th>
                        <td>kenji kondo <a moz-do-not-send="true"
                            href="mailto:kkay.jp@gmail.com"
                            target="_blank">&lt;kkay.jp@gmail.com&gt;</a></td>
                      </tr>
                      <tr>
                        <th align="RIGHT" nowrap="nowrap"
                          valign="BASELINE">To: </th>
                        <td>Vijaikumar M <a moz-do-not-send="true"
                            href="mailto:vmallika@redhat.com"
                            target="_blank">&lt;vmallika@redhat.com&gt;</a></td>
                      </tr>
                    </tbody>
                  </table>
                  <br>
                  <br>
                  <div dir="ltr">
                    <p class="MsoNormal"><span
                        style="font-size:13.3333330154419px">Hi Vijay,</span></p>
                    <p class="MsoNormal"><span
                        style="font-size:13.3333330154419px">Thanks for
                        your comments.</span></p>
                    <p class="MsoNormal"><br>
                    </p>
                    <p class="MsoNormal"><span
                        style="font-size:13.3333330154419px">The lock
                        error occurs at one server it's called
                        "gluster13".</span></p>
                    <p class="MsoNormal"><span
                        style="font-size:13.3333330154419px">In the
                        gluster13, I tried to create new volume and
                        start quota. But it failed as below,</span></p>
                    <p class="MsoNormal"><br>
                    </p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US">In both host
                        gluster10 and gluster13, ran below</span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US">$ sudo mkdir
                        /export11/testbrick1</span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US">$ sudo mkdir
                        /export11/testbrick2</span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US"> </span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US">In
                        gluster13, ran below</span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US">$ sudo
                        /usr/sbin/gluster volume create testvol2
                        gluster13:/export11/testbrick1
                        gluster13:/export11/testbrick2</span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US">volume
                        create: testvol2: failed: Locking failed on
                        gluster13. Please check log file for details.</span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US"> </span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US">$ sudo
                        /usr/sbin/gluster volume create testvol2
                        gluster10:/export11/testbrick1
                        gluster10:/export11/testbrick2</span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US">volume
                        create: testvol2: failed: Locking failed on
                        gluster13. Please check log file for details.</span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US"> </span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US">But I
                        recived error messages above.</span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US">On the other
                        hand, in gluster10, it was success.</span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US"> </span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US">Again, in
                        gluster13, I tried to run quota, but it failed
                        as below.</span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US">$ sudo
                        /usr/sbin/gluster volume quota testvol2 enable</span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US">quota
                        command failed : Locking failed on gluster13.
                        Please check log file for details.</span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US"><br>
                      </span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US">Could you
                        find attached?</span></p>
                    <p class="MsoNormal"><span
                        style="font-size:13.3333330154419px">We can find
                        error messages in the log of gluster13.</span></p>
                    <p class="MsoNormal"><span
                        style="font-size:13.3333330154419px"><br>
                      </span></p>
                    <p class="MsoNormal"><span
                        style="font-size:13.3333330154419px">Best
                        regards,</span></p>
                    <p class="MsoNormal"><span
                        style="font-size:13.3333330154419px">Kondo</span></p>
                    <p class="MsoNormal" style="font-size:14px"><span
                        style="font-size:10pt" lang="EN-US"><br>
                      </span></p>
                  </div>
                  <div class="gmail_extra"><br>
                    <div class="gmail_quote">2015-04-13 19:38 GMT+09:00
                      Vijaikumar M <span dir="ltr">&lt;<a
                          moz-do-not-send="true"
                          href="mailto:vmallika@redhat.com"
                          target="_blank">vmallika@redhat.com</a>&gt;</span>:<br>
                      <blockquote class="gmail_quote" style="margin:0 0
                        0 .8ex;border-left:1px #ccc
                        solid;padding-left:1ex">
                        <div bgcolor="#FFFFFF" text="#000000"> <tt>Hi
                            Kondo,</tt><tt><br>
                          </tt><tt><br>
                          </tt><tt>The lock error you mentioned is
                            because, another operation is still running
                            on the volume and hence not able to acquire
                            the lock.</tt><tt><br>
                          </tt><tt>This is bug of not displaying proper
                            error message, we are working on fixing this
                            issue.</tt><tt><br>
                          </tt><tt><br>
                          </tt><tt>I was not able to find any clue on
                            why quotad is not running.<br>
                            <br>
                            I wanted to check, if we can manually start
                            quotad something like below:</tt><tt><br>
                          </tt><tt><br>
                          </tt><tt># /usr/local/sbin/glusterfs -s
                            localhost --volfile-id gluster/quotad -p
                            /var/lib/glusterd/quotad/run/quotad.pid -l
                            /var/log/glusterfs/quotad.log -S
                            /var/run/gluster/myquotad.socket
                            --xlator-option
                            *replicate*.data-self-heal=off
                            --xlator-option
                            *replicate*.metadata-self-heal=off
                            --xlator-option
                            *replicate*.entry-self-heal=off</tt><tt><br>
                          </tt><tt><br>
                          </tt><tt>or</tt><tt><br>
                          </tt><tt><br>
                          </tt><tt>create a new temporary volume, and
                            enable quota on this volume. (quotad will be
                            same for all the volume which has quota
                            enabled)</tt><tt><br>
                          </tt><tt><br>
                          </tt><tt><br>
                          </tt><tt>Thanks,</tt><tt><br>
                          </tt><tt>Vijay</tt><tt><br>
                          </tt><br>
                          <br>
                          <div>On Sunday 12 April 2015 07:05 PM, kenji
                            kondo wrote:<br>
                          </div>
                          <blockquote type="cite">
                            <div dir="ltr">Hi Vijay,
                              <div><br>
                              </div>
                              <div>Thank you for your suggestion. But
                                I'm sorry, it's difficult to access from
                                outside because my glusterfs system is
                                closed.</div>
                              <div>I will give up if there is no clue
                                information in attached log.</div>
                              <div><br>
                              </div>
                              <div>Best regards,</div>
                              <div>Kondo</div>
                              <div><br>
                              </div>
                            </div>
                            <div class="gmail_extra"><br>
                              <div class="gmail_quote">2015-04-09 15:40
                                GMT+09:00 Vijaikumar M <span dir="ltr">&lt;<a
                                    moz-do-not-send="true"
                                    href="mailto:vmallika@redhat.com"
                                    target="_blank">vmallika@redhat.com</a>&gt;</span>:<br>
                                <blockquote class="gmail_quote"
                                  style="margin:0 0 0
                                  .8ex;border-left:1px #ccc
                                  solid;padding-left:1ex">
                                  <div bgcolor="#FFFFFF" text="#000000">
                                    <br>
                                    <br>
                                    <div>On Thursday 09 April 2015 11:58
                                      AM, Vijaikumar M wrote:<br>
                                    </div>
                                    <blockquote type="cite"> <br>
                                      <br>
                                      <div>On Wednesday 08 April 2015
                                        09:57 PM, kenji kondo wrote:<br>
                                      </div>
                                      <blockquote type="cite">
                                        <div dir="ltr">Hi Vijay,
                                          <div><br>
                                          </div>
                                          <div>I checked the all of the
                                            setting.</div>
                                          <div>The all are
                                            'features.quota=on' when I
                                            set quota enable and the all
                                            are 'features.quota=off'
                                            when I set quota disable.</div>
                                          <div><br>
                                          </div>
                                          <div>But I could find new
                                            issue.</div>
                                          <div>When I checked a volume
                                            status for all server, in
                                            one of the servers I
                                            received the error message
                                            as below.</div>
                                          <div><br>
                                          </div>
                                          <div>$ sudo /usr/sbin/gluster
                                            volume status testvol</div>
                                          <div>Locking failed on
                                            gluster13. Please check log
                                            file for details.</div>
                                          <div><br>
                                          </div>
                                          <div>In
                                            etc-glusterfs-glusterd.vol.log
                                            of problem server, I found
                                            error messages as below.</div>
                                          <div>
                                            <div> </div>
                                            <div>[2015-04-08
                                              08:40:04.782644] I
                                              [mem-pool.c:545:mem_pool_destroy]
                                              0-management: size=588
                                              max=0 total=0</div>
                                            <div>[2015-04-08
                                              08:40:04.782685] I
                                              [mem-pool.c:545:mem_pool_destroy]
                                              0-management: size=124
                                              max=0 total=0</div>
                                            <div>[2015-04-08
                                              08:40:04.782848] W
                                              [socket.c:611:__socket_rwv]
                                              0-management: readv on
                                              /var/run/14b05cd492843e6e288e290c2d63093c.socket
                                              failed (Invalid arguments)</div>
                                            <div>[2015-04-08
                                              08:40:04.805407] I [MSGID:
                                              106006]
                                              [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify]
                                              0-management: nfs has
                                              disconnected from
                                              glusterd.</div>
                                            <div>[2015-04-08
                                              08:43:02.439001] I
                                              [glusterd-handler.c:3803:__glusterd_handle_status_volume]
                                              0-management: Received
                                              status volume req for
                                              volume testvol</div>
                                            <div>[2015-04-08
                                              08:43:02.460581] E
                                              [glusterd-utils.c:148:glusterd_lock]
                                              0-management: Unable to
                                              get lock for uuid:
                                              03a32bce-ec63-4dc3-a287-4901a55dd8c9,
                                              lock held by:
                                              03a32bce-ec63-4dc3-a287-4901a55dd8c9</div>
                                            <div>[2015-04-08
                                              08:43:02.460632] E
                                              [glusterd-op-sm.c:6584:glusterd_op_sm]
                                              0-management: handler
                                              returned: -1</div>
                                            <div>[2015-04-08
                                              08:43:02.460654] E
                                              [glusterd-syncop.c:105:gd_collate_errors]
                                              0-: Locking failed on
                                              gluster13. Please check
                                              log file for details.</div>
                                            <div>[2015-04-08
                                              08:43:02.461409] E
                                              [glusterd-syncop.c:1602:gd_sync_task_begin]
                                              0-management: Locking
                                              Peers Failed.</div>
                                            <div>[2015-04-08
                                              08:43:43.698168] I
                                              [glusterd-handler.c:3803:__glusterd_handle_status_volume]
                                              0-management: Received
                                              status volume req for
                                              volume testvol</div>
                                            <div>[2015-04-08
                                              08:43:43.698813] E
                                              [glusterd-utils.c:148:glusterd_lock]
                                              0-management: Unable to
                                              get lock for uuid:
                                              03a32bce-ec63-4dc3-a287-4901a55dd8c9,
                                              lock held by:
                                              03a32bce-ec63-4dc3-a287-4901a55dd8c9</div>
                                            <div>[2015-04-08
                                              08:43:43.698898] E
                                              [glusterd-op-sm.c:6584:glusterd_op_sm]
                                              0-management: handler
                                              returned: -1</div>
                                            <div>[2015-04-08
                                              08:43:43.698994] E
                                              [glusterd-syncop.c:105:gd_collate_errors]
                                              0-: Locking failed on
                                              gluster13. Please check
                                              log file for details.</div>
                                            <div>[2015-04-08
                                              08:43:43.702126] E
                                              [glusterd-syncop.c:1602:gd_sync_task_begin]
                                              0-management: Locking
                                              Peers Failed.</div>
                                            <div>[2015-04-08
                                              08:44:01.277139] I
                                              [glusterd-handler.c:3803:__glusterd_handle_status_volume]
                                              0-management: Received
                                              status volume req for
                                              volume testvol</div>
                                            <div>[2015-04-08
                                              08:44:01.277560] E
                                              [glusterd-utils.c:148:glusterd_lock]
                                              0-management: Unable to
                                              get lock for uuid:
                                              03a32bce-ec63-4dc3-a287-4901a55dd8c9,
                                              lock held by:
                                              03a32bce-ec63-4dc3-a287-4901a55dd8c9</div>
                                            <div>[2015-04-08
                                              08:44:01.277639] E
                                              [glusterd-op-sm.c:6584:glusterd_op_sm]
                                              0-management: handler
                                              returned: -1</div>
                                            <div>[2015-04-08
                                              08:44:01.277676] E
                                              [glusterd-syncop.c:105:gd_collate_errors]
                                              0-: Locking failed on
                                              gluster13. Please check
                                              log file for details.</div>
                                            <div>[2015-04-08
                                              08:44:01.281514] E
                                              [glusterd-syncop.c:1602:gd_sync_task_begin]
                                              0-management: Locking
                                              Peers Failed.</div>
                                            <div>[2015-04-08
                                              08:45:42.599796] I
                                              [glusterd-handler.c:3803:__glusterd_handle_status_volume]
                                              0-management: Received
                                              status volume req for
                                              volume testvol</div>
                                            <div>[2015-04-08
                                              08:45:42.600343] E
                                              [glusterd-utils.c:148:glusterd_lock]
                                              0-management: Unable to
                                              get lock for uuid:
                                              03a32bce-ec63-4dc3-a287-4901a55dd8c9,
                                              lock held by:
                                              03a32bce-ec63-4dc3-a287-4901a55dd8c9</div>
                                            <div>[2015-04-08
                                              08:45:42.600417] E
                                              [glusterd-op-sm.c:6584:glusterd_op_sm]
                                              0-management: handler
                                              returned: -1</div>
                                            <div>[2015-04-08
                                              08:45:42.600482] E
                                              [glusterd-syncop.c:105:gd_collate_errors]
                                              0-: Locking failed on
                                              gluster13. Please check
                                              log file for details.</div>
                                            <div>[2015-04-08
                                              08:45:42.601039] E
                                              [glusterd-syncop.c:1602:gd_sync_task_begin]
                                              0-management: Locking
                                              Peers Failed.</div>
                                          </div>
                                          <div><br>
                                          </div>
                                          <div>Does this situation
                                            relate to my quota problems?<br>
                                          </div>
                                          <div><br>
                                          </div>
                                        </div>
                                      </blockquote>
                                      <tt><br>
                                      </tt><tt>This is a glusterd
                                        different issue. Can we get the
                                        glusterd logs from gluster13?</tt><tt><br>
                                      </tt><tt>Can get access to these
                                        machines, so that we can debug
                                        live?</tt><tt><br>
                                      </tt><tt><br>
                                      </tt><tt>Thanks,</tt><tt><br>
                                      </tt><tt>Vijay</tt><br>
                                      <tt><br>
                                      </tt></blockquote>
                                    <tt>Regarding quota issue, quota
                                      feature is enabled successfully. I
                                      am wondering why quotad is not
                                      started. </tt><tt><br>
                                      If we get the access to the
                                      machine, it will be easier to
                                      debug the issue.</tt><tt><br>
                                    </tt><tt><br>
                                    </tt><tt>Thanks,</tt><tt><br>
                                    </tt><tt>Vijay</tt><br>
                                    <br>
                                    <br>
                                    <blockquote type="cite">
                                      <blockquote type="cite">
                                        <div dir="ltr">
                                          <div><br>
                                          </div>
                                          <div>Best regards,</div>
                                          <div>Kondo</div>
                                          <div><br>
                                          </div>
                                          <div> </div>
                                        </div>
                                        <div class="gmail_extra"><br>
                                          <div class="gmail_quote">2015-04-08

                                            15:14 GMT+09:00 Vijaikumar M
                                            <span dir="ltr">&lt;<a
                                                moz-do-not-send="true"
                                                href="mailto:vmallika@redhat.com"
                                                target="_blank">vmallika@redhat.com</a>&gt;</span>:<br>
                                            <blockquote
                                              class="gmail_quote"
                                              style="margin:0 0 0
                                              .8ex;border-left:1px #ccc
                                              solid;padding-left:1ex">
                                              <div bgcolor="#FFFFFF"
                                                text="#000000"> <tt>Hi
                                                  Kondo,</tt><tt><br>
                                                </tt><tt><br>
                                                </tt><tt>I suspect, in
                                                  one of the node quota
                                                  feature is not set for
                                                  some reason and hence
                                                  quotad is not
                                                  starting.</tt><tt><br>
                                                </tt><tt><br>
                                                  On all the nodes can
                                                  you check if below
                                                  option is set to 'on'<br>
                                                  <br>
                                                </tt><tt># grep quota
                                                  /var/lib/glusterd/vols/&lt;volname&gt;/info
                                                </tt><tt><br>
                                                </tt><tt>features.quota=on</tt><tt><br>
                                                </tt><tt><br>
                                                  <br>
                                                  Also can I get brick
                                                  logs from all the
                                                  nodes?<br>
                                                  <br>
                                                  Also can you create a
                                                  temporary volume and
                                                  enable the quota here
                                                  and see if see quota
                                                  works fine with this
                                                  volume?<br>
                                                  <br>
                                                  <br>
                                                </tt><tt>Thanks,</tt><tt><br>
                                                </tt><tt>Vijay</tt><br>
                                                <br>
                                                <div>On Tuesday 07 April
                                                  2015 08:34 PM, kenji
                                                  kondo wrote:<br>
                                                </div>
                                                <blockquote type="cite">
                                                  <div dir="ltr">Hi
                                                    Vijay,
                                                    <div><br>
                                                    </div>
                                                    <div>Could you find
                                                      attached?</div>
                                                    <div>I got logs of
                                                      server and client.</div>
                                                    <div>As same as
                                                      before, I could
                                                      not create a file
                                                      after quota
                                                      usage-limit
                                                      setting.</div>
                                                    <div><br>
                                                    </div>
                                                    <div>Best regards,</div>
                                                    <div>Kondo</div>
                                                    <div><br>
                                                    </div>
                                                  </div>
                                                  <div
                                                    class="gmail_extra"><br>
                                                    <div
                                                      class="gmail_quote">2015-04-07


                                                      18:34 GMT+09:00
                                                      Vijaikumar M <span
                                                        dir="ltr">&lt;<a
moz-do-not-send="true" href="mailto:vmallika@redhat.com" target="_blank">vmallika@redhat.com</a>&gt;</span>:<br>
                                                      <blockquote
                                                        class="gmail_quote"
                                                        style="margin:0
                                                        0 0
                                                        .8ex;border-left:1px
                                                        #ccc
                                                        solid;padding-left:1ex">
                                                        <div
                                                          bgcolor="#FFFFFF"
                                                          text="#000000">
                                                          <tt>Hi Konda,</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>Can
                                                          we get all the
                                                          log files?</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt># </tt><tt>gluster


                                                          volume quota
                                                          &lt;volname&gt;
                                                          disable</tt><tt><br>
                                                          </tt><tt> </tt><tt>#
                                                          </tt><tt>gluster

                                                          volume quota
                                                          &lt;volname&gt;
                                                          enable<br>
                                                          <br>
                                                          <br>
                                                          Now copy all
                                                          the logs
                                                          files.<br>
                                                          <br>
                                                          Thanks,<br>
                                                          Vijay<br>
                                                          <br>
                                                          </tt><tt> </tt><br>
                                                          <br>
                                                          <div>On
                                                          Tuesday 07
                                                          April 2015
                                                          12:39 PM,
                                                          K.Kondo wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <div>Thank you
                                                          very much !
                                                          Vijay</div>
                                                          <div>I want to
                                                          use a quota
                                                          because each
                                                          volume became
                                                          too big.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Best
                                                          regard</div>
                                                          <div>Kondo</div>
                                                          <div><br>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          <div><br>
                                                          2015/04/07
                                                          15:18、Vijaikumar
                                                          M &lt;<a
                                                          moz-do-not-send="true"
href="mailto:vmallika@redhat.com" target="_blank">vmallika@redhat.com</a>&gt;







                                                          のメッセージ:<br>
                                                          <br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <div> <tt>Hi
                                                          Kondo,</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>I
                                                          couldn’t find
                                                          clue from the
                                                          logs. I will
                                                          discuss about
                                                          this issue
                                                          with my
                                                          colleagues
                                                          today.<br>
                                                          <br>
                                                          </tt><tt><br>
                                                          </tt><tt>Thanks,</tt><tt><br>
                                                          </tt><tt>Vijay</tt><br>
                                                          <br>
                                                          <br>
                                                          <br>
                                                          <div>On Monday
                                                          06 April 2015
                                                          10:56 PM,
                                                          kenji kondo
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <div dir="ltr">Hello


                                                          Vijay,
                                                          <div>Is there
                                                          something idea
                                                          for this?</div>
                                                          <div>Best
                                                          regards,</div>
                                                          <div>Kondo</div>
                                                          </div>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <div
                                                          class="gmail_quote">2015-03-31




                                                          22:46
                                                          GMT+09:00
                                                          kenji kondo <span
                                                          dir="ltr">&lt;<a
moz-do-not-send="true" href="mailto:kkay.jp@gmail.com" target="_blank">kkay.jp@gmail.com</a>&gt;</span>:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0

                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">Hi



                                                          Vijay,
                                                          <div><br>
                                                          </div>
                                                          <div>I'm sorry
                                                          for late
                                                          reply.</div>
                                                          <div>I could
                                                          get the debug
                                                          mode log as
                                                          attached.</div>
                                                          <div>
                                                          <div>In this
                                                          test,
                                                          unfortunately
                                                          the quota did
                                                          not work as
                                                          same as
                                                          before.</div>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Could you
                                                          find the cause
                                                          of my problem?</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Best
                                                          regards,</div>
                                                          <div>Kondo</div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          <div>
                                                          <div>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <div
                                                          class="gmail_quote">2015-03-25






                                                          17:20
                                                          GMT+09:00
                                                          Vijaikumar M <span
                                                          dir="ltr">&lt;<a
moz-do-not-send="true" href="mailto:vmallika@redhat.com" target="_blank">vmallika@redhat.com</a>&gt;</span>:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0

                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div
                                                          bgcolor="#FFFFFF"
                                                          text="#000000">
                                                          <tt>Hi Kondo,</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>For
                                                          some reason
                                                          quota enable
                                                          was not
                                                          successful. We
                                                          may have
                                                          re-try
                                                          enabling
                                                          quota.<br>
                                                          <br>
                                                          </tt><tt><br>
                                                          </tt><tt>Thanks,</tt><tt><br>
                                                          </tt><tt>Vijay</tt>
                                                          <div>
                                                          <div><br>
                                                          <br>
                                                          <br>
                                                          <div>On
                                                          Tuesday 24
                                                          March 2015
                                                          07:08 PM,
                                                          kenji kondo
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <div dir="ltr">
                                                          <div>Hi Vijay,</div>
                                                          <div>
                                                          <div>Thanks
                                                          for your
                                                          checking.
                                                          Unfortunately,
                                                          currently I
                                                          can't stop the
                                                          service
                                                          because many
                                                          users are
                                                          using.</div>
                                                          <div>But, I
                                                          want to know
                                                          this cause of
                                                          this trouble,
                                                          so I will plan
                                                          to stop.
                                                          Please wait to
                                                          get the log.</div>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Best
                                                          regards,</div>
                                                          <div>Kondo</div>
                                                          </div>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <div
                                                          class="gmail_quote">2015-03-24








                                                          17:01
                                                          GMT+09:00
                                                          Vijaikumar M <span
                                                          dir="ltr">&lt;<a
moz-do-not-send="true" href="mailto:vmallika@redhat.com" target="_blank">vmallika@redhat.com</a>&gt;</span>:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0

                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div
                                                          bgcolor="#FFFFFF"
                                                          text="#000000">
                                                          <tt>Hi Kondo,</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>I
                                                          couldn't find
                                                          much clue in
                                                          the glusterd
                                                          logs, other
                                                          than the error
                                                          message you
                                                          mentioned
                                                          below.</tt><tt><br>
                                                          </tt><tt>Can
                                                          you try
                                                          disabling and
                                                          enabling the
                                                          quota again
                                                          and see if
                                                          this start
                                                          quotad?</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>Try
                                                          below command:</tt><tt><br>
                                                          </tt><tt>#
                                                          gluster volume
                                                          quota
                                                          &lt;volname&gt;
                                                          disable</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>wait
                                                          for all quota
                                                          process to
                                                          terminate</tt><tt><br>
                                                          </tt><tt>#ps
                                                          -ef | </tt><tt>quota</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>#
                                                          service
                                                          glusterd stop</tt><tt><br>
                                                          </tt><tt>#
                                                          glusterd
                                                          -LDEBUG</tt><tt><br>
                                                          </tt><tt>#
                                                          gluster volume
                                                          quota
                                                          &lt;volname&gt;
                                                          enable</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>Now
                                                          verify if
                                                          quotad is
                                                          running</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>Thanks,</tt><tt><br>
                                                          </tt><tt>Vijay</tt>
                                                          <div>
                                                          <div><tt><br>
                                                          </tt><tt><br>
                                                          </tt><br>
                                                          <div>On Monday
                                                          23 March 2015
                                                          06:24 PM,
                                                          kenji kondo
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <div dir="ltr">Hi







                                                          Vijay,
                                                          <div>As you
                                                          pointed out,
                                                          the quotad is
                                                          not running in
                                                          the all of
                                                          server.</div>
                                                          <div>I checked
                                                          the volume
                                                          status and got
                                                          following log.</div>
                                                          <div>
                                                          <div><br>
                                                          </div>
                                                          <div>Quota
                                                          Daemon on
                                                          gluster25<span
style="white-space:pre-wrap"> </span>N/A<span
                                                          style="white-space:pre-wrap">
                                                          </span>N<span
style="white-space:pre-wrap"> </span>N/A<br>
                                                          </div>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>So, I
                                                          attached
                                                          requested log
'etc-glusterfs-glusterd.vol.log'.</div>
                                                          <div>The error
                                                          messages can
                                                          be found in
                                                          the log.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>[2015-03-19







                                                          11:51:07.457697]






                                                          E
                                                          [glusterd-quota.c:1467:glusterd_op_stage_quota]
                                                          0-management:
                                                          Quota is
                                                          disabled,
                                                          please enable
                                                          quota<br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>If you
                                                          want more some
                                                          information to
                                                          solve this
                                                          problems,
                                                          please ask me.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Best
                                                          regards,</div>
                                                          <div>Kondo</div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <div
                                                          class="gmail_quote">2015-03-23








                                                          16:04
                                                          GMT+09:00
                                                          Vijaikumar M <span
                                                          dir="ltr">&lt;<a
moz-do-not-send="true" href="mailto:vmallika@redhat.com" target="_blank">vmallika@redhat.com</a>&gt;</span>:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0

                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div
                                                          bgcolor="#FFFFFF"
                                                          text="#000000">
                                                          <tt>Hi Kondo,</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>Can
                                                          you please
                                                          verify if
                                                          quotad is
                                                          running</tt><tt>?</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>root@rh1:~








                                                          </tt><tt><b>#
                                                          gluster volume
                                                          status</b></tt><tt><br>
                                                          </tt><tt>Status







                                                          of volume:
                                                          vol1</tt><tt><br>
                                                          </tt><tt>Gluster








                                                          process                            








                                                          TCP Port  RDMA
                                                          Port  Online 
                                                          Pid</tt><tt><br>
                                                          </tt><tt>------------------------------------------------------------------------------</tt><tt><br>
                                                          </tt><tt>Brick
                                                          rh1:/var/opt/gluster/bricks/b1/dir   








                                                          49152    
                                                          0         
                                                          Y       1858 </tt><tt><br>
                                                          </tt><tt>NFS
                                                          Server on
                                                          localhost                    
                                                          2049     
                                                          0         
                                                          Y       1879 </tt><tt><br>
                                                          </tt><tt><b>Quota








                                                          Daemon on
                                                          localhost                  
                                                          N/A      
                                                          N/A       
                                                          Y       1914 </b></tt><tt><b><br>
                                                          </b></tt><tt> </tt><tt><br>
                                                          </tt><tt>Task
                                                          Status of
                                                          Volume vol1</tt><tt><br>
                                                          </tt><tt>------------------------------------------------------------------------------</tt><tt><br>
                                                          </tt><tt>There
                                                          are no active
                                                          volume tasks</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>root@rh1:~








                                                          # </tt><tt><b>ps








                                                          -ef | grep
                                                          quotad</b></tt><tt><br>
                                                          </tt><tt>root     








                                                          1914     1  0
                                                          12:29 ?       
                                                          00:00:00
                                                          /usr/local/sbin/glusterfs
                                                          -s localhost
                                                          --volfile-id
                                                          gluster/quotad
                                                          -p
                                                          /var/lib/glusterd/quotad/run/quotad.pid
                                                          -l </tt><tt><b>/var/log/glusterfs/quotad.log</b></tt><tt>
                                                          -S
                                                          /var/run/gluster/bb6ab82f70f555fd5c0e188fa4e09584.socket
--xlator-option
*replicate*.data-self-heal=off
--xlator-option
*replicate*.metadata-self-heal=off
--xlator-option
*replicate*.entry-self-heal=off</tt><tt><br>
                                                          </tt><tt>root     








                                                          1970  1511  0
                                                          12:31 pts/1   
                                                          00:00:00 grep
                                                          quotad</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>root@rh1:~








                                                          # </tt><tt><b>gluster









                                                          volume info</b></tt><tt><br>
                                                          </tt><tt>Volume








                                                          Name: vol1</tt><tt><br>
                                                          </tt><tt>Type:
                                                          Distribute</tt><tt><br>
                                                          </tt><tt>Volume







                                                          ID:
a55519ec-65d1-4741-9ad3-f94020fc9b21</tt><tt><br>
                                                          </tt><tt>Status:








                                                          Started</tt><tt><br>
                                                          </tt><tt>Number







                                                          of Bricks: 1</tt><tt><br>
                                                          </tt><tt>Transport-type:








                                                          tcp</tt><tt><br>
                                                          </tt><tt>Bricks:</tt><tt><br>
                                                          </tt><tt>Brick1:
rh1:/var/opt/gluster/bricks/b1/dir</tt><tt><br>
                                                          </tt><tt>Options








                                                          Reconfigured:</tt><tt><br>
                                                          </tt><tt><b>features.quota:








                                                          on</b></tt><tt><b><br>
                                                          </b></tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>If
                                                          quotad is not
                                                          running, can
                                                          you please
                                                          provide
                                                          glusterd logs
                                                          'usr-local-etc-glusterfs-glusterd.vol.log'.








                                                          I will check
                                                          is there are
                                                          any issues
                                                          starting
                                                          quotad</tt><tt>.<br>
                                                          </tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>Thanks,</tt><tt><br>
                                                          </tt><tt>Vihay</tt>
                                                          <div>
                                                          <div><br>
                                                          <br>
                                                          <br>
                                                          <br>
                                                          <br>
                                                          <br>
                                                          <br>
                                                          <div>On Monday
                                                          23 March 2015
                                                          11:54 AM,
                                                          K.Kondo wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <div>Hi Vijay,<br>
                                                          I could not
                                                          find
                                                          the"quotad.log"
                                                          in directory
                                                          /var/log/glusterfs
                                                          in both
                                                          servers and
                                                          client. But
                                                          other test
                                                          server has the
                                                          log.</div>
                                                          <div>Do you
                                                          know why there
                                                          is no the
                                                          file?</div>
                                                          <div>Thanks,</div>
                                                          <div>Kondo</div>
                                                          <div><br>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          <div><br>
                                                          2015/03/23
                                                          13:41、Vijaikumar
                                                          M &lt;<a
                                                          moz-do-not-send="true"
href="mailto:vmallika@redhat.com" target="_blank">vmallika@redhat.com</a>&gt;










                                                          のメッセージ:<br>
                                                          <br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <div> <tt>Hi
                                                          Kondo,</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>log
                                                          file
                                                          'quotad.log'
                                                          is missing in
                                                          the
                                                          attachment.</tt><tt>
                                                          Can you
                                                          provide this
                                                          log file as
                                                          well?</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>Thanks,</tt><tt><br>
                                                          </tt><tt>Vijay</tt><br>
                                                          <br>
                                                          <br>
                                                          <div>On Monday
                                                          23 March 2015
                                                          09:50 AM,
                                                          kenji kondo
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <div dir="ltr">Hi








                                                          Vijay,
                                                          <div>Could you
                                                          find the
                                                          attached?<br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Best
                                                          regards,</div>
                                                          <div>Kondo</div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <div
                                                          class="gmail_quote">2015-03-23









                                                          12:53
                                                          GMT+09:00
                                                          Vijaikumar M <span
                                                          dir="ltr">&lt;<a
moz-do-not-send="true" href="mailto:vmallika@redhat.com" target="_blank">vmallika@redhat.com</a>&gt;</span>:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0

                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div
                                                          bgcolor="#FFFFFF"
                                                          text="#000000">
                                                          <tt>Hi Kondo,</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>Can
                                                          you please
                                                          provide below
                                                          mentioned
                                                          gluterfs logs?<br>
                                                              client
                                                          logs (name of
                                                          this log will
                                                          be prefixed
                                                          with
                                                          mount-point
                                                          dirname)<br>
                                                              brick logs<br>
                                                              quotad
                                                          logs<br>
                                                          <br>
                                                          <br>
                                                          </tt><tt>Thanks,</tt><tt><br>
                                                          </tt><tt>Vijay</tt>
                                                          <div>
                                                          <div><br>
                                                          <br>
                                                          <br>
                                                          <br>
                                                          <div>On Friday
                                                          20 March 2015
                                                          06:31 PM,
                                                          kenji kondo
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <div dir="ltr">Hi,









                                                          Vijay and
                                                          Peter
                                                          <div>Thanks
                                                          for your
                                                          reply.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>I create
                                                          new volume
                                                          "testvol" with
                                                          two bricks and
                                                          set quota to
                                                          simplify this
                                                          problem.</div>
                                                          <div>I got the
                                                          glusterfs log
                                                          as following
                                                          after try to
                                                          create a
                                                          directory and
                                                          file.</div>
                                                          <div>BTW, my
                                                          glusterd was
                                                          upgraded from
                                                          older version,
                                                          although I
                                                          don't know
                                                          related to it.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Best
                                                          regards,</div>
                                                          <div>Kondo</div>
                                                          <div><br>
                                                          </div>
                                                          <div>
                                                          <div>[2015-03-20










                                                          03:42:52.931016]









                                                          I [MSGID:
                                                          100030]
                                                          [glusterfsd.c:1998:main]
                                                          0-/usr/sbin/glusterfs:










                                                          Started
                                                          running
                                                          /usr/sbin/glusterfs
                                                          version
                                                          3.6.0.29
                                                          (args:
                                                          /usr/sbin/glusterfs
                                                          --volfile-server=gluster10










                                                          --volfile-id=testvol










                                                          testvol)</div>
                                                          <div>[2015-03-20










                                                          03:42:52.944850]









                                                          I
                                                          [dht-shared.c:337:dht_init_regex]
                                                          0-testvol-dht:
                                                          using regex
                                                          rsync-hash-regex
                                                          =
                                                          ^\.(.+)\.[^.]+$</div>
                                                          <div>[2015-03-20










                                                          03:42:52.946256]









                                                          I
                                                          [client.c:2280:notify]
                                                          0-testvol-client-0:









                                                          parent
                                                          translators
                                                          are ready,
                                                          attempting
                                                          connect on
                                                          transport</div>
                                                          <div>[2015-03-20










                                                          03:42:52.950674]









                                                          I
                                                          [client.c:2280:notify]
                                                          0-testvol-client-1:









                                                          parent
                                                          translators
                                                          are ready,
                                                          attempting
                                                          connect on
                                                          transport</div>
                                                          <div>Final
                                                          graph:</div>
                                                          <div>+------------------------------------------------------------------------------+</div>
                                                          <div>  1:
                                                          volume
                                                          testvol-client-0</div>
                                                          <div>  2:    
                                                          type
                                                          protocol/client</div>
                                                          <div>  3:    
                                                          option
                                                          ping-timeout
                                                          42</div>
                                                          <div>  4:    
                                                          option
                                                          remote-host
                                                          gluster24</div>
                                                          <div>  5:    
                                                          option
                                                          remote-subvolume
/export25/brick</div>
                                                          <div>  6:    
                                                          option
                                                          transport-type
                                                          socket</div>
                                                          <div>  7:    
                                                          option
                                                          send-gids true</div>
                                                          <div>  8:
                                                          end-volume</div>
                                                          <div>  9:  </div>
                                                          <div> 10:
                                                          volume
                                                          testvol-client-1</div>
                                                          <div> 11:    
                                                          type
                                                          protocol/client</div>
                                                          <div> 12:    
                                                          option
                                                          ping-timeout
                                                          42</div>
                                                          <div> 13:    
                                                          option
                                                          remote-host
                                                          gluster25</div>
                                                          <div> 14:    
                                                          option
                                                          remote-subvolume
/export25/brick</div>
                                                          <div> 15:    
                                                          option
                                                          transport-type
                                                          socket</div>
                                                          <div> 16:    
                                                          option
                                                          send-gids true</div>
                                                          <div> 17:
                                                          end-volume</div>
                                                          <div> 18:  </div>
                                                          <div> 19:
                                                          volume
                                                          testvol-dht</div>
                                                          <div> 20:    
                                                          type
                                                          cluster/distribute</div>
                                                          <div> 21:    
                                                          subvolumes
                                                          testvol-client-0
testvol-client-1</div>
                                                          <div> 22:
                                                          end-volume</div>
                                                          <div> 23:  </div>
                                                          <div> 24:
                                                          volume
                                                          testvol-write-behind</div>
                                                          <div> 25:    
                                                          type
                                                          performance/write-behind</div>
                                                          <div> 26:    
                                                          subvolumes
                                                          testvol-dht</div>
                                                          <div> 27:
                                                          end-volume</div>
                                                          <div> 28:  </div>
                                                          <div> 29:
                                                          volume
                                                          testvol-read-ahead</div>
                                                          <div> 30:    
                                                          type
                                                          performance/read-ahead</div>
                                                          <div> 31:    
                                                          subvolumes
                                                          testvol-write-behind</div>
                                                          <div> 32:
                                                          end-volume</div>
                                                          <div> 33:  </div>
                                                          <div> 34:
                                                          volume
                                                          testvol-io-cache</div>
                                                          <div> 35:    
                                                          type
                                                          performance/io-cache</div>
                                                          <div> 36:    
                                                          subvolumes
                                                          testvol-read-ahead</div>
                                                          <div> 37:
                                                          end-volume</div>
                                                          <div> 38:  </div>
                                                          <div> 39:
                                                          volume
                                                          testvol-quick-read</div>
                                                          <div> 40:    
                                                          type
                                                          performance/quick-read</div>
                                                          <div> 41:    
                                                          subvolumes
                                                          testvol-io-cache</div>
                                                          <div> 42:
                                                          end-volume</div>
                                                          <div> 43:  </div>
                                                          <div> 44:
                                                          volume
                                                          testvol-md-cache</div>
                                                          <div> 45:    
                                                          type
                                                          performance/md-cache</div>
                                                          <div> 46:    
                                                          subvolumes
                                                          testvol-quick-read</div>
                                                          <div> 47:
                                                          end-volume</div>
                                                          <div> 48:  </div>
                                                          <div> 49:
                                                          volume testvol</div>
                                                          <div> 50:    
                                                          type
                                                          debug/io-stats</div>
                                                          <div> 51:    
                                                          option
                                                          latency-measurement
                                                          off</div>
                                                          <div> 52:    
                                                          option
                                                          count-fop-hits
                                                          off</div>
                                                          <div> 53:    
                                                          subvolumes
                                                          testvol-md-cache</div>
                                                          <div> 54:
                                                          end-volume</div>
                                                          <div> 55:  </div>
                                                          <div> 56:
                                                          volume
                                                          meta-autoload</div>
                                                          <div> 57:    
                                                          type meta</div>
                                                          <div> 58:    
                                                          subvolumes
                                                          testvol</div>
                                                          <div> 59:
                                                          end-volume</div>
                                                          <div> 60:  </div>
                                                          <div>+------------------------------------------------------------------------------+</div>
                                                          <div>[2015-03-20










                                                          03:42:52.955337]









                                                          I
                                                          [rpc-clnt.c:1759:rpc_clnt_reconfig]
                                                          0-testvol-client-0:










                                                          changing port
                                                          to 49155 (from
                                                          0)</div>
                                                          <div>[2015-03-20










                                                          03:42:52.957549]









                                                          I
                                                          [rpc-clnt.c:1759:rpc_clnt_reconfig]
                                                          0-testvol-client-1:










                                                          changing port
                                                          to 49155 (from
                                                          0)</div>
                                                          <div>[2015-03-20










                                                          03:42:52.959889]









                                                          I
                                                          [client-handshake.c:1415:select_server_supported_programs]
                                                          0-testvol-client-0:









                                                          Using Program
                                                          GlusterFS 3.3,
                                                          Num (1298437),
                                                          Version (330)</div>
                                                          <div>[2015-03-20










                                                          03:42:52.960090]









                                                          I
                                                          [client-handshake.c:1415:select_server_supported_programs]
                                                          0-testvol-client-1:









                                                          Using Program
                                                          GlusterFS 3.3,
                                                          Num (1298437),
                                                          Version (330)</div>
                                                          <div>[2015-03-20










                                                          03:42:52.960376]









                                                          I
                                                          [client-handshake.c:1200:client_setvolume_cbk]
                                                          0-testvol-client-0:










                                                          Connected to
                                                          testvol-client-0,
                                                          attached to
                                                          remote volume
'/export25/brick'.</div>
                                                          <div>[2015-03-20










                                                          03:42:52.960405]









                                                          I
                                                          [client-handshake.c:1212:client_setvolume_cbk]
                                                          0-testvol-client-0:









                                                          Server and
                                                          Client
                                                          lk-version
                                                          numbers are
                                                          not same,
                                                          reopening the
                                                          fds</div>
                                                          <div>[2015-03-20










                                                          03:42:52.960471]









                                                          I
                                                          [client-handshake.c:1200:client_setvolume_cbk]
                                                          0-testvol-client-1:










                                                          Connected to
                                                          testvol-client-1,
                                                          attached to
                                                          remote volume
'/export25/brick'.</div>
                                                          <div>[2015-03-20










                                                          03:42:52.960478]









                                                          I
                                                          [client-handshake.c:1212:client_setvolume_cbk]
                                                          0-testvol-client-1:









                                                          Server and
                                                          Client
                                                          lk-version
                                                          numbers are
                                                          not same,
                                                          reopening the
                                                          fds</div>
                                                          <div>[2015-03-20










                                                          03:42:52.962288]









                                                          I
                                                          [fuse-bridge.c:5042:fuse_graph_setup]
                                                          0-fuse:
                                                          switched to
                                                          graph 0</div>
                                                          <div>[2015-03-20










                                                          03:42:52.962351]









                                                          I
                                                          [client-handshake.c:188:client_set_lk_version_cbk]
                                                          0-testvol-client-1:









                                                          Server lk
                                                          version = 1</div>
                                                          <div>[2015-03-20










                                                          03:42:52.962362]









                                                          I
                                                          [client-handshake.c:188:client_set_lk_version_cbk]
                                                          0-testvol-client-0:









                                                          Server lk
                                                          version = 1</div>
                                                          <div>[2015-03-20










                                                          03:42:52.962424]









                                                          I
                                                          [fuse-bridge.c:3971:fuse_init]
                                                          0-glusterfs-fuse:









                                                          FUSE inited
                                                          with protocol
                                                          versions:
                                                          glusterfs 7.22
                                                          kernel 7.14</div>
                                                          <div>[2015-03-20










                                                          03:47:13.352234]









                                                          I
                                                          [glusterfsd-mgmt.c:56:mgmt_cbk_spec]
                                                          0-mgmt: Volume
                                                          file changed</div>
                                                          <div>[2015-03-20










                                                          03:47:15.518667]









                                                          I
                                                          [dht-shared.c:337:dht_init_regex]
                                                          2-testvol-dht:
                                                          using regex
                                                          rsync-hash-regex
                                                          =
                                                          ^\.(.+)\.[^.]+$</div>
                                                          <div>[2015-03-20










                                                          03:47:15.520034]









                                                          W
                                                          [graph.c:344:_log_if_unknown_option]
                                                          2-testvol-quota:









                                                          option
                                                          'timeout' is
                                                          not recognized</div>
                                                          <div>[2015-03-20










                                                          03:47:15.520091]









                                                          I
                                                          [client.c:2280:notify]
                                                          2-testvol-client-0:









                                                          parent
                                                          translators
                                                          are ready,
                                                          attempting
                                                          connect on
                                                          transport</div>
                                                          <div>[2015-03-20










                                                          03:47:15.524546]









                                                          I
                                                          [client.c:2280:notify]
                                                          2-testvol-client-1:









                                                          parent
                                                          translators
                                                          are ready,
                                                          attempting
                                                          connect on
                                                          transport</div>
                                                          <div>Final
                                                          graph:</div>
                                                          <div>+------------------------------------------------------------------------------+</div>
                                                          <div>  1:
                                                          volume
                                                          testvol-client-0</div>
                                                          <div>  2:    
                                                          type
                                                          protocol/client</div>
                                                          <div>  3:    
                                                          option
                                                          ping-timeout
                                                          42</div>
                                                          <div>  4:    
                                                          option
                                                          remote-host
                                                          gluster24</div>
                                                          <div>  5:    
                                                          option
                                                          remote-subvolume
/export25/brick</div>
                                                          <div>  6:    
                                                          option
                                                          transport-type
                                                          socket</div>
                                                          <div>  7:    
                                                          option
                                                          send-gids true</div>
                                                          <div>  8:
                                                          end-volume</div>
                                                          <div>  9:  </div>
                                                          <div> 10:
                                                          volume
                                                          testvol-client-1</div>
                                                          <div> 11:    
                                                          type
                                                          protocol/client</div>
                                                          <div> 12:    
                                                          option
                                                          ping-timeout
                                                          42</div>
                                                          <div> 13:    
                                                          option
                                                          remote-host
                                                          gluster25</div>
                                                          <div> 14:    
                                                          option
                                                          remote-subvolume
/export25/brick</div>
                                                          <div> 15:    
                                                          option
                                                          transport-type
                                                          socket</div>
                                                          <div> 16:    
                                                          option
                                                          send-gids true</div>
                                                          <div> 17:
                                                          end-volume</div>
                                                          <div> 18:  </div>
                                                          <div> 19:
                                                          volume
                                                          testvol-dht</div>
                                                          <div> 20:    
                                                          type
                                                          cluster/distribute</div>
                                                          <div> 21:    
                                                          subvolumes
                                                          testvol-client-0
testvol-client-1</div>
                                                          <div> 22:
                                                          end-volume</div>
                                                          <div> 23:  </div>
                                                          <div> 24:
                                                          volume
                                                          testvol-quota</div>
                                                          <div> 25:    
                                                          type
                                                          features/quota</div>
                                                          <div> 26:    
                                                          option timeout
                                                          0</div>
                                                          <div> 27:    
                                                          option
                                                          deem-statfs
                                                          off</div>
                                                          <div> 28:    
                                                          subvolumes
                                                          testvol-dht</div>
                                                          <div> 29:
                                                          end-volume</div>
                                                          <div> 30:  </div>
                                                          <div> 31:
                                                          volume
                                                          testvol-write-behind</div>
                                                          <div> 32:    
                                                          type
                                                          performance/write-behind</div>
                                                          <div> 33:    
                                                          subvolumes
                                                          testvol-quota</div>
                                                          <div> 34:
                                                          end-volume</div>
                                                          <div> 35:  </div>
                                                          <div> 36:
                                                          volume
                                                          testvol-read-ahead</div>
                                                          <div> 37:    
                                                          type
                                                          performance/read-ahead</div>
                                                          <div> 38:    
                                                          subvolumes
                                                          testvol-write-behind</div>
                                                          <div> 39:
                                                          end-volume</div>
                                                          <div> 40:  </div>
                                                          <div> 41:
                                                          volume
                                                          testvol-io-cache</div>
                                                          <div> 42:    
                                                          type
                                                          performance/io-cache</div>
                                                          <div> 43:    
                                                          subvolumes
                                                          testvol-read-ahead</div>
                                                          <div> 44:
                                                          end-volume</div>
                                                          <div> 45:  </div>
                                                          <div> 46:
                                                          volume
                                                          testvol-quick-read</div>
                                                          <div> 47:    
                                                          type
                                                          performance/quick-read</div>
                                                          <div> 48:    
                                                          subvolumes
                                                          testvol-io-cache</div>
                                                          <div> 49:
                                                          end-volume</div>
                                                          <div> 50:  </div>
                                                          <div> 51:
                                                          volume
                                                          testvol-md-cache</div>
                                                          <div> 52:    
                                                          type
                                                          performance/md-cache</div>
                                                          <div> 53:    
                                                          subvolumes
                                                          testvol-quick-read</div>
                                                          <div> 54:
                                                          end-volume</div>
                                                          <div> 55:  </div>
                                                          <div> 56:
                                                          volume testvol</div>
                                                          <div> 57:    
                                                          type
                                                          debug/io-stats</div>
                                                          <div> 58:    
                                                          option
                                                          latency-measurement
                                                          off</div>
                                                          <div> 59:    
                                                          option
                                                          count-fop-hits
                                                          off</div>
                                                          <div> 60:    
                                                          subvolumes
                                                          testvol-md-cache</div>
                                                          <div> 61:
                                                          end-volume</div>
                                                          <div> 62:  </div>
                                                          <div> 63:
                                                          volume
                                                          meta-autoload</div>
                                                          <div> 64:    
                                                          type meta</div>
                                                          <div> 65:    
                                                          subvolumes
                                                          testvol</div>
                                                          <div> 66:
                                                          end-volume</div>
                                                          <div> 67:  </div>
                                                          <div>+------------------------------------------------------------------------------+</div>
                                                          <div>[2015-03-20










                                                          03:47:15.530005]









                                                          I
                                                          [rpc-clnt.c:1759:rpc_clnt_reconfig]
                                                          2-testvol-client-1:










                                                          changing port
                                                          to 49155 (from
                                                          0)</div>
                                                          <div>[2015-03-20










                                                          03:47:15.530047]









                                                          I
                                                          [rpc-clnt.c:1759:rpc_clnt_reconfig]
                                                          2-testvol-client-0:










                                                          changing port
                                                          to 49155 (from
                                                          0)</div>
                                                          <div>[2015-03-20










                                                          03:47:15.539062]









                                                          I
                                                          [client-handshake.c:1415:select_server_supported_programs]
                                                          2-testvol-client-1:









                                                          Using Program
                                                          GlusterFS 3.3,
                                                          Num (1298437),
                                                          Version (330)</div>
                                                          <div>[2015-03-20










                                                          03:47:15.539299]









                                                          I
                                                          [client-handshake.c:1415:select_server_supported_programs]
                                                          2-testvol-client-0:









                                                          Using Program
                                                          GlusterFS 3.3,
                                                          Num (1298437),
                                                          Version (330)</div>
                                                          <div>[2015-03-20










                                                          03:47:15.539462]









                                                          I
                                                          [client-handshake.c:1200:client_setvolume_cbk]
                                                          2-testvol-client-1:










                                                          Connected to
                                                          testvol-client-1,
                                                          attached to
                                                          remote volume
'/export25/brick'.</div>
                                                          <div>[2015-03-20










                                                          03:47:15.539485]









                                                          I
                                                          [client-handshake.c:1212:client_setvolume_cbk]
                                                          2-testvol-client-1:









                                                          Server and
                                                          Client
                                                          lk-version
                                                          numbers are
                                                          not same,
                                                          reopening the
                                                          fds</div>
                                                          <div>[2015-03-20










                                                          03:47:15.539729]









                                                          I
                                                          [client-handshake.c:1200:client_setvolume_cbk]
                                                          2-testvol-client-0:










                                                          Connected to
                                                          testvol-client-0,
                                                          attached to
                                                          remote volume
'/export25/brick'.</div>
                                                          <div>[2015-03-20










                                                          03:47:15.539751]









                                                          I
                                                          [client-handshake.c:1212:client_setvolume_cbk]
                                                          2-testvol-client-0:









                                                          Server and
                                                          Client
                                                          lk-version
                                                          numbers are
                                                          not same,
                                                          reopening the
                                                          fds</div>
                                                          <div>[2015-03-20










                                                          03:47:15.542878]









                                                          I
                                                          [fuse-bridge.c:5042:fuse_graph_setup]
                                                          0-fuse:
                                                          switched to
                                                          graph 2</div>
                                                          <div>[2015-03-20










                                                          03:47:15.542959]









                                                          I
                                                          [client-handshake.c:188:client_set_lk_version_cbk]
                                                          2-testvol-client-1:









                                                          Server lk
                                                          version = 1</div>
                                                          <div>[2015-03-20










                                                          03:47:15.542987]









                                                          I
                                                          [client-handshake.c:188:client_set_lk_version_cbk]
                                                          2-testvol-client-0:









                                                          Server lk
                                                          version = 1</div>
                                                          <div>[2015-03-20










                                                          03:48:04.586291]









                                                          I
                                                          [client.c:2289:notify]
                                                          0-testvol-client-0:










                                                          current graph
                                                          is no longer
                                                          active,
                                                          destroying
                                                          rpc_client </div>
                                                          <div>[2015-03-20










                                                          03:48:04.586360]









                                                          I
                                                          [client.c:2289:notify]
                                                          0-testvol-client-1:










                                                          current graph
                                                          is no longer
                                                          active,
                                                          destroying
                                                          rpc_client </div>
                                                          <div>[2015-03-20










                                                          03:48:04.586378]









                                                          I
                                                          [client.c:2215:client_rpc_notify]
                                                          0-testvol-client-0:










                                                          disconnected
                                                          from
                                                          testvol-client-0.
                                                          Client process
                                                          will keep
                                                          trying to
                                                          connect to
                                                          glusterd until
                                                          brick's port
                                                          is available</div>
                                                          <div>[2015-03-20










                                                          03:48:04.586430]









                                                          I
                                                          [client.c:2215:client_rpc_notify]
                                                          0-testvol-client-1:










                                                          disconnected
                                                          from
                                                          testvol-client-1.
                                                          Client process
                                                          will keep
                                                          trying to
                                                          connect to
                                                          glusterd until
                                                          brick's port
                                                          is available</div>
                                                          <div>[2015-03-20










                                                          03:48:04.589552]









                                                          W
                                                          [client-rpc-fops.c:306:client3_3_mkdir_cbk]
                                                          2-testvol-client-0:









                                                          remote
                                                          operation
                                                          failed:
                                                          Transport
                                                          endpoint is
                                                          not connected.
                                                          Path: /test/a</div>
                                                          <div>[2015-03-20










                                                          03:48:04.589608]









                                                          W
                                                          [fuse-bridge.c:481:fuse_entry_cbk]
                                                          0-glusterfs-fuse:









                                                          78: MKDIR()
                                                          /test/a =&gt;
                                                          -1 (Transport
                                                          endpoint is
                                                          not connected)</div>
                                                          <div>[2015-03-20










                                                          03:48:11.073349]









                                                          W
                                                          [client-rpc-fops.c:2212:client3_3_create_cbk]
                                                          2-testvol-client-1:









                                                          remote
                                                          operation
                                                          failed:
                                                          Transport
                                                          endpoint is
                                                          not connected.
                                                          Path: /test/f</div>
                                                          <div>[2015-03-20










                                                          03:48:11.073419]









                                                          W
                                                          [fuse-bridge.c:1937:fuse_create_cbk]
                                                          0-glusterfs-fuse:









                                                          82: /test/f
                                                          =&gt; -1
                                                          (Transport
                                                          endpoint is
                                                          not connected)</div>
                                                          </div>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <div
                                                          class="gmail_quote">2015-03-20










                                                          11:27
                                                          GMT+09:00
                                                          Vijaikumar M <span
                                                          dir="ltr">&lt;<a
moz-do-not-send="true" href="mailto:vmallika@redhat.com" target="_blank">vmallika@redhat.com</a>&gt;</span>:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0

                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div
                                                          bgcolor="#FFFFFF"
                                                          text="#000000">
                                                          <tt>Hi Kondo,</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>Can
                                                          you please
                                                          provide all
                                                          the glusterfs
                                                          log files?</tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt>Thanks,</tt><tt><br>
                                                          </tt><tt>Vijay</tt>
                                                          <br>
                                                          <div>
                                                          <div> <br>
                                                          <br>
                                                          <div>On Friday
                                                          20 March 2015
                                                          07:33 AM,
                                                          K.Kondo wrote:<br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <div>
                                                          <div>
                                                          <div>Hello,
                                                          experts</div>
                                                          <div><br>
                                                          </div>
                                                          <div>I had a
                                                          trouble about
                                                          quota.</div>
                                                          <div>I set
                                                          quota to one
                                                          distributed
                                                          volume "vol12"
                                                          as bellow.<br>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US">gluster&gt;
                                                          volume quota
                                                          vol12 enable</span></p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US">volume quota :
                                                          success</span></p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US">gluster&gt;
                                                          volume quota
                                                          vol12
                                                          limit-usage
                                                          /test 10GB</span></p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US">volume quota :
                                                          success</span></p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US"> </span></p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US">But I couldn't
                                                          create a file
                                                          and directory
                                                          with below
                                                          error
                                                          message. </span></p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US"> </span></p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US">In a client
                                                          host,</span></p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US">$cd test
                                                          (mounted using
                                                          fuse)</span></p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US">$mkdir a</span></p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US">mkdir: cannot
                                                          create
                                                          directory `a':
                                                          Transport
                                                          endpoint is
                                                          not connected</span></p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US"> </span></p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US">Additionally,
                                                          I couldn't
                                                          check quota
                                                          status using
                                                          gluster
                                                          command.</span></p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US"> </span></p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US">gluster&gt;
                                                          volume quota
                                                          vol12 list</span></p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US">                 












                                                          Path                  











                                                          Hard-limit
                                                          Soft-limit  
                                                          Used 
                                                          Available 
                                                          Soft-limit
                                                          exceeded?
                                                          Hard-limit
                                                          exceeded?</span></p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US">---------------------------------------------------------------------------------------------------------------------------</span></p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US"> Here, this
                                                          command stops,
                                                          so I have to
                                                          do Ctrl-C.</span></p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt">Gluster











                                                          version is
                                                          3.6.1 and
                                                          3.6.0.29 for
                                                          server and
                                                          client
                                                          respectively.</p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><br>
                                                          </p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt">Any










                                                          idea for this?</p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><br>
                                                          </p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt">Best











                                                          regards,</p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt">K.










                                                          Kondo</p>
                                                          <p
                                                          class="MsoNormal"
                                                          style="text-align:start;margin:0mm

                                                          0mm 0.0001pt"><span
style="background-color:rgba(255,255,255,0)" lang="EN-US"> </span></p>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          <fieldset></fieldset>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          <pre>_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
                                                          </blockquote>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                          </blockquote>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                          </blockquote>
                                                          <br>
                                                        </div>
                                                      </blockquote>
                                                    </div>
                                                    <br>
                                                  </div>
                                                </blockquote>
                                                <br>
                                              </div>
                                            </blockquote>
                                          </div>
                                          <br>
                                        </div>
                                      </blockquote>
                                      <br>
                                    </blockquote>
                                    <br>
                                  </div>
                                </blockquote>
                              </div>
                              <br>
                            </div>
                          </blockquote>
                          <br>
                        </div>
                      </blockquote>
                    </div>
                    <br>
                  </div>
                  <br>
                </div>
                <br>
              </div>
            </blockquote>
          </div>
          <br>
        </div>
      </blockquote>
      <br>
    </blockquote>
    <br>
  </body>
</html>