<html>
  <head>
    <meta content="text/html; charset=windows-1252"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <p>Which application is filling memory? <br>
      <br>
      If it's a brick (glusterfsd) then stopping and starting a brick
      ("kill" and "gluster volume start ... force") will not waste
      cycles re-healing any files that are healthy. Any heals of an
      individual file that were not complete will be restarted as well
      as any files that were changed while the brick was offline.<br>
    </p>
    If it's glusterd, that can be restarted at any time without
    interfering with the volume.<br>
    <br>
    If it's glustershd ("/usr/bin/glusterfs -s localhost --volfile-id
    gluster/glustershd ..."), you can restart that with "gluster volume
    start ... force" (even if the volume is already started).<br>
    <br>
    <br>
    <div class="moz-cite-prefix">On 12/29/2016 02:27 PM, Jackie Tung
      wrote:<br>
    </div>
    <blockquote cite="mid:5FAD1025-9B21-4DF2-A30F-6837920929AB@drive.ai"
      type="cite">
      <meta http-equiv="Content-Type" content="text/html;
        charset=windows-1252">
      <div class="">Ravi,</div>
      <div class=""><br class="">
      </div>
      Got it thanks.  I’ve kicked this off, it seems be doing OK.
      <div class=""><br class="">
      </div>
      <div class="">I am a little concerned about a slow creep of memory
        usage:</div>
      <div class=""><br class="">
      </div>
      <div class="">* swap (64GB) completed filled up on server_1</div>
      <div class="">* general memory usage creeping up slowly over time.</div>
      <div class=""><br class="">
      </div>
      <div class="">
        <div style="margin: 0px; font-size: 10px; line-height: normal;
          font-family: Monaco; color: rgb(204, 204, 204);
          background-color: rgba(0, 0, 0, 0.85098);" class=""><span
            style="font-variant-ligatures: no-common-ligatures" class="">$
            free -m</span></div>
        <div style="margin: 0px; font-size: 10px; line-height: normal;
          font-family: Monaco; color: rgb(204, 204, 204);
          background-color: rgba(0, 0, 0, 0.85098);" class=""><span
            style="font-variant-ligatures: no-common-ligatures" class=""> 
                        total        used        free      shared 
            buff/cache   available</span></div>
        <div style="margin: 0px; font-size: 10px; line-height: normal;
          font-family: Monaco; color: rgb(204, 204, 204);
          background-color: rgba(0, 0, 0, 0.85098);" class=""><span
            style="font-variant-ligatures: no-common-ligatures" class="">Mem:
                    128829       55596         614          53      
            72618       71783</span></div>
        <div style="margin: 0px; font-size: 10px; line-height: normal;
          font-family: Monaco; color: rgb(204, 204, 204);
          background-color: rgba(0, 0, 0, 0.85098);" class=""><span
            style="font-variant-ligatures: no-common-ligatures" class="">Swap:
                    61034       61034           0</span></div>
      </div>
      <div class=""><br class="">
      </div>
      <div class="">Similar issue on server_2, though lower starting
        memory usage:</div>
      <div class=""><br class="">
      </div>
      <div class="">The “available” number is slowly going down - at
        this rate, probably will go to 0 before heal is done.</div>
      <div class=""><br class="">
      </div>
      <div class="">We are actually running 3.8.6, I’d like to try to
        pause the heal, upgrade to 3.8.7, and resume.  Is this possible
        heal suspend/resume possible or advisable?</div>
      <div class=""><br class="">
      </div>
      <div class="">The upgrade idea came from this on bugzilla (not
        100% if it will help my leak):</div>
      <div class=""><br class="">
      </div>
      <div class=""><a moz-do-not-send="true"
          href="https://bugzilla.redhat.com/show_bug.cgi?id=1400927"
          class="">https://bugzilla.redhat.com/show_bug.cgi?id=1400927</a></div>
      <div class=""><br class="">
      </div>
      <div class="">Even without doing the upgrade, I may need to
        restart glusterfs-server anyway to reset memory usage.</div>
      <div class=""><br class="">
      </div>
      <div class="">Thanks,</div>
      <div class="">Jackie</div>
      <div class=""><br class="">
        <div>
          <blockquote type="cite" class="">
            <div class="">On Dec 28, 2016, at 9:40 PM, Ravishankar N
              &lt;<a moz-do-not-send="true"
                href="mailto:ravishankar@redhat.com" class="">ravishankar@redhat.com</a>&gt;
              wrote:</div>
            <br class="Apple-interchange-newline">
            <div class="">
              <meta content="text/html; charset=windows-1252"
                http-equiv="Content-Type" class="">
              <div bgcolor="#FFFFFF" text="#000000" class="">
                <div class="moz-cite-prefix">On 12/29/2016 10:46 AM,
                  Jackie Tung wrote:<br class="">
                </div>
                <blockquote
cite="mid:CAG0reA6KWA3C8xxjw6iUFyvRuYuFED7-XyqU1x5MppNifZ=7jQ@mail.gmail.com"
                  type="cite" class="">
                  <div dir="auto" class="">Thanks very much for the
                    advice.
                    <div dir="auto" class=""><br class="">
                    </div>
                    <div dir="auto" class="">Would you mind elaborating
                      on the "no io" recommendation?  It's somewhat hard
                      for me to guarantee this without a long
                      maintenance window.</div>
                    <div dir="auto" class=""><br class="">
                    </div>
                    <div dir="auto" class="">What is the consequence of
                      having IO at point of add-brick, and for the heal
                      period afterwards?</div>
                  </div>
                </blockquote>
                <br class="">
                Sorry I wasn't clear. Since you're running  16
                distribute legs (16x2), a lot of self-heals would be
                running and there is a chance that clients might
                experience slowness due to the self-heals. Other than
                that it should be fine.<br class="">
                Thanks,<br class="">
                Ravi<br class="">
                <br class="">
                <blockquote
cite="mid:CAG0reA6KWA3C8xxjw6iUFyvRuYuFED7-XyqU1x5MppNifZ=7jQ@mail.gmail.com"
                  type="cite" class="">
                  <div dir="auto" class="">
                    <div dir="auto" class=""><br class="">
                    </div>
                    <div dir="auto" class=""><br class="">
                    </div>
                  </div>
                  <div class="gmail_extra"><br class="">
                    <div class="gmail_quote">On Dec 28, 2016 8:27 PM,
                      "Ravishankar N" &lt;<a moz-do-not-send="true"
                        href="mailto:ravishankar@redhat.com" class="">ravishankar@redhat.com</a>&gt;
                      wrote:<br type="attribution" class="">
                      <blockquote class="quote" style="margin:0 0 0
                        .8ex;border-left:1px #ccc
                        solid;padding-left:1ex">
                        <div bgcolor="#FFFFFF" text="#000000" class="">
                          <div class="elided-text">
                            <div
                              class="m_5051585420273836920moz-cite-prefix">On
                              12/29/2016 07:30 AM, Jackie Tung wrote:<br
                                class="">
                            </div>
                            <blockquote type="cite" class="">
                              <div dir="auto" class="">Version is 3.8.7
                                on Ubuntu xenial.</div>
                              <div class="gmail_extra"><br class="">
                                <div class="gmail_quote">On Dec 28, 2016
                                  5:56 PM, "Jackie Tung" &lt;<a
                                    moz-do-not-send="true"
                                    href="mailto:jackie@drive.ai"
                                    target="_blank" class="">jackie@drive.ai</a>&gt;
                                  wrote:<br type="attribution" class="">
                                  <blockquote class="gmail_quote"
                                    style="margin:0 0 0
                                    .8ex;border-left:1px #ccc
                                    solid;padding-left:1ex">
                                    <div dir="auto" class="">If someone
                                      has experience to share in this
                                      area, i'd be grateful.  I have an
                                      existing distributed replicated
                                      volume, 2x16.
                                      <div dir="auto" class=""><br
                                          class="">
                                      </div>
                                      <div dir="auto" class="">We have a
                                        third server ready to go. 
                                        Redhat docs say just run add
                                        brick replica 3, then run
                                        rebalance.</div>
                                      <div dir="auto" class=""><br
                                          class="">
                                      </div>
                                      <div dir="auto" class="">The
                                        rebalance step feels a bit off
                                        to me.  Isn't some kind of heal
                                        operation in order rather than
                                        rebalance?</div>
                                      <div dir="auto" class=""><br
                                          class="">
                                      </div>
                                      <div dir="auto" class="">No
                                        additional usable space will be
                                        introduced, only replica count
                                        increase from 2 to 3.</div>
                                    </div>
                                  </blockquote>
                                </div>
                              </div>
                            </blockquote>
                            <br class="">
                          </div>
                          You don't need to run re-balance for
                          increasing the replica count. Heals should
                          automatically be triggered when you run
                          'gluster vol add-brick &lt;volname&gt; replica
                          3 &lt;list of bricks for the 3rd replica&gt;`.
                          It is advisable to do this when there is no
                          I/O happening  on the volume. You can verify
                          that files are getting populated in the newly
                          added bricks post running the command. <br
                            class="">
                          <br class="">
                          -Ravi<br class="">
                          <blockquote type="cite" class="">
                            <div class="gmail_extra">
                              <div class="gmail_quote">
                                <blockquote class="gmail_quote"
                                  style="margin:0 0 0
                                  .8ex;border-left:1px #ccc
                                  solid;padding-left:1ex">
                                  <div dir="auto" class="">
                                    <div dir="auto" class=""><br
                                        class="">
                                    </div>
                                    <div dir="auto" class="">Thanks</div>
                                    <div dir="auto" class="">Jackie</div>
                                  </div>
                                </blockquote>
                              </div>
                            </div>
                            <br class="">
                            <p class=""><font class="" face="Courier
                                New" color="#808080" size="1">The
                                information in this email is
                                confidential and may be legally
                                privileged. It is intended solely for
                                the addressee. Access to this email by
                                anyone else is unauthorized. If you are
                                not the intended recipient, any
                                disclosure, copying, distribution or any
                                action taken or omitted to be taken in
                                reliance on it, is prohibited and may be
                                unlawful.</font></p>
                            <br class="">
                            <fieldset
                              class="m_5051585420273836920mimeAttachmentHeader"></fieldset>
                            <br class="">
                            <pre class="">______________________________<wbr class="">_________________
Gluster-users mailing list
<a moz-do-not-send="true" class="m_5051585420273836920moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="m_5051585420273836920moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/<wbr class="">mailman/listinfo/gluster-users</a></pre>
    </blockquote><div class="">

    
</div>
  </div>

</blockquote></div>
</div><p class=""><font class="" face="Courier New" color="#808080" size="1">The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful.</font></p>


</blockquote><div class="">

</div></div></div></blockquote></div>
</div>










<p><font face="Courier New" color="#808080" size="1">The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful.</font></p>

<fieldset class="mimeAttachmentHeader"></fieldset>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>

</blockquote>
</body></html>