<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Ravi,</div><div class=""><br class=""></div>Got it thanks. &nbsp;I’ve kicked this off, it seems be doing OK.<div class=""><br class=""></div><div class="">I am a little concerned about a slow creep of memory usage:</div><div class=""><br class=""></div><div class="">* swap (64GB) completed filled up on server_1</div><div class="">* general memory usage creeping up slowly over time.</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(204, 204, 204); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">$ free -m</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(204, 204, 204); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; total&nbsp; &nbsp; &nbsp; &nbsp; used&nbsp; &nbsp; &nbsp; &nbsp; free&nbsp; &nbsp; &nbsp; shared&nbsp; buff/cache &nbsp; available</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(204, 204, 204); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Mem: &nbsp; &nbsp; &nbsp; &nbsp; 128829 &nbsp; &nbsp; &nbsp; 55596 &nbsp; &nbsp; &nbsp; &nbsp; 614&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 53 &nbsp; &nbsp; &nbsp; 72618 &nbsp; &nbsp; &nbsp; 71783</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(204, 204, 204); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Swap: &nbsp; &nbsp; &nbsp; &nbsp; 61034 &nbsp; &nbsp; &nbsp; 61034 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0</span></div></div><div class=""><br class=""></div><div class="">Similar issue on server_2, though lower starting memory usage:</div><div class=""><br class=""></div><div class="">The “available” number is slowly going down - at this rate, probably will go to 0 before heal is done.</div><div class=""><br class=""></div><div class="">We are actually running 3.8.6, I’d like to try to pause the heal, upgrade to 3.8.7, and resume. &nbsp;Is this possible heal suspend/resume possible or advisable?</div><div class=""><br class=""></div><div class="">The upgrade idea came from this on bugzilla (not 100% if it will help my leak):</div><div class=""><br class=""></div><div class=""><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1400927" class="">https://bugzilla.redhat.com/show_bug.cgi?id=1400927</a></div><div class=""><br class=""></div><div class="">Even without doing the upgrade, I may need to restart glusterfs-server anyway to reset memory usage.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Jackie</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Dec 28, 2016, at 9:40 PM, Ravishankar N &lt;<a href="mailto:ravishankar@redhat.com" class="">ravishankar@redhat.com</a>&gt; wrote:</div><br class="Apple-interchange-newline"><div class="">
  
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type" class="">
  
  <div bgcolor="#FFFFFF" text="#000000" class="">
    <div class="moz-cite-prefix">On 12/29/2016 10:46 AM, Jackie Tung
      wrote:<br class="">
    </div>
    <blockquote cite="mid:CAG0reA6KWA3C8xxjw6iUFyvRuYuFED7-XyqU1x5MppNifZ=7jQ@mail.gmail.com" type="cite" class="">
      <div dir="auto" class="">Thanks very much for the advice.
        <div dir="auto" class=""><br class="">
        </div>
        <div dir="auto" class="">Would you mind elaborating on the "no io"
          recommendation?&nbsp; It's somewhat hard for me to guarantee this
          without a long maintenance window.</div>
        <div dir="auto" class=""><br class="">
        </div>
        <div dir="auto" class="">What is the consequence of having IO at point of
          add-brick, and for the heal period afterwards?</div>
      </div>
    </blockquote>
    <br class="">
    Sorry I wasn't clear. Since you're running&nbsp; 16 distribute legs
    (16x2), a lot of self-heals would be running and there is a chance
    that clients might experience slowness due to the self-heals. Other
    than that it should be fine.<br class="">
    Thanks,<br class="">
    Ravi<br class="">
    <br class="">
    <blockquote cite="mid:CAG0reA6KWA3C8xxjw6iUFyvRuYuFED7-XyqU1x5MppNifZ=7jQ@mail.gmail.com" type="cite" class="">
      <div dir="auto" class="">
        <div dir="auto" class=""><br class="">
        </div>
        <div dir="auto" class=""><br class="">
        </div>
      </div>
      <div class="gmail_extra"><br class="">
        <div class="gmail_quote">On Dec 28, 2016 8:27 PM, "Ravishankar
          N" &lt;<a moz-do-not-send="true" href="mailto:ravishankar@redhat.com" class="">ravishankar@redhat.com</a>&gt;
          wrote:<br type="attribution" class="">
          <blockquote class="quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div bgcolor="#FFFFFF" text="#000000" class="">
              <div class="elided-text">
                <div class="m_5051585420273836920moz-cite-prefix">On
                  12/29/2016 07:30 AM, Jackie Tung wrote:<br class="">
                </div>
                <blockquote type="cite" class="">
                  <div dir="auto" class="">Version is 3.8.7 on Ubuntu xenial.</div>
                  <div class="gmail_extra"><br class="">
                    <div class="gmail_quote">On Dec 28, 2016 5:56 PM,
                      "Jackie Tung" &lt;<a moz-do-not-send="true" href="mailto:jackie@drive.ai" target="_blank" class="">jackie@drive.ai</a>&gt;
                      wrote:<br type="attribution" class="">
                      <blockquote class="gmail_quote" style="margin:0 0
                        0 .8ex;border-left:1px #ccc
                        solid;padding-left:1ex">
                        <div dir="auto" class="">If someone has experience to
                          share in this area, i'd be grateful.&nbsp; I have
                          an existing distributed replicated volume,
                          2x16.
                          <div dir="auto" class=""><br class="">
                          </div>
                          <div dir="auto" class="">We have a third server ready
                            to go.&nbsp; Redhat docs say just run add brick
                            replica 3, then run rebalance.</div>
                          <div dir="auto" class=""><br class="">
                          </div>
                          <div dir="auto" class="">The rebalance step feels a bit
                            off to me.&nbsp; Isn't some kind of heal
                            operation in order rather than rebalance?</div>
                          <div dir="auto" class=""><br class="">
                          </div>
                          <div dir="auto" class="">No additional usable space
                            will be introduced, only replica count
                            increase from 2 to 3.</div>
                        </div>
                      </blockquote>
                    </div>
                  </div>
                </blockquote>
                <br class="">
              </div>
              You don't need to run re-balance for increasing the
              replica count. Heals should automatically be triggered
              when you run 'gluster vol add-brick &lt;volname&gt;
              replica 3 &lt;list of bricks for the 3rd replica&gt;`. It
              is advisable to do this when there is no I/O happening&nbsp; on
              the volume. You can verify that files are getting
              populated in the newly added bricks post running the
              command. <br class="">
              <br class="">
              -Ravi<br class="">
              <blockquote type="cite" class="">
                <div class="gmail_extra">
                  <div class="gmail_quote">
                    <blockquote class="gmail_quote" style="margin:0 0 0
                      .8ex;border-left:1px #ccc solid;padding-left:1ex">
                      <div dir="auto" class="">
                        <div dir="auto" class=""><br class="">
                        </div>
                        <div dir="auto" class="">Thanks</div>
                        <div dir="auto" class="">Jackie</div>
                      </div>
                    </blockquote>
                  </div>
                </div>
                <br class=""><p class=""><font color="#808080" face="Courier New" size="1" class="">The
                    information in this email is confidential and may be
                    legally privileged. It is intended solely for the
                    addressee. Access to this email by anyone else is
                    unauthorized. If you are not the intended recipient,
                    any disclosure, copying, distribution or any action
                    taken or omitted to be taken in reliance on it, is
                    prohibited and may be unlawful.</font></p>
                <br class="">
                <fieldset class="m_5051585420273836920mimeAttachmentHeader"></fieldset>
                <br class="">
                <pre class="">______________________________<wbr class="">_________________
Gluster-users mailing list
<a moz-do-not-send="true" class="m_5051585420273836920moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="m_5051585420273836920moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/<wbr class="">mailman/listinfo/gluster-users</a></pre>
    </blockquote><div class="">

    <br class="webkit-block-placeholder"></div>
  </div>

</blockquote></div>
</div><p class=""><font color="#808080" face="Courier New" size="1" class="">The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful.</font></p>


</blockquote><div class="">
<br class="webkit-block-placeholder"></div></div></div></blockquote></div><br class=""></div></body></html>
<br>








<p><font color="#808080" face="Courier New" size="1">The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful.</font></p>