<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Ravi,</div><div class=""><br class=""></div>Got it thanks. I’ve kicked this off, it seems be doing OK.<div class=""><br class=""></div><div class="">I am a little concerned about a slow creep of memory usage:</div><div class=""><br class=""></div><div class="">* swap (64GB) completed filled up on server_1</div><div class="">* general memory usage creeping up slowly over time.</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(204, 204, 204); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">$ free -m</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(204, 204, 204); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total used free shared buff/cache available</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(204, 204, 204); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Mem: 128829 55596 614 53 72618 71783</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(204, 204, 204); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Swap: 61034 61034 0</span></div></div><div class=""><br class=""></div><div class="">Similar issue on server_2, though lower starting memory usage:</div><div class=""><br class=""></div><div class="">The “available” number is slowly going down - at this rate, probably will go to 0 before heal is done.</div><div class=""><br class=""></div><div class="">We are actually running 3.8.6, I’d like to try to pause the heal, upgrade to 3.8.7, and resume. Is this possible heal suspend/resume possible or advisable?</div><div class=""><br class=""></div><div class="">The upgrade idea came from this on bugzilla (not 100% if it will help my leak):</div><div class=""><br class=""></div><div class=""><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1400927" class="">https://bugzilla.redhat.com/show_bug.cgi?id=1400927</a></div><div class=""><br class=""></div><div class="">Even without doing the upgrade, I may need to restart glusterfs-server anyway to reset memory usage.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Jackie</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Dec 28, 2016, at 9:40 PM, Ravishankar N <<a href="mailto:ravishankar@redhat.com" class="">ravishankar@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type" class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
<div class="moz-cite-prefix">On 12/29/2016 10:46 AM, Jackie Tung
wrote:<br class="">
</div>
<blockquote cite="mid:CAG0reA6KWA3C8xxjw6iUFyvRuYuFED7-XyqU1x5MppNifZ=7jQ@mail.gmail.com" type="cite" class="">
<div dir="auto" class="">Thanks very much for the advice.
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">Would you mind elaborating on the "no io"
recommendation? It's somewhat hard for me to guarantee this
without a long maintenance window.</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">What is the consequence of having IO at point of
add-brick, and for the heal period afterwards?</div>
</div>
</blockquote>
<br class="">
Sorry I wasn't clear. Since you're running 16 distribute legs
(16x2), a lot of self-heals would be running and there is a chance
that clients might experience slowness due to the self-heals. Other
than that it should be fine.<br class="">
Thanks,<br class="">
Ravi<br class="">
<br class="">
<blockquote cite="mid:CAG0reA6KWA3C8xxjw6iUFyvRuYuFED7-XyqU1x5MppNifZ=7jQ@mail.gmail.com" type="cite" class="">
<div dir="auto" class="">
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class=""><br class="">
</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Dec 28, 2016 8:27 PM, "Ravishankar
N" <<a moz-do-not-send="true" href="mailto:ravishankar@redhat.com" class="">ravishankar@redhat.com</a>>
wrote:<br type="attribution" class="">
<blockquote class="quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000" class="">
<div class="elided-text">
<div class="m_5051585420273836920moz-cite-prefix">On
12/29/2016 07:30 AM, Jackie Tung wrote:<br class="">
</div>
<blockquote type="cite" class="">
<div dir="auto" class="">Version is 3.8.7 on Ubuntu xenial.</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Dec 28, 2016 5:56 PM,
"Jackie Tung" <<a moz-do-not-send="true" href="mailto:jackie@drive.ai" target="_blank" class="">jackie@drive.ai</a>>
wrote:<br type="attribution" class="">
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="auto" class="">If someone has experience to
share in this area, i'd be grateful. I have
an existing distributed replicated volume,
2x16.
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">We have a third server ready
to go. Redhat docs say just run add brick
replica 3, then run rebalance.</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">The rebalance step feels a bit
off to me. Isn't some kind of heal
operation in order rather than rebalance?</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">No additional usable space
will be introduced, only replica count
increase from 2 to 3.</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br class="">
</div>
You don't need to run re-balance for increasing the
replica count. Heals should automatically be triggered
when you run 'gluster vol add-brick <volname>
replica 3 <list of bricks for the 3rd replica>`. It
is advisable to do this when there is no I/O happening on
the volume. You can verify that files are getting
populated in the newly added bricks post running the
command. <br class="">
<br class="">
-Ravi<br class="">
<blockquote type="cite" class="">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto" class="">
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">Thanks</div>
<div dir="auto" class="">Jackie</div>
</div>
</blockquote>
</div>
</div>
<br class=""><p class=""><font color="#808080" face="Courier New" size="1" class="">The
information in this email is confidential and may be
legally privileged. It is intended solely for the
addressee. Access to this email by anyone else is
unauthorized. If you are not the intended recipient,
any disclosure, copying, distribution or any action
taken or omitted to be taken in reliance on it, is
prohibited and may be unlawful.</font></p>
<br class="">
<fieldset class="m_5051585420273836920mimeAttachmentHeader"></fieldset>
<br class="">
<pre class="">______________________________<wbr class="">_________________
Gluster-users mailing list
<a moz-do-not-send="true" class="m_5051585420273836920moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="m_5051585420273836920moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/<wbr class="">mailman/listinfo/gluster-users</a></pre>
</blockquote><div class="">
<br class="webkit-block-placeholder"></div>
</div>
</blockquote></div>
</div><p class=""><font color="#808080" face="Courier New" size="1" class="">The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful.</font></p>
</blockquote><div class="">
<br class="webkit-block-placeholder"></div></div></div></blockquote></div><br class=""></div></body></html>
<br>
<p><font color="#808080" face="Courier New" size="1">The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful.</font></p>