<div dir="auto">Thank you.<div dir="auto"><br></div><div dir="auto">It was glusterd and the bricks.<div dir="auto"><br></div><div dir="auto">I sent TERM to glusterd, then restarted the systemd service. This actually caused all bricks to be restarted as well. Is this a right way to do this?</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Dec 29, 2016 3:01 PM, "Joe Julian" <<a href="mailto:joe@julianfamily.org">joe@julianfamily.org</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Which application is filling memory? <br>
<br>
If it's a brick (glusterfsd) then stopping and starting a brick
("kill" and "gluster volume start ... force") will not waste
cycles re-healing any files that are healthy. Any heals of an
individual file that were not complete will be restarted as well
as any files that were changed while the brick was offline.<br>
</p>
If it's glusterd, that can be restarted at any time without
interfering with the volume.<br>
<br>
If it's glustershd ("/usr/bin/glusterfs -s localhost --volfile-id
gluster/glustershd ..."), you can restart that with "gluster volume
start ... force" (even if the volume is already started).<br>
<br>
<br>
<div class="m_7264050636128832157moz-cite-prefix">On 12/29/2016 02:27 PM, Jackie Tung
wrote:<br>
</div>
<blockquote type="cite">
<div>Ravi,</div>
<div><br>
</div>
Got it thanks. I’ve kicked this off, it seems be doing OK.
<div><br>
</div>
<div>I am a little concerned about a slow creep of memory
usage:</div>
<div><br>
</div>
<div>* swap (64GB) completed filled up on server_1</div>
<div>* general memory usage creeping up slowly over time.</div>
<div><br>
</div>
<div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(204,204,204);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">$
free -m</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(204,204,204);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">
total used free shared
buff/cache available</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(204,204,204);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">Mem:
128829 55596 614 53
72618 71783</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(204,204,204);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">Swap:
61034 61034 0</span></div>
</div>
<div><br>
</div>
<div>Similar issue on server_2, though lower starting
memory usage:</div>
<div><br>
</div>
<div>The “available” number is slowly going down - at
this rate, probably will go to 0 before heal is done.</div>
<div><br>
</div>
<div>We are actually running 3.8.6, I’d like to try to
pause the heal, upgrade to 3.8.7, and resume. Is this possible
heal suspend/resume possible or advisable?</div>
<div><br>
</div>
<div>The upgrade idea came from this on bugzilla (not
100% if it will help my leak):</div>
<div><br>
</div>
<div><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1400927" target="_blank">https://bugzilla.redhat.com/<wbr>show_bug.cgi?id=1400927</a></div>
<div><br>
</div>
<div>Even without doing the upgrade, I may need to
restart glusterfs-server anyway to reset memory usage.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Jackie</div>
<div><br>
<div>
<blockquote type="cite">
<div>On Dec 28, 2016, at 9:40 PM, Ravishankar N
<<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>>
wrote:</div>
<br class="m_7264050636128832157Apple-interchange-newline">
<div>
<div bgcolor="#FFFFFF" text="#000000">
<div class="m_7264050636128832157moz-cite-prefix">On 12/29/2016 10:46 AM,
Jackie Tung wrote:<br>
</div>
<blockquote type="cite">
<div dir="auto">Thanks very much for the
advice.
<div dir="auto"><br>
</div>
<div dir="auto">Would you mind elaborating
on the "no io" recommendation? It's somewhat hard
for me to guarantee this without a long
maintenance window.</div>
<div dir="auto"><br>
</div>
<div dir="auto">What is the consequence of
having IO at point of add-brick, and for the heal
period afterwards?</div>
</div>
</blockquote>
<br>
Sorry I wasn't clear. Since you're running 16
distribute legs (16x2), a lot of self-heals would be
running and there is a chance that clients might
experience slowness due to the self-heals. Other than
that it should be fine.<br>
Thanks,<br>
Ravi<br>
<br>
<blockquote type="cite">
<div dir="auto">
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Dec 28, 2016 8:27 PM,
"Ravishankar N" <<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>>
wrote:<br type="attribution">
<blockquote class="m_7264050636128832157quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div class="m_7264050636128832157elided-text">
<div class="m_7264050636128832157m_5051585420273836920moz-cite-prefix">On
12/29/2016 07:30 AM, Jackie Tung wrote:<br>
</div>
<blockquote type="cite">
<div dir="auto">Version is 3.8.7
on Ubuntu xenial.</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Dec 28, 2016
5:56 PM, "Jackie Tung" <<a href="mailto:jackie@drive.ai" target="_blank">jackie@drive.ai</a>>
wrote:<br type="attribution">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">If someone
has experience to share in this
area, i'd be grateful. I have an
existing distributed replicated
volume, 2x16.
<div dir="auto"><br>
</div>
<div dir="auto">We have a
third server ready to go.
Redhat docs say just run add
brick replica 3, then run
rebalance.</div>
<div dir="auto"><br>
</div>
<div dir="auto">The
rebalance step feels a bit off
to me. Isn't some kind of heal
operation in order rather than
rebalance?</div>
<div dir="auto"><br>
</div>
<div dir="auto">No
additional usable space will be
introduced, only replica count
increase from 2 to 3.</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
</div>
You don't need to run re-balance for
increasing the replica count. Heals should
automatically be triggered when you run
'gluster vol add-brick <volname> replica
3 <list of bricks for the 3rd replica>`.
It is advisable to do this when there is no
I/O happening on the volume. You can verify
that files are getting populated in the newly
added bricks post running the command. <br>
<br>
-Ravi<br>
<blockquote type="cite">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div dir="auto"><br>
</div>
<div dir="auto">Thanks</div>
<div dir="auto">Jackie</div>
</div>
</blockquote>
</div>
</div>
<br>
<p><font face="Courier
New" color="#808080" size="1">The
information in this email is
confidential and may be legally
privileged. It is intended solely for
the addressee. Access to this email by
anyone else is unauthorized. If you are
not the intended recipient, any
disclosure, copying, distribution or any
action taken or omitted to be taken in
reliance on it, is prohibited and may be
unlawful.</font></p>
<br>
<fieldset class="m_7264050636128832157m_5051585420273836920mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>_________________
Gluster-users mailing list
<a class="m_7264050636128832157m_5051585420273836920moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a class="m_7264050636128832157m_5051585420273836920moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a></pre>
</blockquote><div>
</div>
</div>
</blockquote></div>
</div><p><font face="Courier New" color="#808080" size="1">The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful.</font></p>
</blockquote><div>
</div></div></div></blockquote></div>
</div>
<p><font face="Courier New" color="#808080" size="1">The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful.</font></p>
<fieldset class="m_7264050636128832157mimeAttachmentHeader"></fieldset>
<pre>______________________________<wbr>_________________
Gluster-users mailing list
<a class="m_7264050636128832157moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a class="m_7264050636128832157moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-users</a></pre>
</blockquote>
</div><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div></div>
<br>
<p><font color="#808080" face="Courier New" size="1">The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful.</font></p>