<div dir="ltr">My test servers have been running for about 3 hours now (with the while loop to constantly write and delete files) and it looks like the memory usage of the arbiter brick process has not increased in the past hour. Before it was constantly increasing, so it looks like adding the "<span style="font-size:12.8px">GF_FREE (ctx->iattbuf);" line in arbiter.c fixed the issue. If anything changes overnight I will post an update, but I believe that the fix worked!</span><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Once this patch makes it into the master branch, how long does it usually take to get released back to 3.8?</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Thanks!</span></div><div><span style="font-size:12.8px">Ben</span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 23, 2016 at 2:18 PM, Benjamin Edgar <span dir="ltr"><<a href="mailto:benedgar8@gmail.com" target="_blank">benedgar8@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Ravi,<div><br></div><div>I saw that you updated the patch today (<span style="font-size:12.8px">@ </span><a href="http://review.gluster.org/#/c/15289/" style="font-size:12.8px" target="_blank">http://review.gluster.org/#<wbr>/c/15289/</a>). I built an RPM of the first iteration you had of the patch (just changing the one line in arbiter.c "GF_FREE (ctx->iattbuf);") and am running that on some test servers now to see if the memory of the arbiter brick gets out of control.</div><div><br></div><div>Ben</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 23, 2016 at 3:38 AM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hi Benjamin<span><br>
<br>
On 08/23/2016 06:41 AM, Benjamin Edgar wrote:<br>
</span></div><span>
<blockquote type="cite">
<div dir="ltr">I've attached a statedump of the problem brick
process. Let me know if there are any other logs you need.</div>
</blockquote>
<br></span>
Thanks for the report! I've sent a fix @
<a href="http://review.gluster.org/#/c/15289/" target="_blank">http://review.gluster.org/#/c/<wbr>15289/</a> . It would be nice if you can
verify if the patch fixes the issue for you.<br>
<br>
Thanks,<br>
Ravi<div><div><br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Thanks a lot,</div>
<div>Ben</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Aug 22, 2016 at 5:03 PM,
Pranith Kumar Karampuri <span dir="ltr"><<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>Could you collect statedump of the brick process by
following: <a href="https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump" target="_blank">https://gluster.readthedocs.io<wbr>/en/latest/Troubleshooting/sta<wbr>tedump</a><br>
<br>
</div>
That should help us identify which datatype is causing
leaks and fix it.<br>
<br>
</div>
Thanks!<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">
<div>
<div>On Tue, Aug 23, 2016 at 2:22 AM,
Benjamin Edgar <span dir="ltr"><<a href="mailto:benedgar8@gmail.com" target="_blank">benedgar8@gmail.com</a>></span>
wrote:<br>
</div>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div>
<div dir="ltr">Hi,
<div><br>
</div>
<div>I appear to have a memory leak with a
replica 3 arbiter 1 configuration of gluster.
I have a data brick and an arbiter brick on
one server, and another server with the last
data brick. The more I write files to gluster
in this configuration, the more memory the
arbiter brick process takes up.</div>
<div><br>
</div>
<div>I am able to reproduce this issue by first
setting up a replica 3 arbiter 1 configuration
and then using the following bash script to
create 10,000 200kB files, delete those files,
and run forever:</div>
<div><br>
</div>
<div>
<div>while true ; do</div>
<div> for i in {1..10000} ; do</div>
<div> dd if=/dev/urandom bs=200K count=1
of=$TEST_FILES_DIR/file$i</div>
<div> done</div>
<div> rm -rf $TEST_FILES_DIR/*<br>
</div>
<div>done<br>
</div>
</div>
<div><br>
</div>
<div>$TEST_FILES_DIR is a location on my gluster
mount.</div>
<div><br>
</div>
<div>After about 3 days of this script running
on one of my clusters, this is what the output
of "top" looks like:</div>
<div>
<div> PID USER PR NI VIRT
RES SHR S %CPU %MEM TIME+
COMMAND</div>
<div>16039 root 20 0 1397220
77720 3948 S 20.6 1.0
860:01.53 glusterfsd</div>
<div>13174 root 20 0 1395824
112728 3692 S 19.6 1.5
806:07.17 glusterfs</div>
<div>19961 root 20 0 2967204 <b>2.145g</b>
3896 S 17.3 29.0 752:10.70
glusterfsd</div>
</div>
<div><br>
</div>
<div>As you can see one of the brick processes
is using over 2 gigabytes of memory.</div>
<div><br>
</div>
<div>One work-around for this is to kill the
arbiter brick process and restart the gluster
daemon. This restarts arbiter brick process
and its memory usage goes back down to a
reasonable level. However I would rather not
kill the arbiter brick every week for
production environments.</div>
<div><br>
</div>
<div>Has anyone seen this issue before and is
there a known work-around/fix?</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Ben</div>
</div>
<br>
</div>
</div>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a><span><font color="#888888"><br>
</font></span></blockquote>
</div>
<span><font color="#888888"><br>
<br clear="all">
<br>
-- <br>
<div data-smartmail="gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div data-smartmail="gmail_signature">
<div dir="ltr"><span style="font-family:arial;font-size:small">Benjamin
Edgar</span>
<div style="font-family:arial;font-size:small">Computer
Science</div>
<div style="font-family:arial;font-size:small">University of
Virginia 2015</div>
<div style="font-family:arial;font-size:small">(571)
338-0878</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>______________________________<wbr>_________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a></pre>
</blockquote>
<p><br>
</p>
</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><span style="font-family:arial;font-size:small">Benjamin Edgar</span><div style="font-family:arial;font-size:small">Computer Science</div><div style="font-family:arial;font-size:small">University of Virginia 2015</div><div style="font-family:arial;font-size:small"><a href="tel:%28571%29%20338-0878" value="+15713380878" target="_blank">(571) 338-0878</a></div></div></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><span style="font-family:arial;font-size:small">Benjamin Edgar</span><div style="font-family:arial;font-size:small">Computer Science</div><div style="font-family:arial;font-size:small">University of Virginia 2015</div><div style="font-family:arial;font-size:small">(571) 338-0878</div></div></div>
</div>