<div dir="ltr">Hi Ravi,<div><br></div><div>I saw that you updated the patch today (<span style="font-size:12.8px">@ </span><a href="http://review.gluster.org/#/c/15289/" target="_blank" style="font-size:12.8px">http://review.gluster.org/#/c/<wbr>15289/</a>). I built an RPM of the first iteration you had of the patch (just changing the one line in arbiter.c "GF_FREE (ctx->iattbuf);") and am running that on some test servers now to see if the memory of the arbiter brick gets out of control.</div><div><br></div><div>Ben</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 23, 2016 at 3:38 AM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hi Benjamin<span class=""><br>
<br>
On 08/23/2016 06:41 AM, Benjamin Edgar wrote:<br>
</span></div><span class="">
<blockquote type="cite">
<div dir="ltr">I've attached a statedump of the problem brick
process. Let me know if there are any other logs you need.</div>
</blockquote>
<br></span>
Thanks for the report! I've sent a fix @
<a href="http://review.gluster.org/#/c/15289/" target="_blank">http://review.gluster.org/#/c/<wbr>15289/</a> . It would be nice if you can
verify if the patch fixes the issue for you.<br>
<br>
Thanks,<br>
Ravi<div><div class="h5"><br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Thanks a lot,</div>
<div>Ben</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Aug 22, 2016 at 5:03 PM,
Pranith Kumar Karampuri <span dir="ltr"><<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>Could you collect statedump of the brick process by
following: <a href="https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump" target="_blank">https://gluster.readthedocs.io<wbr>/en/latest/Troubleshooting/sta<wbr>tedump</a><br>
<br>
</div>
That should help us identify which datatype is causing
leaks and fix it.<br>
<br>
</div>
Thanks!<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">
<div>
<div>On Tue, Aug 23, 2016 at 2:22 AM,
Benjamin Edgar <span dir="ltr"><<a href="mailto:benedgar8@gmail.com" target="_blank">benedgar8@gmail.com</a>></span>
wrote:<br>
</div>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div>
<div dir="ltr">Hi,
<div><br>
</div>
<div>I appear to have a memory leak with a
replica 3 arbiter 1 configuration of gluster.
I have a data brick and an arbiter brick on
one server, and another server with the last
data brick. The more I write files to gluster
in this configuration, the more memory the
arbiter brick process takes up.</div>
<div><br>
</div>
<div>I am able to reproduce this issue by first
setting up a replica 3 arbiter 1 configuration
and then using the following bash script to
create 10,000 200kB files, delete those files,
and run forever:</div>
<div><br>
</div>
<div>
<div>while true ; do</div>
<div> for i in {1..10000} ; do</div>
<div> dd if=/dev/urandom bs=200K count=1
of=$TEST_FILES_DIR/file$i</div>
<div> done</div>
<div> rm -rf $TEST_FILES_DIR/*<br>
</div>
<div>done<br>
</div>
</div>
<div><br>
</div>
<div>$TEST_FILES_DIR is a location on my gluster
mount.</div>
<div><br>
</div>
<div>After about 3 days of this script running
on one of my clusters, this is what the output
of "top" looks like:</div>
<div>
<div> PID USER PR NI VIRT
RES SHR S %CPU %MEM TIME+
COMMAND</div>
<div>16039 root 20 0 1397220
77720 3948 S 20.6 1.0
860:01.53 glusterfsd</div>
<div>13174 root 20 0 1395824
112728 3692 S 19.6 1.5
806:07.17 glusterfs</div>
<div>19961 root 20 0 2967204 <b>2.145g</b>
3896 S 17.3 29.0 752:10.70
glusterfsd</div>
</div>
<div><br>
</div>
<div>As you can see one of the brick processes
is using over 2 gigabytes of memory.</div>
<div><br>
</div>
<div>One work-around for this is to kill the
arbiter brick process and restart the gluster
daemon. This restarts arbiter brick process
and its memory usage goes back down to a
reasonable level. However I would rather not
kill the arbiter brick every week for
production environments.</div>
<div><br>
</div>
<div>Has anyone seen this issue before and is
there a known work-around/fix?</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Ben</div>
</div>
<br>
</div>
</div>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a><span><font color="#888888"><br>
</font></span></blockquote>
</div>
<span><font color="#888888"><br>
<br clear="all">
<br>
-- <br>
<div data-smartmail="gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div data-smartmail="gmail_signature">
<div dir="ltr"><span style="font-family:arial;font-size:small">Benjamin
Edgar</span>
<div style="font-family:arial;font-size:small">Computer
Science</div>
<div style="font-family:arial;font-size:small">University of
Virginia 2015</div>
<div style="font-family:arial;font-size:small">(571)
338-0878</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>______________________________<wbr>_________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-users</a></pre>
</blockquote>
<p><br>
</p>
</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><span style="font-family:arial;font-size:small">Benjamin Edgar</span><div style="font-family:arial;font-size:small">Computer Science</div><div style="font-family:arial;font-size:small">University of Virginia 2015</div><div style="font-family:arial;font-size:small">(571) 338-0878</div></div></div>
</div>