<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 08/24/2016 02:12 AM, Benjamin Edgar
wrote:<br>
</div>
<blockquote
cite="mid:CAN1QS65jWYWqq48vSv+AUP4X=Va6xZCBhCD8WRnTNGC4X6uYJg@mail.gmail.com"
type="cite">
<div dir="ltr">My test servers have been running for about 3 hours
now (with the while loop to constantly write and delete files)
and it looks like the memory usage of the arbiter brick process
has not increased in the past hour. Before it was constantly
increasing, so it looks like adding the "<span
style="font-size:12.8px">GF_FREE (ctx->iattbuf);" line in
arbiter.c fixed the issue. If anything changes overnight I
will post an update, but I believe that the fix worked!</span>
<div><span style="font-size:12.8px"><br>
</span></div>
<div><span style="font-size:12.8px">Once this patch makes it
into the master branch, how long does it usually take to get
released back to 3.8?</span></div>
<div><span style="font-size:12.8px"><br>
</span></div>
</div>
</blockquote>
Hi Ben,<br>
Thanks for testing. The minor release schedule [1] for 3.8.x is on
the 10th of every month. But an out of order 3.8.3 release was just
made. So maybe 3.8.4 would take a bit longer.<br>
<br>
Thanks,<br>
Ravi<br>
<br>
[1] <a class="moz-txt-link-freetext" href="https://www.gluster.org/community/release-schedule/">https://www.gluster.org/community/release-schedule/</a>
<blockquote
cite="mid:CAN1QS65jWYWqq48vSv+AUP4X=Va6xZCBhCD8WRnTNGC4X6uYJg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><span style="font-size:12.8px">Thanks!</span></div>
<div><span style="font-size:12.8px">Ben</span></div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Aug 23, 2016 at 2:18 PM,
Benjamin Edgar <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:benedgar8@gmail.com" target="_blank">benedgar8@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Ravi,
<div><br>
</div>
<div>I saw that you updated the patch today (<span
style="font-size:12.8px">@ </span><a
moz-do-not-send="true"
href="http://review.gluster.org/#/c/15289/"
style="font-size:12.8px" target="_blank">http://review.gluster.org/#<wbr>/c/15289/</a>).
I built an RPM of the first iteration you had of the
patch (just changing the one line in arbiter.c "GF_FREE
(ctx->iattbuf);") and am running that on some test
servers now to see if the memory of the arbiter brick
gets out of control.</div>
<div><br>
</div>
<div>Ben</div>
</div>
<div class="HOEnZb">
<div class="h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Aug 23, 2016 at 3:38
AM, Ravishankar N <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:ravishankar@redhat.com"
target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hi Benjamin<span><br>
<br>
On 08/23/2016 06:41 AM, Benjamin Edgar
wrote:<br>
</span></div>
<span>
<blockquote type="cite">
<div dir="ltr">I've attached a statedump of
the problem brick process. Let me know if
there are any other logs you need.</div>
</blockquote>
<br>
</span> Thanks for the report! I've sent a fix @
<a moz-do-not-send="true"
href="http://review.gluster.org/#/c/15289/"
target="_blank">http://review.gluster.org/#/c/<wbr>15289/</a>
. It would be nice if you can verify if the
patch fixes the issue for you.<br>
<br>
Thanks,<br>
Ravi
<div>
<div><br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Thanks a lot,</div>
<div>Ben</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Aug 22,
2016 at 5:03 PM, Pranith Kumar
Karampuri <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:pkarampu@redhat.com"
target="_blank">pkarampu@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>Could you collect statedump
of the brick process by
following: <a
moz-do-not-send="true"
href="https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump"
target="_blank">https://gluster.readthedocs.io<wbr>/en/latest/Troubleshooting/sta<wbr>tedump</a><br>
<br>
</div>
That should help us identify
which datatype is causing leaks
and fix it.<br>
<br>
</div>
Thanks!<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">
<div>
<div>On Tue, Aug 23, 2016 at
2:22 AM, Benjamin Edgar <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:benedgar8@gmail.com"
target="_blank">benedgar8@gmail.com</a>></span>
wrote:<br>
</div>
</div>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div>
<div>
<div dir="ltr">Hi,
<div><br>
</div>
<div>I appear to have a
memory leak with a
replica 3 arbiter 1
configuration of
gluster. I have a data
brick and an arbiter
brick on one server,
and another server
with the last data
brick. The more I
write files to gluster
in this configuration,
the more memory the
arbiter brick process
takes up.</div>
<div><br>
</div>
<div>I am able to
reproduce this issue
by first setting up a
replica 3 arbiter 1
configuration and then
using the following
bash script to create
10,000 200kB files,
delete those files,
and run forever:</div>
<div><br>
</div>
<div>
<div>while true ; do</div>
<div> for i in
{1..10000} ; do</div>
<div> dd
if=/dev/urandom
bs=200K count=1
of=$TEST_FILES_DIR/file$i</div>
<div> done</div>
<div> rm -rf
$TEST_FILES_DIR/*<br>
</div>
<div>done<br>
</div>
</div>
<div><br>
</div>
<div>$TEST_FILES_DIR is
a location on my
gluster mount.</div>
<div><br>
</div>
<div>After about 3 days
of this script running
on one of my clusters,
this is what the
output of "top" looks
like:</div>
<div>
<div> PID USER
PR NI VIRT
RES SHR S
%CPU %MEM TIME+
COMMAND</div>
<div>16039 root
20 0
1397220 77720
3948 S 20.6 1.0
860:01.53
glusterfsd</div>
<div>13174 root
20 0
1395824 112728
3692 S 19.6 1.5
806:07.17
glusterfs</div>
<div>19961 root
20 0
2967204 <b>2.145g</b>
3896 S 17.3
29.0
752:10.70
glusterfsd</div>
</div>
<div><br>
</div>
<div>As you can see one
of the brick processes
is using over 2
gigabytes of memory.</div>
<div><br>
</div>
<div>One work-around for
this is to kill the
arbiter brick process
and restart the
gluster daemon. This
restarts arbiter brick
process and its memory
usage goes back down
to a reasonable level.
However I would rather
not kill the arbiter
brick every week for
production
environments.</div>
<div><br>
</div>
<div>Has anyone seen
this issue before and
is there a known
work-around/fix?</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Ben</div>
</div>
<br>
</div>
</div>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org"
target="_blank">Gluster-users@gluster.org</a><br>
<a moz-do-not-send="true"
href="http://www.gluster.org/mailman/listinfo/gluster-users"
rel="noreferrer"
target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a><span><font
color="#888888"><br>
</font></span></blockquote>
</div>
<span><font color="#888888"><br>
<br clear="all">
<br>
-- <br>
<div
data-smartmail="gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div data-smartmail="gmail_signature">
<div dir="ltr"><span
style="font-family:arial;font-size:small">Benjamin
Edgar</span>
<div
style="font-family:arial;font-size:small">Computer
Science</div>
<div
style="font-family:arial;font-size:small">University
of Virginia 2015</div>
<div
style="font-family:arial;font-size:small">(571)
338-0878</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>______________________________<wbr>_________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a></pre>
</blockquote>
<p>
</p>
</div></div></div>
</blockquote></div>
<div>
</div>--
<div data-smartmail="gmail_signature"><div dir="ltr"><span style="font-family:arial;font-size:small">Benjamin Edgar</span><div style="font-family:arial;font-size:small">Computer Science</div><div style="font-family:arial;font-size:small">University of Virginia 2015</div><div style="font-family:arial;font-size:small"><a moz-do-not-send="true" href="tel:%28571%29%20338-0878" value="+15713380878" target="_blank">(571) 338-0878</a></div></div></div>
</div>
</div></div></blockquote></div>
<div>
</div>--
<div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><span style="font-family:arial;font-size:small">Benjamin Edgar</span><div style="font-family:arial;font-size:small">Computer Science</div><div style="font-family:arial;font-size:small">University of Virginia 2015</div><div style="font-family:arial;font-size:small">(571) 338-0878</div></div></div>
</div>
</blockquote><p>
</p></body></html>