<div dir="ltr">I've attached a statedump of the problem brick process. Let me know if there are any other logs you need.<div><br></div><div>Thanks a lot,</div><div>Ben</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 22, 2016 at 5:03 PM, Pranith Kumar Karampuri <span dir="ltr"><<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Could you collect statedump of the brick process by following: <a href="https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump" target="_blank">https://gluster.readthedocs.<wbr>io/en/latest/Troubleshooting/<wbr>statedump</a><br><br></div>That should help us identify which datatype is causing leaks and fix it.<br><br></div>Thanks!<br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Tue, Aug 23, 2016 at 2:22 AM, Benjamin Edgar <span dir="ltr"><<a href="mailto:benedgar8@gmail.com" target="_blank">benedgar8@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr">Hi,<div><br></div><div>I appear to have a memory leak with a replica 3 arbiter 1 configuration of gluster. I have a data brick and an arbiter brick on one server, and another server with the last data brick. The more I write files to gluster in this configuration, the more memory the arbiter brick process takes up.</div><div><br></div><div>I am able to reproduce this issue by first setting up a replica 3 arbiter 1 configuration and then using the following bash script to create 10,000 200kB files, delete those files, and run forever:</div><div><br></div><div><div>while true ; do</div><div> for i in {1..10000} ; do</div><div> dd if=/dev/urandom bs=200K count=1 of=$TEST_FILES_DIR/file$i</div><div> done</div><div> rm -rf $TEST_FILES_DIR/*<br></div><div>done<br></div></div><div><br></div><div>$TEST_FILES_DIR is a location on my gluster mount.</div><div><br></div><div>After about 3 days of this script running on one of my clusters, this is what the output of "top" looks like:</div><div><div> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND</div><div>16039 root 20 0 1397220 77720 3948 S 20.6 1.0 860:01.53 glusterfsd</div><div>13174 root 20 0 1395824 112728 3692 S 19.6 1.5 806:07.17 glusterfs</div><div>19961 root 20 0 2967204 <b>2.145g</b> 3896 S 17.3 29.0 752:10.70 glusterfsd</div></div><div><br></div><div>As you can see one of the brick processes is using over 2 gigabytes of memory.</div><div><br></div><div>One work-around for this is to kill the arbiter brick process and restart the gluster daemon. This restarts arbiter brick process and its memory usage goes back down to a reasonable level. However I would rather not kill the arbiter brick every week for production environments.</div><div><br></div><div>Has anyone seen this issue before and is there a known work-around/fix?</div><div><br></div><div>Thanks,</div><div>Ben</div></div>
<br></div></div>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a><span class="HOEnZb"><font color="#888888"><br></font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><span style="font-family:arial;font-size:small">Benjamin Edgar</span><div style="font-family:arial;font-size:small">Computer Science</div><div style="font-family:arial;font-size:small">University of Virginia 2015</div><div style="font-family:arial;font-size:small">(571) 338-0878</div></div></div>
</div>