<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Le 29/07/2016 à 18:39, Pranith Kumar
Karampuri a écrit :<br>
</div>
<blockquote
cite="mid:CAOgeEnbwwq26Nwy=w4sthuY_xJXnA7t97XMOHdZpdp6qF94h2w@mail.gmail.com"
type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Jul 29, 2016 at 2:26 PM,
Yannick Perret <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:yannick.perret@liris.cnrs.fr"
target="_blank">yannick.perret@liris.cnrs.fr</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">Ok, last try:<br>
after investigating more versions I found that FUSE client
leaks memory on all of them.<br>
I tested:<br>
- 3.6.7 client on debian 7 32bit and on debian 8 64bit
(with 3.6.7 serveurs on debian 8 64bit)<br>
- 3.6.9 client on debian 7 32bit and on debian 8 64bit
(with 3.6.7 serveurs on debian 8 64bit)<br>
- 3.7.13 client on debian 8 64bit (with 3.8.1 serveurs on
debian 8 64bit)<br>
- 3.8.1 client on debian 8 64bit (with 3.8.1 serveurs on
debian 8 64bit)<br>
In all cases compiled from sources, appart for 3.8.1 where
.deb were used (due to a configure runtime error).<br>
For 3.7 it was compiled with --disable-tiering. I also
tried to compile with --disable-fusermount (no change).<br>
<br>
In all of these cases the memory (resident & virtual)
of glusterfs process on client grows on each activity and
never reach a max (and never reduce).<br>
"Activity" for these tests is cp -Rp and ls -lR.<br>
The client I let grows the most overreached ~4Go RAM. On
smaller machines it ends by OOM killer killing glusterfs
process or glusterfs dying due to allocation error.<br>
<br>
In 3.6 mem seems to grow continusly, whereas in 3.8.1 it
grows by "steps" (430400 ko → 629144 (~1min) → 762324
(~1min) → 827860…).<br>
<br>
All tests performed on a single test volume used only by
my test client. Volume in a basic x2 replica. The only
parameters I changed on this volume (without any effect)
are diagnostics.client-log-level set to ERROR and
network.inode-lru-limit set to 1024.<br>
</blockquote>
<div><br>
</div>
<div>Could you attach statedumps of your runs?<br>
The following link has steps to capture this(<a
moz-do-not-send="true"
href="https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/">https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/</a>
). We basically need to see what are the memory types that
are increasing. If you could help find the issue, we can
send the fixes for your workload. There is a 3.8.2 release
in around 10 days I think. We can probably target this
issue for that?<br>
</div>
</div>
</div>
</div>
</blockquote>
Here are statedumps.<br>
Steps:<br>
1. mount -t glusterfs ldap1.my.domain:SHARE /root/MNT/ (here VSZ and
RSS are 381896 35828)<br>
2. take a dump with kill -USR1 <pid-of-glusterfs-process>
(file glusterdump.n1.dump.1470042769)<br>
3. perform a 'ls -lR /root/MNT | wc -l' (btw result of wc -l is
518396 :)) and a 'cp -Rp /usr/* /root/MNT/boo' (VSZ/RSS are
1301536/711992 at end of these operations)<br>
4. take a dump with kill -USR1 <pid-of-glusterfs-process>
(file glusterdump.n2.dump.1470043929)<br>
5. do 'cp -Rp * /root/MNT/toto/', so on an other directory (VSZ/RSS
are 1432608/909968 at end of this operation)<br>
6. take a dump with kill -USR1 <pid-of-glusterfs-process>
(file glusterdump.n3.dump.)<br>
<br>
Dump files are gzip'ed because they are very large.<br>
Dump files are here (too big for email):<br>
<a class="moz-txt-link-freetext" href="http://wikisend.com/download/623430/glusterdump.n1.dump.1470042769.gz">http://wikisend.com/download/623430/glusterdump.n1.dump.1470042769.gz</a><br>
<a class="moz-txt-link-freetext" href="http://wikisend.com/download/771220/glusterdump.n2.dump.1470043929.gz">http://wikisend.com/download/771220/glusterdump.n2.dump.1470043929.gz</a><br>
<a class="moz-txt-link-freetext" href="http://wikisend.com/download/428752/glusterdump.n3.dump.1470045181.gz">http://wikisend.com/download/428752/glusterdump.n3.dump.1470045181.gz</a><br>
(I keep the files if someone whats them in an other format)<br>
<br>
Client and servers are installed from .deb files
(glusterfs-client_3.8.1-1_amd64.deb and
glusterfs-common_3.8.1-1_amd64.deb on client side).<br>
They are all Debian 8 64bit. Servers are test machines that serve
only one volume to this sole client. Volume is a simple x2 replica.
I just changed for test network.inode-lru-limit value to 1024. Mount
point /root/MNT is only used for these tests.<br>
<br>
--<br>
Y.<br>
<br>
<br>
</body>
</html>