<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 8, 2016 at 12:33 PM, Oleksandr Natalenko <span dir="ltr"><<a href="mailto:oleksandr@natalenko.name" target="_blank">oleksandr@natalenko.name</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Yup, I can do that, but please note that RSS does not change. Will statedump show VIRT values?<br>
<br>
Also, I'm looking at the numbers now, and see that on each reconnect VIRT grows by ~24M (once per ~10–15 mins). Probably, that could give you some idea what is going wrong.<br></blockquote><div><br></div><div>That's interesting. Never saw something like this happen. I would still like to see if there are any clues in statedump when all this happens. May be what you said will be confirmed that nothing new is allocated but I would just like to confirm.<br><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
08.06.2016 09:50, Pranith Kumar Karampuri написав:<div class="HOEnZb"><div class="h5"><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Oleksandr,<br>
Could you take statedump of the shd process once in 5-10 minutes and<br>
send may be 5 samples of them when it starts to increase? This will<br>
help us find what datatypes are being allocated a lot and can lead to<br>
coming up with possible theories for the increase.<br>
<br>
On Wed, Jun 8, 2016 at 12:03 PM, Oleksandr Natalenko<br>
<<a href="mailto:oleksandr@natalenko.name" target="_blank">oleksandr@natalenko.name</a>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Also, I've checked shd log files, and found out that for some reason<br>
shd constantly reconnects to bricks: [1]<br>
<br>
Please note that suggested fix [2] by Pranith does not help, VIRT<br>
value still grows:<br>
<br>
===<br>
root 1010 0.0 9.6 7415248 374688 ? Ssl чер07 0:14<br>
/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p<br>
/var/lib/glusterd/glustershd/run/glustershd.pid -l<br>
/var/log/glusterfs/glustershd.log -S<br>
/var/run/gluster/7848e17764dd4ba80f4623aecb91b07a.socket<br>
--xlator-option<br>
*replicate*.node-uuid=80bc95e1-2027-4a96-bb66-d9c8ade624d7<br>
===<br>
<br>
I do not know the reason why it is reconnecting, but I suspect leak<br>
to happen on that reconnect.<br>
<br>
CCing Pranith.<br>
<br>
[1] <a href="http://termbin.com/brob" rel="noreferrer" target="_blank">http://termbin.com/brob</a><br>
[2] <a href="http://review.gluster.org/#/c/14053/" rel="noreferrer" target="_blank">http://review.gluster.org/#/c/14053/</a><br>
<br>
06.06.2016 12:21, Kaushal M написав:<br>
Has multi-threaded SHD been merged into 3.7.* by any chance? If<br>
not,<br>
<br>
what I'm saying below doesn't apply.<br>
<br>
We saw problems when encrypted transports were used, because the RPC<br>
layer was not reaping threads (doing pthread_join) when a connection<br>
ended. This lead to similar observations of huge VIRT and relatively<br>
small RSS.<br>
<br>
I'm not sure how multi-threaded shd works, but it could be leaking<br>
threads in a similar way.<br>
<br>
On Mon, Jun 6, 2016 at 1:54 PM, Oleksandr Natalenko<br>
<<a href="mailto:oleksandr@natalenko.name" target="_blank">oleksandr@natalenko.name</a>> wrote:<br>
Hello.<br>
<br>
We use v3.7.11, replica 2 setup between 2 nodes + 1 dummy node for<br>
keeping<br>
volumes metadata.<br>
<br>
Now we observe huge VSZ (VIRT) usage by glustershd on dummy node:<br>
<br>
===<br>
root 15109 0.0 13.7 76552820 535272 ? Ssl тра26 2:11<br>
/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p<br>
/var/lib/glusterd/glustershd/run/glustershd.pid -l<br>
/var/log/glusterfs/glustershd.log -S<br>
/var/run/gluster/7848e17764dd4ba80f4623aecb91b07a.socket<br>
--xlator-option<br>
*replicate*.node-uuid=80bc95e1-2027-4a96-bb66-d9c8ade624d7<br>
===<br>
<br>
that is ~73G. RSS seems to be OK (~522M). Here is the statedump of<br>
glustershd process: [1]<br>
<br>
Also, here is sum of sizes, presented in statedump:<br>
<br>
===<br>
# cat /var/run/gluster/glusterdump.15109.dump.1465200139 | awk -F<br>
'=' 'BEGIN<br>
{sum=0} /^size=/ {sum+=$2} END {print sum}'<br>
353276406<br>
===<br>
<br>
That is ~337 MiB.<br>
<br>
Also, here are VIRT values from 2 replica nodes:<br>
<br>
===<br>
root 24659 0.0 0.3 5645836 451796 ? Ssl тра24 3:28<br>
/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p<br>
/var/lib/glusterd/glustershd/run/glustershd.pid -l<br>
/var/log/glusterfs/glustershd.log -S<br>
/var/run/gluster/44ec3f29003eccedf894865107d5db90.socket<br>
--xlator-option<br>
*replicate*.node-uuid=a19afcc2-e26c-43ce-bca6-d27dc1713e87<br>
root 18312 0.0 0.3 6137500 477472 ? Ssl тра19 6:37<br>
/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p<br>
/var/lib/glusterd/glustershd/run/glustershd.pid -l<br>
/var/log/glusterfs/glustershd.log -S<br>
/var/run/gluster/1670a3abbd1eea968126eb6f5be20322.socket<br>
--xlator-option<br>
*replicate*.node-uuid=52dca21b-c81c-48b5-9de2-1ed37987fbc2<br>
===<br>
<br>
Those are 5 to 6G, which is much less than dummy node has, but still<br>
look<br>
too big for us.<br>
<br>
Should we care about huge VIRT value on dummy node? Also, how one<br>
would<br>
debug that?<br>
<br>
Regards,<br>
Oleksandr.<br>
<br>
[1] <a href="https://gist.github.com/d2cfa25251136512580220fcdb8a6ce6" rel="noreferrer" target="_blank">https://gist.github.com/d2cfa25251136512580220fcdb8a6ce6</a><br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
</blockquote>
<br>
--<br>
<br>
Pranith<br>
</blockquote>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>