<div dir="ltr"><div class="gmail_quote"><br><div dir="ltr"><p class="MsoNormal"> </p>
<p class="MsoNormal">We are using GlusterFS 3.7.11 (upgraded from 3.7.6 last week)
on RHEL 7.x in AWS EC2.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">We continue to see memory utilization going up once every 2
days. The memory utilization of the server demon(glusterd) in NFS server is keep on increasing. In about 30+
hours the Memory utilization of glusterd service alone will reach 70% of memory
available. Since we have alarms for this threshold, we get notified and only
way to stop it so far is to restart the glusterd. </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">This happens even where there’s not a lot of load in GlusterFS.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">The GlusterFS is configured in the two server nodes with two
mount location.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">$ df -i</p>
<p class="MsoNormal">Filesystem
Inodes IUsed IFree IUse% Mounted on</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">/dev/xvdf
125829120 120186 125708934 1%
/nfs_app1</p>
<p class="MsoNormal">/dev/xvdg 125829120 142937 125686183 1% /nfs_app2</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">As part of debugging, we tried the following:</p>
<p class="MsoNormal"> </p>
<p>1.<span style="font-stretch:normal;font-size:7pt;font-family:'Times New Roman'">
</span>From the client side, in the mount location, we tried
to read and write around 1000 files (each of 4MB size). There was no marked
spike in memory utilization during this time. </p>
<p>2.<span style="font-stretch:normal;font-size:7pt;font-family:'Times New Roman'">
</span>We were using GlusterFS 3.7.6 and moved to 3.7.11
and despite that the problem persists.</p>
<p>3.<span style="font-stretch:normal;font-size:7pt;font-family:'Times New Roman'">
</span>We created a dump of the volume in question. The
dump file is attached. Some of memory allocations such as
gf_common_mt_asptinlf_memoryusage has huge total_allocs. Specifically 3 of them
that are listed below.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal" style="margin-left:0.5in">[global.glusterfs - usage-type
gf_common_mt_asprintf memusage]</p>
<p class="MsoNormal" style="margin-left:0.5in">size=260</p>
<p class="MsoNormal" style="margin-left:0.5in">num_allocs=12</p>
<p class="MsoNormal" style="margin-left:0.5in">max_size=2464</p>
<p class="MsoNormal" style="margin-left:0.5in">max_num_allocs=294</p>
<p class="MsoNormal" style="margin-left:0.5in">total_allocs=927964</p>
<p class="MsoNormal" style="margin-left:0.5in"> </p>
<p class="MsoNormal" style="margin-left:0.5in">[global.glusterfs - usage-type
gf_common_mt_char memusage]</p>
<p class="MsoNormal" style="margin-left:0.5in">size=6388</p>
<p class="MsoNormal" style="margin-left:0.5in">num_allocs=164</p>
<p class="MsoNormal" style="margin-left:0.5in">max_size=30134</p>
<p class="MsoNormal" style="margin-left:0.5in">max_num_allocs=645</p>
<p class="MsoNormal" style="margin-left:0.5in">total_allocs=1424017</p>
<p class="MsoNormal" style="margin-left:0.5in"> </p>
<p class="MsoNormal" style="margin-left:0.5in">[protocol/server.xyz-server -
usage-type gf_common_mt_strdup memusage]</p>
<p class="MsoNormal" style="margin-left:0.5in">size=26055</p>
<p class="MsoNormal" style="margin-left:0.5in">num_allocs=2795</p>
<p class="MsoNormal" style="margin-left:0.5in">max_size=27198</p>
<p class="MsoNormal" style="margin-left:0.5in">max_num_allocs=2828</p>
<p class="MsoNormal" style="margin-left:0.5in">total_allocs=135503</p>
<p class="MsoNormal"> </p>
<p>4.<span style="font-stretch:normal;font-size:7pt;font-family:'Times New Roman'">
</span>We also noticed that the mempool has nr_files as
a negative number. Not sure if this is also a cause of the problem.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal" style="margin-left:0.5in">[mempool]</p>
<p class="MsoNormal" style="margin-left:0.5in">[storage/posix.xyz-posix]</p>
<p class="MsoNormal" style="margin-left:0.5in">base_path=/nfs_xyz/abc</p>
<p class="MsoNormal" style="margin-left:0.5in">base_path_length=25</p>
<p class="MsoNormal" style="margin-left:0.5in">max_read=44215866</p>
<p class="MsoNormal" style="margin-left:0.5in">max_write=104925485</p>
<p class="MsoNormal" style="margin-left:0.5in">nr_files=-418</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Attached the complete dump file for your reference. </p><p class="MsoNormal"><br></p>
<p class="MsoNormal">This is happening in Prod and as expected generates a lot of
problems. </p>
<p class="MsoNormal">Has anybody seen this before? Any insights into what we can
try would be greatly appreciated.</p><p class="MsoNormal"><br></p><p class="MsoNormal">Awaiting for the response.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p><p class="MsoNormal">Regards,</p><p class="MsoNormal">Nagendran N</p><p class="MsoNormal">MailId - <a href="mailto:uganit@gmail.com" target="_blank">uganit@gmail.com</a></p></div>
</div><br></div>