<div dir="ltr"><div class="gmail_signature"><div dir="ltr"><div><div><p class="MsoNormal">Hello,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">It’s been a while since I last ran GlusterFS so I thought I
might give it another try here at home in my lab. I am using the 3.7 branch on 2 systems with a
3<sup>rd</sup> being an arbiter node.
Much like the last time I tried GlusterFS, I keep running into issues
with the glusterfsd process eating up so many resources that the systems
sometimes become all but unusable. A
quick Google search tells me I am not the only one to run into this issue but I
have yet to find a cure. The last time I
ran GlusterFS, it was to host web sites and I just chalked the problem up to a
large number of small files. This time,
I am using it to host VM’s and there are only 7 of them and while they are
running, they are not doing anything else.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">When I run ‘gluster volume heal gv0 info’, it almost always
shows a few files and sometimes they say “Possibly undergoing heal”. Next time, it will be different files. So I am wondering if the problems are heal
related and I’m hoping there is a way I can tune for this.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">The systems are dual-quad core processors with 72 GB of
memory and Gigabit Nics. Network speed
doesn't seem to be the issue as when I check it with iperf, I can consistently
get over 850 Mbits/sec. Processor and/or
drive speed seems to be where the issues are coming from but not sure which
comes first or what to do as I would think I should have ample resources.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">When I don’t have Gluster running, I can run “hdparm –Tt
/dev/sdb” and consistently get over 110MB/sec.
Granted I am using 7200 rpm SATA3 drives but for 7 VMs that are all very
quiet, I find it hard to believe they are not fast enough. When I run GlusterFS with the bricks on that
drive (and nothing else running on those drives), I have seen that speed slow
down to 20MB/sec. As I type this, one of
the systems has 5 VMs up on it and the other has 2 and both experience the same
thing. A quick look at the output of
“top” will show load average of 15-20 and sometimes even higher. Again, the VMs are running but I have checked
and they are not doing much of anything other than running.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">When I originally set this up, I didn't have any extra
config settings in place but as you can see from the output, I do now, although
none have helped thus far. I am
connecting to GlusterFS utilizing the built-in NFS server (v3 but that doesn’t
seem to matter either) with options noatime, nodiratime, rsize=65536, and
wsize=65536 but they didn't seem to make much of a difference either.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">So to sum up, the problem seems to be all the heals going
on. What can I do to get this back under
control? Luckily it’s in a lab
environment so I can experiment a bit.
Here is some additional info that may be of assistance and thank you in
advance:</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">gluster volume info gv0:</p>
<p class="MsoNormal">Volume Name: gv0</p>
<p class="MsoNormal">Type: Replicate</p>
<p class="MsoNormal">Volume ID: 14e7bb9c-aa5e-4386-8dd2-83a88d93dc54</p>
<p class="MsoNormal">Status: Started</p>
<p class="MsoNormal">Number of Bricks: 1 x 3 = 3</p>
<p class="MsoNormal">Transport-type: tcp</p>
<p class="MsoNormal">Bricks:</p>
<p class="MsoNormal">Brick1: server1:/export/brick1</p>
<p class="MsoNormal">Brick2: server2:/export/brick1</p>
<p class="MsoNormal">Brick3: kvm:/export/brick1</p>
<p class="MsoNormal">Options Reconfigured:</p>
<p class="MsoNormal">nfs.acl: off</p>
<p class="MsoNormal">performance.readdir-ahead: on</p>
<p class="MsoNormal">performance.quick-read: off</p>
<p class="MsoNormal">performance.read-ahead: off</p>
<p class="MsoNormal">performance.io-cache: off</p>
<p class="MsoNormal">performance.stat-prefetch: off</p>
<p class="MsoNormal">cluster.eager-lock: enable</p>
<p class="MsoNormal">network.remote-dio: enable</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">--- </p>
<p class="MsoNormal">I gathered this with load averages of approximately 12 so it
has been much higher</p>
<p class="MsoNormal">gluster volume status gv0 info:</p>
<p class="MsoNormal">Status of volume: gv0</p>
<p class="MsoNormal">------------------------------------------------------------------------------</p>
<p class="MsoNormal">Brick :
Brick server1:/export/brick1</p>
<p class="MsoNormal">TCP Port :
49152</p>
<p class="MsoNormal">RDMA Port :
0</p>
<p class="MsoNormal">Online : Y</p>
<p class="MsoNormal">Pid :
4409</p>
<p class="MsoNormal">File System :
ext3</p>
<p class="MsoNormal">Device :
/dev/sdb1</p>
<p class="MsoNormal">Mount Options :
rw</p>
<p class="MsoNormal">Inode Size :
128</p>
<p class="MsoNormal">Disk Space Free :
1.7TB</p>
<p class="MsoNormal">Total Disk Space :
1.8TB</p>
<p class="MsoNormal">Inode Count :
244203520</p>
<p class="MsoNormal">Free Inodes : 244203413</p>
<p class="MsoNormal">------------------------------------------------------------------------------</p>
<p class="MsoNormal">Brick :
Brick server2:/export/brick1</p>
<p class="MsoNormal">TCP Port :
49152</p>
<p class="MsoNormal">RDMA Port :
0</p>
<p class="MsoNormal">Online :
Y</p>
<p class="MsoNormal">Pid :
4535</p>
<p class="MsoNormal">File System :
ext3</p>
<p class="MsoNormal">Device :
/dev/sdb1</p>
<p class="MsoNormal">Mount Options :
rw</p>
<p class="MsoNormal">Inode Size :
128</p>
<p class="MsoNormal">Disk Space Free :
1.7TB</p>
<p class="MsoNormal">Total Disk Space :
1.8TB</p>
<p class="MsoNormal">Inode Count :
244203520</p>
<p class="MsoNormal">Free Inodes :
244203405</p>
<p class="MsoNormal">---</p>
<p class="MsoNormal">gluster volume provile gv0 as follows:</p>
<p class="MsoNormal">Brick: server1:/export/brick1</p>
<p class="MsoNormal">--------------------------------</p>
<p class="MsoNormal">Cumulative Stats:</p>
<p class="MsoNormal"> Block Size: 16b+ 512b+ 1024b+</p>
<p class="MsoNormal">No. of Reads: 3 1319 39679</p>
<p class="MsoNormal">No. of Writes: 3 10804 917</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Block Size: 2048b+ 4096b+ 8192b+</p>
<p class="MsoNormal">No. of Reads: 673 33872 7199</p>
<p class="MsoNormal">No. of Writes: 3642 219192 133087</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Block Size: 16384b+ 32768b+ 65536b+</p>
<p class="MsoNormal">No. of Reads:
8808 20790 2275</p>
<p class="MsoNormal">No. of Writes:
52815 180145 4910</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Block Size: 131072b+</p>
<p class="MsoNormal">No. of Reads:
952554</p>
<p class="MsoNormal">No. of Writes:
5558597</p>
<p class="MsoNormal">%-latency
Avg-latency Min-Latency Max-Latency
No. of calls Fop</p>
<p class="MsoNormal">--------- ----------- -----------
----------- ------------ ----</p>
<p class="MsoNormal"> 0.00 0.00 us 0.00 us 0.00 us 5 FORGET</p>
<p class="MsoNormal"> 0.00 0.00 us 0.00 us 0.00 us 755 RELEASE</p>
<p class="MsoNormal"> 0.00 0.00 us 0.00 us
0.00 us 214 RELEASEDIR</p>
<p class="MsoNormal"> 0.00 4.00 us 4.00 us 4.00 us 2 OPENDIR</p>
<p class="MsoNormal"> 0.00 168.67 us 31.00 us 406.00 us 3 READDIR</p>
<p class="MsoNormal"> 0.00 535.40 us 84.00 us 1612.00 us 5 OPEN</p>
<p class="MsoNormal"> 0.00 975.33 us 916.00 us 1045.00 us 3 FTRUNCATE</p>
<p class="MsoNormal"> 0.00 305.36 us 15.00 us
14784.00 us 59 FLUSH</p>
<p class="MsoNormal"> 0.00 14988.50 us 9326.00 us
20651.00 us 2
XATTROP</p>
<p class="MsoNormal"> 0.00 16257.40 us 61.00 us
58866.00 us 10 LOOKUP</p>
<p class="MsoNormal"> 0.01 41136.33 us 37.00 us
219392.00 us 6 GETXATTR</p>
<p class="MsoNormal"> 0.01 146916.50 us
51093.00 us 242740.00 us 2 SETATTR</p>
<p class="MsoNormal"> 0.01 637.58 us 24.00 us
124014.00 us 651 READ</p>
<p class="MsoNormal"> 0.02 241.17 us 6.00 us
75127.00 us 2553 ACCESS</p>
<p class="MsoNormal"> 0.02 220.31 us 10.00 us
77807.00 us 3235 STAT</p>
<p class="MsoNormal"> 0.10 223016.80 us
72200.00 us 681213.00 us 15 FSYNC</p>
<p class="MsoNormal"> 0.79 1466.11 us 29.00 us
236061.00 us 18414 FXATTROP</p>
<p class="MsoNormal"> 1.20 1392.82 us 13.00 us
842550.00 us 29201 FINODELK</p>
<p class="MsoNormal"> 12.66 3079.72 us 19.00 us 1161311.00 us 139747 INODELK</p>
<p class="MsoNormal"> 85.17 34610.43 us 56.00 us 2401944.00 us 83630 WRITE</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Duration: 81829
seconds</p>
<p class="MsoNormal"> Data Read:
126295937593 bytes</p>
<p class="MsoNormal">Data Written: 740051667513 bytes</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Interval 7 Stats:</p>
<p class="MsoNormal"> Block Size: 512b+ 1024b+ 2048b+</p>
<p class="MsoNormal">No. of Reads: 0 234 0</p>
<p class="MsoNormal">No. of Writes: 171 7 16</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Block Size: 4096b+ 8192b+ 16384b+</p>
<p class="MsoNormal">No. of Reads: 0 0 0</p>
<p class="MsoNormal">No. of Writes: 1162 472 278</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Block Size: 32768b+ 65536b+ 131072b+</p>
<p class="MsoNormal">No. of Reads: 1 0 0</p>
<p class="MsoNormal">No. of Writes: 1593 30 36991</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">%-latency
Avg-latency Min-Latency Max-Latency
No. of calls Fop</p>
<p class="MsoNormal">---------
----------- ----------- -----------
------------ ----</p>
<p class="MsoNormal"> 0.00 0.00 us 0.00 us 0.00 us 1 RELEASE</p>
<p class="MsoNormal"> 0.00 4.00 us 4.00 us 4.00 us 1 OPENDIR</p>
<p class="MsoNormal"> 0.00 406.00 us 406.00 us 406.00 us 1 READDIR</p>
<p class="MsoNormal"> 0.00 979.50 us 347.00 us 1612.00 us 2 OPEN</p>
<p class="MsoNormal"> 0.00 1005.00 us 965.00 us 1045.00 us 2 FTRUNCATE</p>
<p class="MsoNormal"> 0.00 9326.00 us
9326.00 us 9326.00 us 1 XATTROP</p>
<p class="MsoNormal"> 0.00 656.32 us 27.00 us
14784.00 us 25 FLUSH</p>
<p class="MsoNormal"> 0.00 51093.00 us
51093.00 us 51093.00 us 1 SETATTR</p>
<p class="MsoNormal"> 0.01 16878.60 us 164.00 us
47828.00 us 5 LOOKUP</p>
<p class="MsoNormal"> 0.01 363.44 us 24.00 us
47319.00 us 235 READ</p>
<p class="MsoNormal"> 0.01 112.50 us 10.00 us
18678.00 us 1162 STAT</p>
<p class="MsoNormal"> 0.02 78723.67 us 38.00 us
219392.00 us 3 GETXATTR</p>
<p class="MsoNormal"> 0.03 387.51 us 7.00 us
75127.00 us 918 ACCESS</p>
<p class="MsoNormal"> 0.16 426656.75 us
217375.00 us 681213.00 us 4 FSYNC</p>
<p class="MsoNormal"> 0.51 1440.22 us 37.00 us
175074.00 us 3820
FXATTROP</p>
<p class="MsoNormal"> 1.00 1311.97 us 16.00 us
842550.00 us 8218 FINODELK</p>
<p class="MsoNormal"> 13.44 1951.10 us 23.00 us 1161311.00 us 73993 INODELK</p>
<p class="MsoNormal"> 84.79 22359.52 us 56.00 us 2401944.00 us 40723 WRITE</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Duration: 394
seconds</p>
<p class="MsoNormal"> Data Read: 392192
bytes</p>
<p class="MsoNormal">Data Written: 4934906368 bytes</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Brick: kvm:/export/brick1</p>
<p class="MsoNormal">--------------------------</p>
<p class="MsoNormal">Cumulative Stats:</p>
<p class="MsoNormal"> Block Size: 16b+ 512b+ 1024b+</p>
<p class="MsoNormal">No. of Reads: 0 0 0</p>
<p class="MsoNormal">No. of Writes: 5 10288 849</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Block Size: 2048b+ 4096b+ 8192b+</p>
<p class="MsoNormal">No. of Reads:
0 0 0</p>
<p class="MsoNormal">No. of Writes: 3399 206329 128093</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Block Size: 16384b+ 32768b+ 65536b+</p>
<p class="MsoNormal">No. of Reads: 0 0 0</p>
<p class="MsoNormal">No. of Writes:
50427 175784 4811</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Block Size: 131072b+</p>
<p class="MsoNormal">No. of Reads: 0</p>
<p class="MsoNormal">No. of Writes:
6260106</p>
<p class="MsoNormal">%-latency
Avg-latency Min-Latency Max-Latency
No. of calls Fop</p>
<p class="MsoNormal">---------
----------- ----------- -----------
------------ ----</p>
<p class="MsoNormal"> 0.00 0.00 us 0.00 us 0.00 us 250 RELEASE</p>
<p class="MsoNormal"> 0.00 0.00 us 0.00 us 0.00 us 144
RELEASEDIR</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Duration: 72903
seconds</p>
<p class="MsoNormal"> Data Read: 0 bytes</p>
<p class="MsoNormal">Data Written: 831653482591 bytes</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Interval 7 Stats:</p>
<p class="MsoNormal"> Block Size: 512b+ 1024b+ 2048b+</p>
<p class="MsoNormal">No. of Reads: 0 0 0</p>
<p class="MsoNormal">No. of Writes: 171 7 16</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Block Size: 4096b+ 8192b+ 16384b+</p>
<p class="MsoNormal">No. of Reads: 0 0 0</p>
<p class="MsoNormal">No. of Writes: 1162 472 278</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Block Size: 32768b+ 65536b+ 131072b+</p>
<p class="MsoNormal">No. of Reads: 0 0 0</p>
<p class="MsoNormal">No. of Writes: 1593 30 36068</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">%-latency
Avg-latency Min-Latency Max-Latency
No. of calls Fop</p>
<p class="MsoNormal">---------
----------- ----------- -----------
------------ ----</p>
<p class="MsoNormal"> 0.00 0.00 us
0.00 us 0.00 us 1 RELEASE</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Duration: 394
seconds</p>
<p class="MsoNormal"> Data Read: 0 bytes</p>
<p class="MsoNormal">Data Written: 4813926912 bytes</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Brick: server2:/export/brick1</p>
<p class="MsoNormal">--------------------------------</p>
<p class="MsoNormal">Cumulative Stats:</p>
<p class="MsoNormal"> Block Size: 16b+ 512b+ 1024b+</p>
<p class="MsoNormal">No. of Reads: 2 1554 187</p>
<p class="MsoNormal">No. of Writes: 3 10813 920</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Block Size: 2048b+ 4096b+ 8192b+</p>
<p class="MsoNormal">No. of Reads: 0 205139 61200</p>
<p class="MsoNormal">No. of Writes: 3642 219205 133087</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Block Size: 16384b+ 32768b+ 65536b+</p>
<p class="MsoNormal">No. of Reads:
41437 311428 9278</p>
<p class="MsoNormal">No. of Writes:
52809 180141 4853</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Block Size: 131072b+</p>
<p class="MsoNormal">No. of Reads:
5558571</p>
<p class="MsoNormal">No. of Writes:
292354</p>
<p class="MsoNormal">%-latency
Avg-latency Min-Latency Max-Latency
No. of calls Fop</p>
<p class="MsoNormal">---------
----------- ----------- -----------
------------ ----</p>
<p class="MsoNormal"> 0.00 0.00 us 0.00 us 0.00 us 5 FORGET</p>
<p class="MsoNormal"> 0.00 0.00 us 0.00 us 0.00 us 795 RELEASE</p>
<p class="MsoNormal"> 0.00 0.00 us 0.00 us 0.00 us 196
RELEASEDIR</p>
<p class="MsoNormal"> 0.00 94.49 us 10.00 us 2660.00 us 59 FLUSH</p>
<p class="MsoNormal"> 0.00
6859.40 us 78.00 us 16605.00 us 5 OPEN</p>
<p class="MsoNormal"> 0.00 22617.50 us
11067.00 us 34168.00 us 2 XATTROP</p>
<p class="MsoNormal"> 0.01 20768.00 us 112.00 us
81026.00 us 8 LOOKUP</p>
<p class="MsoNormal"> 0.04 37279.69 us
19436.00 us 86934.00 us 13 FSYNC</p>
<p class="MsoNormal"> 0.85 385.97 us 10.00 us
212031.00 us 28913 FINODELK</p>
<p class="MsoNormal"> 1.33 951.03 us 34.00 us
221747.00 us 18400 FXATTROP</p>
<p class="MsoNormal"> 1.62 303.60 us 22.00 us
231359.00 us 70024 READ</p>
<p class="MsoNormal"> 9.25 868.44 us 16.00 us
407424.00 us 139714 INODELK</p>
<p class="MsoNormal"> 86.89 82862.96 us 59.00 us
638518.00 us 13759 WRITE</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Duration: 82915
seconds</p>
<p class="MsoNormal"> Data Read:
744922109478 bytes</p>
<p class="MsoNormal">Data Written: 49788871225 bytes</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Interval 7 Stats:</p>
<p class="MsoNormal"> Block Size: 512b+ 1024b+ 2048b+</p>
<p class="MsoNormal">No. of Reads: 30 0 0</p>
<p class="MsoNormal">No. of Writes: 171 7 16</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Block Size: 4096b+ 8192b+ 16384b+</p>
<p class="MsoNormal">No. of Reads: 55 4 25</p>
<p class="MsoNormal">No. of Writes: 1162 472 278</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Block Size: 32768b+ 65536b+ 131072b+</p>
<p class="MsoNormal">No. of Reads: 30 0 37001</p>
<p class="MsoNormal">No. of Writes: 1593 30 0</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">%-latency
Avg-latency Min-Latency Max-Latency
No. of calls Fop</p>
<p class="MsoNormal">---------
----------- ----------- -----------
------------ ----</p>
<p class="MsoNormal"> 0.00 0.00 us 0.00 us 0.00 us 1
RELEASE</p>
<p class="MsoNormal"> 0.00 45.52 us 10.00 us 91.00 us 25 FLUSH</p>
<p class="MsoNormal"> 0.00 11067.00 us
11067.00 us 11067.00 us 1 XATTROP</p>
<p class="MsoNormal"> 0.00 9621.00 us 2637.00 us
16605.00 us 2 OPEN</p>
<p class="MsoNormal"> 0.03
44736.33 us 19436.00 us 86934.00 us 3 FSYNC</p>
<p class="MsoNormal"> 0.03 33648.00 us 129.00 us
81026.00 us 4 LOOKUP</p>
<p class="MsoNormal"> 1.21 1351.63 us 35.00 us
123188.00 us 3820 FXATTROP</p>
<p class="MsoNormal"> 1.21 634.89 us 10.00 us
212031.00 us 8156 FINODELK</p>
<p class="MsoNormal"> 2.96 339.80 us 22.00 us
231359.00 us 37145 READ</p>
<p class="MsoNormal"> 12.90 743.44 us 17.00 us
407424.00 us 74014 INODELK</p>
<p class="MsoNormal"> 81.64 93359.19 us 64.00 us
442704.00 us 3729 WRITE</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Duration: 394
seconds</p>
<p class="MsoNormal"> Data Read:
4851727360 bytes</p>
<p class="MsoNormal">Data Written: 86422016 bytes</p><p class="MsoNormal"><br></p><p class="MsoNormal">Thank you.</p><p class="MsoNormal"><br></p></div></div></div></div>
</div>