<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
So the nodes 3 and 6 seem to indicate inode-locks and lookups are of
the highest latency. This only seems to confirm self-heals are
happening. <br>
If you are unable to use the system because of this, you could try
killing the self-heal daemons on both these nodes (kill `pgrep -f
glustershd`) to stop heals. You can then do a lookup of the files
from the mount, which will also trigger heals.<br>
Restart the selfheal daemons (with `gluster vol start volname
force`) when you think you can spare the volume for heals again. The
sooner the better though.<br>
For the brick log errors, we are suspecting it could be something
related to selinux.<br>
<br>
Can you tell what kind of data is there in your volume? - no. of
files, avg. file size, brick size, network connection speed etc?
Perhaps we can try to reproduce the issue and identify the bottle
neck.<br>
<br>
Thanks,<br>
Ravi<br>
<br>
<div class="moz-cite-prefix">On 08/07/2015 01:27 PM, Prasun Gera
wrote:<br>
</div>
<blockquote
cite="mid:CAFLz+B=OZpNxZy_zjbRTkv=matJGuQudoX4BVXoY-gjLKaXmew@mail.gmail.com"
type="cite">
<div dir="ltr">All the volume commands are taking several minutes
to complete. Here's the profiler's output:
<div>Node3's brick is the one that was replaced. It's replica is
node6</div>
<div><br>
<div><br>
</div>
<div>
<div>Brick: node1:/bricks/brickname</div>
<div>---------------------------------------------------</div>
<div>Cumulative Stats:</div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 0
13 2 </div>
<div>No. of Writes: 60
0 141 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 3
26 17 </div>
<div>No. of Writes: 325
87 738 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 99
114 222 </div>
<div>No. of Writes: 877
343 128 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 110
5 2 </div>
<div>No. of Writes: 29401
78829 1448 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 19
33 34679 </div>
<div>No. of Writes: 6233
22903 41202 </div>
<div> </div>
<div> Block Size: 262144b+
524288b+ 1048576b+ </div>
<div> No. of Reads: 0
1 513 </div>
<div>No. of Writes: 1
0 105 </div>
<div> </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
126138 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
141671 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
117718 RELEASEDIR</div>
<div> 0.02 43.00 us 21.00 us 80.00 us
17 STAT</div>
<div> 0.02 51.62 us 27.00 us 131.00 us
21 STATFS</div>
<div> 0.09 42.78 us 11.00 us 1640.00 us
95 FLUSH</div>
<div> 0.21 50.01 us 23.00 us 567.00 us
189 ENTRYLK</div>
<div> 0.21 50.01 us 19.00 us 291.00 us
190 FINODELK</div>
<div> 0.56 8578.33 us 53.00 us 25625.00 us
3 GETXATTR</div>
<div> 0.62 148.89 us 71.00 us 2761.00 us
190 XATTROP</div>
<div> 0.70 168.11 us 83.00 us 1019.00 us
190 FXATTROP</div>
<div> 0.91 219.84 us 47.00 us 13732.00 us
190 SETATTR</div>
<div> 1.21 71.74 us 16.00 us 11516.00 us
775 INODELK</div>
<div> 1.47 354.88 us 56.00 us 22669.00 us
190 REMOVEXATTR</div>
<div> 2.60 1254.69 us 122.00 us 12514.00 us
95 WRITE</div>
<div> 3.25 194.10 us 51.00 us 48823.00 us
770 LOOKUP</div>
<div> 88.15 43068.81 us 265.00 us 418819.00 us
94 CREATE</div>
<div> </div>
<div> Duration: 644070 seconds</div>
<div> Data Read: 5089537361 bytes</div>
<div>Data Written: 9083513756 bytes</div>
<div> </div>
<div>Interval 0 Stats:</div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 0
13 2 </div>
<div>No. of Writes: 60
0 141 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 3
26 17 </div>
<div>No. of Writes: 325
87 738 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 99
114 222 </div>
<div>No. of Writes: 877
343 128 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 110
5 2 </div>
<div>No. of Writes: 29401
78829 1448 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 19
33 34679 </div>
<div>No. of Writes: 6233
22903 41202 </div>
<div> </div>
<div> Block Size: 262144b+
524288b+ 1048576b+ </div>
<div> No. of Reads: 0
1 513 </div>
<div>No. of Writes: 1
0 105 </div>
<div> </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
126138 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
141671 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
117718 RELEASEDIR</div>
<div> 0.02 43.00 us 21.00 us 80.00 us
17 STAT</div>
<div> 0.02 51.62 us 27.00 us 131.00 us
21 STATFS</div>
<div> 0.09 42.78 us 11.00 us 1640.00 us
95 FLUSH</div>
<div> 0.21 50.01 us 23.00 us 567.00 us
189 ENTRYLK</div>
<div> 0.21 50.01 us 19.00 us 291.00 us
190 FINODELK</div>
<div> 0.56 8578.33 us 53.00 us 25625.00 us
3 GETXATTR</div>
<div> 0.62 148.89 us 71.00 us 2761.00 us
190 XATTROP</div>
<div> 0.70 168.11 us 83.00 us 1019.00 us
190 FXATTROP</div>
<div> 0.91 219.84 us 47.00 us 13732.00 us
190 SETATTR</div>
<div> 1.21 71.74 us 16.00 us 11516.00 us
775 INODELK</div>
<div> 1.47 354.88 us 56.00 us 22669.00 us
190 REMOVEXATTR</div>
<div> 2.60 1254.69 us 122.00 us 12514.00 us
95 WRITE</div>
<div> 3.25 194.10 us 51.00 us 48823.00 us
770 LOOKUP</div>
<div> 88.15 43068.81 us 265.00 us 418819.00 us
94 CREATE</div>
<div> </div>
<div> Duration: 644070 seconds</div>
<div> Data Read: 5089537361 bytes</div>
<div>Data Written: 9083513756 bytes</div>
<div> </div>
<div>Brick: node2:/bricks/brickname</div>
<div>-----------------------------------------------</div>
<div>Cumulative Stats:</div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 0
1 23 </div>
<div>No. of Writes: 60
0 141 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 45
47 363 </div>
<div>No. of Writes: 325
87 738 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 515
37 42 </div>
<div>No. of Writes: 877
343 128 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 17
0 1 </div>
<div>No. of Writes: 29401
78829 1448 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 15
39 67031 </div>
<div>No. of Writes: 6233
22903 41202 </div>
<div> </div>
<div> Block Size: 262144b+
1048576b+ </div>
<div> No. of Reads: 1
105 </div>
<div>No. of Writes: 1
105 </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
126136 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
141671 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
117718 RELEASEDIR</div>
<div> 0.02 74.00 us 74.00 us 74.00 us
1 STAT</div>
<div> 0.15 196.00 us 141.00 us 288.00 us
3 GETXATTR</div>
<div> 0.59 105.23 us 44.00 us 146.00 us
22 STATFS</div>
<div> 2.05 83.88 us 11.00 us 137.00 us
96 FLUSH</div>
<div> 4.79 98.61 us 20.00 us 146.00 us
191 ENTRYLK</div>
<div> 5.03 102.93 us 22.00 us 158.00 us
192 FINODELK</div>
<div> 5.52 226.07 us 136.00 us 295.00 us
96 WRITE</div>
<div> 6.31 261.08 us 150.00 us 345.00 us
95 CREATE</div>
<div> 6.72 137.53 us 50.00 us 214.00 us
192 SETATTR</div>
<div> 7.68 157.14 us 75.00 us 237.00 us
192 REMOVEXATTR</div>
<div> 8.13 166.49 us 81.00 us 282.00 us
192 XATTROP</div>
<div> 8.26 169.01 us 76.00 us 275.00 us
192 FXATTROP</div>
<div> 17.22 86.46 us 16.00 us 216.00 us
783 INODELK</div>
<div> 27.54 138.09 us 43.00 us 266.00 us
784 LOOKUP</div>
<div> </div>
<div> Duration: 644071 seconds</div>
<div> Data Read: 8902589511 bytes</div>
<div>Data Written: 9083513756 bytes</div>
<div> </div>
<div>Interval 0 Stats:</div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 0
1 23 </div>
<div>No. of Writes: 60
0 141 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 45
47 363 </div>
<div>No. of Writes: 325
87 738 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 515
37 42 </div>
<div>No. of Writes: 877
343 128 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 17
0 1 </div>
<div>No. of Writes: 29401
78829 1448 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 15
39 67031 </div>
<div>No. of Writes: 6233
22903 41202 </div>
<div> </div>
<div> Block Size: 262144b+
1048576b+ </div>
<div> No. of Reads: 1
105 </div>
<div>No. of Writes: 1
105 </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
126136 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
141671 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
117718 RELEASEDIR</div>
<div> 0.02 74.00 us 74.00 us 74.00 us
1 STAT</div>
<div> 0.15 196.00 us 141.00 us 288.00 us
3 GETXATTR</div>
<div> 0.59 105.23 us 44.00 us 146.00 us
22 STATFS</div>
<div> 2.05 83.88 us 11.00 us 137.00 us
96 FLUSH</div>
<div> 4.79 98.61 us 20.00 us 146.00 us
191 ENTRYLK</div>
<div> 5.03 102.93 us 22.00 us 158.00 us
192 FINODELK</div>
<div> 5.52 226.07 us 136.00 us 295.00 us
96 WRITE</div>
<div> 6.31 261.08 us 150.00 us 345.00 us
95 CREATE</div>
<div> 6.72 137.53 us 50.00 us 214.00 us
192 SETATTR</div>
<div> 7.68 157.14 us 75.00 us 237.00 us
192 REMOVEXATTR</div>
<div> 8.13 166.49 us 81.00 us 282.00 us
192 XATTROP</div>
<div> 8.26 169.01 us 76.00 us 275.00 us
192 FXATTROP</div>
<div> 17.22 86.46 us 16.00 us 216.00 us
783 INODELK</div>
<div> 27.54 138.09 us 43.00 us 266.00 us
784 LOOKUP</div>
<div> </div>
<div> Duration: 644071 seconds</div>
<div> Data Read: 8902589511 bytes</div>
<div>Data Written: 9083513756 bytes</div>
<div> </div>
<div>Brick: node3(sink):/bricks/brickname</div>
<div>------------------------------------------------</div>
<div>Cumulative Stats:</div>
<div> Block Size: 1b+
2b+ 4b+ </div>
<div> No. of Reads: 0
0 0 </div>
<div>No. of Writes: 11
26 125 </div>
<div> </div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 0
0 0 </div>
<div>No. of Writes: 829
2341 9599 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 0
0 0 </div>
<div>No. of Writes: 12674
9229 27346 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 2
10 0 </div>
<div>No. of Writes: 23414
28727 18372 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 1
0 0 </div>
<div>No. of Writes: 48347
92134 9675 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 2
11 50 </div>
<div>No. of Writes: 11717
24948 1022216 </div>
<div> </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
13805186 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
17674891 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
218068 RELEASEDIR</div>
<div> 0.00 24.00 us 24.00 us 24.00 us
1 OPENDIR</div>
<div> 0.00 13.06 us 8.00 us 36.00 us
16 STAT</div>
<div> 0.00 19.22 us 9.00 us 46.00 us
18 STATFS</div>
<div> 0.00 45.54 us 22.00 us 82.00 us
13 SETXATTR</div>
<div> 0.00 120.93 us 77.00 us 156.00 us
14 XATTROP</div>
<div> 0.00 11.01 us 7.00 us 68.00 us
156 ENTRYLK</div>
<div> 0.01 283.15 us 246.00 us 504.00 us
59 READDIR</div>
<div> 0.02 899.19 us 39.00 us 17518.00 us
26 SETATTR</div>
<div> 0.02 2004.85 us 38.00 us 10406.00 us
13 WRITE</div>
<div> 0.02 2022.77 us 24.00 us 21677.00 us
13 REMOVEXATTR</div>
<div> 0.03 2965.85 us 34.00 us 37695.00 us
13 FTRUNCATE</div>
<div> 0.04 3691.62 us 31.00 us 18386.00 us
13 FLUSH</div>
<div> 0.31 2105.65 us 23.00 us 57417.00 us
177 OPEN</div>
<div> 0.43 2603.12 us 57.00 us 73929.00 us
202 FXATTROP</div>
<div> 0.46 3030.94 us 7.00 us 87892.00 us
186 FSTAT</div>
<div> 1.07 33.17 us 18.00 us 17545.00 us
39491 GETXATTR</div>
<div> 1.31 123033.46 us 75610.00 us 269227.00 us
13 FSYNC</div>
<div> 46.14 704.24 us 6.00 us 268597.00 us
79866 INODELK</div>
<div> 50.13 699.93 us 20.00 us 267607.00 us
87307 LOOKUP</div>
<div> </div>
<div> Duration: 112674 seconds</div>
<div> Data Read: 7441454 bytes</div>
<div>Data Written: 138577629032 bytes</div>
<div> </div>
<div>Interval 0 Stats:</div>
<div> Block Size: 1b+
2b+ 4b+ </div>
<div> No. of Reads: 0
0 0 </div>
<div>No. of Writes: 11
26 125 </div>
<div> </div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 0
0 0 </div>
<div>No. of Writes: 829
2341 9599 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 0
0 0 </div>
<div>No. of Writes: 12674
9229 27346 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 2
10 0 </div>
<div>No. of Writes: 23414
28727 18372 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 1
0 0 </div>
<div>No. of Writes: 48347
92134 9675 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 2
11 50 </div>
<div>No. of Writes: 11717
24948 1022216 </div>
<div> </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
13805186 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
17674862 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
218068 RELEASEDIR</div>
<div> 0.00 24.00 us 24.00 us 24.00 us
1 OPENDIR</div>
<div> 0.00 13.06 us 8.00 us 36.00 us
16 STAT</div>
<div> 0.00 19.22 us 9.00 us 46.00 us
18 STATFS</div>
<div> 0.00 45.54 us 22.00 us 82.00 us
13 SETXATTR</div>
<div> 0.00 120.93 us 77.00 us 156.00 us
14 XATTROP</div>
<div> 0.00 11.01 us 7.00 us 68.00 us
156 ENTRYLK</div>
<div> 0.01 283.15 us 246.00 us 504.00 us
59 READDIR</div>
<div> 0.02 899.19 us 39.00 us 17518.00 us
26 SETATTR</div>
<div> 0.02 2004.85 us 38.00 us 10406.00 us
13 WRITE</div>
<div> 0.02 2022.77 us 24.00 us 21677.00 us
13 REMOVEXATTR</div>
<div> 0.03 2965.85 us 34.00 us 37695.00 us
13 FTRUNCATE</div>
<div> 0.04 3691.62 us 31.00 us 18386.00 us
13 FLUSH</div>
<div> 0.31 2105.65 us 23.00 us 57417.00 us
177 OPEN</div>
<div> 0.43 2603.12 us 57.00 us 73929.00 us
202 FXATTROP</div>
<div> 0.46 3030.94 us 7.00 us 87892.00 us
186 FSTAT</div>
<div> 1.07 33.17 us 18.00 us 17545.00 us
39491 GETXATTR</div>
<div> 1.31 123033.46 us 75610.00 us 269227.00 us
13 FSYNC</div>
<div> 46.14 704.24 us 6.00 us 268597.00 us
79866 INODELK</div>
<div> 50.13 699.93 us 20.00 us 267607.00 us
87307 LOOKUP</div>
<div> </div>
<div> Duration: 112674 seconds</div>
<div> Data Read: 7441454 bytes</div>
<div>Data Written: 138577629032 bytes</div>
<div> </div>
<div>Brick: node4:/bricks/brickname</div>
<div>-----------------------------------------------</div>
<div>Cumulative Stats:</div>
<div> Block Size: 8b+
32b+ 64b+ </div>
<div> No. of Reads: 0
9 24 </div>
<div>No. of Writes: 62
128 335 </div>
<div> </div>
<div> Block Size: 128b+
256b+ 512b+ </div>
<div> No. of Reads: 21
177 257 </div>
<div>No. of Writes: 186
779 885 </div>
<div> </div>
<div> Block Size: 1024b+
2048b+ 4096b+ </div>
<div> No. of Reads: 30
14 7 </div>
<div>No. of Writes: 286
101 29410 </div>
<div> </div>
<div> Block Size: 8192b+
16384b+ 32768b+ </div>
<div> No. of Reads: 0
9 0 </div>
<div>No. of Writes: 79662
1379 6187 </div>
<div> </div>
<div> Block Size: 65536b+
131072b+ 262144b+ </div>
<div> No. of Reads: 29
3924 0 </div>
<div>No. of Writes: 22467
32424 1 </div>
<div> </div>
<div> Block Size: 1048576b+ </div>
<div> No. of Reads: 0 </div>
<div>No. of Writes: 105 </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
126295 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
141875 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
117220 RELEASEDIR</div>
<div> 0.12 119.50 us 102.00 us 147.00 us
4 GETXATTR</div>
<div> 0.19 68.18 us 42.00 us 109.00 us
11 STAT</div>
<div> 0.44 92.16 us 19.00 us 141.00 us
19 STATFS</div>
<div> 1.20 68.03 us 11.00 us 120.00 us
71 FLUSH</div>
<div> 2.93 83.02 us 18.00 us 136.00 us
142 ENTRYLK</div>
<div> 3.15 89.18 us 16.00 us 160.00 us
142 FINODELK</div>
<div> 3.40 192.82 us 76.00 us 271.00 us
71 WRITE</div>
<div> 4.04 114.43 us 35.00 us 204.00 us
142 SETATTR</div>
<div> 4.87 138.05 us 49.00 us 222.00 us
142 REMOVEXATTR</div>
<div> 5.63 159.52 us 56.00 us 262.00 us
142 FXATTROP</div>
<div> 10.68 73.81 us 11.00 us 202.00 us
582 INODELK</div>
<div> 19.90 1127.35 us 116.00 us 27717.00 us
71 CREATE</div>
<div> 21.67 613.82 us 46.00 us 65260.00 us
142 XATTROP</div>
<div> 21.80 130.68 us 35.00 us 241.00 us
671 LOOKUP</div>
<div> </div>
<div> Duration: 458509 seconds</div>
<div> Data Read: 517180943 bytes</div>
<div>Data Written: 7895152670 bytes</div>
<div> </div>
<div>Interval 0 Stats:</div>
<div> Block Size: 8b+
32b+ 64b+ </div>
<div> No. of Reads: 0
9 24 </div>
<div>No. of Writes: 62
128 335 </div>
<div> </div>
<div> Block Size: 128b+
256b+ 512b+ </div>
<div> No. of Reads: 21
177 257 </div>
<div>No. of Writes: 186
779 885 </div>
<div> </div>
<div> Block Size: 1024b+
2048b+ 4096b+ </div>
<div> No. of Reads: 30
14 7 </div>
<div>No. of Writes: 286
101 29410 </div>
<div> </div>
<div> Block Size: 8192b+
16384b+ 32768b+ </div>
<div> No. of Reads: 0
9 0 </div>
<div>No. of Writes: 79662
1379 6187 </div>
<div> </div>
<div> Block Size: 65536b+
131072b+ 262144b+ </div>
<div> No. of Reads: 29
3924 0 </div>
<div>No. of Writes: 22467
32424 1 </div>
<div> </div>
<div> Block Size: 1048576b+ </div>
<div> No. of Reads: 0 </div>
<div>No. of Writes: 105 </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
126295 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
141875 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
117220 RELEASEDIR</div>
<div> 0.12 119.50 us 102.00 us 147.00 us
4 GETXATTR</div>
<div> 0.19 68.18 us 42.00 us 109.00 us
11 STAT</div>
<div> 0.44 92.16 us 19.00 us 141.00 us
19 STATFS</div>
<div> 1.20 68.03 us 11.00 us 120.00 us
71 FLUSH</div>
<div> 2.93 83.02 us 18.00 us 136.00 us
142 ENTRYLK</div>
<div> 3.15 89.18 us 16.00 us 160.00 us
142 FINODELK</div>
<div> 3.40 192.82 us 76.00 us 271.00 us
71 WRITE</div>
<div> 4.04 114.43 us 35.00 us 204.00 us
142 SETATTR</div>
<div> 4.87 138.05 us 49.00 us 222.00 us
142 REMOVEXATTR</div>
<div> 5.63 159.52 us 56.00 us 262.00 us
142 FXATTROP</div>
<div> 10.68 73.81 us 11.00 us 202.00 us
582 INODELK</div>
<div> 19.90 1127.35 us 116.00 us 27717.00 us
71 CREATE</div>
<div> 21.67 613.82 us 46.00 us 65260.00 us
142 XATTROP</div>
<div> 21.80 130.68 us 35.00 us 241.00 us
671 LOOKUP</div>
<div> </div>
<div> Duration: 458509 seconds</div>
<div> Data Read: 517180943 bytes</div>
<div>Data Written: 7895152670 bytes</div>
<div> </div>
<div>Brick: node5:/bricks/brickname</div>
<div>------------------------------------------------</div>
<div>Cumulative Stats:</div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 0
14 9 </div>
<div>No. of Writes: 62
0 128 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 23
56 225 </div>
<div>No. of Writes: 335
186 779 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 357
106 233 </div>
<div>No. of Writes: 885
286 102 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 128
11 15 </div>
<div>No. of Writes: 29410
79662 1379 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 16
34 28965 </div>
<div>No. of Writes: 6191
22467 32424 </div>
<div> </div>
<div> Block Size: 262144b+
524288b+ 1048576b+ </div>
<div> No. of Reads: 5
3 984 </div>
<div>No. of Writes: 1
0 105 </div>
<div> </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
126301 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
141880 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
117718 RELEASEDIR</div>
<div> 0.01 51.75 us 41.00 us 69.00 us
4 GETXATTR</div>
<div> 0.02 59.50 us 35.00 us 108.00 us
4 STAT</div>
<div> 0.05 44.06 us 28.00 us 118.00 us
16 STATFS</div>
<div> 0.10 24.78 us 15.00 us 99.00 us
59 FLUSH</div>
<div> 0.34 41.01 us 24.00 us 107.00 us
118 ENTRYLK</div>
<div> 0.37 44.43 us 25.00 us 156.00 us
118 FINODELK</div>
<div> 0.76 90.88 us 70.00 us 183.00 us
118 SETATTR</div>
<div> 0.81 193.88 us 162.00 us 283.00 us
59 WRITE</div>
<div> 0.98 116.89 us 85.00 us 212.00 us
118 REMOVEXATTR</div>
<div> 1.10 131.18 us 86.00 us 219.00 us
118 FXATTROP</div>
<div> 1.52 44.02 us 24.00 us 1004.00 us
487 INODELK</div>
<div> 3.35 86.29 us 52.00 us 183.00 us
549 LOOKUP</div>
<div> 4.21 504.03 us 83.00 us 43099.00 us
118 XATTROP</div>
<div> 86.39 20696.02 us 207.00 us 69802.00 us
59 CREATE</div>
<div> </div>
<div> Duration: 644071 seconds</div>
<div> Data Read: 4837930222 bytes</div>
<div>Data Written: 7895351133 bytes</div>
<div> </div>
<div>Interval 0 Stats:</div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 0
14 9 </div>
<div>No. of Writes: 62
0 128 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 23
56 225 </div>
<div>No. of Writes: 335
186 779 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 357
106 233 </div>
<div>No. of Writes: 885
286 102 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 128
11 15 </div>
<div>No. of Writes: 29410
79662 1379 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 16
34 28965 </div>
<div>No. of Writes: 6191
22467 32424 </div>
<div> </div>
<div> Block Size: 262144b+
524288b+ 1048576b+ </div>
<div> No. of Reads: 5
3 984 </div>
<div>No. of Writes: 1
0 105 </div>
<div> </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
126301 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
141880 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
117718 RELEASEDIR</div>
<div> 0.01 51.75 us 41.00 us 69.00 us
4 GETXATTR</div>
<div> 0.02 59.50 us 35.00 us 108.00 us
4 STAT</div>
<div> 0.05 44.06 us 28.00 us 118.00 us
16 STATFS</div>
<div> 0.10 24.78 us 15.00 us 99.00 us
59 FLUSH</div>
<div> 0.34 41.01 us 24.00 us 107.00 us
118 ENTRYLK</div>
<div> 0.37 44.43 us 25.00 us 156.00 us
118 FINODELK</div>
<div> 0.76 90.88 us 70.00 us 183.00 us
118 SETATTR</div>
<div> 0.81 193.88 us 162.00 us 283.00 us
59 WRITE</div>
<div> 0.98 116.89 us 85.00 us 212.00 us
118 REMOVEXATTR</div>
<div> 1.10 131.18 us 86.00 us 219.00 us
118 FXATTROP</div>
<div> 1.52 44.02 us 24.00 us 1004.00 us
487 INODELK</div>
<div> 3.35 86.29 us 52.00 us 183.00 us
549 LOOKUP</div>
<div> 4.21 504.03 us 83.00 us 43099.00 us
118 XATTROP</div>
<div> 86.39 20696.02 us 207.00 us 69802.00 us
59 CREATE</div>
<div> </div>
<div> Duration: 644071 seconds</div>
<div> Data Read: 4837930222 bytes</div>
<div>Data Written: 7895351133 bytes</div>
<div> </div>
<div>Brick: node6(source):/bricks/brickname</div>
<div>--------------------------------------------------</div>
<div>Cumulative Stats:</div>
<div> Block Size: 1b+
2b+ 4b+ </div>
<div> No. of Reads: 7
18 89 </div>
<div>No. of Writes: 4
8 37 </div>
<div> </div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 727
2325 9459 </div>
<div>No. of Writes: 108
54 188 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 12419
9313 27616 </div>
<div>No. of Writes: 360
85 772 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 23708
28691 18594 </div>
<div>No. of Writes: 847
313 138 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 19484
12596 8458 </div>
<div>No. of Writes: 29185
79632 1431 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 5695
5755 1062899 </div>
<div>No. of Writes: 6168
19435 32017 </div>
<div> </div>
<div> Block Size: 262144b+
1048576b+ </div>
<div> No. of Reads: 0
0 </div>
<div>No. of Writes: 1
105 </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
13806534 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
17813646 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
223324 RELEASEDIR</div>
<div> 0.00 560.00 us 560.00 us 560.00 us
1 ENTRYLK</div>
<div> 0.00 3901.00 us 3901.00 us 3901.00 us
1 SETXATTR</div>
<div> 0.00 4010.00 us 4010.00 us 4010.00 us
1 REMOVEXATTR</div>
<div> 0.01 62446.08 us 8.00 us 365433.00 us
13 FLUSH</div>
<div> 0.01 93887.77 us 52.00 us 588566.00 us
13 SETATTR</div>
<div> 0.03 10772.83 us 28.00 us 1121761.00 us
253 GETXATTR</div>
<div> 0.04 3190096.00 us 3190096.00 us 3190096.00 us
1 READDIR</div>
<div> 0.09 558307.69 us 179931.00 us 3188951.00 us
13 READ</div>
<div> 0.11 616756.00 us 74.00 us 7307745.00 us
14 XATTROP</div>
<div> 0.12 4754785.50 us 48.00 us 9509523.00 us
2 OPENDIR</div>
<div> 0.15 1799185.00 us 2310.00 us 5023537.00 us
7 STATFS</div>
<div> 0.16 68757.98 us 10.00 us 872148.00 us
189 FSTAT</div>
<div> 0.31 143533.93 us 42.00 us 7002195.00 us
174 OPEN</div>
<div> 0.40 160262.95 us 661.00 us 2825083.00 us
202 READDIRP</div>
<div> 1.55 624450.87 us 31.00 us 7397432.00 us
203 FXATTROP</div>
<div> 22.43 212161.62 us 12.00 us 7397413.00 us
8639 INODELK</div>
<div> 74.60 541421.09 us 63.00 us 14463033.00 us
11261 LOOKUP</div>
<div> </div>
<div> Duration: 644071 seconds</div>
<div> Data Read: 140706386722 bytes</div>
<div>Data Written: 7549422894 bytes</div>
<div> </div>
<div>Interval 0 Stats:</div>
<div> Block Size: 1b+
2b+ 4b+ </div>
<div> No. of Reads: 7
18 89 </div>
<div>No. of Writes: 4
8 37 </div>
<div> </div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 727
2325 9459 </div>
<div>No. of Writes: 108
54 188 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 12419
9313 27616 </div>
<div>No. of Writes: 360
85 772 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 23708
28691 18594 </div>
<div>No. of Writes: 847
313 138 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 19484
12596 8458 </div>
<div>No. of Writes: 29185
79632 1431 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 5695
5755 1062899 </div>
<div>No. of Writes: 6168
19435 32017 </div>
<div> </div>
<div> Block Size: 262144b+
1048576b+ </div>
<div> No. of Reads: 0
0 </div>
<div>No. of Writes: 1
105 </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
13806534 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
17813657 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
223324 RELEASEDIR</div>
<div> 0.00 560.00 us 560.00 us 560.00 us
1 ENTRYLK</div>
<div> 0.00 3901.00 us 3901.00 us 3901.00 us
1 SETXATTR</div>
<div> 0.00 4010.00 us 4010.00 us 4010.00 us
1 REMOVEXATTR</div>
<div> 0.01 62446.08 us 8.00 us 365433.00 us
13 FLUSH</div>
<div> 0.01 93887.77 us 52.00 us 588566.00 us
13 SETATTR</div>
<div> 0.03 10772.83 us 28.00 us 1121761.00 us
253 GETXATTR</div>
<div> 0.04 3190096.00 us 3190096.00 us 3190096.00 us
1 READDIR</div>
<div> 0.09 558307.69 us 179931.00 us 3188951.00 us
13 READ</div>
<div> 0.11 616756.00 us 74.00 us 7307745.00 us
14 XATTROP</div>
<div> 0.12 4754785.50 us 48.00 us 9509523.00 us
2 OPENDIR</div>
<div> 0.15 1799185.00 us 2310.00 us 5023537.00 us
7 STATFS</div>
<div> 0.16 68757.98 us 10.00 us 872148.00 us
189 FSTAT</div>
<div> 0.31 143533.93 us 42.00 us 7002195.00 us
174 OPEN</div>
<div> 0.40 160262.95 us 661.00 us 2825083.00 us
202 READDIRP</div>
<div> 1.55 624450.87 us 31.00 us 7397432.00 us
203 FXATTROP</div>
<div> 22.43 212161.62 us 12.00 us 7397413.00 us
8639 INODELK</div>
<div> 74.60 541421.09 us 63.00 us 14463033.00 us
11261 LOOKUP</div>
<div> </div>
<div> Duration: 644071 seconds</div>
<div> Data Read: 140706386722 bytes</div>
<div>Data Written: 7549422894 bytes</div>
<div> </div>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Aug 7, 2015 at 12:17 AM,
Ravishankar N <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span class=""> <br>
<br>
<div>On 08/07/2015 12:11 PM, Prasun Gera wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">No, no noticeable difference. Still
very high, possibly higher than before. </div>
</blockquote>
<br>
</span> I was guessing that the cpu usage could be because
of the diff algorithm which computes checksums (which is
a cpu intensive task). That doesn't seem to be the case.
Could you do a volume profile and see the FOPS that are
happening on the bricks and share the result?<br>
1.gluster volume profile <volname> start<br>
2. gluster volume profile <volname> info<br>
3. wait 10-15 seconds<br>
4.gluster volume profile <volname> info<span
class=""><br>
<br>
<br>
<br>
<blockquote type="cite">
<div dir="ltr">The system has come down to a crawl.
It's difficult to even ssh or run any commands on
the terminal. Do you make anything of the logs ? The
brick log is just a giant alternating stream of
those two lines I mentioned earlier. <br>
</div>
</blockquote>
<br>
<br>
<blockquote type="cite">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Aug 6, 2015 at
10:10 PM, Ravishankar N <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:ravishankar@redhat.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:ravishankar@redhat.com">ravishankar@redhat.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex"><span><br>
<br>
On 08/07/2015 01:33 AM, Prasun Gera wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex"> I replaced the
brick in a node in my 3x2 dist+repl volume
(RHS 3). I'm seeing that the heal process,
which should essentially be a dump from the
working replica to the newly added one is
taking exceptionally long. It has moved ~100
G over a day on a 1Gigabit network. The CPU
usage on both the nodes of the replica has
been pretty high. <br>
</blockquote>
<br>
</span> Does setting
`cluster.data-self-heal-algorithm` to full make
a difference in the cpu usage?
<div>
<div><br>
<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex"> I also think
that nagios is making it worse. The heal
is slow enough as it is, and nagios keeps
triggering heal info, which I think never
completes. I also see my logs filling up
These are some of the log contents which I
got by running tail on them:<br>
</blockquote>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</span></div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>