<div dir="ltr">So I let the heal complete, and it sped up later on. The total data that needed to be transferred to the brick was about 400G. It took about 2.5 days to finish. However, most of the time was spent in transferring a few GBs. Once it was through the rough patch, the rest of it transferred at acceptable speeds. That also corelates with the errors in the brick logs. It was really slow with high CPU usage when those errors were thrown in the brick log. Later one, the errors went away and the speed also became normal. Each brick is 1.8 TB. All the nodes have 2 TB SATA hard drives with 200GB reserved for OS, and rest as bricks. Some of the systems are old with low memory (4 GB). Not sure if that played a part in the heal. I did see spikes for kswapd0 when the CPU was high. The usage is a regular file server with most files ranging in the KBs to low MBs range. The network is a stock gigabit network without any tweaks for bonding, MTU etc. I can generate more specific stats if there are commands. </div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Aug 7, 2015 at 3:04 AM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
So the nodes 3 and 6 seem to indicate inode-locks and lookups are of
the highest latency. This only seems to confirm self-heals are
happening. <br>
If you are unable to use the system because of this, you could try
killing the self-heal daemons on both these nodes (kill `pgrep -f
glustershd`) to stop heals. You can then do a lookup of the files
from the mount, which will also trigger heals.<br>
Restart the selfheal daemons (with `gluster vol start volname
force`) when you think you can spare the volume for heals again. The
sooner the better though.<br>
For the brick log errors, we are suspecting it could be something
related to selinux.<br>
<br>
Can you tell what kind of data is there in your volume? - no. of
files, avg. file size, brick size, network connection speed etc?
Perhaps we can try to reproduce the issue and identify the bottle
neck.<br>
<br>
Thanks,<br>
Ravi<div><div class="h5"><br>
<br>
<div>On 08/07/2015 01:27 PM, Prasun Gera
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">All the volume commands are taking several minutes
to complete. Here's the profiler's output:
<div>Node3's brick is the one that was replaced. It's replica is
node6</div>
<div><br>
<div><br>
</div>
<div>
<div>Brick: node1:/bricks/brickname</div>
<div>---------------------------------------------------</div>
<div>Cumulative Stats:</div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 0
13 2 </div>
<div>No. of Writes: 60
0 141 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 3
26 17 </div>
<div>No. of Writes: 325
87 738 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 99
114 222 </div>
<div>No. of Writes: 877
343 128 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 110
5 2 </div>
<div>No. of Writes: 29401
78829 1448 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 19
33 34679 </div>
<div>No. of Writes: 6233
22903 41202 </div>
<div> </div>
<div> Block Size: 262144b+
524288b+ 1048576b+ </div>
<div> No. of Reads: 0
1 513 </div>
<div>No. of Writes: 1
0 105 </div>
<div> </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
126138 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
141671 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
117718 RELEASEDIR</div>
<div> 0.02 43.00 us 21.00 us 80.00 us
17 STAT</div>
<div> 0.02 51.62 us 27.00 us 131.00 us
21 STATFS</div>
<div> 0.09 42.78 us 11.00 us 1640.00 us
95 FLUSH</div>
<div> 0.21 50.01 us 23.00 us 567.00 us
189 ENTRYLK</div>
<div> 0.21 50.01 us 19.00 us 291.00 us
190 FINODELK</div>
<div> 0.56 8578.33 us 53.00 us 25625.00 us
3 GETXATTR</div>
<div> 0.62 148.89 us 71.00 us 2761.00 us
190 XATTROP</div>
<div> 0.70 168.11 us 83.00 us 1019.00 us
190 FXATTROP</div>
<div> 0.91 219.84 us 47.00 us 13732.00 us
190 SETATTR</div>
<div> 1.21 71.74 us 16.00 us 11516.00 us
775 INODELK</div>
<div> 1.47 354.88 us 56.00 us 22669.00 us
190 REMOVEXATTR</div>
<div> 2.60 1254.69 us 122.00 us 12514.00 us
95 WRITE</div>
<div> 3.25 194.10 us 51.00 us 48823.00 us
770 LOOKUP</div>
<div> 88.15 43068.81 us 265.00 us 418819.00 us
94 CREATE</div>
<div> </div>
<div> Duration: 644070 seconds</div>
<div> Data Read: <a href="tel:5089537361" value="+15089537361" target="_blank">5089537361</a> bytes</div>
<div>Data Written: <a href="tel:9083513756" value="+19083513756" target="_blank">9083513756</a> bytes</div>
<div> </div>
<div>Interval 0 Stats:</div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 0
13 2 </div>
<div>No. of Writes: 60
0 141 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 3
26 17 </div>
<div>No. of Writes: 325
87 738 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 99
114 222 </div>
<div>No. of Writes: 877
343 128 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 110
5 2 </div>
<div>No. of Writes: 29401
78829 1448 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 19
33 34679 </div>
<div>No. of Writes: 6233
22903 41202 </div>
<div> </div>
<div> Block Size: 262144b+
524288b+ 1048576b+ </div>
<div> No. of Reads: 0
1 513 </div>
<div>No. of Writes: 1
0 105 </div>
<div> </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
126138 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
141671 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
117718 RELEASEDIR</div>
<div> 0.02 43.00 us 21.00 us 80.00 us
17 STAT</div>
<div> 0.02 51.62 us 27.00 us 131.00 us
21 STATFS</div>
<div> 0.09 42.78 us 11.00 us 1640.00 us
95 FLUSH</div>
<div> 0.21 50.01 us 23.00 us 567.00 us
189 ENTRYLK</div>
<div> 0.21 50.01 us 19.00 us 291.00 us
190 FINODELK</div>
<div> 0.56 8578.33 us 53.00 us 25625.00 us
3 GETXATTR</div>
<div> 0.62 148.89 us 71.00 us 2761.00 us
190 XATTROP</div>
<div> 0.70 168.11 us 83.00 us 1019.00 us
190 FXATTROP</div>
<div> 0.91 219.84 us 47.00 us 13732.00 us
190 SETATTR</div>
<div> 1.21 71.74 us 16.00 us 11516.00 us
775 INODELK</div>
<div> 1.47 354.88 us 56.00 us 22669.00 us
190 REMOVEXATTR</div>
<div> 2.60 1254.69 us 122.00 us 12514.00 us
95 WRITE</div>
<div> 3.25 194.10 us 51.00 us 48823.00 us
770 LOOKUP</div>
<div> 88.15 43068.81 us 265.00 us 418819.00 us
94 CREATE</div>
<div> </div>
<div> Duration: 644070 seconds</div>
<div> Data Read: <a href="tel:5089537361" value="+15089537361" target="_blank">5089537361</a> bytes</div>
<div>Data Written: <a href="tel:9083513756" value="+19083513756" target="_blank">9083513756</a> bytes</div>
<div> </div>
<div>Brick: node2:/bricks/brickname</div>
<div>-----------------------------------------------</div>
<div>Cumulative Stats:</div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 0
1 23 </div>
<div>No. of Writes: 60
0 141 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 45
47 363 </div>
<div>No. of Writes: 325
87 738 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 515
37 42 </div>
<div>No. of Writes: 877
343 128 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 17
0 1 </div>
<div>No. of Writes: 29401
78829 1448 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 15
39 67031 </div>
<div>No. of Writes: 6233
22903 41202 </div>
<div> </div>
<div> Block Size: 262144b+
1048576b+ </div>
<div> No. of Reads: 1
105 </div>
<div>No. of Writes: 1
105 </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
126136 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
141671 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
117718 RELEASEDIR</div>
<div> 0.02 74.00 us 74.00 us 74.00 us
1 STAT</div>
<div> 0.15 196.00 us 141.00 us 288.00 us
3 GETXATTR</div>
<div> 0.59 105.23 us 44.00 us 146.00 us
22 STATFS</div>
<div> 2.05 83.88 us 11.00 us 137.00 us
96 FLUSH</div>
<div> 4.79 98.61 us 20.00 us 146.00 us
191 ENTRYLK</div>
<div> 5.03 102.93 us 22.00 us 158.00 us
192 FINODELK</div>
<div> 5.52 226.07 us 136.00 us 295.00 us
96 WRITE</div>
<div> 6.31 261.08 us 150.00 us 345.00 us
95 CREATE</div>
<div> 6.72 137.53 us 50.00 us 214.00 us
192 SETATTR</div>
<div> 7.68 157.14 us 75.00 us 237.00 us
192 REMOVEXATTR</div>
<div> 8.13 166.49 us 81.00 us 282.00 us
192 XATTROP</div>
<div> 8.26 169.01 us 76.00 us 275.00 us
192 FXATTROP</div>
<div> 17.22 86.46 us 16.00 us 216.00 us
783 INODELK</div>
<div> 27.54 138.09 us 43.00 us 266.00 us
784 LOOKUP</div>
<div> </div>
<div> Duration: 644071 seconds</div>
<div> Data Read: 8902589511 bytes</div>
<div>Data Written: <a href="tel:9083513756" value="+19083513756" target="_blank">9083513756</a> bytes</div>
<div> </div>
<div>Interval 0 Stats:</div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 0
1 23 </div>
<div>No. of Writes: 60
0 141 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 45
47 363 </div>
<div>No. of Writes: 325
87 738 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 515
37 42 </div>
<div>No. of Writes: 877
343 128 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 17
0 1 </div>
<div>No. of Writes: 29401
78829 1448 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 15
39 67031 </div>
<div>No. of Writes: 6233
22903 41202 </div>
<div> </div>
<div> Block Size: 262144b+
1048576b+ </div>
<div> No. of Reads: 1
105 </div>
<div>No. of Writes: 1
105 </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
126136 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
141671 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
117718 RELEASEDIR</div>
<div> 0.02 74.00 us 74.00 us 74.00 us
1 STAT</div>
<div> 0.15 196.00 us 141.00 us 288.00 us
3 GETXATTR</div>
<div> 0.59 105.23 us 44.00 us 146.00 us
22 STATFS</div>
<div> 2.05 83.88 us 11.00 us 137.00 us
96 FLUSH</div>
<div> 4.79 98.61 us 20.00 us 146.00 us
191 ENTRYLK</div>
<div> 5.03 102.93 us 22.00 us 158.00 us
192 FINODELK</div>
<div> 5.52 226.07 us 136.00 us 295.00 us
96 WRITE</div>
<div> 6.31 261.08 us 150.00 us 345.00 us
95 CREATE</div>
<div> 6.72 137.53 us 50.00 us 214.00 us
192 SETATTR</div>
<div> 7.68 157.14 us 75.00 us 237.00 us
192 REMOVEXATTR</div>
<div> 8.13 166.49 us 81.00 us 282.00 us
192 XATTROP</div>
<div> 8.26 169.01 us 76.00 us 275.00 us
192 FXATTROP</div>
<div> 17.22 86.46 us 16.00 us 216.00 us
783 INODELK</div>
<div> 27.54 138.09 us 43.00 us 266.00 us
784 LOOKUP</div>
<div> </div>
<div> Duration: 644071 seconds</div>
<div> Data Read: 8902589511 bytes</div>
<div>Data Written: <a href="tel:9083513756" value="+19083513756" target="_blank">9083513756</a> bytes</div>
<div> </div>
<div>Brick: node3(sink):/bricks/brickname</div>
<div>------------------------------------------------</div>
<div>Cumulative Stats:</div>
<div> Block Size: 1b+
2b+ 4b+ </div>
<div> No. of Reads: 0
0 0 </div>
<div>No. of Writes: 11
26 125 </div>
<div> </div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 0
0 0 </div>
<div>No. of Writes: 829
2341 9599 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 0
0 0 </div>
<div>No. of Writes: 12674
9229 27346 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 2
10 0 </div>
<div>No. of Writes: 23414
28727 18372 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 1
0 0 </div>
<div>No. of Writes: 48347
92134 9675 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 2
11 50 </div>
<div>No. of Writes: 11717
24948 1022216 </div>
<div> </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
13805186 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
17674891 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
218068 RELEASEDIR</div>
<div> 0.00 24.00 us 24.00 us 24.00 us
1 OPENDIR</div>
<div> 0.00 13.06 us 8.00 us 36.00 us
16 STAT</div>
<div> 0.00 19.22 us 9.00 us 46.00 us
18 STATFS</div>
<div> 0.00 45.54 us 22.00 us 82.00 us
13 SETXATTR</div>
<div> 0.00 120.93 us 77.00 us 156.00 us
14 XATTROP</div>
<div> 0.00 11.01 us 7.00 us 68.00 us
156 ENTRYLK</div>
<div> 0.01 283.15 us 246.00 us 504.00 us
59 READDIR</div>
<div> 0.02 899.19 us 39.00 us 17518.00 us
26 SETATTR</div>
<div> 0.02 2004.85 us 38.00 us 10406.00 us
13 WRITE</div>
<div> 0.02 2022.77 us 24.00 us 21677.00 us
13 REMOVEXATTR</div>
<div> 0.03 2965.85 us 34.00 us 37695.00 us
13 FTRUNCATE</div>
<div> 0.04 3691.62 us 31.00 us 18386.00 us
13 FLUSH</div>
<div> 0.31 2105.65 us 23.00 us 57417.00 us
177 OPEN</div>
<div> 0.43 2603.12 us 57.00 us 73929.00 us
202 FXATTROP</div>
<div> 0.46 3030.94 us 7.00 us 87892.00 us
186 FSTAT</div>
<div> 1.07 33.17 us 18.00 us 17545.00 us
39491 GETXATTR</div>
<div> 1.31 123033.46 us 75610.00 us 269227.00 us
13 FSYNC</div>
<div> 46.14 704.24 us 6.00 us 268597.00 us
79866 INODELK</div>
<div> 50.13 699.93 us 20.00 us 267607.00 us
87307 LOOKUP</div>
<div> </div>
<div> Duration: 112674 seconds</div>
<div> Data Read: 7441454 bytes</div>
<div>Data Written: 138577629032 bytes</div>
<div> </div>
<div>Interval 0 Stats:</div>
<div> Block Size: 1b+
2b+ 4b+ </div>
<div> No. of Reads: 0
0 0 </div>
<div>No. of Writes: 11
26 125 </div>
<div> </div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 0
0 0 </div>
<div>No. of Writes: 829
2341 9599 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 0
0 0 </div>
<div>No. of Writes: 12674
9229 27346 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 2
10 0 </div>
<div>No. of Writes: 23414
28727 18372 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 1
0 0 </div>
<div>No. of Writes: 48347
92134 9675 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 2
11 50 </div>
<div>No. of Writes: 11717
24948 1022216 </div>
<div> </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
13805186 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
17674862 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
218068 RELEASEDIR</div>
<div> 0.00 24.00 us 24.00 us 24.00 us
1 OPENDIR</div>
<div> 0.00 13.06 us 8.00 us 36.00 us
16 STAT</div>
<div> 0.00 19.22 us 9.00 us 46.00 us
18 STATFS</div>
<div> 0.00 45.54 us 22.00 us 82.00 us
13 SETXATTR</div>
<div> 0.00 120.93 us 77.00 us 156.00 us
14 XATTROP</div>
<div> 0.00 11.01 us 7.00 us 68.00 us
156 ENTRYLK</div>
<div> 0.01 283.15 us 246.00 us 504.00 us
59 READDIR</div>
<div> 0.02 899.19 us 39.00 us 17518.00 us
26 SETATTR</div>
<div> 0.02 2004.85 us 38.00 us 10406.00 us
13 WRITE</div>
<div> 0.02 2022.77 us 24.00 us 21677.00 us
13 REMOVEXATTR</div>
<div> 0.03 2965.85 us 34.00 us 37695.00 us
13 FTRUNCATE</div>
<div> 0.04 3691.62 us 31.00 us 18386.00 us
13 FLUSH</div>
<div> 0.31 2105.65 us 23.00 us 57417.00 us
177 OPEN</div>
<div> 0.43 2603.12 us 57.00 us 73929.00 us
202 FXATTROP</div>
<div> 0.46 3030.94 us 7.00 us 87892.00 us
186 FSTAT</div>
<div> 1.07 33.17 us 18.00 us 17545.00 us
39491 GETXATTR</div>
<div> 1.31 123033.46 us 75610.00 us 269227.00 us
13 FSYNC</div>
<div> 46.14 704.24 us 6.00 us 268597.00 us
79866 INODELK</div>
<div> 50.13 699.93 us 20.00 us 267607.00 us
87307 LOOKUP</div>
<div> </div>
<div> Duration: 112674 seconds</div>
<div> Data Read: 7441454 bytes</div>
<div>Data Written: 138577629032 bytes</div>
<div> </div>
<div>Brick: node4:/bricks/brickname</div>
<div>-----------------------------------------------</div>
<div>Cumulative Stats:</div>
<div> Block Size: 8b+
32b+ 64b+ </div>
<div> No. of Reads: 0
9 24 </div>
<div>No. of Writes: 62
128 335 </div>
<div> </div>
<div> Block Size: 128b+
256b+ 512b+ </div>
<div> No. of Reads: 21
177 257 </div>
<div>No. of Writes: 186
779 885 </div>
<div> </div>
<div> Block Size: 1024b+
2048b+ 4096b+ </div>
<div> No. of Reads: 30
14 7 </div>
<div>No. of Writes: 286
101 29410 </div>
<div> </div>
<div> Block Size: 8192b+
16384b+ 32768b+ </div>
<div> No. of Reads: 0
9 0 </div>
<div>No. of Writes: 79662
1379 6187 </div>
<div> </div>
<div> Block Size: 65536b+
131072b+ 262144b+ </div>
<div> No. of Reads: 29
3924 0 </div>
<div>No. of Writes: 22467
32424 1 </div>
<div> </div>
<div> Block Size: 1048576b+ </div>
<div> No. of Reads: 0 </div>
<div>No. of Writes: 105 </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
126295 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
141875 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
117220 RELEASEDIR</div>
<div> 0.12 119.50 us 102.00 us 147.00 us
4 GETXATTR</div>
<div> 0.19 68.18 us 42.00 us 109.00 us
11 STAT</div>
<div> 0.44 92.16 us 19.00 us 141.00 us
19 STATFS</div>
<div> 1.20 68.03 us 11.00 us 120.00 us
71 FLUSH</div>
<div> 2.93 83.02 us 18.00 us 136.00 us
142 ENTRYLK</div>
<div> 3.15 89.18 us 16.00 us 160.00 us
142 FINODELK</div>
<div> 3.40 192.82 us 76.00 us 271.00 us
71 WRITE</div>
<div> 4.04 114.43 us 35.00 us 204.00 us
142 SETATTR</div>
<div> 4.87 138.05 us 49.00 us 222.00 us
142 REMOVEXATTR</div>
<div> 5.63 159.52 us 56.00 us 262.00 us
142 FXATTROP</div>
<div> 10.68 73.81 us 11.00 us 202.00 us
582 INODELK</div>
<div> 19.90 1127.35 us 116.00 us 27717.00 us
71 CREATE</div>
<div> 21.67 613.82 us 46.00 us 65260.00 us
142 XATTROP</div>
<div> 21.80 130.68 us 35.00 us 241.00 us
671 LOOKUP</div>
<div> </div>
<div> Duration: 458509 seconds</div>
<div> Data Read: 517180943 bytes</div>
<div>Data Written: 7895152670 bytes</div>
<div> </div>
<div>Interval 0 Stats:</div>
<div> Block Size: 8b+
32b+ 64b+ </div>
<div> No. of Reads: 0
9 24 </div>
<div>No. of Writes: 62
128 335 </div>
<div> </div>
<div> Block Size: 128b+
256b+ 512b+ </div>
<div> No. of Reads: 21
177 257 </div>
<div>No. of Writes: 186
779 885 </div>
<div> </div>
<div> Block Size: 1024b+
2048b+ 4096b+ </div>
<div> No. of Reads: 30
14 7 </div>
<div>No. of Writes: 286
101 29410 </div>
<div> </div>
<div> Block Size: 8192b+
16384b+ 32768b+ </div>
<div> No. of Reads: 0
9 0 </div>
<div>No. of Writes: 79662
1379 6187 </div>
<div> </div>
<div> Block Size: 65536b+
131072b+ 262144b+ </div>
<div> No. of Reads: 29
3924 0 </div>
<div>No. of Writes: 22467
32424 1 </div>
<div> </div>
<div> Block Size: 1048576b+ </div>
<div> No. of Reads: 0 </div>
<div>No. of Writes: 105 </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
126295 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
141875 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
117220 RELEASEDIR</div>
<div> 0.12 119.50 us 102.00 us 147.00 us
4 GETXATTR</div>
<div> 0.19 68.18 us 42.00 us 109.00 us
11 STAT</div>
<div> 0.44 92.16 us 19.00 us 141.00 us
19 STATFS</div>
<div> 1.20 68.03 us 11.00 us 120.00 us
71 FLUSH</div>
<div> 2.93 83.02 us 18.00 us 136.00 us
142 ENTRYLK</div>
<div> 3.15 89.18 us 16.00 us 160.00 us
142 FINODELK</div>
<div> 3.40 192.82 us 76.00 us 271.00 us
71 WRITE</div>
<div> 4.04 114.43 us 35.00 us 204.00 us
142 SETATTR</div>
<div> 4.87 138.05 us 49.00 us 222.00 us
142 REMOVEXATTR</div>
<div> 5.63 159.52 us 56.00 us 262.00 us
142 FXATTROP</div>
<div> 10.68 73.81 us 11.00 us 202.00 us
582 INODELK</div>
<div> 19.90 1127.35 us 116.00 us 27717.00 us
71 CREATE</div>
<div> 21.67 613.82 us 46.00 us 65260.00 us
142 XATTROP</div>
<div> 21.80 130.68 us 35.00 us 241.00 us
671 LOOKUP</div>
<div> </div>
<div> Duration: 458509 seconds</div>
<div> Data Read: 517180943 bytes</div>
<div>Data Written: 7895152670 bytes</div>
<div> </div>
<div>Brick: node5:/bricks/brickname</div>
<div>------------------------------------------------</div>
<div>Cumulative Stats:</div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 0
14 9 </div>
<div>No. of Writes: 62
0 128 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 23
56 225 </div>
<div>No. of Writes: 335
186 779 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 357
106 233 </div>
<div>No. of Writes: 885
286 102 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 128
11 15 </div>
<div>No. of Writes: 29410
79662 1379 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 16
34 28965 </div>
<div>No. of Writes: 6191
22467 32424 </div>
<div> </div>
<div> Block Size: 262144b+
524288b+ 1048576b+ </div>
<div> No. of Reads: 5
3 984 </div>
<div>No. of Writes: 1
0 105 </div>
<div> </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
126301 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
141880 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
117718 RELEASEDIR</div>
<div> 0.01 51.75 us 41.00 us 69.00 us
4 GETXATTR</div>
<div> 0.02 59.50 us 35.00 us 108.00 us
4 STAT</div>
<div> 0.05 44.06 us 28.00 us 118.00 us
16 STATFS</div>
<div> 0.10 24.78 us 15.00 us 99.00 us
59 FLUSH</div>
<div> 0.34 41.01 us 24.00 us 107.00 us
118 ENTRYLK</div>
<div> 0.37 44.43 us 25.00 us 156.00 us
118 FINODELK</div>
<div> 0.76 90.88 us 70.00 us 183.00 us
118 SETATTR</div>
<div> 0.81 193.88 us 162.00 us 283.00 us
59 WRITE</div>
<div> 0.98 116.89 us 85.00 us 212.00 us
118 REMOVEXATTR</div>
<div> 1.10 131.18 us 86.00 us 219.00 us
118 FXATTROP</div>
<div> 1.52 44.02 us 24.00 us 1004.00 us
487 INODELK</div>
<div> 3.35 86.29 us 52.00 us 183.00 us
549 LOOKUP</div>
<div> 4.21 504.03 us 83.00 us 43099.00 us
118 XATTROP</div>
<div> 86.39 20696.02 us 207.00 us 69802.00 us
59 CREATE</div>
<div> </div>
<div> Duration: 644071 seconds</div>
<div> Data Read: 4837930222 bytes</div>
<div>Data Written: 7895351133 bytes</div>
<div> </div>
<div>Interval 0 Stats:</div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 0
14 9 </div>
<div>No. of Writes: 62
0 128 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 23
56 225 </div>
<div>No. of Writes: 335
186 779 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 357
106 233 </div>
<div>No. of Writes: 885
286 102 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 128
11 15 </div>
<div>No. of Writes: 29410
79662 1379 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 16
34 28965 </div>
<div>No. of Writes: 6191
22467 32424 </div>
<div> </div>
<div> Block Size: 262144b+
524288b+ 1048576b+ </div>
<div> No. of Reads: 5
3 984 </div>
<div>No. of Writes: 1
0 105 </div>
<div> </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
126301 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
141880 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
117718 RELEASEDIR</div>
<div> 0.01 51.75 us 41.00 us 69.00 us
4 GETXATTR</div>
<div> 0.02 59.50 us 35.00 us 108.00 us
4 STAT</div>
<div> 0.05 44.06 us 28.00 us 118.00 us
16 STATFS</div>
<div> 0.10 24.78 us 15.00 us 99.00 us
59 FLUSH</div>
<div> 0.34 41.01 us 24.00 us 107.00 us
118 ENTRYLK</div>
<div> 0.37 44.43 us 25.00 us 156.00 us
118 FINODELK</div>
<div> 0.76 90.88 us 70.00 us 183.00 us
118 SETATTR</div>
<div> 0.81 193.88 us 162.00 us 283.00 us
59 WRITE</div>
<div> 0.98 116.89 us 85.00 us 212.00 us
118 REMOVEXATTR</div>
<div> 1.10 131.18 us 86.00 us 219.00 us
118 FXATTROP</div>
<div> 1.52 44.02 us 24.00 us 1004.00 us
487 INODELK</div>
<div> 3.35 86.29 us 52.00 us 183.00 us
549 LOOKUP</div>
<div> 4.21 504.03 us 83.00 us 43099.00 us
118 XATTROP</div>
<div> 86.39 20696.02 us 207.00 us 69802.00 us
59 CREATE</div>
<div> </div>
<div> Duration: 644071 seconds</div>
<div> Data Read: 4837930222 bytes</div>
<div>Data Written: 7895351133 bytes</div>
<div> </div>
<div>Brick: node6(source):/bricks/brickname</div>
<div>--------------------------------------------------</div>
<div>Cumulative Stats:</div>
<div> Block Size: 1b+
2b+ 4b+ </div>
<div> No. of Reads: 7
18 89 </div>
<div>No. of Writes: 4
8 37 </div>
<div> </div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 727
2325 9459 </div>
<div>No. of Writes: 108
54 188 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 12419
9313 27616 </div>
<div>No. of Writes: 360
85 772 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 23708
28691 18594 </div>
<div>No. of Writes: 847
313 138 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 19484
12596 8458 </div>
<div>No. of Writes: 29185
79632 1431 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 5695
5755 1062899 </div>
<div>No. of Writes: 6168
19435 32017 </div>
<div> </div>
<div> Block Size: 262144b+
1048576b+ </div>
<div> No. of Reads: 0
0 </div>
<div>No. of Writes: 1
105 </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
13806534 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
17813646 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
223324 RELEASEDIR</div>
<div> 0.00 560.00 us 560.00 us 560.00 us
1 ENTRYLK</div>
<div> 0.00 3901.00 us 3901.00 us 3901.00 us
1 SETXATTR</div>
<div> 0.00 4010.00 us 4010.00 us 4010.00 us
1 REMOVEXATTR</div>
<div> 0.01 62446.08 us 8.00 us 365433.00 us
13 FLUSH</div>
<div> 0.01 93887.77 us 52.00 us 588566.00 us
13 SETATTR</div>
<div> 0.03 10772.83 us 28.00 us 1121761.00 us
253 GETXATTR</div>
<div> 0.04 3190096.00 us 3190096.00 us 3190096.00 us
1 READDIR</div>
<div> 0.09 558307.69 us 179931.00 us 3188951.00 us
13 READ</div>
<div> 0.11 616756.00 us 74.00 us 7307745.00 us
14 XATTROP</div>
<div> 0.12 4754785.50 us 48.00 us 9509523.00 us
2 OPENDIR</div>
<div> 0.15 1799185.00 us 2310.00 us 5023537.00 us
7 STATFS</div>
<div> 0.16 68757.98 us 10.00 us 872148.00 us
189 FSTAT</div>
<div> 0.31 143533.93 us 42.00 us 7002195.00 us
174 OPEN</div>
<div> 0.40 160262.95 us 661.00 us 2825083.00 us
202 READDIRP</div>
<div> 1.55 624450.87 us 31.00 us 7397432.00 us
203 FXATTROP</div>
<div> 22.43 212161.62 us 12.00 us 7397413.00 us
8639 INODELK</div>
<div> 74.60 541421.09 us 63.00 us 14463033.00 us
11261 LOOKUP</div>
<div> </div>
<div> Duration: 644071 seconds</div>
<div> Data Read: 140706386722 bytes</div>
<div>Data Written: <a href="tel:7549422894" value="+17549422894" target="_blank">7549422894</a> bytes</div>
<div> </div>
<div>Interval 0 Stats:</div>
<div> Block Size: 1b+
2b+ 4b+ </div>
<div> No. of Reads: 7
18 89 </div>
<div>No. of Writes: 4
8 37 </div>
<div> </div>
<div> Block Size: 8b+
16b+ 32b+ </div>
<div> No. of Reads: 727
2325 9459 </div>
<div>No. of Writes: 108
54 188 </div>
<div> </div>
<div> Block Size: 64b+
128b+ 256b+ </div>
<div> No. of Reads: 12419
9313 27616 </div>
<div>No. of Writes: 360
85 772 </div>
<div> </div>
<div> Block Size: 512b+
1024b+ 2048b+ </div>
<div> No. of Reads: 23708
28691 18594 </div>
<div>No. of Writes: 847
313 138 </div>
<div> </div>
<div> Block Size: 4096b+
8192b+ 16384b+ </div>
<div> No. of Reads: 19484
12596 8458 </div>
<div>No. of Writes: 29185
79632 1431 </div>
<div> </div>
<div> Block Size: 32768b+
65536b+ 131072b+ </div>
<div> No. of Reads: 5695
5755 1062899 </div>
<div>No. of Writes: 6168
19435 32017 </div>
<div> </div>
<div> Block Size: 262144b+
1048576b+ </div>
<div> No. of Reads: 0
0 </div>
<div>No. of Writes: 1
105 </div>
<div> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop</div>
<div> --------- ----------- ----------- -----------
------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
13806534 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
17813657 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us
223324 RELEASEDIR</div>
<div> 0.00 560.00 us 560.00 us 560.00 us
1 ENTRYLK</div>
<div> 0.00 3901.00 us 3901.00 us 3901.00 us
1 SETXATTR</div>
<div> 0.00 4010.00 us 4010.00 us 4010.00 us
1 REMOVEXATTR</div>
<div> 0.01 62446.08 us 8.00 us 365433.00 us
13 FLUSH</div>
<div> 0.01 93887.77 us 52.00 us 588566.00 us
13 SETATTR</div>
<div> 0.03 10772.83 us 28.00 us 1121761.00 us
253 GETXATTR</div>
<div> 0.04 3190096.00 us 3190096.00 us 3190096.00 us
1 READDIR</div>
<div> 0.09 558307.69 us 179931.00 us 3188951.00 us
13 READ</div>
<div> 0.11 616756.00 us 74.00 us 7307745.00 us
14 XATTROP</div>
<div> 0.12 4754785.50 us 48.00 us 9509523.00 us
2 OPENDIR</div>
<div> 0.15 1799185.00 us 2310.00 us 5023537.00 us
7 STATFS</div>
<div> 0.16 68757.98 us 10.00 us 872148.00 us
189 FSTAT</div>
<div> 0.31 143533.93 us 42.00 us 7002195.00 us
174 OPEN</div>
<div> 0.40 160262.95 us 661.00 us 2825083.00 us
202 READDIRP</div>
<div> 1.55 624450.87 us 31.00 us 7397432.00 us
203 FXATTROP</div>
<div> 22.43 212161.62 us 12.00 us 7397413.00 us
8639 INODELK</div>
<div> 74.60 541421.09 us 63.00 us 14463033.00 us
11261 LOOKUP</div>
<div> </div>
<div> Duration: 644071 seconds</div>
<div> Data Read: 140706386722 bytes</div>
<div>Data Written: <a href="tel:7549422894" value="+17549422894" target="_blank">7549422894</a> bytes</div>
<div> </div>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Aug 7, 2015 at 12:17 AM,
Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span> <br>
<br>
<div>On 08/07/2015 12:11 PM, Prasun Gera wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">No, no noticeable difference. Still
very high, possibly higher than before. </div>
</blockquote>
<br>
</span> I was guessing that the cpu usage could be because
of the diff algorithm which computes checksums (which is
a cpu intensive task). That doesn't seem to be the case.
Could you do a volume profile and see the FOPS that are
happening on the bricks and share the result?<br>
1.gluster volume profile <volname> start<br>
2. gluster volume profile <volname> info<br>
3. wait 10-15 seconds<br>
4.gluster volume profile <volname> info<span><br>
<br>
<br>
<br>
<blockquote type="cite">
<div dir="ltr">The system has come down to a crawl.
It's difficult to even ssh or run any commands on
the terminal. Do you make anything of the logs ? The
brick log is just a giant alternating stream of
those two lines I mentioned earlier. <br>
</div>
</blockquote>
<br>
<br>
<blockquote type="cite">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Aug 6, 2015 at
10:10 PM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank"></a><a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><br>
<br>
On 08/07/2015 01:33 AM, Prasun Gera wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> I replaced the
brick in a node in my 3x2 dist+repl volume
(RHS 3). I'm seeing that the heal process,
which should essentially be a dump from the
working replica to the newly added one is
taking exceptionally long. It has moved ~100
G over a day on a 1Gigabit network. The CPU
usage on both the nodes of the replica has
been pretty high. <br>
</blockquote>
<br>
</span> Does setting
`cluster.data-self-heal-algorithm` to full make
a difference in the cpu usage?
<div>
<div><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> I also think
that nagios is making it worse. The heal
is slow enough as it is, and nagios keeps
triggering heal info, which I think never
completes. I also see my logs filling up
These are some of the log contents which I
got by running tail on them:<br>
</blockquote>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</span></div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div>