<div dir="ltr">All the volume commands are taking several minutes to complete. Here's the profiler's output:<div>Node3's brick is the one that was replaced. It's replica is node6</div><div><br><div><br></div><div><div>Brick: node1:/bricks/brickname</div><div>---------------------------------------------------</div><div>Cumulative Stats:</div><div> Block Size: 8b+ 16b+ 32b+ </div><div> No. of Reads: 0 13 2 </div><div>No. of Writes: 60 0 141 </div><div> </div><div> Block Size: 64b+ 128b+ 256b+ </div><div> No. of Reads: 3 26 17 </div><div>No. of Writes: 325 87 738 </div><div> </div><div> Block Size: 512b+ 1024b+ 2048b+ </div><div> No. of Reads: 99 114 222 </div><div>No. of Writes: 877 343 128 </div><div> </div><div> Block Size: 4096b+ 8192b+ 16384b+ </div><div> No. of Reads: 110 5 2 </div><div>No. of Writes: 29401 78829 1448 </div><div> </div><div> Block Size: 32768b+ 65536b+ 131072b+ </div><div> No. of Reads: 19 33 34679 </div><div>No. of Writes: 6233 22903 41202 </div><div> </div><div> Block Size: 262144b+ 524288b+ 1048576b+ </div><div> No. of Reads: 0 1 513 </div><div>No. of Writes: 1 0 105 </div><div> </div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 126138 FORGET</div><div> 0.00 0.00 us 0.00 us 0.00 us 141671 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 117718 RELEASEDIR</div><div> 0.02 43.00 us 21.00 us 80.00 us 17 STAT</div><div> 0.02 51.62 us 27.00 us 131.00 us 21 STATFS</div><div> 0.09 42.78 us 11.00 us 1640.00 us 95 FLUSH</div><div> 0.21 50.01 us 23.00 us 567.00 us 189 ENTRYLK</div><div> 0.21 50.01 us 19.00 us 291.00 us 190 FINODELK</div><div> 0.56 8578.33 us 53.00 us 25625.00 us 3 GETXATTR</div><div> 0.62 148.89 us 71.00 us 2761.00 us 190 XATTROP</div><div> 0.70 168.11 us 83.00 us 1019.00 us 190 FXATTROP</div><div> 0.91 219.84 us 47.00 us 13732.00 us 190 SETATTR</div><div> 1.21 71.74 us 16.00 us 11516.00 us 775 INODELK</div><div> 1.47 354.88 us 56.00 us 22669.00 us 190 REMOVEXATTR</div><div> 2.60 1254.69 us 122.00 us 12514.00 us 95 WRITE</div><div> 3.25 194.10 us 51.00 us 48823.00 us 770 LOOKUP</div><div> 88.15 43068.81 us 265.00 us 418819.00 us 94 CREATE</div><div> </div><div> Duration: 644070 seconds</div><div> Data Read: 5089537361 bytes</div><div>Data Written: 9083513756 bytes</div><div> </div><div>Interval 0 Stats:</div><div> Block Size: 8b+ 16b+ 32b+ </div><div> No. of Reads: 0 13 2 </div><div>No. of Writes: 60 0 141 </div><div> </div><div> Block Size: 64b+ 128b+ 256b+ </div><div> No. of Reads: 3 26 17 </div><div>No. of Writes: 325 87 738 </div><div> </div><div> Block Size: 512b+ 1024b+ 2048b+ </div><div> No. of Reads: 99 114 222 </div><div>No. of Writes: 877 343 128 </div><div> </div><div> Block Size: 4096b+ 8192b+ 16384b+ </div><div> No. of Reads: 110 5 2 </div><div>No. of Writes: 29401 78829 1448 </div><div> </div><div> Block Size: 32768b+ 65536b+ 131072b+ </div><div> No. of Reads: 19 33 34679 </div><div>No. of Writes: 6233 22903 41202 </div><div> </div><div> Block Size: 262144b+ 524288b+ 1048576b+ </div><div> No. of Reads: 0 1 513 </div><div>No. of Writes: 1 0 105 </div><div> </div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 126138 FORGET</div><div> 0.00 0.00 us 0.00 us 0.00 us 141671 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 117718 RELEASEDIR</div><div> 0.02 43.00 us 21.00 us 80.00 us 17 STAT</div><div> 0.02 51.62 us 27.00 us 131.00 us 21 STATFS</div><div> 0.09 42.78 us 11.00 us 1640.00 us 95 FLUSH</div><div> 0.21 50.01 us 23.00 us 567.00 us 189 ENTRYLK</div><div> 0.21 50.01 us 19.00 us 291.00 us 190 FINODELK</div><div> 0.56 8578.33 us 53.00 us 25625.00 us 3 GETXATTR</div><div> 0.62 148.89 us 71.00 us 2761.00 us 190 XATTROP</div><div> 0.70 168.11 us 83.00 us 1019.00 us 190 FXATTROP</div><div> 0.91 219.84 us 47.00 us 13732.00 us 190 SETATTR</div><div> 1.21 71.74 us 16.00 us 11516.00 us 775 INODELK</div><div> 1.47 354.88 us 56.00 us 22669.00 us 190 REMOVEXATTR</div><div> 2.60 1254.69 us 122.00 us 12514.00 us 95 WRITE</div><div> 3.25 194.10 us 51.00 us 48823.00 us 770 LOOKUP</div><div> 88.15 43068.81 us 265.00 us 418819.00 us 94 CREATE</div><div> </div><div> Duration: 644070 seconds</div><div> Data Read: 5089537361 bytes</div><div>Data Written: 9083513756 bytes</div><div> </div><div>Brick: node2:/bricks/brickname</div><div>-----------------------------------------------</div><div>Cumulative Stats:</div><div> Block Size: 8b+ 16b+ 32b+ </div><div> No. of Reads: 0 1 23 </div><div>No. of Writes: 60 0 141 </div><div> </div><div> Block Size: 64b+ 128b+ 256b+ </div><div> No. of Reads: 45 47 363 </div><div>No. of Writes: 325 87 738 </div><div> </div><div> Block Size: 512b+ 1024b+ 2048b+ </div><div> No. of Reads: 515 37 42 </div><div>No. of Writes: 877 343 128 </div><div> </div><div> Block Size: 4096b+ 8192b+ 16384b+ </div><div> No. of Reads: 17 0 1 </div><div>No. of Writes: 29401 78829 1448 </div><div> </div><div> Block Size: 32768b+ 65536b+ 131072b+ </div><div> No. of Reads: 15 39 67031 </div><div>No. of Writes: 6233 22903 41202 </div><div> </div><div> Block Size: 262144b+ 1048576b+ </div><div> No. of Reads: 1 105 </div><div>No. of Writes: 1 105 </div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 126136 FORGET</div><div> 0.00 0.00 us 0.00 us 0.00 us 141671 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 117718 RELEASEDIR</div><div> 0.02 74.00 us 74.00 us 74.00 us 1 STAT</div><div> 0.15 196.00 us 141.00 us 288.00 us 3 GETXATTR</div><div> 0.59 105.23 us 44.00 us 146.00 us 22 STATFS</div><div> 2.05 83.88 us 11.00 us 137.00 us 96 FLUSH</div><div> 4.79 98.61 us 20.00 us 146.00 us 191 ENTRYLK</div><div> 5.03 102.93 us 22.00 us 158.00 us 192 FINODELK</div><div> 5.52 226.07 us 136.00 us 295.00 us 96 WRITE</div><div> 6.31 261.08 us 150.00 us 345.00 us 95 CREATE</div><div> 6.72 137.53 us 50.00 us 214.00 us 192 SETATTR</div><div> 7.68 157.14 us 75.00 us 237.00 us 192 REMOVEXATTR</div><div> 8.13 166.49 us 81.00 us 282.00 us 192 XATTROP</div><div> 8.26 169.01 us 76.00 us 275.00 us 192 FXATTROP</div><div> 17.22 86.46 us 16.00 us 216.00 us 783 INODELK</div><div> 27.54 138.09 us 43.00 us 266.00 us 784 LOOKUP</div><div> </div><div> Duration: 644071 seconds</div><div> Data Read: 8902589511 bytes</div><div>Data Written: 9083513756 bytes</div><div> </div><div>Interval 0 Stats:</div><div> Block Size: 8b+ 16b+ 32b+ </div><div> No. of Reads: 0 1 23 </div><div>No. of Writes: 60 0 141 </div><div> </div><div> Block Size: 64b+ 128b+ 256b+ </div><div> No. of Reads: 45 47 363 </div><div>No. of Writes: 325 87 738 </div><div> </div><div> Block Size: 512b+ 1024b+ 2048b+ </div><div> No. of Reads: 515 37 42 </div><div>No. of Writes: 877 343 128 </div><div> </div><div> Block Size: 4096b+ 8192b+ 16384b+ </div><div> No. of Reads: 17 0 1 </div><div>No. of Writes: 29401 78829 1448 </div><div> </div><div> Block Size: 32768b+ 65536b+ 131072b+ </div><div> No. of Reads: 15 39 67031 </div><div>No. of Writes: 6233 22903 41202 </div><div> </div><div> Block Size: 262144b+ 1048576b+ </div><div> No. of Reads: 1 105 </div><div>No. of Writes: 1 105 </div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 126136 FORGET</div><div> 0.00 0.00 us 0.00 us 0.00 us 141671 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 117718 RELEASEDIR</div><div> 0.02 74.00 us 74.00 us 74.00 us 1 STAT</div><div> 0.15 196.00 us 141.00 us 288.00 us 3 GETXATTR</div><div> 0.59 105.23 us 44.00 us 146.00 us 22 STATFS</div><div> 2.05 83.88 us 11.00 us 137.00 us 96 FLUSH</div><div> 4.79 98.61 us 20.00 us 146.00 us 191 ENTRYLK</div><div> 5.03 102.93 us 22.00 us 158.00 us 192 FINODELK</div><div> 5.52 226.07 us 136.00 us 295.00 us 96 WRITE</div><div> 6.31 261.08 us 150.00 us 345.00 us 95 CREATE</div><div> 6.72 137.53 us 50.00 us 214.00 us 192 SETATTR</div><div> 7.68 157.14 us 75.00 us 237.00 us 192 REMOVEXATTR</div><div> 8.13 166.49 us 81.00 us 282.00 us 192 XATTROP</div><div> 8.26 169.01 us 76.00 us 275.00 us 192 FXATTROP</div><div> 17.22 86.46 us 16.00 us 216.00 us 783 INODELK</div><div> 27.54 138.09 us 43.00 us 266.00 us 784 LOOKUP</div><div> </div><div> Duration: 644071 seconds</div><div> Data Read: 8902589511 bytes</div><div>Data Written: 9083513756 bytes</div><div> </div><div>Brick: node3(sink):/bricks/brickname</div><div>------------------------------------------------</div><div>Cumulative Stats:</div><div> Block Size: 1b+ 2b+ 4b+ </div><div> No. of Reads: 0 0 0 </div><div>No. of Writes: 11 26 125 </div><div> </div><div> Block Size: 8b+ 16b+ 32b+ </div><div> No. of Reads: 0 0 0 </div><div>No. of Writes: 829 2341 9599 </div><div> </div><div> Block Size: 64b+ 128b+ 256b+ </div><div> No. of Reads: 0 0 0 </div><div>No. of Writes: 12674 9229 27346 </div><div> </div><div> Block Size: 512b+ 1024b+ 2048b+ </div><div> No. of Reads: 2 10 0 </div><div>No. of Writes: 23414 28727 18372 </div><div> </div><div> Block Size: 4096b+ 8192b+ 16384b+ </div><div> No. of Reads: 1 0 0 </div><div>No. of Writes: 48347 92134 9675 </div><div> </div><div> Block Size: 32768b+ 65536b+ 131072b+ </div><div> No. of Reads: 2 11 50 </div><div>No. of Writes: 11717 24948 1022216 </div><div> </div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 13805186 FORGET</div><div> 0.00 0.00 us 0.00 us 0.00 us 17674891 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 218068 RELEASEDIR</div><div> 0.00 24.00 us 24.00 us 24.00 us 1 OPENDIR</div><div> 0.00 13.06 us 8.00 us 36.00 us 16 STAT</div><div> 0.00 19.22 us 9.00 us 46.00 us 18 STATFS</div><div> 0.00 45.54 us 22.00 us 82.00 us 13 SETXATTR</div><div> 0.00 120.93 us 77.00 us 156.00 us 14 XATTROP</div><div> 0.00 11.01 us 7.00 us 68.00 us 156 ENTRYLK</div><div> 0.01 283.15 us 246.00 us 504.00 us 59 READDIR</div><div> 0.02 899.19 us 39.00 us 17518.00 us 26 SETATTR</div><div> 0.02 2004.85 us 38.00 us 10406.00 us 13 WRITE</div><div> 0.02 2022.77 us 24.00 us 21677.00 us 13 REMOVEXATTR</div><div> 0.03 2965.85 us 34.00 us 37695.00 us 13 FTRUNCATE</div><div> 0.04 3691.62 us 31.00 us 18386.00 us 13 FLUSH</div><div> 0.31 2105.65 us 23.00 us 57417.00 us 177 OPEN</div><div> 0.43 2603.12 us 57.00 us 73929.00 us 202 FXATTROP</div><div> 0.46 3030.94 us 7.00 us 87892.00 us 186 FSTAT</div><div> 1.07 33.17 us 18.00 us 17545.00 us 39491 GETXATTR</div><div> 1.31 123033.46 us 75610.00 us 269227.00 us 13 FSYNC</div><div> 46.14 704.24 us 6.00 us 268597.00 us 79866 INODELK</div><div> 50.13 699.93 us 20.00 us 267607.00 us 87307 LOOKUP</div><div> </div><div> Duration: 112674 seconds</div><div> Data Read: 7441454 bytes</div><div>Data Written: 138577629032 bytes</div><div> </div><div>Interval 0 Stats:</div><div> Block Size: 1b+ 2b+ 4b+ </div><div> No. of Reads: 0 0 0 </div><div>No. of Writes: 11 26 125 </div><div> </div><div> Block Size: 8b+ 16b+ 32b+ </div><div> No. of Reads: 0 0 0 </div><div>No. of Writes: 829 2341 9599 </div><div> </div><div> Block Size: 64b+ 128b+ 256b+ </div><div> No. of Reads: 0 0 0 </div><div>No. of Writes: 12674 9229 27346 </div><div> </div><div> Block Size: 512b+ 1024b+ 2048b+ </div><div> No. of Reads: 2 10 0 </div><div>No. of Writes: 23414 28727 18372 </div><div> </div><div> Block Size: 4096b+ 8192b+ 16384b+ </div><div> No. of Reads: 1 0 0 </div><div>No. of Writes: 48347 92134 9675 </div><div> </div><div> Block Size: 32768b+ 65536b+ 131072b+ </div><div> No. of Reads: 2 11 50 </div><div>No. of Writes: 11717 24948 1022216 </div><div> </div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 13805186 FORGET</div><div> 0.00 0.00 us 0.00 us 0.00 us 17674862 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 218068 RELEASEDIR</div><div> 0.00 24.00 us 24.00 us 24.00 us 1 OPENDIR</div><div> 0.00 13.06 us 8.00 us 36.00 us 16 STAT</div><div> 0.00 19.22 us 9.00 us 46.00 us 18 STATFS</div><div> 0.00 45.54 us 22.00 us 82.00 us 13 SETXATTR</div><div> 0.00 120.93 us 77.00 us 156.00 us 14 XATTROP</div><div> 0.00 11.01 us 7.00 us 68.00 us 156 ENTRYLK</div><div> 0.01 283.15 us 246.00 us 504.00 us 59 READDIR</div><div> 0.02 899.19 us 39.00 us 17518.00 us 26 SETATTR</div><div> 0.02 2004.85 us 38.00 us 10406.00 us 13 WRITE</div><div> 0.02 2022.77 us 24.00 us 21677.00 us 13 REMOVEXATTR</div><div> 0.03 2965.85 us 34.00 us 37695.00 us 13 FTRUNCATE</div><div> 0.04 3691.62 us 31.00 us 18386.00 us 13 FLUSH</div><div> 0.31 2105.65 us 23.00 us 57417.00 us 177 OPEN</div><div> 0.43 2603.12 us 57.00 us 73929.00 us 202 FXATTROP</div><div> 0.46 3030.94 us 7.00 us 87892.00 us 186 FSTAT</div><div> 1.07 33.17 us 18.00 us 17545.00 us 39491 GETXATTR</div><div> 1.31 123033.46 us 75610.00 us 269227.00 us 13 FSYNC</div><div> 46.14 704.24 us 6.00 us 268597.00 us 79866 INODELK</div><div> 50.13 699.93 us 20.00 us 267607.00 us 87307 LOOKUP</div><div> </div><div> Duration: 112674 seconds</div><div> Data Read: 7441454 bytes</div><div>Data Written: 138577629032 bytes</div><div> </div><div>Brick: node4:/bricks/brickname</div><div>-----------------------------------------------</div><div>Cumulative Stats:</div><div> Block Size: 8b+ 32b+ 64b+ </div><div> No. of Reads: 0 9 24 </div><div>No. of Writes: 62 128 335 </div><div> </div><div> Block Size: 128b+ 256b+ 512b+ </div><div> No. of Reads: 21 177 257 </div><div>No. of Writes: 186 779 885 </div><div> </div><div> Block Size: 1024b+ 2048b+ 4096b+ </div><div> No. of Reads: 30 14 7 </div><div>No. of Writes: 286 101 29410 </div><div> </div><div> Block Size: 8192b+ 16384b+ 32768b+ </div><div> No. of Reads: 0 9 0 </div><div>No. of Writes: 79662 1379 6187 </div><div> </div><div> Block Size: 65536b+ 131072b+ 262144b+ </div><div> No. of Reads: 29 3924 0 </div><div>No. of Writes: 22467 32424 1 </div><div> </div><div> Block Size: 1048576b+ </div><div> No. of Reads: 0 </div><div>No. of Writes: 105 </div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 126295 FORGET</div><div> 0.00 0.00 us 0.00 us 0.00 us 141875 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 117220 RELEASEDIR</div><div> 0.12 119.50 us 102.00 us 147.00 us 4 GETXATTR</div><div> 0.19 68.18 us 42.00 us 109.00 us 11 STAT</div><div> 0.44 92.16 us 19.00 us 141.00 us 19 STATFS</div><div> 1.20 68.03 us 11.00 us 120.00 us 71 FLUSH</div><div> 2.93 83.02 us 18.00 us 136.00 us 142 ENTRYLK</div><div> 3.15 89.18 us 16.00 us 160.00 us 142 FINODELK</div><div> 3.40 192.82 us 76.00 us 271.00 us 71 WRITE</div><div> 4.04 114.43 us 35.00 us 204.00 us 142 SETATTR</div><div> 4.87 138.05 us 49.00 us 222.00 us 142 REMOVEXATTR</div><div> 5.63 159.52 us 56.00 us 262.00 us 142 FXATTROP</div><div> 10.68 73.81 us 11.00 us 202.00 us 582 INODELK</div><div> 19.90 1127.35 us 116.00 us 27717.00 us 71 CREATE</div><div> 21.67 613.82 us 46.00 us 65260.00 us 142 XATTROP</div><div> 21.80 130.68 us 35.00 us 241.00 us 671 LOOKUP</div><div> </div><div> Duration: 458509 seconds</div><div> Data Read: 517180943 bytes</div><div>Data Written: 7895152670 bytes</div><div> </div><div>Interval 0 Stats:</div><div> Block Size: 8b+ 32b+ 64b+ </div><div> No. of Reads: 0 9 24 </div><div>No. of Writes: 62 128 335 </div><div> </div><div> Block Size: 128b+ 256b+ 512b+ </div><div> No. of Reads: 21 177 257 </div><div>No. of Writes: 186 779 885 </div><div> </div><div> Block Size: 1024b+ 2048b+ 4096b+ </div><div> No. of Reads: 30 14 7 </div><div>No. of Writes: 286 101 29410 </div><div> </div><div> Block Size: 8192b+ 16384b+ 32768b+ </div><div> No. of Reads: 0 9 0 </div><div>No. of Writes: 79662 1379 6187 </div><div> </div><div> Block Size: 65536b+ 131072b+ 262144b+ </div><div> No. of Reads: 29 3924 0 </div><div>No. of Writes: 22467 32424 1 </div><div> </div><div> Block Size: 1048576b+ </div><div> No. of Reads: 0 </div><div>No. of Writes: 105 </div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 126295 FORGET</div><div> 0.00 0.00 us 0.00 us 0.00 us 141875 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 117220 RELEASEDIR</div><div> 0.12 119.50 us 102.00 us 147.00 us 4 GETXATTR</div><div> 0.19 68.18 us 42.00 us 109.00 us 11 STAT</div><div> 0.44 92.16 us 19.00 us 141.00 us 19 STATFS</div><div> 1.20 68.03 us 11.00 us 120.00 us 71 FLUSH</div><div> 2.93 83.02 us 18.00 us 136.00 us 142 ENTRYLK</div><div> 3.15 89.18 us 16.00 us 160.00 us 142 FINODELK</div><div> 3.40 192.82 us 76.00 us 271.00 us 71 WRITE</div><div> 4.04 114.43 us 35.00 us 204.00 us 142 SETATTR</div><div> 4.87 138.05 us 49.00 us 222.00 us 142 REMOVEXATTR</div><div> 5.63 159.52 us 56.00 us 262.00 us 142 FXATTROP</div><div> 10.68 73.81 us 11.00 us 202.00 us 582 INODELK</div><div> 19.90 1127.35 us 116.00 us 27717.00 us 71 CREATE</div><div> 21.67 613.82 us 46.00 us 65260.00 us 142 XATTROP</div><div> 21.80 130.68 us 35.00 us 241.00 us 671 LOOKUP</div><div> </div><div> Duration: 458509 seconds</div><div> Data Read: 517180943 bytes</div><div>Data Written: 7895152670 bytes</div><div> </div><div>Brick: node5:/bricks/brickname</div><div>------------------------------------------------</div><div>Cumulative Stats:</div><div> Block Size: 8b+ 16b+ 32b+ </div><div> No. of Reads: 0 14 9 </div><div>No. of Writes: 62 0 128 </div><div> </div><div> Block Size: 64b+ 128b+ 256b+ </div><div> No. of Reads: 23 56 225 </div><div>No. of Writes: 335 186 779 </div><div> </div><div> Block Size: 512b+ 1024b+ 2048b+ </div><div> No. of Reads: 357 106 233 </div><div>No. of Writes: 885 286 102 </div><div> </div><div> Block Size: 4096b+ 8192b+ 16384b+ </div><div> No. of Reads: 128 11 15 </div><div>No. of Writes: 29410 79662 1379 </div><div> </div><div> Block Size: 32768b+ 65536b+ 131072b+ </div><div> No. of Reads: 16 34 28965 </div><div>No. of Writes: 6191 22467 32424 </div><div> </div><div> Block Size: 262144b+ 524288b+ 1048576b+ </div><div> No. of Reads: 5 3 984 </div><div>No. of Writes: 1 0 105 </div><div> </div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 126301 FORGET</div><div> 0.00 0.00 us 0.00 us 0.00 us 141880 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 117718 RELEASEDIR</div><div> 0.01 51.75 us 41.00 us 69.00 us 4 GETXATTR</div><div> 0.02 59.50 us 35.00 us 108.00 us 4 STAT</div><div> 0.05 44.06 us 28.00 us 118.00 us 16 STATFS</div><div> 0.10 24.78 us 15.00 us 99.00 us 59 FLUSH</div><div> 0.34 41.01 us 24.00 us 107.00 us 118 ENTRYLK</div><div> 0.37 44.43 us 25.00 us 156.00 us 118 FINODELK</div><div> 0.76 90.88 us 70.00 us 183.00 us 118 SETATTR</div><div> 0.81 193.88 us 162.00 us 283.00 us 59 WRITE</div><div> 0.98 116.89 us 85.00 us 212.00 us 118 REMOVEXATTR</div><div> 1.10 131.18 us 86.00 us 219.00 us 118 FXATTROP</div><div> 1.52 44.02 us 24.00 us 1004.00 us 487 INODELK</div><div> 3.35 86.29 us 52.00 us 183.00 us 549 LOOKUP</div><div> 4.21 504.03 us 83.00 us 43099.00 us 118 XATTROP</div><div> 86.39 20696.02 us 207.00 us 69802.00 us 59 CREATE</div><div> </div><div> Duration: 644071 seconds</div><div> Data Read: 4837930222 bytes</div><div>Data Written: 7895351133 bytes</div><div> </div><div>Interval 0 Stats:</div><div> Block Size: 8b+ 16b+ 32b+ </div><div> No. of Reads: 0 14 9 </div><div>No. of Writes: 62 0 128 </div><div> </div><div> Block Size: 64b+ 128b+ 256b+ </div><div> No. of Reads: 23 56 225 </div><div>No. of Writes: 335 186 779 </div><div> </div><div> Block Size: 512b+ 1024b+ 2048b+ </div><div> No. of Reads: 357 106 233 </div><div>No. of Writes: 885 286 102 </div><div> </div><div> Block Size: 4096b+ 8192b+ 16384b+ </div><div> No. of Reads: 128 11 15 </div><div>No. of Writes: 29410 79662 1379 </div><div> </div><div> Block Size: 32768b+ 65536b+ 131072b+ </div><div> No. of Reads: 16 34 28965 </div><div>No. of Writes: 6191 22467 32424 </div><div> </div><div> Block Size: 262144b+ 524288b+ 1048576b+ </div><div> No. of Reads: 5 3 984 </div><div>No. of Writes: 1 0 105 </div><div> </div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 126301 FORGET</div><div> 0.00 0.00 us 0.00 us 0.00 us 141880 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 117718 RELEASEDIR</div><div> 0.01 51.75 us 41.00 us 69.00 us 4 GETXATTR</div><div> 0.02 59.50 us 35.00 us 108.00 us 4 STAT</div><div> 0.05 44.06 us 28.00 us 118.00 us 16 STATFS</div><div> 0.10 24.78 us 15.00 us 99.00 us 59 FLUSH</div><div> 0.34 41.01 us 24.00 us 107.00 us 118 ENTRYLK</div><div> 0.37 44.43 us 25.00 us 156.00 us 118 FINODELK</div><div> 0.76 90.88 us 70.00 us 183.00 us 118 SETATTR</div><div> 0.81 193.88 us 162.00 us 283.00 us 59 WRITE</div><div> 0.98 116.89 us 85.00 us 212.00 us 118 REMOVEXATTR</div><div> 1.10 131.18 us 86.00 us 219.00 us 118 FXATTROP</div><div> 1.52 44.02 us 24.00 us 1004.00 us 487 INODELK</div><div> 3.35 86.29 us 52.00 us 183.00 us 549 LOOKUP</div><div> 4.21 504.03 us 83.00 us 43099.00 us 118 XATTROP</div><div> 86.39 20696.02 us 207.00 us 69802.00 us 59 CREATE</div><div> </div><div> Duration: 644071 seconds</div><div> Data Read: 4837930222 bytes</div><div>Data Written: 7895351133 bytes</div><div> </div><div>Brick: node6(source):/bricks/brickname</div><div>--------------------------------------------------</div><div>Cumulative Stats:</div><div> Block Size: 1b+ 2b+ 4b+ </div><div> No. of Reads: 7 18 89 </div><div>No. of Writes: 4 8 37 </div><div> </div><div> Block Size: 8b+ 16b+ 32b+ </div><div> No. of Reads: 727 2325 9459 </div><div>No. of Writes: 108 54 188 </div><div> </div><div> Block Size: 64b+ 128b+ 256b+ </div><div> No. of Reads: 12419 9313 27616 </div><div>No. of Writes: 360 85 772 </div><div> </div><div> Block Size: 512b+ 1024b+ 2048b+ </div><div> No. of Reads: 23708 28691 18594 </div><div>No. of Writes: 847 313 138 </div><div> </div><div> Block Size: 4096b+ 8192b+ 16384b+ </div><div> No. of Reads: 19484 12596 8458 </div><div>No. of Writes: 29185 79632 1431 </div><div> </div><div> Block Size: 32768b+ 65536b+ 131072b+ </div><div> No. of Reads: 5695 5755 1062899 </div><div>No. of Writes: 6168 19435 32017 </div><div> </div><div> Block Size: 262144b+ 1048576b+ </div><div> No. of Reads: 0 0 </div><div>No. of Writes: 1 105 </div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 13806534 FORGET</div><div> 0.00 0.00 us 0.00 us 0.00 us 17813646 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 223324 RELEASEDIR</div><div> 0.00 560.00 us 560.00 us 560.00 us 1 ENTRYLK</div><div> 0.00 3901.00 us 3901.00 us 3901.00 us 1 SETXATTR</div><div> 0.00 4010.00 us 4010.00 us 4010.00 us 1 REMOVEXATTR</div><div> 0.01 62446.08 us 8.00 us 365433.00 us 13 FLUSH</div><div> 0.01 93887.77 us 52.00 us 588566.00 us 13 SETATTR</div><div> 0.03 10772.83 us 28.00 us 1121761.00 us 253 GETXATTR</div><div> 0.04 3190096.00 us 3190096.00 us 3190096.00 us 1 READDIR</div><div> 0.09 558307.69 us 179931.00 us 3188951.00 us 13 READ</div><div> 0.11 616756.00 us 74.00 us 7307745.00 us 14 XATTROP</div><div> 0.12 4754785.50 us 48.00 us 9509523.00 us 2 OPENDIR</div><div> 0.15 1799185.00 us 2310.00 us 5023537.00 us 7 STATFS</div><div> 0.16 68757.98 us 10.00 us 872148.00 us 189 FSTAT</div><div> 0.31 143533.93 us 42.00 us 7002195.00 us 174 OPEN</div><div> 0.40 160262.95 us 661.00 us 2825083.00 us 202 READDIRP</div><div> 1.55 624450.87 us 31.00 us 7397432.00 us 203 FXATTROP</div><div> 22.43 212161.62 us 12.00 us 7397413.00 us 8639 INODELK</div><div> 74.60 541421.09 us 63.00 us 14463033.00 us 11261 LOOKUP</div><div> </div><div> Duration: 644071 seconds</div><div> Data Read: 140706386722 bytes</div><div>Data Written: 7549422894 bytes</div><div> </div><div>Interval 0 Stats:</div><div> Block Size: 1b+ 2b+ 4b+ </div><div> No. of Reads: 7 18 89 </div><div>No. of Writes: 4 8 37 </div><div> </div><div> Block Size: 8b+ 16b+ 32b+ </div><div> No. of Reads: 727 2325 9459 </div><div>No. of Writes: 108 54 188 </div><div> </div><div> Block Size: 64b+ 128b+ 256b+ </div><div> No. of Reads: 12419 9313 27616 </div><div>No. of Writes: 360 85 772 </div><div> </div><div> Block Size: 512b+ 1024b+ 2048b+ </div><div> No. of Reads: 23708 28691 18594 </div><div>No. of Writes: 847 313 138 </div><div> </div><div> Block Size: 4096b+ 8192b+ 16384b+ </div><div> No. of Reads: 19484 12596 8458 </div><div>No. of Writes: 29185 79632 1431 </div><div> </div><div> Block Size: 32768b+ 65536b+ 131072b+ </div><div> No. of Reads: 5695 5755 1062899 </div><div>No. of Writes: 6168 19435 32017 </div><div> </div><div> Block Size: 262144b+ 1048576b+ </div><div> No. of Reads: 0 0 </div><div>No. of Writes: 1 105 </div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 13806534 FORGET</div><div> 0.00 0.00 us 0.00 us 0.00 us 17813657 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 223324 RELEASEDIR</div><div> 0.00 560.00 us 560.00 us 560.00 us 1 ENTRYLK</div><div> 0.00 3901.00 us 3901.00 us 3901.00 us 1 SETXATTR</div><div> 0.00 4010.00 us 4010.00 us 4010.00 us 1 REMOVEXATTR</div><div> 0.01 62446.08 us 8.00 us 365433.00 us 13 FLUSH</div><div> 0.01 93887.77 us 52.00 us 588566.00 us 13 SETATTR</div><div> 0.03 10772.83 us 28.00 us 1121761.00 us 253 GETXATTR</div><div> 0.04 3190096.00 us 3190096.00 us 3190096.00 us 1 READDIR</div><div> 0.09 558307.69 us 179931.00 us 3188951.00 us 13 READ</div><div> 0.11 616756.00 us 74.00 us 7307745.00 us 14 XATTROP</div><div> 0.12 4754785.50 us 48.00 us 9509523.00 us 2 OPENDIR</div><div> 0.15 1799185.00 us 2310.00 us 5023537.00 us 7 STATFS</div><div> 0.16 68757.98 us 10.00 us 872148.00 us 189 FSTAT</div><div> 0.31 143533.93 us 42.00 us 7002195.00 us 174 OPEN</div><div> 0.40 160262.95 us 661.00 us 2825083.00 us 202 READDIRP</div><div> 1.55 624450.87 us 31.00 us 7397432.00 us 203 FXATTROP</div><div> 22.43 212161.62 us 12.00 us 7397413.00 us 8639 INODELK</div><div> 74.60 541421.09 us 63.00 us 14463033.00 us 11261 LOOKUP</div><div> </div><div> Duration: 644071 seconds</div><div> Data Read: 140706386722 bytes</div><div>Data Written: 7549422894 bytes</div><div> </div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Aug 7, 2015 at 12:17 AM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span class="">
<br>
<br>
<div>On 08/07/2015 12:11 PM, Prasun Gera
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">No, no noticeable difference. Still very high,
possibly higher than before. </div>
</blockquote>
<br></span>
I was guessing that the cpu usage could be because of the diff
algorithm which computes checksums (which is a cpu intensive task).
That doesn't seem to be the case. Could you do a volume profile and
see the FOPS that are happening on the bricks and share the result?<br>
1.gluster volume profile <volname> start<br>
2. gluster volume profile <volname> info<br>
3. wait 10-15 seconds<br>
4.gluster volume profile <volname> info<span class=""><br>
<br>
<br>
<br>
<blockquote type="cite">
<div dir="ltr">The system has come down to a crawl. It's difficult
to even ssh or run any commands on the terminal. Do you make
anything of the logs ? The brick log is just a giant alternating
stream of those two lines I mentioned earlier. <br>
</div>
</blockquote>
<br>
<br>
<blockquote type="cite">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Aug 6, 2015 at 10:10 PM,
Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><br>
<br>
On 08/07/2015 01:33 AM, Prasun Gera wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I replaced the brick in a node in my 3x2 dist+repl
volume (RHS 3). I'm seeing that the heal process, which
should essentially be a dump from the working replica to
the newly added one is taking exceptionally long. It has
moved ~100 G over a day on a 1Gigabit network. The CPU
usage on both the nodes of the replica has been pretty
high. <br>
</blockquote>
<br>
</span>
Does setting `cluster.data-self-heal-algorithm` to full make
a difference in the cpu usage?
<div>
<div><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I also think that nagios is making it worse. The heal
is slow enough as it is, and nagios keeps triggering
heal info, which I think never completes. I also see
my logs filling up These are some of the log contents
which I got by running tail on them:<br>
</blockquote>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</span></div>
</blockquote></div><br></div>