<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>Ho jiri</div><div><br></div><div>Looking in the logs I also think that a self heal is triggered as I see numerous entries with dht_heal regarding volume relay out activities.</div><div><br></div><div>So yes that might be the cause, but my personal impression was that this would only be started by a rebalance, so I expect that it's caused because the application "owncloud" is constantly scanning the file system once users log in; currently something like 2500 a day ....</div><div><br></div><div>Thanks for the input<br><br><span style="background-color: rgba(255, 255, 255, 0);">Met vriendelijke groet / kind regards,<br><br>Sander Zijlstra<br><br>| Linux Engineer | SURFsara | <a href="x-apple-data-detectors://1" x-apple-data-detectors="true" x-apple-data-detectors-type="address" x-apple-data-detectors-result="1">Science Park 140</a> | 1098XG Amsterdam | </span><div><span style="background-color: rgba(255, 255, 255, 0);">T <a href="tel:+31%20(0)6%2043%2099%2012%2047" x-apple-data-detectors="true" x-apple-data-detectors-type="telephone" x-apple-data-detectors-result="2">+31(0)6 43 99 12 47</a> | <a href="mailto:sander.zijlstra@surfsara.nl" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="3">sander.zijlstra@surfsara.nl</a> | <a href="http://www.surfsara.nl/" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="4">www.surfsara.nl</a> |</span><br style="font-family: UICTFontTextStyleBody; -webkit-text-size-adjust: auto;"></div></div><div><br>Op 09 Apr 2015 om 15:15 heeft Jiri Hoogeveen <<a href="mailto:j.hoogeveen@bluebillywig.com">j.hoogeveen@bluebillywig.com</a>> het volgende geschreven:<br><br></div><blockquote type="cite"><div><meta http-equiv="Content-Type" content="text/html charset=utf-8">Hi Sander,<div class=""><br class=""></div><div class="">It sounds to me, that it triggered the self-healing, which will do a scan on the bricks. Depending on the number of files on the brick, it can use a lot of CPU.</div><div class=""><br class=""></div><div class="">Does the logs say anything useful?</div><div class=""> </div><div class="">Grtz, </div><div class="">Jiri Hoogeveen</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 09 Apr 2015, at 14:18, Sander Zijlstra <<a href="mailto:sander.zijlstra@surfsara.nl" class="">sander.zijlstra@surfsara.nl</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">LS,<div class=""><br class=""></div><div class="">We have a GlusterFS cluster which consists of 4 nodes with one brick each and a distributed-replicated volume of 72 TB.</div><div class=""><br class=""></div><div class="">Today I extended the cluster to 8 machines and added new bricks to the volume, so it now contains 8 bricks.</div><div class=""><br class=""></div><div class="">I didn’t start the rebalance yet to limit the impact during the day but to my surprise all glusterfsd process went sky high and performance was really really bad. So effectively I cause downtime to our storage service while I didn’t anticipated this, hence I didn’t do any rebalance yet.</div><div class=""><br class=""></div><div class="">Can somebody explain to me why adding bricks to a volume causes this high CPU usage? I can imagine the meta data needed to be synced but if this is so heavy, why can’t I tune this?</div><div class=""><br class=""></div><div class="">This is my current volume setup:</div><div class=""><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">Volume Name: gv0</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">Type: Distributed-Replicate</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">Volume ID: 0322f20f-e507-492b-91db-cb4c953a24eb</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">Status: Started</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">Number of Bricks: 4 x 2 = 8</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">Transport-type: tcp</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">Bricks:</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">Brick1: s-s35-06:/glusterfs/bricks/brick1/brick</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">Brick2: s-s35-07:/glusterfs/bricks/brick1/brick</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">Brick3: s-s35-08:/glusterfs/bricks/brick1/brick</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">Brick4: s-s35-09:/glusterfs/bricks/brick1/brick</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">Brick5: v39-app-01:/glusterfs/bricks/brick1/gv0</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">Brick6: v39-app-02:/glusterfs/bricks/brick1/gv0</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">Brick7: v39-app-03:/glusterfs/bricks/brick1/gv0</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">Brick8: v39-app-04:/glusterfs/bricks/brick1/gv0</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">Options Reconfigured:</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">performance.cache-size: 256MB</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">nfs.disable: on</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">geo-replication.indexing: off</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">geo-replication.ignore-pid-check: on</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">changelog.changelog: on</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">performance.io-thread-count: 32</div><div style="margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);" class="">performance.write-behind-window-size: 5MB</div></div><div class=""><div class=""><div class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class="Apple-interchange-newline">Met vriendelijke groet / kind regards,</div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><b class="">Sander Zijlstra</b></div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">| Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | <a href="mailto:sander.zijlstra@surfsara.nl" class="">sander.zijlstra@surfsara.nl</a> | <a href="http://www.surfsara.nl/" class="">www.surfsara.nl</a> |<br class=""><br class=""><i class="">Regular day off on friday</i></div>
</div>
<br class=""></div></div></div>_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class=""><a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></div></blockquote></div><br class=""></div></div></blockquote></body></html>