<html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><style>body { line-height: 1.5; }blockquote { margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em; }body { font-size: 10.5pt; font-family: 'Microsoft YaHei UI'; color: rgb(0, 0, 0); line-height: 1.5; }</style></head><body>
<div><span></span>Hi Susant,</div><div><br></div><div>You are right, the rebalance process itself is normal now. But the writing brick keeps increasing during rebalancing. Current task has been running for 16 hours, here is the top info.</div><div><br></div><div>===================== top ===========================</div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">top - 08:58:27 up 3 days, 12:08, 1 user, load average: 1.33, 1.18, 1.21<br>Tasks: 173 total, 1 running, 172 sleeping, 0 stopped, 0 zombie<br>Cpu(s): 13.0%us, 16.9%sy, 0.0%ni, 65.7%id, 2.7%wa, 0.0%hi, 1.8%si, 0.0%st<br>Mem: 8060900k total, 7923204k used, 137696k free, 4528380k buffers<br>Swap: 0k total, 0k used, 0k free, 393444k cached<br><br> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br> 8555 root 20 0 950m 143m 1728 S 154.7 1.8 875:01.07 glusterfs<br> 8479 root 20 0 1284m 139m 1892 S 69.8 1.8 443:25.88 glusterfsd<br> 8497 root 20 0 2628m 1.8g 1892 S 68.2 23.0 485:31.42 glusterfsd<br> 874 root 20 0 0 0 0 S 2.3 0.0 65:34.68 jbd2/vdb1-8<br> 58 root 20 0 0 0 0 S 0.7 0.0 44:44.37 kblockd/0<br> 99 root 20 0 0 0 0 S 0.7 0.0 39:17.63 kswapd0<br> 39 root 20 0 0 0 0 S 0.3 0.0 0:16.90 events/4<br></span></div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">=====================================================</span></div><div><span style="font-family: ''; font-size: 10.5pt; line-height: 1.5; background-color: window;">As you can see, the PID 8497 takes 1.8g mem now. </span></div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><br></span></div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">I have taken some state dumps. Later dumps are much bigger than the earlier.</span></div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">================ ls -lh /var/run/gluster/*dump* ================</span></div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">-rw------- 1 root root 4.1M Dec 17 17:52 mnt-b1-brick.8497.dump.1450345948<br>-rw------- 1 root root 292M Dec 18 09:08 mnt-b1-brick.8497.dump.1450400909<br>-rw------- 1 root root 297M Dec 18 09:15 mnt-b1-brick.8497.dump.1450401273<br></span></span></div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">=====================================================</span></span></div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><br></span></span></div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">You can download these state dumps (gziped) from this url:</span></span></div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">http://pan.baidu.com/s/1jHuZCMU</span></span></span></div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><br></span></div>
<div><br></div><hr style="width: 210px; height: 1px;" color="#b5c4df" size="1" align="left">
<div><span><div style="MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt"><div>PuYun</div></div></span></div>
<blockquote style="margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em;"><div> </div><div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"><div style="PADDING-RIGHT: 8px; PADDING-LEFT: 8px; FONT-SIZE: 12px;FONT-FAMILY:tahoma;COLOR:#000000; BACKGROUND: #efefef; PADDING-BOTTOM: 8px; PADDING-TOP: 8px"><div><b>From:</b> <a href="mailto:spalai@redhat.com">Susant Palai</a></div><div><b>Date:</b> 2015-12-17 20:23</div><div><b>To:</b> <a href="mailto:cloudor@126.com">PuYun</a></div><div><b>CC:</b> <a href="mailto:gluster-users@gluster.org">gluster-users</a></div><div><b>Subject:</b> Re: [Gluster-users] How to diagnose volume rebalance failure?</div></div></div><div><div>Ok from your reply rebalance seems to be fine. </div>
<div>So what you can do is check whether the mem-usage of brick process keeps increasing constantly. If that is the case take multiple state-dumps intermittently.</div>
<div> </div>
<div>Regards,</div>
<div>Susant </div>
<div> </div>
<div>----- Original Message -----</div>
<div>From: "PuYun" <cloudor@126.com></div>
<div>To: "gluster-users" <gluster-users@gluster.org></div>
<div>Cc: "gluster-users" <gluster-users@gluster.org></div>
<div>Sent: Thursday, 17 December, 2015 3:57:12 PM</div>
<div>Subject: Re: [Gluster-users] How to diagnose volume rebalance failure?</div>
<div> </div>
<div> </div>
<div> </div>
<div>Hi Susant, </div>
<div> </div>
<div> </div>
<div>Thank you for your instructions. I'll do that. </div>
<div> </div>
<div> </div>
<div>My volume contains more than 2 million end sub directories. Most of the end sub directories contains 10~30 small files. Current total size is about 900G. Two bricks, each one is 1T. Current ram size is 8G. </div>
<div> </div>
<div> </div>
<div>Previously I saw 3 processes, one is glusterfs for rebalance and 2 glusterfsd for bricks. Only 1 glusterfsd occupied very large mem and it is related to the newly added brick. The other 2 processes seems normal. If that happens again, I will send you the state dump. </div>
<div> </div>
<div> </div>
<div>Thank you. </div>
<div> </div>
<div>PuYun </div>
<div> </div>
<div> </div>
<div> </div>
<div> </div>
<div> </div>
</div></blockquote>
</body></html>