<html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><style>body { line-height: 1.5; }blockquote { margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em; }body { font-size: 10.5pt; font-family: 'Microsoft YaHei UI'; color: rgb(0, 0, 0); line-height: 1.5; }</style></head><body>
<div><span></span>Hi Susant,</div><div><br></div><div><span style="font-size: 10.5pt; line-height: 1.5; background-color: window;">Thank you for your instructions. </span>I'll do that. </div><div><br></div><div>My volume contains more than 2 million end sub directories. Most of the end sub directories contains 10~30 small files. Current total size is about 900G. Two bricks, each one is 1T. Current ram size is 8G.</div><div><br></div><div>Previously I saw 3 processes, one is glusterfs for rebalance and 2 glusterfsd for bricks. Only 1 glusterfsd occupied very large mem and it is related to the newly added brick. The other 2 processes seems normal. If that happens again, I will send you the state dump.</div><div><br></div><div>Thank you.</div>
<div><br></div><hr style="width: 210px; height: 1px;" color="#b5c4df" size="1" align="left">
<div><span><div style="MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt">PuYun</div></span></div>
<blockquote style="margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em;"><div> </div><div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"><div style="PADDING-RIGHT: 8px; PADDING-LEFT: 8px; FONT-SIZE: 12px;FONT-FAMILY:tahoma;COLOR:#000000; BACKGROUND: #efefef; PADDING-BOTTOM: 8px; PADDING-TOP: 8px"><div><b>From:</b> <a href="mailto:spalai@redhat.com">Susant Palai</a></div><div><b>Date:</b> 2015-12-17 14:50</div><div><b>To:</b> <a href="mailto:cloudor@126.com">PuYun</a></div><div><b>CC:</b> <a href="mailto:gluster-users@gluster.org">gluster-users</a></div><div><b>Subject:</b> Re: [Gluster-users] How to diagnose volume rebalance failure?</div></div></div><div><div>Hi PuYun,</div>
<div> Would you be able to run rebalance again and take state-dumps in intervals when you see high mem-usages. Here is the details.</div>
<div>##How to generate statedump</div>
<div>We can find the directory where statedump files are created using 'gluster --print-statedumpdir' command.</div>
<div>Create that directory if not already present based on the type of installation.</div>
<div>Lets call this directory `statedump-directory`.</div>
<div> </div>
<div>We can generate statedump using 'kill -USR1 <pid-of-gluster-process>'.</div>
<div>gluster-process is nothing but glusterd/glusterfs/glusterfsd process.</div>
<div> </div>
<div>I would like to know some more information.</div>
<div> </div>
<div>1) How big is your file system? [no. of files/dirs]</div>
<div>2) What is the vm RAM size?</div>
<div> </div>
<div> </div>
<div>Regards,</div>
<div>Susant </div>
<div> </div>
<div>----- Original Message -----</div>
<div>From: "PuYun" <cloudor@126.com></div>
<div>To: "gluster-users" <gluster-users@gluster.org></div>
<div>Sent: Wednesday, 16 December, 2015 8:30:57 PM</div>
<div>Subject: Re: [Gluster-users] How to diagnose volume rebalance failure?</div>
<div> </div>
<div> </div>
<div> </div>
<div>Hi, </div>
<div> </div>
<div> </div>
<div>I have upgraded all my server/client gluster packages to version 3.7.6 and started reblance task again. It had been running much longer than before, but it got oom and failed again. </div>
<div> </div>
<div> </div>
<div>===================== /var/log/messages ================== </div>
<div>Dec 16 20:06:41 d001 kernel: glusterfsd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 </div>
<div>Dec 16 20:06:41 d001 kernel: glusterfsd cpuset=/ mems_allowed=0 </div>
<div>Dec 16 20:06:41 d001 kernel: Pid: 4843, comm: glusterfsd Not tainted 2.6.32-431.23.3.el6.x86_64 #1 </div>
<div>Dec 16 20:06:41 d001 kernel: Call Trace: </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff810d0431>] ? cpuset_print_task_mems_allowed+0x91/0xb0 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff81122810>] ? dump_header+0x90/0x1b0 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8122833c>] ? security_real_capable_noaudit+0x3c/0x70 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff81122c92>] ? oom_kill_process+0x82/0x2a0 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff81122bd1>] ? select_bad_process+0xe1/0x120 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff811230d0>] ? out_of_memory+0x220/0x3c0 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8112f9ef>] ? __alloc_pages_nodemask+0x89f/0x8d0 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff811678ea>] ? alloc_pages_current+0xaa/0x110 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8111fc07>] ? __page_cache_alloc+0x87/0x90 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8111f5ee>] ? find_get_page+0x1e/0xa0 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff81120ba7>] ? filemap_fault+0x1a7/0x500 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff81149ed4>] ? __do_fault+0x54/0x530 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8114a4a7>] ? handle_pte_fault+0xf7/0xb00 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff810aee5e>] ? futex_wake+0x10e/0x120 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8114b0da>] ? handle_mm_fault+0x22a/0x300 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8104a8d8>] ? __do_page_fault+0x138/0x480 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8103f9d8>] ? pvclock_clocksource_read+0x58/0xd0 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8152e74e>] ? do_page_fault+0x3e/0xa0 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8152bb05>] ? page_fault+0x25/0x30 </div>
<div>Dec 16 20:06:41 d001 kernel: Mem-Info: </div>
<div> </div>
<div>Dec 16 20:06:41 d001 kernel: Node 0 DMA per-cpu: </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 0: hi: 0, btch: 1 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 1: hi: 0, btch: 1 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 2: hi: 0, btch: 1 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 3: hi: 0, btch: 1 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 4: hi: 0, btch: 1 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 5: hi: 0, btch: 1 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 6: hi: 0, btch: 1 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 7: hi: 0, btch: 1 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: Node 0 DMA32 per-cpu: </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 0: hi: 186, btch: 31 usd: 14 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 1: hi: 186, btch: 31 usd: 152 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 2: hi: 186, btch: 31 usd: 108 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 3: hi: 186, btch: 31 usd: 70 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 4: hi: 186, btch: 31 usd: 152 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 5: hi: 186, btch: 31 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 6: hi: 186, btch: 31 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 7: hi: 186, btch: 31 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: Node 0 Normal per-cpu: </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 0: hi: 186, btch: 31 usd: 145 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 1: hi: 186, btch: 31 usd: 19 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 2: hi: 186, btch: 31 usd: 33 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 3: hi: 186, btch: 31 usd: 20 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 4: hi: 186, btch: 31 usd: 165 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 5: hi: 186, btch: 31 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 6: hi: 186, btch: 31 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 7: hi: 186, btch: 31 usd: 0 </div>
<div> </div>
<div>Dec 16 20:06:41 d001 kernel: active_anon:1955964 inactive_anon:38 isolated_anon:0 </div>
<div>Dec 16 20:06:41 d001 kernel: active_file:312 inactive_file:1262 isolated_file:0 </div>
<div>Dec 16 20:06:41 d001 kernel: unevictable:0 dirty:1 writeback:3 unstable:0 </div>
<div>Dec 16 20:06:41 d001 kernel: free:25745 slab_reclaimable:2412 slab_unreclaimable:7815 </div>
<div>Dec 16 20:06:41 d001 kernel: mapped:208 shmem:43 pagetables:4679 bounce:0 </div>
<div>Dec 16 20:06:41 d001 kernel: Node 0 DMA free:15752kB min:124kB low:152kB high:184kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15364kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes </div>
<div>Dec 16 20:06:41 d001 kernel: lowmem_reserve[]: 0 3000 8050 8050 </div>
<div>Dec 16 20:06:41 d001 kernel: Node 0 DMA32 free:45044kB min:25140kB low:31424kB high:37708kB active_anon:2740816kB inactive_anon:0kB active_file:896kB inactive_file:4176kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:0kB writeback:4kB mapped:816kB shmem:0kB slab_reclaimable:1636kB slab_unreclaimable:1888kB kernel_stack:128kB pagetables:5204kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1664 all_unreclaimable? yes </div>
<div>Dec 16 20:06:41 d001 kernel: lowmem_reserve[]: 0 0 5050 5050 </div>
<div>Dec 16 20:06:41 d001 kernel: Node 0 Normal free:42184kB min:42316kB low:52892kB high:63472kB active_anon:5083040kB inactive_anon:152kB active_file:352kB inactive_file:872kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:5171200kB mlocked:0kB dirty:4kB writeback:8kB mapped:16kB shmem:172kB slab_reclaimable:8012kB slab_unreclaimable:29372kB kernel_stack:2240kB pagetables:13512kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1131 all_unreclaimable? yes </div>
<div>Dec 16 20:06:41 d001 kernel: lowmem_reserve[]: 0 0 0 0 </div>
<div>Dec 16 20:06:41 d001 kernel: Node 0 DMA: 2*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15752kB </div>
<div>Dec 16 20:06:41 d001 kernel: Node 0 DMA32: 11044*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44176kB </div>
<div>Dec 16 20:06:41 d001 kernel: Node 0 Normal: 10515*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 42060kB </div>
<div>Dec 16 20:06:41 d001 kernel: 1782 total pagecache pages </div>
<div> </div>
<div>Dec 16 20:06:41 d001 kernel: 0 pages in swap cache </div>
<div>Dec 16 20:06:41 d001 kernel: Swap cache stats: add 0, delete 0, find 0/0 </div>
<div>Dec 16 20:06:41 d001 kernel: Free swap = 0kB </div>
<div>Dec 16 20:06:41 d001 kernel: Total swap = 0kB </div>
<div>Dec 16 20:06:41 d001 kernel: 2097151 pages RAM </div>
<div>Dec 16 20:06:41 d001 kernel: 81926 pages reserved </div>
<div>Dec 16 20:06:41 d001 kernel: 924 pages shared </div>
<div>Dec 16 20:06:41 d001 kernel: 1984896 pages non-shared </div>
<div>Dec 16 20:06:41 d001 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name </div>
<div>Dec 16 20:06:41 d001 kernel: [ 477] 0 477 2662 105 0 -17 -1000 udevd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 916] 0 916 374 48 0 0 0 aliyun-service </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1156] 0 1156 62798 192 0 0 0 rsyslogd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1178] 32 1178 4744 62 0 0 0 rpcbind </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1198] 29 1198 5837 112 1 0 0 rpc.statd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1382] 28 1382 157544 113 1 0 0 nscd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1414] 0 1414 118751 699 0 0 0 AliYunDunUpdate </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1448] 0 1448 16657 178 0 -17 -1000 sshd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1463] 38 1463 6683 152 0 0 0 ntpd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1473] 0 1473 29325 154 0 0 0 crond </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1516] 0 1516 1016 19 1 0 0 mingetty </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1518] 0 1518 1016 17 3 0 0 mingetty </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1520] 0 1520 1016 18 5 0 0 mingetty </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1522] 0 1522 2661 104 1 -17 -1000 udevd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1523] 0 1523 2661 104 4 -17 -1000 udevd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1524] 0 1524 1016 18 2 0 0 mingetty </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1526] 0 1526 1016 19 4 0 0 mingetty </div>
<div> </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1528] 0 1528 1016 19 1 0 0 mingetty </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1652] 0 1652 191799 1191 0 0 0 AliYunDun </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1670] 0 1670 249011 1149 0 0 0 AliHids </div>
<div>Dec 16 20:06:41 d001 kernel: [ 4546] 0 4546 185509 4817 1 0 0 glusterd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 4697] 0 4697 429110 35780 1 0 0 glusterfsd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 4715] 0 4715 2149944 1788310 0 0 0 glusterfsd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 4830] 0 4830 137846 6463 0 0 0 glusterfs </div>
<div>Dec 16 20:06:41 d001 kernel: [ 4940] 0 4940 341517 116710 1 0 0 glusterfs </div>
<div>Dec 16 20:06:41 d001 kernel: Out of memory: Kill process 4715 (glusterfsd) score 859 or sacrifice child </div>
<div>Dec 16 20:06:41 d001 kernel: Killed process 4715, UID 0, (glusterfsd) total-vm:8599776kB, anon-rss:7152896kB, file-rss:344kB </div>
<div>Dec 16 20:06:41 d001 kernel: glusterfsd invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0 </div>
<div>Dec 16 20:06:41 d001 kernel: glusterfsd cpuset=/ mems_allowed=0 </div>
<div>Dec 16 20:06:41 d001 kernel: Pid: 4717, comm: glusterfsd Not tainted 2.6.32-431.23.3.el6.x86_64 #1 </div>
<div>Dec 16 20:06:41 d001 kernel: Call Trace: </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff810d0431>] ? cpuset_print_task_mems_allowed+0x91/0xb0 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff81122810>] ? dump_header+0x90/0x1b0 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8122833c>] ? security_real_capable_noaudit+0x3c/0x70 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff81122c92>] ? oom_kill_process+0x82/0x2a0 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff81122bd1>] ? select_bad_process+0xe1/0x120 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff811230d0>] ? out_of_memory+0x220/0x3c0 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8112f9ef>] ? __alloc_pages_nodemask+0x89f/0x8d0 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8116e2d2>] ? kmem_getpages+0x62/0x170 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8116eeea>] ? fallback_alloc+0x1ba/0x270 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8116e93f>] ? cache_grow+0x2cf/0x320 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8116ec69>] ? ____cache_alloc_node+0x99/0x160 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8116fbeb>] ? kmem_cache_alloc+0x11b/0x190 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff810efb75>] ? taskstats_exit+0x305/0x390 </div>
<div> </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff81076c27>] ? do_exit+0x157/0x870 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff81060aa3>] ? perf_event_task_sched_out+0x33/0x70 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff81077398>] ? do_group_exit+0x58/0xd0 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8108cd46>] ? get_signal_to_deliver+0x1f6/0x460 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8100a265>] ? do_signal+0x75/0x800 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8108c85a>] ? dequeue_signal+0xda/0x170 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8108cb40>] ? sys_rt_sigtimedwait+0x250/0x260 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff81077087>] ? do_exit+0x5b7/0x870 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8100aa80>] ? do_notify_resume+0x90/0xc0 </div>
<div>Dec 16 20:06:41 d001 kernel: [<ffffffff8100b341>] ? int_signal+0x12/0x17 </div>
<div>Dec 16 20:06:41 d001 kernel: Mem-Info: </div>
<div>Dec 16 20:06:41 d001 kernel: Node 0 DMA per-cpu: </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 0: hi: 0, btch: 1 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 1: hi: 0, btch: 1 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 2: hi: 0, btch: 1 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 3: hi: 0, btch: 1 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 4: hi: 0, btch: 1 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 5: hi: 0, btch: 1 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 6: hi: 0, btch: 1 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 7: hi: 0, btch: 1 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: Node 0 DMA32 per-cpu: </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 0: hi: 186, btch: 31 usd: 14 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 1: hi: 186, btch: 31 usd: 152 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 2: hi: 186, btch: 31 usd: 108 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 3: hi: 186, btch: 31 usd: 70 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 4: hi: 186, btch: 31 usd: 152 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 5: hi: 186, btch: 31 usd: 0 </div>
<div> </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 6: hi: 186, btch: 31 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 7: hi: 186, btch: 31 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: Node 0 Normal per-cpu: </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 0: hi: 186, btch: 31 usd: 145 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 1: hi: 186, btch: 31 usd: 19 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 2: hi: 186, btch: 31 usd: 33 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 3: hi: 186, btch: 31 usd: 50 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 4: hi: 186, btch: 31 usd: 165 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 5: hi: 186, btch: 31 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 6: hi: 186, btch: 31 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: CPU 7: hi: 186, btch: 31 usd: 0 </div>
<div>Dec 16 20:06:41 d001 kernel: active_anon:1955964 inactive_anon:38 isolated_anon:0 </div>
<div>Dec 16 20:06:41 d001 kernel: active_file:312 inactive_file:1262 isolated_file:0 </div>
<div>Dec 16 20:06:41 d001 kernel: unevictable:0 dirty:1 writeback:3 unstable:0 </div>
<div>Dec 16 20:06:41 d001 kernel: free:25745 slab_reclaimable:2412 slab_unreclaimable:7815 </div>
<div>Dec 16 20:06:41 d001 kernel: mapped:208 shmem:43 pagetables:4679 bounce:0 </div>
<div>Dec 16 20:06:41 d001 kernel: Node 0 DMA free:15752kB min:124kB low:152kB high:184kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15364kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes </div>
<div>Dec 16 20:06:41 d001 kernel: lowmem_reserve[]: 0 3000 8050 8050 </div>
<div>Dec 16 20:06:41 d001 kernel: Node 0 DMA32 free:45044kB min:25140kB low:31424kB high:37708kB active_anon:2740816kB inactive_anon:0kB active_file:896kB inactive_file:4176kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:0kB writeback:4kB mapped:816kB shmem:0kB slab_reclaimable:1636kB slab_unreclaimable:1888kB kernel_stack:128kB pagetables:5204kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1664 all_unreclaimable? yes </div>
<div>Dec 16 20:06:41 d001 kernel: lowmem_reserve[]: 0 0 5050 5050 </div>
<div> </div>
<div>Dec 16 20:06:41 d001 kernel: Node 0 Normal free:42184kB min:42316kB low:52892kB high:63472kB active_anon:5083040kB inactive_anon:152kB active_file:352kB inactive_file:872kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:5171200kB mlocked:0kB dirty:4kB writeback:8kB mapped:16kB shmem:172kB slab_reclaimable:8012kB slab_unreclaimable:29372kB kernel_stack:2240kB pagetables:13512kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1131 all_unreclaimable? yes </div>
<div>Dec 16 20:06:41 d001 kernel: lowmem_reserve[]: 0 0 0 0 </div>
<div>Dec 16 20:06:41 d001 kernel: Node 0 DMA: 2*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15752kB </div>
<div>Dec 16 20:06:41 d001 kernel: Node 0 DMA32: 11044*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44176kB </div>
<div>Dec 16 20:06:41 d001 kernel: Node 0 Normal: 10484*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41936kB </div>
<div>Dec 16 20:06:41 d001 kernel: 1782 total pagecache pages </div>
<div>Dec 16 20:06:41 d001 kernel: 0 pages in swap cache </div>
<div>Dec 16 20:06:41 d001 kernel: Swap cache stats: add 0, delete 0, find 0/0 </div>
<div>Dec 16 20:06:41 d001 kernel: Free swap = 0kB </div>
<div>Dec 16 20:06:41 d001 kernel: Total swap = 0kB </div>
<div>Dec 16 20:06:41 d001 kernel: 2097151 pages RAM </div>
<div>Dec 16 20:06:41 d001 kernel: 81926 pages reserved </div>
<div>Dec 16 20:06:41 d001 kernel: 931 pages shared </div>
<div>Dec 16 20:06:41 d001 kernel: 1984884 pages non-shared </div>
<div>Dec 16 20:06:41 d001 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name </div>
<div>Dec 16 20:06:41 d001 kernel: [ 477] 0 477 2662 105 0 -17 -1000 udevd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 916] 0 916 374 48 0 0 0 aliyun-service </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1156] 0 1156 62798 192 0 0 0 rsyslogd </div>
<div> </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1178] 32 1178 4744 62 0 0 0 rpcbind </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1198] 29 1198 5837 112 1 0 0 rpc.statd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1382] 28 1382 157544 113 1 0 0 nscd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1414] 0 1414 118751 699 0 0 0 AliYunDunUpdate </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1448] 0 1448 16657 178 0 -17 -1000 sshd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1463] 38 1463 6683 152 0 0 0 ntpd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1473] 0 1473 29325 154 0 0 0 crond </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1516] 0 1516 1016 19 1 0 0 mingetty </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1518] 0 1518 1016 17 3 0 0 mingetty </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1520] 0 1520 1016 18 5 0 0 mingetty </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1522] 0 1522 2661 104 1 -17 -1000 udevd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1523] 0 1523 2661 104 4 -17 -1000 udevd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1524] 0 1524 1016 18 2 0 0 mingetty </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1526] 0 1526 1016 19 4 0 0 mingetty </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1528] 0 1528 1016 19 1 0 0 mingetty </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1652] 0 1652 191799 1203 0 0 0 AliYunDun </div>
<div>Dec 16 20:06:41 d001 kernel: [ 1670] 0 1670 249011 1160 0 0 0 AliHids </div>
<div>Dec 16 20:06:41 d001 kernel: [ 4546] 0 4546 185509 4817 1 0 0 glusterd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 4697] 0 4697 429110 35780 1 0 0 glusterfsd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 4717] 0 4715 2149944 1788310 4 0 0 glusterfsd </div>
<div>Dec 16 20:06:41 d001 kernel: [ 4830] 0 4830 137846 6463 0 0 0 glusterfs </div>
<div>Dec 16 20:06:41 d001 kernel: [ 4940] 0 4940 341517 116710 1 0 0 glusterfs </div>
<div>===================== <EOF> ================================== </div>
<div> </div>
<div> </div>
<div> </div>
<div>PuYun </div>
<div> </div>
<div> </div>
<div> </div>
<div> </div>
<div> </div>
<div>From: PuYun </div>
<div>Date: 2015-12-15 22:10 </div>
<div>To: gluster-users </div>
<div>Subject: Re: [Gluster-users] How to diagnose volume rebalance failure? </div>
<div> </div>
<div> </div>
<div>Hi, </div>
<div> </div>
<div> </div>
<div>I find this bug link https://bugzilla.redhat.com/show_bug.cgi?id=1261234 . My version is 3.7.4 which is older than the fixed version 3.7.5. </div>
<div>I'll upgrade my gluster version and try again later. </div>
<div> </div>
<div> </div>
<div>Thank you. </div>
<div> </div>
<div> </div>
<div> </div>
<div>PuYun </div>
<div> </div>
<div> </div>
<div> </div>
<div> </div>
<div> </div>
<div> </div>
<div> </div>
<div> </div>
<div> </div>
<div> </div>
<div> </div>
<div> </div>
<div>_______________________________________________</div>
<div>Gluster-users mailing list</div>
<div>Gluster-users@gluster.org</div>
<div>http://www.gluster.org/mailman/listinfo/gluster-users</div>
</div></blockquote>
</body></html>