<html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><style>body { line-height: 1.5; }blockquote { margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em; }div.foxdiv20151216224726118204 { }body { font-size: 10.5pt; font-family: 'Microsoft YaHei UI'; color: rgb(0, 0, 0); line-height: 1.5; }</style></head><body>
<div><span></span>Hi,</div><div><br></div><div>I have upgraded all my server/client gluster packages to version 3.7.6 and started reblance task again. It had been running much longer than before, but it got oom and failed again.</div><div><br></div><div>===================== /var/log/messages ==================</div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">Dec 16 20:06:41 d001 kernel: glusterfsd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0<br>Dec 16 20:06:41 d001 kernel: glusterfsd cpuset=/ mems_allowed=0<br>Dec 16 20:06:41 d001 kernel: Pid: 4843, comm: glusterfsd Not tainted 2.6.32-431.23.3.el6.x86_64 #1<br>Dec 16 20:06:41 d001 kernel: Call Trace:<br>Dec 16 20:06:41 d001 kernel: [<ffffffff810d0431>] ? cpuset_print_task_mems_allowed+0x91/0xb0<br>Dec 16 20:06:41 d001 kernel: [<ffffffff81122810>] ? dump_header+0x90/0x1b0<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8122833c>] ? security_real_capable_noaudit+0x3c/0x70<br>Dec 16 20:06:41 d001 kernel: [<ffffffff81122c92>] ? oom_kill_process+0x82/0x2a0<br>Dec 16 20:06:41 d001 kernel: [<ffffffff81122bd1>] ? select_bad_process+0xe1/0x120<br>Dec 16 20:06:41 d001 kernel: [<ffffffff811230d0>] ? out_of_memory+0x220/0x3c0<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8112f9ef>] ? __alloc_pages_nodemask+0x89f/0x8d0<br>Dec 16 20:06:41 d001 kernel: [<ffffffff811678ea>] ? alloc_pages_current+0xaa/0x110<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8111fc07>] ? __page_cache_alloc+0x87/0x90<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8111f5ee>] ? find_get_page+0x1e/0xa0<br>Dec 16 20:06:41 d001 kernel: [<ffffffff81120ba7>] ? filemap_fault+0x1a7/0x500<br>Dec 16 20:06:41 d001 kernel: [<ffffffff81149ed4>] ? __do_fault+0x54/0x530<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8114a4a7>] ? handle_pte_fault+0xf7/0xb00<br>Dec 16 20:06:41 d001 kernel: [<ffffffff810aee5e>] ? futex_wake+0x10e/0x120<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8114b0da>] ? handle_mm_fault+0x22a/0x300<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8104a8d8>] ? __do_page_fault+0x138/0x480<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8103f9d8>] ? pvclock_clocksource_read+0x58/0xd0<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8152e74e>] ? do_page_fault+0x3e/0xa0<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8152bb05>] ? page_fault+0x25/0x30<br>Dec 16 20:06:41 d001 kernel: Mem-Info:<br></span></div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">Dec 16 20:06:41 d001 kernel: Node 0 DMA per-cpu:<br>Dec 16 20:06:41 d001 kernel: CPU 0: hi: 0, btch: 1 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 1: hi: 0, btch: 1 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 2: hi: 0, btch: 1 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 3: hi: 0, btch: 1 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 4: hi: 0, btch: 1 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 5: hi: 0, btch: 1 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 6: hi: 0, btch: 1 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 7: hi: 0, btch: 1 usd: 0<br>Dec 16 20:06:41 d001 kernel: Node 0 DMA32 per-cpu:<br>Dec 16 20:06:41 d001 kernel: CPU 0: hi: 186, btch: 31 usd: 14<br>Dec 16 20:06:41 d001 kernel: CPU 1: hi: 186, btch: 31 usd: 152<br>Dec 16 20:06:41 d001 kernel: CPU 2: hi: 186, btch: 31 usd: 108<br>Dec 16 20:06:41 d001 kernel: CPU 3: hi: 186, btch: 31 usd: 70<br>Dec 16 20:06:41 d001 kernel: CPU 4: hi: 186, btch: 31 usd: 152<br>Dec 16 20:06:41 d001 kernel: CPU 5: hi: 186, btch: 31 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 6: hi: 186, btch: 31 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 7: hi: 186, btch: 31 usd: 0<br>Dec 16 20:06:41 d001 kernel: Node 0 Normal per-cpu:<br>Dec 16 20:06:41 d001 kernel: CPU 0: hi: 186, btch: 31 usd: 145<br>Dec 16 20:06:41 d001 kernel: CPU 1: hi: 186, btch: 31 usd: 19<br>Dec 16 20:06:41 d001 kernel: CPU 2: hi: 186, btch: 31 usd: 33<br>Dec 16 20:06:41 d001 kernel: CPU 3: hi: 186, btch: 31 usd: 20<br>Dec 16 20:06:41 d001 kernel: CPU 4: hi: 186, btch: 31 usd: 165<br>Dec 16 20:06:41 d001 kernel: CPU 5: hi: 186, btch: 31 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 6: hi: 186, btch: 31 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 7: hi: 186, btch: 31 usd: 0<br></span></div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">Dec 16 20:06:41 d001 kernel: active_anon:1955964 inactive_anon:38 isolated_anon:0<br>Dec 16 20:06:41 d001 kernel: active_file:312 inactive_file:1262 isolated_file:0<br>Dec 16 20:06:41 d001 kernel: unevictable:0 dirty:1 writeback:3 unstable:0<br>Dec 16 20:06:41 d001 kernel: free:25745 slab_reclaimable:2412 slab_unreclaimable:7815<br>Dec 16 20:06:41 d001 kernel: mapped:208 shmem:43 pagetables:4679 bounce:0<br>Dec 16 20:06:41 d001 kernel: Node 0 DMA free:15752kB min:124kB low:152kB high:184kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15364kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes<br>Dec 16 20:06:41 d001 kernel: lowmem_reserve[]: 0 3000 8050 8050<br>Dec 16 20:06:41 d001 kernel: Node 0 DMA32 free:45044kB min:25140kB low:31424kB high:37708kB active_anon:2740816kB inactive_anon:0kB active_file:896kB inactive_file:4176kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:0kB writeback:4kB mapped:816kB shmem:0kB slab_reclaimable:1636kB slab_unreclaimable:1888kB kernel_stack:128kB pagetables:5204kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1664 all_unreclaimable? yes<br>Dec 16 20:06:41 d001 kernel: lowmem_reserve[]: 0 0 5050 5050<br>Dec 16 20:06:41 d001 kernel: Node 0 Normal free:42184kB min:42316kB low:52892kB high:63472kB active_anon:5083040kB inactive_anon:152kB active_file:352kB inactive_file:872kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:5171200kB mlocked:0kB dirty:4kB writeback:8kB mapped:16kB shmem:172kB slab_reclaimable:8012kB slab_unreclaimable:29372kB kernel_stack:2240kB pagetables:13512kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1131 all_unreclaimable? yes<br>Dec 16 20:06:41 d001 kernel: lowmem_reserve[]: 0 0 0 0<br>Dec 16 20:06:41 d001 kernel: Node 0 DMA: 2*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15752kB<br>Dec 16 20:06:41 d001 kernel: Node 0 DMA32: 11044*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44176kB<br>Dec 16 20:06:41 d001 kernel: Node 0 Normal: 10515*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 42060kB<br>Dec 16 20:06:41 d001 kernel: 1782 total pagecache pages<br></span></div>
<div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">Dec 16 20:06:41 d001 kernel: 0 pages in swap cache<br>Dec 16 20:06:41 d001 kernel: Swap cache stats: add 0, delete 0, find 0/0<br>Dec 16 20:06:41 d001 kernel: Free swap = 0kB<br>Dec 16 20:06:41 d001 kernel: Total swap = 0kB<br>Dec 16 20:06:41 d001 kernel: 2097151 pages RAM<br>Dec 16 20:06:41 d001 kernel: 81926 pages reserved<br>Dec 16 20:06:41 d001 kernel: 924 pages shared<br>Dec 16 20:06:41 d001 kernel: 1984896 pages non-shared<br>Dec 16 20:06:41 d001 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name<br>Dec 16 20:06:41 d001 kernel: [ 477] 0 477 2662 105 0 -17 -1000 udevd<br>Dec 16 20:06:41 d001 kernel: [ 916] 0 916 374 48 0 0 0 aliyun-service<br>Dec 16 20:06:41 d001 kernel: [ 1156] 0 1156 62798 192 0 0 0 rsyslogd<br>Dec 16 20:06:41 d001 kernel: [ 1178] 32 1178 4744 62 0 0 0 rpcbind<br>Dec 16 20:06:41 d001 kernel: [ 1198] 29 1198 5837 112 1 0 0 rpc.statd<br>Dec 16 20:06:41 d001 kernel: [ 1382] 28 1382 157544 113 1 0 0 nscd<br>Dec 16 20:06:41 d001 kernel: [ 1414] 0 1414 118751 699 0 0 0 AliYunDunUpdate<br>Dec 16 20:06:41 d001 kernel: [ 1448] 0 1448 16657 178 0 -17 -1000 sshd<br>Dec 16 20:06:41 d001 kernel: [ 1463] 38 1463 6683 152 0 0 0 ntpd<br>Dec 16 20:06:41 d001 kernel: [ 1473] 0 1473 29325 154 0 0 0 crond<br>Dec 16 20:06:41 d001 kernel: [ 1516] 0 1516 1016 19 1 0 0 mingetty<br>Dec 16 20:06:41 d001 kernel: [ 1518] 0 1518 1016 17 3 0 0 mingetty<br>Dec 16 20:06:41 d001 kernel: [ 1520] 0 1520 1016 18 5 0 0 mingetty<br>Dec 16 20:06:41 d001 kernel: [ 1522] 0 1522 2661 104 1 -17 -1000 udevd<br>Dec 16 20:06:41 d001 kernel: [ 1523] 0 1523 2661 104 4 -17 -1000 udevd<br>Dec 16 20:06:41 d001 kernel: [ 1524] 0 1524 1016 18 2 0 0 mingetty<br>Dec 16 20:06:41 d001 kernel: [ 1526] 0 1526 1016 19 4 0 0 mingetty<br></span></div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">Dec 16 20:06:41 d001 kernel: [ 1528] 0 1528 1016 19 1 0 0 mingetty<br>Dec 16 20:06:41 d001 kernel: [ 1652] 0 1652 191799 1191 0 0 0 AliYunDun<br>Dec 16 20:06:41 d001 kernel: [ 1670] 0 1670 249011 1149 0 0 0 AliHids<br>Dec 16 20:06:41 d001 kernel: [ 4546] 0 4546 185509 4817 1 0 0 glusterd<br>Dec 16 20:06:41 d001 kernel: [ 4697] 0 4697 429110 35780 1 0 0 glusterfsd<br>Dec 16 20:06:41 d001 kernel: [ 4715] 0 4715 2149944 1788310 0 0 0 glusterfsd<br>Dec 16 20:06:41 d001 kernel: [ 4830] 0 4830 137846 6463 0 0 0 glusterfs<br>Dec 16 20:06:41 d001 kernel: [ 4940] 0 4940 341517 116710 1 0 0 glusterfs<br>Dec 16 20:06:41 d001 kernel: Out of memory: Kill process 4715 (glusterfsd) score 859 or sacrifice child<br>Dec 16 20:06:41 d001 kernel: Killed process 4715, UID 0, (glusterfsd) total-vm:8599776kB, anon-rss:7152896kB, file-rss:344kB<br>Dec 16 20:06:41 d001 kernel: glusterfsd invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0<br>Dec 16 20:06:41 d001 kernel: glusterfsd cpuset=/ mems_allowed=0<br>Dec 16 20:06:41 d001 kernel: Pid: 4717, comm: glusterfsd Not tainted 2.6.32-431.23.3.el6.x86_64 #1<br>Dec 16 20:06:41 d001 kernel: Call Trace:<br>Dec 16 20:06:41 d001 kernel: [<ffffffff810d0431>] ? cpuset_print_task_mems_allowed+0x91/0xb0<br>Dec 16 20:06:41 d001 kernel: [<ffffffff81122810>] ? dump_header+0x90/0x1b0<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8122833c>] ? security_real_capable_noaudit+0x3c/0x70<br>Dec 16 20:06:41 d001 kernel: [<ffffffff81122c92>] ? oom_kill_process+0x82/0x2a0<br>Dec 16 20:06:41 d001 kernel: [<ffffffff81122bd1>] ? select_bad_process+0xe1/0x120<br>Dec 16 20:06:41 d001 kernel: [<ffffffff811230d0>] ? out_of_memory+0x220/0x3c0<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8112f9ef>] ? __alloc_pages_nodemask+0x89f/0x8d0<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8116e2d2>] ? kmem_getpages+0x62/0x170<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8116eeea>] ? fallback_alloc+0x1ba/0x270<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8116e93f>] ? cache_grow+0x2cf/0x320<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8116ec69>] ? ____cache_alloc_node+0x99/0x160<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8116fbeb>] ? kmem_cache_alloc+0x11b/0x190<br>Dec 16 20:06:41 d001 kernel: [<ffffffff810efb75>] ? taskstats_exit+0x305/0x390<br></span></span></div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">Dec 16 20:06:41 d001 kernel: [<ffffffff81076c27>] ? do_exit+0x157/0x870<br>Dec 16 20:06:41 d001 kernel: [<ffffffff81060aa3>] ? perf_event_task_sched_out+0x33/0x70<br>Dec 16 20:06:41 d001 kernel: [<ffffffff81077398>] ? do_group_exit+0x58/0xd0<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8108cd46>] ? get_signal_to_deliver+0x1f6/0x460<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8100a265>] ? do_signal+0x75/0x800<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8108c85a>] ? dequeue_signal+0xda/0x170<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8108cb40>] ? sys_rt_sigtimedwait+0x250/0x260<br>Dec 16 20:06:41 d001 kernel: [<ffffffff81077087>] ? do_exit+0x5b7/0x870<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8100aa80>] ? do_notify_resume+0x90/0xc0<br>Dec 16 20:06:41 d001 kernel: [<ffffffff8100b341>] ? int_signal+0x12/0x17<br>Dec 16 20:06:41 d001 kernel: Mem-Info:<br>Dec 16 20:06:41 d001 kernel: Node 0 DMA per-cpu:<br>Dec 16 20:06:41 d001 kernel: CPU 0: hi: 0, btch: 1 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 1: hi: 0, btch: 1 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 2: hi: 0, btch: 1 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 3: hi: 0, btch: 1 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 4: hi: 0, btch: 1 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 5: hi: 0, btch: 1 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 6: hi: 0, btch: 1 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 7: hi: 0, btch: 1 usd: 0<br>Dec 16 20:06:41 d001 kernel: Node 0 DMA32 per-cpu:<br>Dec 16 20:06:41 d001 kernel: CPU 0: hi: 186, btch: 31 usd: 14<br>Dec 16 20:06:41 d001 kernel: CPU 1: hi: 186, btch: 31 usd: 152<br>Dec 16 20:06:41 d001 kernel: CPU 2: hi: 186, btch: 31 usd: 108<br>Dec 16 20:06:41 d001 kernel: CPU 3: hi: 186, btch: 31 usd: 70<br>Dec 16 20:06:41 d001 kernel: CPU 4: hi: 186, btch: 31 usd: 152<br>Dec 16 20:06:41 d001 kernel: CPU 5: hi: 186, btch: 31 usd: 0<br></span></span></span></div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">Dec 16 20:06:41 d001 kernel: CPU 6: hi: 186, btch: 31 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 7: hi: 186, btch: 31 usd: 0<br>Dec 16 20:06:41 d001 kernel: Node 0 Normal per-cpu:<br>Dec 16 20:06:41 d001 kernel: CPU 0: hi: 186, btch: 31 usd: 145<br>Dec 16 20:06:41 d001 kernel: CPU 1: hi: 186, btch: 31 usd: 19<br>Dec 16 20:06:41 d001 kernel: CPU 2: hi: 186, btch: 31 usd: 33<br>Dec 16 20:06:41 d001 kernel: CPU 3: hi: 186, btch: 31 usd: 50<br>Dec 16 20:06:41 d001 kernel: CPU 4: hi: 186, btch: 31 usd: 165<br>Dec 16 20:06:41 d001 kernel: CPU 5: hi: 186, btch: 31 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 6: hi: 186, btch: 31 usd: 0<br>Dec 16 20:06:41 d001 kernel: CPU 7: hi: 186, btch: 31 usd: 0<br>Dec 16 20:06:41 d001 kernel: active_anon:1955964 inactive_anon:38 isolated_anon:0<br>Dec 16 20:06:41 d001 kernel: active_file:312 inactive_file:1262 isolated_file:0<br>Dec 16 20:06:41 d001 kernel: unevictable:0 dirty:1 writeback:3 unstable:0<br>Dec 16 20:06:41 d001 kernel: free:25745 slab_reclaimable:2412 slab_unreclaimable:7815<br>Dec 16 20:06:41 d001 kernel: mapped:208 shmem:43 pagetables:4679 bounce:0<br>Dec 16 20:06:41 d001 kernel: Node 0 DMA free:15752kB min:124kB low:152kB high:184kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15364kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes<br>Dec 16 20:06:41 d001 kernel: lowmem_reserve[]: 0 3000 8050 8050<br>Dec 16 20:06:41 d001 kernel: Node 0 DMA32 free:45044kB min:25140kB low:31424kB high:37708kB active_anon:2740816kB inactive_anon:0kB active_file:896kB inactive_file:4176kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:0kB writeback:4kB mapped:816kB shmem:0kB slab_reclaimable:1636kB slab_unreclaimable:1888kB kernel_stack:128kB pagetables:5204kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1664 all_unreclaimable? yes<br>Dec 16 20:06:41 d001 kernel: lowmem_reserve[]: 0 0 5050 5050<br></span></span></span></span></div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">Dec 16 20:06:41 d001 kernel: Node 0 Normal free:42184kB min:42316kB low:52892kB high:63472kB active_anon:5083040kB inactive_anon:152kB active_file:352kB inactive_file:872kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:5171200kB mlocked:0kB dirty:4kB writeback:8kB mapped:16kB shmem:172kB slab_reclaimable:8012kB slab_unreclaimable:29372kB kernel_stack:2240kB pagetables:13512kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1131 all_unreclaimable? yes<br>Dec 16 20:06:41 d001 kernel: lowmem_reserve[]: 0 0 0 0<br>Dec 16 20:06:41 d001 kernel: Node 0 DMA: 2*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15752kB<br>Dec 16 20:06:41 d001 kernel: Node 0 DMA32: 11044*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44176kB<br>Dec 16 20:06:41 d001 kernel: Node 0 Normal: 10484*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41936kB<br>Dec 16 20:06:41 d001 kernel: 1782 total pagecache pages<br>Dec 16 20:06:41 d001 kernel: 0 pages in swap cache<br>Dec 16 20:06:41 d001 kernel: Swap cache stats: add 0, delete 0, find 0/0<br>Dec 16 20:06:41 d001 kernel: Free swap = 0kB<br>Dec 16 20:06:41 d001 kernel: Total swap = 0kB<br>Dec 16 20:06:41 d001 kernel: 2097151 pages RAM<br>Dec 16 20:06:41 d001 kernel: 81926 pages reserved<br>Dec 16 20:06:41 d001 kernel: 931 pages shared<br>Dec 16 20:06:41 d001 kernel: 1984884 pages non-shared<br>Dec 16 20:06:41 d001 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name<br>Dec 16 20:06:41 d001 kernel: [ 477] 0 477 2662 105 0 -17 -1000 udevd<br>Dec 16 20:06:41 d001 kernel: [ 916] 0 916 374 48 0 0 0 aliyun-service<br>Dec 16 20:06:41 d001 kernel: [ 1156] 0 1156 62798 192 0 0 0 rsyslogd<br></span></span></span></span></span></div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">Dec 16 20:06:41 d001 kernel: [ 1178] 32 1178 4744 62 0 0 0 rpcbind<br>Dec 16 20:06:41 d001 kernel: [ 1198] 29 1198 5837 112 1 0 0 rpc.statd<br>Dec 16 20:06:41 d001 kernel: [ 1382] 28 1382 157544 113 1 0 0 nscd<br>Dec 16 20:06:41 d001 kernel: [ 1414] 0 1414 118751 699 0 0 0 AliYunDunUpdate<br>Dec 16 20:06:41 d001 kernel: [ 1448] 0 1448 16657 178 0 -17 -1000 sshd<br>Dec 16 20:06:41 d001 kernel: [ 1463] 38 1463 6683 152 0 0 0 ntpd<br>Dec 16 20:06:41 d001 kernel: [ 1473] 0 1473 29325 154 0 0 0 crond<br>Dec 16 20:06:41 d001 kernel: [ 1516] 0 1516 1016 19 1 0 0 mingetty<br>Dec 16 20:06:41 d001 kernel: [ 1518] 0 1518 1016 17 3 0 0 mingetty<br>Dec 16 20:06:41 d001 kernel: [ 1520] 0 1520 1016 18 5 0 0 mingetty<br>Dec 16 20:06:41 d001 kernel: [ 1522] 0 1522 2661 104 1 -17 -1000 udevd<br>Dec 16 20:06:41 d001 kernel: [ 1523] 0 1523 2661 104 4 -17 -1000 udevd<br>Dec 16 20:06:41 d001 kernel: [ 1524] 0 1524 1016 18 2 0 0 mingetty<br>Dec 16 20:06:41 d001 kernel: [ 1526] 0 1526 1016 19 4 0 0 mingetty<br>Dec 16 20:06:41 d001 kernel: [ 1528] 0 1528 1016 19 1 0 0 mingetty<br>Dec 16 20:06:41 d001 kernel: [ 1652] 0 1652 191799 1203 0 0 0 AliYunDun<br>Dec 16 20:06:41 d001 kernel: [ 1670] 0 1670 249011 1160 0 0 0 AliHids<br>Dec 16 20:06:41 d001 kernel: [ 4546] 0 4546 185509 4817 1 0 0 glusterd<br>Dec 16 20:06:41 d001 kernel: [ 4697] 0 4697 429110 35780 1 0 0 glusterfsd<br>Dec 16 20:06:41 d001 kernel: [ 4717] 0 4715 2149944 1788310 4 0 0 glusterfsd<br>Dec 16 20:06:41 d001 kernel: [ 4830] 0 4830 137846 6463 0 0 0 glusterfs<br>Dec 16 20:06:41 d001 kernel: [ 4940] 0 4940 341517 116710 1 0 0 glusterfs</span></span></span></span></span></span></div><div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">===================== <EOF> ==================================</span></span></span></span></span></span></div><hr style="width: 210px; height: 1px;" color="#b5c4df" size="1" align="left">
<div><span><div style="MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt"><div>PuYun</div></div></span></div>
<blockquote style="margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em;"><div> </div><div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"><div style="PADDING-RIGHT: 8px; PADDING-LEFT: 8px; FONT-SIZE: 12px;FONT-FAMILY:tahoma;COLOR:#000000; BACKGROUND: #efefef; PADDING-BOTTOM: 8px; PADDING-TOP: 8px"><div><b>From:</b> <a href="mailto:cloudor@126.com">PuYun</a></div><div><b>Date:</b> 2015-12-15 22:10</div><div><b>To:</b> <a href="mailto:gluster-users@gluster.org">gluster-users</a></div><div><b>Subject:</b> Re: [Gluster-users] How to diagnose volume rebalance failure?</div></div></div><div><div class="FoxDiv20151216224726118204">
<div><span></span>Hi,</div><div><br></div><div>I find this bug link <span style="font-family: ''; font-size: 10.5pt; line-height: 1.5; background-color: window;"></span><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1261234" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" font-weight:="" normal;="" font-style:="" normal;'="" style="font-family: ''; font-size: 10.5pt; line-height: 1.5; background-color: window;">https://bugzilla.redhat.com/show_bug.cgi?id=1261234</a><span microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgb(255,="" 255,="" 255);="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="" style="font-family: ''; font-size: 10.5pt; line-height: 1.5; background-color: window;"> . My version is 3.7.4 which is older than the fixed version 3.7.5.</span></div><div><span microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgb(255,="" 255,="" 255);="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="" style="font-family: ''; font-size: 10.5pt; line-height: 1.5; background-color: window;">I'll upgrade my gluster version and try again later.</span></div><div><span microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgb(255,="" 255,="" 255);="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="" style="font-family: ''; font-size: 10.5pt; line-height: 1.5; background-color: window;"><br></span></div><span style="font-family: "" microsoft="" yahei="" ui'";="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgb(255,="" 255,="" 255);="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""></span>
<div>Thank you.</div><hr style="width: 210px; height: 1px;" color="#b5c4df" size="1" align="left">
<div><span><div style="MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt"><div>PuYun</div></div></span></div>
<blockquote style="margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em;"><div> </div><div><div class="FoxDiv20151215220804871171"><blockquote style="margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em;"><div><div class="FoxDiv20151215214505767857"><blockquote style="margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em;"><div>
</div></blockquote>
</div></div></blockquote>
</div></div></blockquote>
</div></div></blockquote>
</body></html>