<html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><style>body { line-height: 1.5; }blockquote { margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em; }body { font-size: 10.5pt; font-family: 'Microsoft YaHei UI'; color: rgb(0, 0, 0); line-height: 1.5; }</style></head><body>
<div><span></span>Hi,</div><div><br></div><div>Thank you for your reply. I don't know how to send you the huge sized rebalance log file which is about 2GB. </div><div><br></div><div>However, I might have found out the reason why the task failed. My gluster server has only 2 cpu cores and carries 2 ssd bricks. When the rebalance task began, top 3 <span style="font-size: 10.5pt; line-height: 1.5; background-color: window;"> </span><span style="font-size: 10.5pt; line-height: 1.5; background-color: window;">processes are 70%~80%, 30%~40 and 30%~40 cpu usage. Others are less than 1%. But after a while, 2 CPU cores are used up totally and I even can't login until the rebalance task failed. </span></div><div><br></div><div>It seems 2 bricks require 4 CPU cores at least. Now I upgrade the virtual server with 8 CPU cores and start rebalance task again. Everything goes well for now.</div><div><br></div><div>I will report again when the current task completed or failed.</div>
<div><br></div><hr style="width: 210px; height: 1px;" color="#b5c4df" size="1" align="left">
<div><span><div style="MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt"><div>PuYun</div></div></span></div>
<blockquote style="margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em;"><div> </div><div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"><div style="PADDING-RIGHT: 8px; PADDING-LEFT: 8px; FONT-SIZE: 12px;FONT-FAMILY:tahoma;COLOR:#000000; BACKGROUND: #efefef; PADDING-BOTTOM: 8px; PADDING-TOP: 8px"><div><b>From:</b> <a href="mailto:nbalacha@redhat.com">Nithya Balachandran</a></div><div><b>Date:</b> 2015-12-14 18:57</div><div><b>To:</b> <a href="mailto:cloudor@126.com">PuYun</a></div><div><b>CC:</b> <a href="mailto:gluster-users@gluster.org">gluster-users</a></div><div><b>Subject:</b> Re: [Gluster-users] How to diagnose volume rebalance failure?</div></div></div><div><div>Hi,</div>
<div> </div>
<div>Can you send us the rebalance log?</div>
<div> </div>
<div>Regards,</div>
<div>Nithya</div>
<div> </div>
<div>----- Original Message -----</div>
<div>> From: "PuYun" <cloudor@126.com></div>
<div>> To: "gluster-users" <gluster-users@gluster.org></div>
<div>> Sent: Monday, December 14, 2015 11:33:40 AM</div>
<div>> Subject: Re: [Gluster-users] How to diagnose volume rebalance failure?</div>
<div>> </div>
<div>> Here is the tail of the failed rebalance log, any clue?</div>
<div>> </div>
<div>> [2015-12-13 21:30:31.527493] I [dht-rebalance.c:2340:gf_defrag_process_dir]</div>
<div>> 0-FastVol-dht: Migration operation on dir</div>
<div>> /for_ybest_fsdir/user/Weixin.oClDcjhe/Ny/5F/1MsH5--BcoGRAJPI took 20.95 secs</div>
<div>> [2015-12-13 21:30:31.528704] I [dht-rebalance.c:1010:dht_migrate_file]</div>
<div>> 0-FastVol-dht:</div>
<div>> /for_ybest_fsdir/user/Weixin.oClDcjhe/Kn/hM/oHcPMp4hKq5Tq2ZQ/flag_finished:</div>
<div>> attempting to move from FastVol-client-0 to FastVol-client-1</div>
<div>> [2015-12-13 21:30:31.543901] I [dht-rebalance.c:1010:dht_migrate_file]</div>
<div>> 0-FastVol-dht:</div>
<div>> /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/userPoint:</div>
<div>> attempting to move from FastVol-client-0 to FastVol-client-1</div>
<div>> [2015-12-13 21:31:37.210496] I [MSGID: 109081]</div>
<div>> [dht-common.c:3780:dht_setxattr] 0-FastVol-dht: fixing the layout of</div>
<div>> /for_ybest_fsdir/user/Weixin.oClDcjhe/Ny/7Q</div>
<div>> [2015-12-13 21:31:37.722825] I [MSGID: 109045]</div>
<div>> [dht-selfheal.c:1508:dht_fix_layout_of_directory] 0-FastVol-dht: subvolume 0</div>
<div>> (FastVol-client-0): 1032124 chunks</div>
<div>> [2015-12-13 21:31:37.722837] I [MSGID: 109045]</div>
<div>> [dht-selfheal.c:1508:dht_fix_layout_of_directory] 0-FastVol-dht: subvolume 1</div>
<div>> (FastVol-client-1): 1032124 chunks</div>
<div>> [2015-12-13 21:33:03.955539] I [MSGID: 109064]</div>
<div>> [dht-layout.c:808:dht_layout_dir_mismatch] 0-FastVol-dht: subvol:</div>
<div>> FastVol-client-0; inode layout - 0 - 2146817919 - 1; disk layout -</div>
<div>> 2146817920 - 4294967295 - 1</div>
<div>> [2015-12-13 21:33:04.069859] I [MSGID: 109018]</div>
<div>> [dht-common.c:806:dht_revalidate_cbk] 0-FastVol-dht: Mismatching layouts for</div>
<div>> /for_ybest_fsdir/user/Weixin.oClDcjhe/Ny/7Q, gfid =</div>
<div>> f38c4ed2-a26a-4d83-adfd-6b0331831738</div>
<div>> [2015-12-13 21:33:04.118800] I [MSGID: 109064]</div>
<div>> [dht-layout.c:808:dht_layout_dir_mismatch] 0-FastVol-dht: subvol:</div>
<div>> FastVol-client-1; inode layout - 2146817920 - 4294967295 - 1; disk layout -</div>
<div>> 0 - 2146817919 - 1</div>
<div>> [2015-12-13 21:33:19.979507] I [MSGID: 109022]</div>
<div>> [dht-rebalance.c:1290:dht_migrate_file] 0-FastVol-dht: completed migration</div>
<div>> of</div>
<div>> /for_ybest_fsdir/user/Weixin.oClDcjhe/Kn/hM/oHcPMp4hKq5Tq2ZQ/flag_finished</div>
<div>> from subvolume FastVol-client-0 to FastVol-client-1</div>
<div>> [2015-12-13 21:33:19.979459] I [MSGID: 109022]</div>
<div>> [dht-rebalance.c:1290:dht_migrate_file] 0-FastVol-dht: completed migration</div>
<div>> of /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/userPoint</div>
<div>> from subvolume FastVol-client-0 to FastVol-client-1</div>
<div>> [2015-12-13 21:33:25.543941] I [dht-rebalance.c:1010:dht_migrate_file]</div>
<div>> 0-FastVol-dht:</div>
<div>> /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/portrait_origin.jpg:</div>
<div>> attempting to move from FastVol-client-0 to FastVol-client-1</div>
<div>> [2015-12-13 21:33:25.962547] I [dht-rebalance.c:1010:dht_migrate_file]</div>
<div>> 0-FastVol-dht:</div>
<div>> /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/portrait_small.jpg:</div>
<div>> attempting to move from FastVol-client-0 to FastVol-client-1</div>
<div>> </div>
<div>> </div>
<div>> Cloudor</div>
<div>> </div>
<div>> </div>
<div>> </div>
<div>> From: Sakshi Bansal</div>
<div>> Date: 2015-12-12 13:02</div>
<div>> To: 蒲云</div>
<div>> CC: gluster-users</div>
<div>> Subject: Re: [Gluster-users] How to diagnose volume rebalance failure?</div>
<div>> In the rebalance log file you can check the file/directory for which the</div>
<div>> rebalance has failed. It can mention what was the fop for whihc the failure</div>
<div>> happened.</div>
<div>> </div>
<div>> _______________________________________________</div>
<div>> Gluster-users mailing list</div>
<div>> Gluster-users@gluster.org</div>
<div>> http://www.gluster.org/mailman/listinfo/gluster-users</div>
</div></blockquote>
</body></html>