<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<br>
<div class="moz-cite-prefix">On 08/17/2015 01:58 AM, Christophe
TREFOIS wrote:<br>
</div>
<blockquote
cite="mid:2EBB29CB9A8F494FB5253F6AF2E6A1981CDC4A88@trip.uni.lux"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
p.banner-container, li.banner-container, div.banner-container
{mso-style-name:banner-container;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle20
{mso-style-type:personal-compose;
font-family:"Times New Roman",serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Dear all,<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">I have successfully
added a new node to our setup, and finally managed to get
a successful fix-layout run as well with no errors.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Now, as per the
documentation, I started a gluster volume rebalance live
start task and I see many skipped files. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">The error log contains
then entires as follows for each skipped file.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:23:30.591161] E [MSGID: 109023]
[dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht:
Migrate file failed:/hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Mea<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">s_05(2013-10-11_17-12-02)/004010008.flex
lookup failed<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:23:30.768391] E [MSGID: 109023]
[dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht:
Migrate file failed:/hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Mea<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">s_05(2013-10-11_17-12-02)/007005003.flex
lookup failed<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:23:30.804811] E [MSGID: 109023]
[dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht:
Migrate file failed:/hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Mea<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">s_05(2013-10-11_17-12-02)/006005009.flex
lookup failed<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:23:30.805201] E [MSGID: 109023]
[dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht:
Migrate file failed:/hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Mea<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">s_05(2013-10-11_17-12-02)/005006011.flex
lookup failed<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:23:30.880037] E [MSGID: 109023]
[dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht:
Migrate file failed:/hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Mea<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">s_05(2013-10-11_17-12-02)/005009012.flex
lookup failed<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:23:31.038236] E [MSGID: 109023]
[dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht:
Migrate file failed:/hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Mea<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">s_05(2013-10-11_17-12-02)/003008007.flex
lookup failed<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:23:31.259762] E [MSGID: 109023]
[dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht:
Migrate file failed:/hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Mea<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">s_05(2013-10-11_17-12-02)/004008006.flex
lookup failed<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:23:31.333764] E [MSGID: 109023]
[dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht:
Migrate file failed:/hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Mea<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">s_05(2013-10-11_17-12-02)/007008001.flex
lookup failed<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:23:31.340190] E [MSGID: 109023]
[dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht:
Migrate file failed:/hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Mea<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">s_05(2013-10-11_17-12-02)/006007004.flex
lookup failed<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Update: one of the
rebalance tasks now failed.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">@Rafi, I got the same
error as Friday except this time with data.</span></p>
</div>
</div>
</blockquote>
<br>
Packets that carrying the ping request could be waiting in the queue
during the whole time-out period, because of the heavy traffic in
the network. I have sent a patch for this. You can track the status
here : <a class="moz-txt-link-freetext" href="http://review.gluster.org/11935">http://review.gluster.org/11935</a><br>
<br>
<br>
<blockquote
cite="mid:2EBB29CB9A8F494FB5253F6AF2E6A1981CDC4A88@trip.uni.lux"
type="cite">
<div class="WordSection1">
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.533167] C
[rpc-clnt-ping.c:161:rpc_clnt_ping_timer_expired]
0-live-client-0: server 192.168.123.104:49164 has not
responded in the last 42 seconds, disconnecting.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.533614] E [rpc-clnt.c:362:saved_frames_unwind]
(-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6]
(--> /lib64/libgfrpc.so.0(saved_frames_unwin<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">d+0x1de)[0x7fa454bb09be]
(-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c]
(--> /lib64/li<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">bgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38]
))))) 0-live-client-0: forced unwinding frame
type(GlusterFS 3.3) op(INODELK(29)) called at 2015-08-16
20:23:51.305640 (xid=0x5dd4da)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.533672] E [MSGID: 114031]
[client-rpc-fops.c:1621:client3_3_inodelk_cbk]
0-live-client-0: remote operation failed [Transport
endpoint is not connected]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.534201] E [rpc-clnt.c:362:saved_frames_unwind]
(-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6]
(--> /lib64/libgfrpc.so.0(saved_frames_unwin<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">d+0x1de)[0x7fa454bb09be]
(-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c]
(--> /lib64/li<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">bgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38]
))))) 0-live-client-0: forced unwinding frame
type(GlusterFS 3.3) op(READ(12)) called at 2015-08-16
20:23:51.303938 (xid=0x5dd4d7)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.534347] E [MSGID: 109023]
[dht-rebalance.c:1124:dht_migrate_file] 0-live-dht:
Migrate file failed: /hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Meas_<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">12(2013-10-12_00-12-55)/007008007.flex:
failed to migrate data<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.534413] E [rpc-clnt.c:362:saved_frames_unwind]
(-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6]
(--> /lib64/libgfrpc.so.0(saved_frames_unwin<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">d+0x1de)[0x7fa454bb09be]
(-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38]
))))) 0-live-client-0: forced unwinding frame
type(GlusterFS 3.3) op(READ(12)) called at 2015-08-16
20:23:51.303969 (xid=0x5dd4d8)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.534579] E [MSGID: 109023]
[dht-rebalance.c:1124:dht_migrate_file] 0-live-dht:
Migrate file failed: /hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Meas_12(2013-10-12_00-12-55)/007009012.flex:
failed to migrate data<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.534676] E [rpc-clnt.c:362:saved_frames_unwind]
(-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6]
(-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa454bb09be]
(-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38]
))))) 0-live-client-0: forced unwinding frame
type(GlusterFS 3.3) op(READ(12)) called at 2015-08-16
20:23:51.313548 (xid=0x5dd4db)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.534745] E [MSGID: 109023]
[dht-rebalance.c:1124:dht_migrate_file] 0-live-dht:
Migrate file failed: /hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Meas_12(2013-10-12_00-12-55)/006008011.flex:
failed to migrate data<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.535199] E [rpc-clnt.c:362:saved_frames_unwind]
(-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6]
(-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa454bb09be]
(-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38]
))))) 0-live-client-0: forced unwinding frame
type(GlusterFS 3.3) op(READ(12)) called at 2015-08-16
20:23:51.326369 (xid=0x5dd4dc)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.535232] E [MSGID: 109023]
[dht-rebalance.c:1124:dht_migrate_file] 0-live-dht:
Migrate file failed: /hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Meas_12(2013-10-12_00-12-55)/005003001.flex:
failed to migrate data<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.535984] E [rpc-clnt.c:362:saved_frames_unwind]
(-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6]
(-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa454bb09be]
(-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38]
))))) 0-live-client-0: forced unwinding frame
type(GlusterFS 3.3) op(READ(12)) called at 2015-08-16
20:23:51.326437 (xid=0x5dd4dd)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.536069] E [MSGID: 109023]
[dht-rebalance.c:1124:dht_migrate_file] 0-live-dht:
Migrate file failed: /hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Meas_12(2013-10-12_00-12-55)/007010012.flex:
failed to migrate data<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.536267] E [rpc-clnt.c:362:saved_frames_unwind]
(-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6]
(-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa454bb09be]
(-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38]
))))) 0-live-client-0: forced unwinding frame
type(GlusterFS 3.3) op(LOOKUP(27)) called at 2015-08-16
20:23:51.337240 (xid=0x5dd4de)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.536339] E [MSGID: 109023]
[dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht:
Migrate file failed:/hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Meas_08(2013-10-11_20-12-25)/002005012.flex
lookup failed<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.536487] E [rpc-clnt.c:362:saved_frames_unwind]
(-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6]
(-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa454bb09be]
(-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38]
))))) 0-live-client-0: forced unwinding frame
type(GlusterFS 3.3) op(LOOKUP(27)) called at 2015-08-16
20:23:51.425254 (xid=0x5dd4df)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.536685] E [rpc-clnt.c:362:saved_frames_unwind]
(-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6]
(-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa454bb09be]
(-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38]
))))) 0-live-client-0: forced unwinding frame
type(GlusterFS 3.3) op(LOOKUP(27)) called at 2015-08-16
20:23:51.738907 (xid=0x5dd4e0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.536891] E [rpc-clnt.c:362:saved_frames_unwind]
(-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6]
(-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa454bb09be]
(-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38]
))))) 0-live-client-0: forced unwinding frame
type(GlusterFS 3.3) op(LOOKUP(27)) called at 2015-08-16
20:23:51.805096 (xid=0x5dd4e1)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.537316] E [rpc-clnt.c:362:saved_frames_unwind]
(-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6]
(-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa454bb09be]
(-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38]
))))) 0-live-client-0: forced unwinding frame
type(GlusterFS 3.3) op(LOOKUP(27)) called at 2015-08-16
20:23:51.805977 (xid=0x5dd4e2)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.537735] E [rpc-clnt.c:362:saved_frames_unwind]
(-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6]
(-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa454bb09be]
(-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38]
))))) 0-live-client-0: forced unwinding frame
type(GF-DUMP) op(NULL(2)) called at 2015-08-16
20:23:52.530107 (xid=0x5dd4e3)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.538475] E [MSGID: 114031]
[client-rpc-fops.c:1621:client3_3_inodelk_cbk]
0-live-client-0: remote operation failed [Transport
endpoint is not connected]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">The message "E [MSGID:
114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk]
0-live-client-0: remote operation failed [Transport
endpoint is not connected]" repeated 4 times between
[2015-08-16 20:24:34.538475] and [2015-08-16
20:24:34.538535]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.538584] E [MSGID: 109023]
[dht-rebalance.c:1617:gf_defrag_migrate_single_file]
0-live-dht: Migrate file failed: 002004003.flex lookup
failed<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.538904] E [MSGID: 109023]
[dht-rebalance.c:1617:gf_defrag_migrate_single_file]
0-live-dht: Migrate file failed: 003009008.flex lookup
failed<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.539724] E [MSGID: 109023]
[dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht:
Migrate file failed:/hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Meas_08(2013-10-11_20-12-25)/005009006.flex
lookup failed<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.539820] E [MSGID: 109016]
[dht-rebalance.c:2554:gf_defrag_fix_layout] 0-live-dht:
Fix layout failed for /hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Meas_08(2013-10-11_20-12-25)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.540031] E [MSGID: 109016]
[dht-rebalance.c:2554:gf_defrag_fix_layout] 0-live-dht:
Fix layout failed for /hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.540691] E [MSGID: 114031]
[client-rpc-fops.c:251:client3_3_mknod_cbk]
0-live-client-0: remote operation failed. Path:
/hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Meas_12(2013-10-12_00-12-55)/002005008.flex
[Transport endpoint is not connected]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.541152] E [MSGID: 114031]
[client-rpc-fops.c:251:client3_3_mknod_cbk]
0-live-client-0: remote operation failed. Path:
/hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Meas_12(2013-10-12_00-12-55)/005004009.flex
[Transport endpoint is not connected]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.541331] E [MSGID: 114031]
[client-rpc-fops.c:251:client3_3_mknod_cbk]
0-live-client-0: remote operation failed. Path:
/hcs/hcs/OperaArchiveCol/SK
20131011_Oligo_Rot_lowConc_P1/Meas_12(2013-10-12_00-12-55)/007005011.flex
[Transport endpoint is not connected]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.541486] E [MSGID: 109016]
[dht-rebalance.c:2554:gf_defrag_fix_layout] 0-live-dht:
Fix layout failed for /hcs/hcs/OperaArchiveCol<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.541572] E [MSGID: 109016]
[dht-rebalance.c:2554:gf_defrag_fix_layout] 0-live-dht:
Fix layout failed for /hcs/hcs<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[2015-08-16
20:24:34.541639] E [MSGID: 109016]
[dht-rebalance.c:2554:gf_defrag_fix_layout] 0-live-dht:
Fix layout failed for /hcs<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Any help would be
greatly appreciated.</span></p>
</div>
</div>
</blockquote>
CCing dht teams to give you better idea about why rebalance failed/
and about huge memory consumption by rebalance process (200GB RAM) .<br>
<br>
Regards<br>
Rafi KC<br>
<br>
<br>
<br>
<blockquote
cite="mid:2EBB29CB9A8F494FB5253F6AF2E6A1981CDC4A88@trip.uni.lux"
type="cite">
<div class="WordSection1">
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">--<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Christophe<o:p></o:p></span></p>
<div id="AppleMailSignature">
<div>
<p style="line-height:12.0pt"><b><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#3D3B3B">Dr
Christophe Trefois, Dipl.-Ing.</span></b><span
class="apple-converted-space"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#212121"> </span></span><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#212121"><br>
</span><span
style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#3D3B3B">Technical
Specialist / Post-Doc</span><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#212121"><o:p></o:p></span></p>
<p style="line-height:12.0pt"><b><span
style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#3D3B3B">UNIVERSITÉ
DU LUXEMBOURG</span></b><span
style="font-size:7.5pt;font-family:"Arial",sans-serif;color:black"><br>
<br>
</span><b><span
style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#3D3B3B">LUXEMBOURG
CENTRE FOR SYSTEMS BIOMEDICINE</span></b><span
style="font-size:7.5pt;font-family:"Arial",sans-serif;color:black"><br>
</span><span
style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#3D3B3B">Campus
Belval | House of Biomedicine<span
class="apple-converted-space"> </span><br>
<span class="apple-converted-space">6, avenue du
Swing </span><br>
L-4367 Belvaux<span class="apple-converted-space"> </span></span><span
style="font-size:7.5pt;font-family:"Arial",sans-serif;color:black"><br>
</span><span
style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#3D3B3B">T:<span
class="apple-converted-space"> </span>+352 46 66 44
6124</span><span class="apple-converted-space"><span
style="font-size:7.5pt;font-family:"Arial",sans-serif;color:black"> </span></span><span
style="font-size:7.5pt;font-family:"Arial",sans-serif;color:black"><br>
</span><span
style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#3D3B3B">F:<span
class="apple-converted-space"> </span>+352 46 66 44
6949</span><span class="apple-converted-space"><span
style="font-size:7.5pt;font-family:"Arial",sans-serif;color:black"> </span></span><span
style="font-size:7.5pt;font-family:"Arial",sans-serif;color:black"><br>
<a moz-do-not-send="true"
href="http://www.uni.lu/lcsb"><span
style="color:#006DBD">http://www.uni.lu/lcsb</span></a><o:p></o:p></span></p>
<p style="line-height:12.0pt"><span
style="font-size:7.0pt;font-family:"Arial",sans-serif;color:#3D3B3B"
lang="EN-US">----<br>
This message is confidential and may contain
privileged information.<span
class="apple-converted-space"> </span><br>
It is intended for the named recipient only.<span
class="apple-converted-space"> </span><br>
If you receive it in error please notify me and
permanently delete the original message and any
copies.<span class="apple-converted-space"> </span><br>
</span><span
style="font-size:7.0pt;font-family:"Arial",sans-serif;color:#3D3B3B">----<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <span
class="apple-converted-space"> </span><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
<br>
</body>
</html>