<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Dear Rafi,
<div class=""><br class="">
</div>
<div class="">Thanks for submitting a patch.</div>
<div class=""><br class="">
</div>
<div class="">@DHT, I have two additional questions / problems.</div>
<div class=""><br class="">
</div>
<div class="">1. When doing a rebalance (with data) RAM consumption on the nodes goes dramatically high, eg out of 196 GB available per node, RAM usage would fill up to 195.6 GB. This seems quite excessive and strange to me. </div>
<div class=""><br class="">
</div>
<div class="">2. As you can see, the rebalance (with data) failed as one endpoint becomes unconnected (even though it still is connected). I’m thinking this could be due to the high RAM usage?</div>
<div class=""><br class="">
</div>
<div class="">Thank you for your help,</div>
<div class=""><br class="">
</div>
<div class="">—</div>
<div class="">Christophe<br class="">
<div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<p style="font-family: Arial, sans-serif; font-size: 10pt; line-height: 16px; color: rgb(33, 33, 33);" class="">
<span style="font-weight: bold; color: rgb(61, 59, 59); display: inline;" class="">Dr Christophe Trefois, Dipl.-Ing.</span><span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span><span style="display: inline;" class=""></span><br class="">
<span style="color: rgb(61, 59, 59); display: inline; font-size: 7.5pt;" class="">Technical Specialist / Post-Doc</span></p>
<p style="font-family: Arial, sans-serif; font-size: 7.5pt; line-height: 16px;" class="">
<span style="font-weight: bold; color: rgb(61, 59, 59); display: inline;" class="">UNIVERSITÉ DU LUXEMBOURG</span><br class="">
<span style="display: inline;" class=""><br class="">
</span><span style="font-weight: bold; color: rgb(61, 59, 59); display: inline;" class="">LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE</span><br class="">
<span style="color: rgb(61, 59, 59); display: inline;" class="">Campus Belval | House of Biomedicine<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span><br class="">
<span class="Apple-converted-space">6, avenue du Swing </span><br class="">
L-4367 Belvaux<span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span></span><br class="">
<span style="color: rgb(61, 59, 59); display: inline;" class="">T:<span class="Apple-converted-space"> </span></span><span style="color: rgb(61, 59, 59); display: inline;" class="">+352 46 66 44 6124</span><span class="Apple-converted-space"> </span><br class="">
<span style="color: rgb(61, 59, 59); display: inline;" class="">F:<span class="Apple-converted-space"> </span></span><span style="color: rgb(61, 59, 59); display: inline;" class="">+352 46 66 44 6949</span><span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span><span style="display: inline;" class=""><br class="">
</span><a href="http://www.uni.lu/lcsb" style="color: rgb(0, 109, 189); display: inline;" class="">http://www.uni.lu/lcsb</a></p>
<p style="font-family: Arial, sans-serif; font-size: 14px; line-height: 16px;" class="">
<a href="https://www.facebook.com/trefex" style="display: inline;" class=""><img width="24" height="24" data-filename="facebook.png" src="https://s3.amazonaws.com/htmlsig-assets/rounded/facebook.png" alt="Facebook" class=""></a><span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span><a href="https://twitter.com/Trefex" style="display: inline;" class=""><img width="24" height="24" data-filename="twitter.png" src="https://s3.amazonaws.com/htmlsig-assets/rounded/twitter.png" alt="Twitter" class=""></a><span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span><a href="https://plus.google.com/+ChristopheTrefois/" style="display: inline;" class=""><img width="24" height="24" data-filename="googleplus.png" src="https://s3.amazonaws.com/htmlsig-assets/rounded/googleplus.png" alt="Google Plus" class=""></a><span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span><a href="https://www.linkedin.com/in/trefoischristophe" style="display: inline;" class=""><img width="24" height="24" data-filename="linkedin.png" src="https://s3.amazonaws.com/htmlsig-assets/rounded/linkedin.png" alt="Linkedin" class=""></a><span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span><a href="http://skype:Trefex?call" style="display: inline;" class=""><img width="24" height="24" data-filename="skype.png" src="https://s3.amazonaws.com/htmlsig-assets/rounded/skype.png" alt="skype" class=""></a></p>
<p class="banner-container" style="font-family: Arial, sans-serif; font-size: 14px; line-height: 16px;">
</p>
<p style="font-family: Arial, sans-serif; color: rgb(61, 59, 59); font-size: 9px; line-height: 16px;" class="">
----<br class="">
This message is confidential and may contain privileged information.<span class="Apple-converted-space"> </span><br class="">
It is intended for the named recipient only.<span class="Apple-converted-space"> </span><br class="">
If you receive it in error please notify me and permanently delete the original message and any copies.<span class="Apple-converted-space"> </span><br class="">
----<br class="">
</p>
<span class="Apple-converted-space"> </span></div>
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 17 Aug 2015, at 11:27, Mohammed Rafi K C <<a href="mailto:rkavunga@redhat.com" class="">rkavunga@redhat.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><br style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<br style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<div class="moz-cite-prefix" style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);">
On 08/17/2015 01:58 AM, Christophe TREFOIS wrote:<br class="">
</div>
<blockquote cite="mid:2EBB29CB9A8F494FB5253F6AF2E6A1981CDC4A88@trip.uni.lux" type="cite" style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<div class="WordSection1" style="page: WordSection1;">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">Dear all,<o:p class=""></o:p></span></div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class=""> </span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">I have successfully added a new node to our setup, and finally managed to get a successful fix-layout run as well with no errors.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class=""> </span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">Now, as per the documentation, I started a gluster volume rebalance live start task and I see many skipped files. <o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">The error log contains then entires as follows for each skipped file.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class=""> </span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:23:30.591161] E [MSGID: 109023] [dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht: Migrate file failed:/hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Mea<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">s_05(2013-10-11_17-12-02)/004010008.flex lookup failed<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:23:30.768391] E [MSGID: 109023] [dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht: Migrate file failed:/hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Mea<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">s_05(2013-10-11_17-12-02)/007005003.flex lookup failed<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:23:30.804811] E [MSGID: 109023] [dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht: Migrate file failed:/hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Mea<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">s_05(2013-10-11_17-12-02)/006005009.flex lookup failed<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:23:30.805201] E [MSGID: 109023] [dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht: Migrate file failed:/hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Mea<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">s_05(2013-10-11_17-12-02)/005006011.flex lookup failed<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:23:30.880037] E [MSGID: 109023] [dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht: Migrate file failed:/hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Mea<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">s_05(2013-10-11_17-12-02)/005009012.flex lookup failed<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:23:31.038236] E [MSGID: 109023] [dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht: Migrate file failed:/hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Mea<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">s_05(2013-10-11_17-12-02)/003008007.flex lookup failed<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:23:31.259762] E [MSGID: 109023] [dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht: Migrate file failed:/hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Mea<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">s_05(2013-10-11_17-12-02)/004008006.flex lookup failed<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:23:31.333764] E [MSGID: 109023] [dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht: Migrate file failed:/hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Mea<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">s_05(2013-10-11_17-12-02)/007008001.flex lookup failed<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:23:31.340190] E [MSGID: 109023] [dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht: Migrate file failed:/hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Mea<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">s_05(2013-10-11_17-12-02)/006007004.flex lookup failed<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class=""> </span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">Update: one of the rebalance tasks now failed.<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class=""> </span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">@Rafi, I got the same error as Friday except this time with data.</span></div>
</div>
</div>
</blockquote>
<br style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">Packets
that carrying the ping request could be waiting in the queue during the whole time-out period, because of the heavy traffic in the network. I have sent a patch for this. You can track the status here :<span class="Apple-converted-space"> </span></span><a class="moz-txt-link-freetext" href="http://review.gluster.org/11935" style="color: purple; text-decoration: underline; font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);">http://review.gluster.org/11935</a><br style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<br style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<br style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<blockquote cite="mid:2EBB29CB9A8F494FB5253F6AF2E6A1981CDC4A88@trip.uni.lux" type="cite" style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<div class="WordSection1" style="page: WordSection1;">
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class=""> </span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.533167] C [rpc-clnt-ping.c:161:rpc_clnt_ping_timer_expired] 0-live-client-0: server 192.168.123.104:49164 has not responded in the last 42 seconds, disconnecting.<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.533614] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6] (--> /lib64/libgfrpc.so.0(saved_frames_unwin<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">d+0x1de)[0x7fa454bb09be] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c] (--> /lib64/li<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">bgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38] ))))) 0-live-client-0: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2015-08-16 20:23:51.305640 (xid=0x5dd4da)<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.533672] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-live-client-0: remote operation failed [Transport endpoint is not connected]<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.534201] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6] (--> /lib64/libgfrpc.so.0(saved_frames_unwin<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">d+0x1de)[0x7fa454bb09be] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c] (--> /lib64/li<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">bgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38] ))))) 0-live-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2015-08-16 20:23:51.303938 (xid=0x5dd4d7)<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.534347] E [MSGID: 109023] [dht-rebalance.c:1124:dht_migrate_file] 0-live-dht: Migrate file failed: /hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Meas_<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">12(2013-10-12_00-12-55)/007008007.flex: failed to migrate data<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.534413] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6] (--> /lib64/libgfrpc.so.0(saved_frames_unwin<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">d+0x1de)[0x7fa454bb09be] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38]
))))) 0-live-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2015-08-16 20:23:51.303969 (xid=0x5dd4d8)<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.534579] E [MSGID: 109023] [dht-rebalance.c:1124:dht_migrate_file] 0-live-dht: Migrate file failed: /hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Meas_12(2013-10-12_00-12-55)/007009012.flex: failed
to migrate data<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.534676] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa454bb09be] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38] ))))) 0-live-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2015-08-16 20:23:51.313548
(xid=0x5dd4db)<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.534745] E [MSGID: 109023] [dht-rebalance.c:1124:dht_migrate_file] 0-live-dht: Migrate file failed: /hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Meas_12(2013-10-12_00-12-55)/006008011.flex: failed
to migrate data<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.535199] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa454bb09be] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38] ))))) 0-live-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2015-08-16 20:23:51.326369
(xid=0x5dd4dc)<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.535232] E [MSGID: 109023] [dht-rebalance.c:1124:dht_migrate_file] 0-live-dht: Migrate file failed: /hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Meas_12(2013-10-12_00-12-55)/005003001.flex: failed
to migrate data<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.535984] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa454bb09be] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38] ))))) 0-live-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2015-08-16 20:23:51.326437
(xid=0x5dd4dd)<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.536069] E [MSGID: 109023] [dht-rebalance.c:1124:dht_migrate_file] 0-live-dht: Migrate file failed: /hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Meas_12(2013-10-12_00-12-55)/007010012.flex: failed
to migrate data<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.536267] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa454bb09be] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38] ))))) 0-live-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2015-08-16 20:23:51.337240
(xid=0x5dd4de)<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.536339] E [MSGID: 109023] [dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht: Migrate file failed:/hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Meas_08(2013-10-11_20-12-25)/002005012.flex lookup
failed<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.536487] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa454bb09be] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38] ))))) 0-live-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2015-08-16 20:23:51.425254
(xid=0x5dd4df)<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.536685] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa454bb09be] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38] ))))) 0-live-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2015-08-16 20:23:51.738907
(xid=0x5dd4e0)<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.536891] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa454bb09be] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38] ))))) 0-live-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2015-08-16 20:23:51.805096
(xid=0x5dd4e1)<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.537316] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa454bb09be] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38] ))))) 0-live-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2015-08-16 20:23:51.805977
(xid=0x5dd4e2)<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.537735] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7fa454de59e6] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa454bb09be] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa454bb0ace]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fa454bb247c] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa454bb2c38] ))))) 0-live-client-0: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2015-08-16 20:23:52.530107 (xid=0x5dd4e3)<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.538475] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-live-client-0: remote operation failed [Transport endpoint is not connected]<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">The message "E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-live-client-0: remote operation failed [Transport endpoint is not connected]" repeated 4 times between [2015-08-16 20:24:34.538475] and [2015-08-16 20:24:34.538535]<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.538584] E [MSGID: 109023] [dht-rebalance.c:1617:gf_defrag_migrate_single_file] 0-live-dht: Migrate file failed: 002004003.flex lookup failed<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.538904] E [MSGID: 109023] [dht-rebalance.c:1617:gf_defrag_migrate_single_file] 0-live-dht: Migrate file failed: 003009008.flex lookup failed<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.539724] E [MSGID: 109023] [dht-rebalance.c:1965:gf_defrag_get_entry] 0-live-dht: Migrate file failed:/hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Meas_08(2013-10-11_20-12-25)/005009006.flex lookup
failed<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.539820] E [MSGID: 109016] [dht-rebalance.c:2554:gf_defrag_fix_layout] 0-live-dht: Fix layout failed for /hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Meas_08(2013-10-11_20-12-25)<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.540031] E [MSGID: 109016] [dht-rebalance.c:2554:gf_defrag_fix_layout] 0-live-dht: Fix layout failed for /hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.540691] E [MSGID: 114031] [client-rpc-fops.c:251:client3_3_mknod_cbk] 0-live-client-0: remote operation failed. Path: /hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Meas_12(2013-10-12_00-12-55)/002005008.flex
[Transport endpoint is not connected]<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.541152] E [MSGID: 114031] [client-rpc-fops.c:251:client3_3_mknod_cbk] 0-live-client-0: remote operation failed. Path: /hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Meas_12(2013-10-12_00-12-55)/005004009.flex
[Transport endpoint is not connected]<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.541331] E [MSGID: 114031] [client-rpc-fops.c:251:client3_3_mknod_cbk] 0-live-client-0: remote operation failed. Path: /hcs/hcs/OperaArchiveCol/SK 20131011_Oligo_Rot_lowConc_P1/Meas_12(2013-10-12_00-12-55)/007005011.flex
[Transport endpoint is not connected]<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.541486] E [MSGID: 109016] [dht-rebalance.c:2554:gf_defrag_fix_layout] 0-live-dht: Fix layout failed for /hcs/hcs/OperaArchiveCol<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.541572] E [MSGID: 109016] [dht-rebalance.c:2554:gf_defrag_fix_layout] 0-live-dht: Fix layout failed for /hcs/hcs<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">[2015-08-16 20:24:34.541639] E [MSGID: 109016] [dht-rebalance.c:2554:gf_defrag_fix_layout] 0-live-dht: Fix layout failed for /hcs<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class=""> </span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">Any help would be greatly appreciated.</span></div>
</div>
</div>
</blockquote>
<span style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">CCing
dht teams to give you better idea about why rebalance failed/ and about huge memory consumption by rebalance process (200GB RAM) .</span><br style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<br style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">Regards</span><br style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">Rafi
KC</span><br style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<br style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<br style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<br style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<blockquote cite="mid:2EBB29CB9A8F494FB5253F6AF2E6A1981CDC4A88@trip.uni.lux" type="cite" style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<div class="WordSection1" style="page: WordSection1;">
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class=""> </span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">Thanks,<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class=""> </span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">--<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span lang="EN-US" class="">Christophe<o:p class=""></o:p></span></div>
<div class="">
<div class="">
<p style="margin-right: 0cm; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; line-height: 12pt;" class="">
<b class=""><span style="font-size: 10pt; font-family: Arial, sans-serif; color: rgb(61, 59, 59);" class="">Dr Christophe Trefois, Dipl.-Ing.</span></b><span class="apple-converted-space"><span style="font-size: 10pt; font-family: Arial, sans-serif; color: rgb(33, 33, 33);" class=""> </span></span><span style="font-size: 10pt; font-family: Arial, sans-serif; color: rgb(33, 33, 33);" class=""><br class="">
</span><span style="font-size: 7.5pt; font-family: Arial, sans-serif; color: rgb(61, 59, 59);" class="">Technical Specialist / Post-Doc</span><span style="font-size: 10pt; font-family: Arial, sans-serif; color: rgb(33, 33, 33);" class=""><o:p class=""></o:p></span></p>
<p style="margin-right: 0cm; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; line-height: 12pt;" class="">
<b class=""><span style="font-size: 7.5pt; font-family: Arial, sans-serif; color: rgb(61, 59, 59);" class="">UNIVERSITÉ DU LUXEMBOURG</span></b><span style="font-size: 7.5pt; font-family: Arial, sans-serif;" class=""><br class="">
<br class="">
</span><b class=""><span style="font-size: 7.5pt; font-family: Arial, sans-serif; color: rgb(61, 59, 59);" class="">LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE</span></b><span style="font-size: 7.5pt; font-family: Arial, sans-serif;" class=""><br class="">
</span><span style="font-size: 7.5pt; font-family: Arial, sans-serif; color: rgb(61, 59, 59);" class="">Campus Belval | House of Biomedicine<span class="apple-converted-space"> </span><br class="">
<span class="apple-converted-space">6, avenue du Swing </span><br class="">
L-4367 Belvaux<span class="apple-converted-space"> </span></span><span style="font-size: 7.5pt; font-family: Arial, sans-serif;" class=""><br class="">
</span><span style="font-size: 7.5pt; font-family: Arial, sans-serif; color: rgb(61, 59, 59);" class="">T:<span class="apple-converted-space"> </span>+352 46 66 44 6124</span><span class="apple-converted-space"><span style="font-size: 7.5pt; font-family: Arial, sans-serif;" class=""> </span></span><span style="font-size: 7.5pt; font-family: Arial, sans-serif;" class=""><br class="">
</span><span style="font-size: 7.5pt; font-family: Arial, sans-serif; color: rgb(61, 59, 59);" class="">F:<span class="apple-converted-space"> </span>+352 46 66 44 6949</span><span class="apple-converted-space"><span style="font-size: 7.5pt; font-family: Arial, sans-serif;" class=""> </span></span><span style="font-size: 7.5pt; font-family: Arial, sans-serif;" class=""><br class="">
<a moz-do-not-send="true" href="http://www.uni.lu/lcsb" style="color: purple; text-decoration: underline;" class=""><span style="color: rgb(0, 109, 189);" class="">http://www.uni.lu/lcsb</span></a><o:p class=""></o:p></span></p>
<p style="margin-right: 0cm; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; line-height: 12pt;" class="">
<span lang="EN-US" style="font-size: 7pt; font-family: Arial, sans-serif; color: rgb(61, 59, 59);" class="">----<br class="">
This message is confidential and may contain privileged information.<span class="apple-converted-space"> </span><br class="">
It is intended for the named recipient only.<span class="apple-converted-space"> </span><br class="">
If you receive it in error please notify me and permanently delete the original message and any copies.<span class="apple-converted-space"> </span><br class="">
</span><span style="font-size: 7pt; font-family: Arial, sans-serif; color: rgb(61, 59, 59);" class="">----<o:p class=""></o:p></span></p>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="" class=""> <span class="apple-converted-space"> </span><o:p class=""></o:p></span></div>
</div>
</div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
</div>
</div>
</blockquote>
<br style="font-family: Helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<br class="Apple-interchange-newline">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>