<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=gb2312">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from rtf -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<font face="Calibri" size="2"><span style="font-size:10.5pt;">
<div align="left" style="text-align:justify;">Hi Gluster experts,</div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;">Good day!</div>
<div align="left" style="text-align:justify;">I am checking one issue related to Glusterfs, and found following issue.</div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;">Description of issue:</div>
<div align="left" style="text-align:justify;">1>our cluster env has three storage node , sn-0(169.254.0.24)and sn-1(169.254.0.23) is deployed as replicate storage nodes. Sn-2 is quorum node.</div>
<div align="left" style="text-align:justify;">2>stop sn-0 node(so sn-0 become unreachable suddenly), and execute mound command: time mount -t glusterfs 169.254.0.24:/log /mnt/test2 -o rw,backupvolfile-server=169.254.0.23</div>
<div align="left" style="text-align:justify;">3> this command will not finish until 5 minutes pass.</div>
<div align="left" style="text-align:justify;">4> wait some time (10 minutes) , execute this mount command again, the same command will finish within several seconds.</div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;">My analysis:</div>
<div align="left" style="text-align:justify;">On the second time the mount command could return quickly, while the first time mount hang 5 minutes, I compare the glusterfs log file, and find </div>
<div align="left" style="text-align:justify;">1>on the first time the rpc client reconnect keeps trying until following prints:</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
[2016-11-17 02:50:23.593340] E [socket.c:2282:socket_connect_finish] 0-log-client-0: connection to 169.254.0.24:24007 failed (Connection timed out)</div>
<div align="left" style="text-indent:31.5pt;text-align:justify;"> [2016-11-17 02:50:23.593360] T [socket.c:738:__socket_disconnect] 0-log-client-0: disconnecting 0xe2a280, state=0 gen=0 sock=12</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
[2016-11-17 02:50:23.593370] D [socket.c:719:__socket_shutdown] 0-log-client-0: shutdown() returned -1. Transport endpoint is not connected</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
[2016-11-17 02:50:23.593378] D [socket.c:2359:socket_event_handler] 0-transport: disconnecting now</div>
<div align="left" style="text-align:justify;padding-left:18pt;">but the begin time of this retry is:</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
[2016-11-17 02:48:20.416900] T [rpc-clnt.c:418:rpc_clnt_reconnect] 0-log-client-0: attempting reconnect</div>
<div align="left" style="text-align:justify;padding-left:18pt;">Glusterfs process try a lot of times and the time elapse is around 2 minutes</div>
<div align="left" style="text-align:justify;">2>On the second time the rpc client also keeps reconnecting but each time it returns very quickly with following prints:</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
[2016-11-17 04:27:06.045700] E [socket.c:2282:socket_connect_finish] 0-log-client-0: connection to 169.254.0.24:24007 failed (No route to host)</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
[2016-11-17 04:27:06.045751] T [socket.c:738:__socket_disconnect] 0-log-client-0: disconnecting 0xffc4a0, state=0 gen=0 sock=12</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
[2016-11-17 04:27:06.045760] D [socket.c:719:__socket_shutdown] 0-log-client-0: shutdown() returned -1. Transport endpoint is not connected</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
[2016-11-17 04:27:06.045766] D [socket.c:2359:socket_event_handler] 0-transport: disconnecting now</div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;">Questions:</div>
<div align="left" style="text-align:justify;"> <b>W</b><b>hy </b><b>socket_connect_finish is called much later for the first time mount? I assume it is because</b><b> in the first mount</b><b> connect() timeout</b><b> </b><b>is slowly </b><b>and the second
time connect() return much faster? </b></div>
<div align="left" style="text-align:justify;"><b> Is there any</b><b> way to control the </b><b>rpc_clnt_reconnect</b><b> times? Because seems this is a waste, no need to retry so many times.</b></div>
<div align="left" style="text-align:justify;padding-left:18pt;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;padding-left:7.45pt;">Attachment: two times mount trace log of glusterfs.</div>
<div align="left" style="text-align:justify;padding-left:7.45pt;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;padding-left:7.45pt;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;"><font face="Arial">Best regards,<br>
<b>Cynthia </b><font face="ËÎÌå"><b>£¨ÖÜÁÕ£©</b></font></font></div>
<div align="left" style="text-align:justify;"><font face="Arial">MBB SM HETRAN SW3 MATRIX </font></div>
<div align="left" style="text-align:justify;"><font face="Arial">Storage <br>
Mobile: +86 (0)18657188311</font></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
</span></font>
</body>
</html>