<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"纯文本 Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.5pt;
font-family:"Calibri","sans-serif";}
span.Char
{mso-style-name:"纯文本 Char";
mso-style-priority:99;
mso-style-link:纯文本;
font-family:"Calibri","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
/* Page Definitions */
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="blue" vlink="purple" style="text-justify-trim:punctuation">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Following is backTrace of the etc-glusterfs-glusterd.vol.log file, thanks<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:32.228232] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c, lock held by: ea17d7f9-d737-4472-ab9a-feed3cfac57c<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:32.228267] E [glusterd-syncop.c:1023:gd_sync_task_begin] 0-management: Unable to acquire lock<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.802737] E [rpc-clnt.c:207:call_bail] 0-management: bailing out frame type(GLUSTERD-DUMP) op(DUMP(1)) xid = 0x280776x sent = 2015-05-07 11:46:31.743195. timeout = 600<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.802772] E [glusterd-handshake.c:1074:__glusterd_peer_dump_version_cbk] 0-: Error through RPC layer, retry again later<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.802836] W [socket.c:514:__socket_rwv] 0-management: readv failed (No data available)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.844795] E [rpc-clnt.c:368:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13d) [0x39d9a0ec8d] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x39d9a0e7f3] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)
[0x39d9a0e70e]))) 0-management: forced unwinding frame type(glusterd mgmt) op(--(1)) called at 2015-05-07 11:47:47.530276 (xid=0x280777x)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.844833] E [rpc-clnt.c:368:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13d) [0x39d9a0ec8d] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x39d9a0e7f3] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)
[0x39d9a0e70e]))) 0-management: forced unwinding frame type(glusterd mgmt) op(--(1)) called at 2015-05-07 11:48:48.544182 (xid=0x280778x)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.844854] E [rpc-clnt.c:368:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13d) [0x39d9a0ec8d] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x39d9a0e7f3] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)
[0x39d9a0e70e]))) 0-management: forced unwinding frame type(glusterd mgmt) op(--(1)) called at 2015-05-07 11:53:48.340948 (xid=0x280779x)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.844861] E [glusterd-syncop.c:715:gd_lock_op_phase] 0-management: Failed to acquire lock<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.844898] E [glusterd-syncop.c:715:gd_lock_op_phase] 0-management: Failed to acquire lock<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.844960] E [rpc-clnt.c:368:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13d) [0x39d9a0ec8d] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x39d9a0e7f3] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)
[0x39d9a0e70e]))) 0-management: forced unwinding frame type(glusterd mgmt) op(--(1)) called at 2015-05-07 11:55:47.461721 (xid=0x280780x)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.848677] I [socket.c:3027:socket_submit_request] 0-management: not connected (priv->connected = 0)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.848697] W [rpc-clnt.c:1488:rpc_clnt_submit] 0-management: failed to submit rpc-request (XID: 0x280781x Program: glusterd mgmt, ProgVers: 2, Proc: 2) to rpc-transport (management)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.848744] W [rpc-clnt.c:1488:rpc_clnt_submit] 0-management: failed to submit rpc-request (XID: 0x280782x Program: glusterd mgmt, ProgVers: 2, Proc: 2) to rpc-transport (management)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.848770] E [glusterd-syncop.c:715:gd_lock_op_phase] 0-management: Failed to acquire lock<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.848812] I [socket.c:3101:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.848822] E [rpcsvc.c:1113:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1x, Program: GlusterD svc cli, ProgVers: 2, Proc: 27) to rpc-transport (socket.management)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.848852] E [glusterd-utils.c:586:glusterd_submit_reply] 0-: Reply submission failed<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.848868] E [glusterd-utils.c:365:glusterd_unlock] 0-management: Cluster lock not held!<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.848885] E [glusterd-syncop.c:715:gd_lock_op_phase] 0-management: Failed to acquire lock<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.848909] I [socket.c:3101:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.848918] E [rpcsvc.c:1113:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1x, Program: GlusterD svc cli, ProgVers: 2, Proc: 27) to rpc-transport (socket.management)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.848933] E [glusterd-utils.c:586:glusterd_submit_reply] 0-: Reply submission failed<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.848942] E [glusterd-utils.c:365:glusterd_unlock] 0-management: Cluster lock not held!<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.849903] I [socket.c:3101:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.849913] E [rpcsvc.c:1113:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1x, Program: GlusterD svc cli, ProgVers: 2, Proc: 27) to rpc-transport (socket.management)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.849924] E [glusterd-utils.c:586:glusterd_submit_reply] 0-: Reply submission failed<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:41.849935] E [glusterd-utils.c:365:glusterd_unlock] 0-management: Cluster lock not held!<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:49.706897] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:50.640610] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:51.689354] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:51.689903] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:51.690286] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:51.690660] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:51.691090] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:51.691519] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:56:51.691868] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:57:40.505580] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: 674a78b5-0590-48d4-8752-d4608832ed1d, lock held by: ea17d7f9-d737-4472-ab9a-feed3cfac57c<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:57:40.505712] E [glusterd-op-sm.c:5445:glusterd_op_sm] 0-management: handler returned: -1<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:57:49.848485] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:57:50.614735] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:57:51.737376] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:57:51.755171] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:57:51.755740] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:57:51.756165] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:57:51.756637] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:57:51.757055] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:57:51.757453] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:58:25.351136] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: 674a78b5-0590-48d4-8752-d4608832ed1d, lock held by: ea17d7f9-d737-4472-ab9a-feed3cfac57c<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:58:25.367614] E [glusterd-op-sm.c:5445:glusterd_op_sm] 0-management: handler returned: -1<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:58:53.655999] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:58:53.675171] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:58:54.068535] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:58:56.079742] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:58:56.092418] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:58:56.101670] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:58:56.922748] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:58:56.929828] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:58:56.939039] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:59:54.303643] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:59:54.316523] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:59:54.324661] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:59:54.334902] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:59:54.346829] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:59:54.358660] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:59:54.359401] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:59:54.365991] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 11:59:54.371101] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 12:00:50.916410] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: 674a78b5-0590-48d4-8752-d4608832ed1d, lock held by: ea17d7f9-d737-4472-ab9a-feed3cfac57c<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 12:00:50.943695] E [glusterd-op-sm.c:5445:glusterd_op_sm] 0-management: handler returned: -1<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 12:00:50.954392] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 12:00:55.394885] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 12:00:55.403735] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 12:00:55.431832] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 12:00:55.441505] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 12:00:55.453781] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 12:00:55.459636] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 12:00:55.467670] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">[2015-05-07 12:00:55.474254] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">Best Regards <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">Young Yang</span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">On 05/08/2015 10:37 AM, vyyy</span><span style="font-family:宋体">杨雨阳</span><span lang="EN-US"> wrote:<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> Hi,<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> We've been using glusterfs for 1 years</span><span style="font-family:宋体">，</span><span lang="EN-US">It</span><span style="font-family:宋体">’</span><span lang="EN-US">s greadt except occasional
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> glusterd crashed, and just need restart<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> But nowadays, Glusterd crashed more frequently, sometimes several
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> glusterds crashed and cause split-brain and gfid different<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> Our environment: Centos 6.4 Gluster 3.4.2<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> [root@SVR4666HW2285 ~]# service glusterd status<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> glusterd dead but pid file exists<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> When glusterd crashed, We can see linux swap exhausted and then
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> released<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">We would require the backtrace and the glusterd log file to further analyze it.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">~Atin<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> Then glustershd.log filled with </span>
<span lang="EN-US" style="font-family:"Courier New"">“</span><span lang="EN-US">readv failed</span><span lang="EN-US" style="font-family:"Courier New"">”</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> [2015-05-07 12:40:53.580605] W [socket.c:514:__socket_rwv]
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> 0-glusterfs: readv failed (No data available)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> [2015-05-07 12:40:53.609217] W <o:p>
</o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> [socket.c:1962:__socket_proto_state_machine] 0-glusterfs: reading from
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> socket failed. Error (No data available), peer (127.0.0.1:24007)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> [2015-05-07 12:41:03.837798] E [socket.c:2157:socket_connect_finish]
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> 0-glusterfs: connection to 127.0.0.1:24007 failed (Connection refused)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> [2015-05-07 12:41:03.837853] W [socket.c:514:__socket_rwv]
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> 0-glusterfs: readv failed (No data available)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> [2015-05-07 12:41:06.844262] W [socket.c:514:__socket_rwv]
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> 0-glusterfs: readv failed (No data available)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> [2015-05-07 12:41:09.847699] W [socket.c:514:__socket_rwv]
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> 0-glusterfs: readv failed (No data available)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> [2015-05-07 12:41:12.854243] W [socket.c:514:__socket_rwv]
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> 0-glusterfs: readv failed (No data available)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> Any help appreciated.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> Best Regards<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> Young Yang<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> _______________________________________________<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> Gluster-users mailing list<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <a href="mailto:Gluster-users@gluster.org">
<span style="color:windowtext;text-decoration:none">Gluster-users@gluster.org</span></a><o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <a href="http://www.gluster.org/mailman/listinfo/gluster-users">
<span style="color:windowtext;text-decoration:none">http://www.gluster.org/mailman/listinfo/gluster-users</span></a><o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">> <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">--<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">~Atin<o:p></o:p></span></p>
</div>
</body>
</html>