<HTML><HEAD>
<STYLE id=eMClientCss>BLOCKQUOTE.cite {
PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px
}
BLOCKQUOTE.cite2 {
MARGIN-TOP: 3px; PADDING-TOP: 0px; PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px
}
.plain PRE {
FONT-SIZE: 100%; FONT-FAMILY: monospace; WHITE-SPACE: pre-wrap; FONT-WEIGHT: normal; FONT-STYLE: normal
}
.plain TT {
FONT-SIZE: 100%; FONT-FAMILY: monospace; WHITE-SPACE: pre-wrap; FONT-WEIGHT: normal; FONT-STYLE: normal
}
A IMG {
BORDER-TOP: 0px; BORDER-RIGHT: 0px; BORDER-BOTTOM: 0px; BORDER-LEFT: 0px
}
#xbc8eca36ab9140d6abdd836d39e48bc8 {
FONT-SIZE: 12pt; FONT-FAMILY: Times New Roman
}
.plain PRE {
FONT-SIZE: 12pt; FONT-FAMILY: Times New Roman
}
.plain TT {
FONT-SIZE: 12pt; FONT-FAMILY: Times New Roman
}
BODY {
FONT-SIZE: 12pt; FONT-FAMILY: Times New Roman
}
#x10f3d3182c59401ab3ec86d7186be92b DIV.MsoNormal {
FONT-SIZE: 11pt; FONT-FAMILY: "Calibri","sans-serif"; MARGIN: 0in 0in 0pt
}
#x10f3d3182c59401ab3ec86d7186be92b LI.MsoNormal {
FONT-SIZE: 11pt; FONT-FAMILY: "Calibri","sans-serif"; MARGIN: 0in 0in 0pt
}
#xbc8eca36ab9140d6abdd836d39e48bc8 DIV.MsoNormal {
FONT-SIZE: 11pt; FONT-FAMILY: "Calibri","sans-serif"; MARGIN: 0in 0in 0pt
}
#xbc8eca36ab9140d6abdd836d39e48bc8 LI.MsoNormal {
FONT-SIZE: 11pt; FONT-FAMILY: "Calibri","sans-serif"; MARGIN: 0in 0in 0pt
}
#x10f3d3182c59401ab3ec86d7186be92b SPAN.MsoHyperlink {
TEXT-DECORATION: underline; COLOR: blue; mso-style-priority: 99
}
#xbc8eca36ab9140d6abdd836d39e48bc8 SPAN.MsoHyperlink {
TEXT-DECORATION: underline; COLOR: blue; mso-style-priority: 99
}
#x10f3d3182c59401ab3ec86d7186be92b SPAN.MsoHyperlinkFollowed {
TEXT-DECORATION: underline; COLOR: purple; mso-style-priority: 99
}
#xbc8eca36ab9140d6abdd836d39e48bc8 SPAN.MsoHyperlinkFollowed {
TEXT-DECORATION: underline; COLOR: purple; mso-style-priority: 99
}
</STYLE>
<STYLE title=signatureStyle>#x99d0c83487814dbda3bd1696eb29eaa0 P.MsoNormal {
FONT-SIZE: 11pt; FONT-FAMILY: "Calibri","sans-serif"; MARGIN: 0in 0in 0pt
}
DIV.MsoNormal {
FONT-SIZE: 11pt; FONT-FAMILY: "Calibri","sans-serif"; MARGIN: 0in 0in 0pt
}
LI.MsoNormal {
FONT-SIZE: 11pt; FONT-FAMILY: "Calibri","sans-serif"; MARGIN: 0in 0in 0pt
}
#x99d0c83487814dbda3bd1696eb29eaa0 A:link {
TEXT-DECORATION: underline; COLOR: blue; mso-style-priority: 99
}
SPAN.MsoHyperlink {
TEXT-DECORATION: underline; COLOR: blue; mso-style-priority: 99
}
#x99d0c83487814dbda3bd1696eb29eaa0 A:visited {
TEXT-DECORATION: underline; COLOR: purple; mso-style-priority: 99
}
SPAN.MsoHyperlinkFollowed {
TEXT-DECORATION: underline; COLOR: purple; mso-style-priority: 99
}
#x99d0c83487814dbda3bd1696eb29eaa0 SPAN.EmailStyle17 {
FONT-FAMILY: "Calibri","sans-serif"; COLOR: windowtext; mso-style-type: personal-compose
}
#x99d0c83487814dbda3bd1696eb29eaa0 .MsoChpDefault {
FONT-FAMILY: "Calibri","sans-serif"; mso-style-type: export-only
}
#x99d0c83487814dbda3bd1696eb29eaa0 DIV.WordSection1 {
page: WordSection1
}
</STYLE>
</HEAD>
<BODY>
<DIV>I upgraded my storage server from 3.6.3 to 3.6.6 and am now having issues. My setup (4x2) is shown below. One of the bricks (gfs01a) has a very high cpu-load even though the load on the other 3-bricks (gfs01b, gfs02a, gfs02b) is almost zero. The FUSE mounted partition is extremely slow and basically unuseable since the upgrade. I am getting a lot of the messages shown below in the logs on gfs01a and gfs01b. Nothing out of the ordinary is showing up on the gfs02a/gfs02b bricks.</DIV>
<DIV> </DIV>
<DIV>Can someone help?</DIV>
<DIV> </DIV>
<DIV><SPAN id=x10f3d3182c59401ab3ec86d7186be92b>[root@gfs01b glusterfs]# gluster volume info homegfs<BR> <BR>Volume Name: homegfs<BR>Type: Distributed-Replicate<BR>Volume ID: 1e32672a-f1b7-4b58-ba94-58c085e59071<BR>Status: Started<BR>Number of Bricks: 4 x 2 = 8<BR>Transport-type: tcp<BR>Bricks:<BR>Brick1: gfsib01a.corvidtec.com:/data/brick01a/homegfs<BR>Brick2: gfsib01b.corvidtec.com:/data/brick01b/homegfs<BR>Brick3: gfsib01a.corvidtec.com:/data/brick02a/homegfs<BR>Brick4: gfsib01b.corvidtec.com:/data/brick02b/homegfs<BR>Brick5: gfsib02a.corvidtec.com:/data/brick01a/homegfs<BR>Brick6: gfsib02b.corvidtec.com:/data/brick01b/homegfs<BR>Brick7: gfsib02a.corvidtec.com:/data/brick02a/homegfs<BR>Brick8: gfsib02b.corvidtec.com:/data/brick02b/homegfs<BR>Options Reconfigured:<BR>changelog.rollover-time: 15<BR>changelog.fsync-interval: 3<BR>changelog.changelog: on<BR>geo-replication.ignore-pid-check: on<BR>geo-replication.indexing: off<BR>storage.owner-gid: 100<BR>network.ping-timeout: 10<BR>server.allow-insecure: on<BR>performance.write-behind-window-size: 128MB<BR>performance.cache-size: 128MB<BR>performance.io-thread-count: 32<BR>server.manage-gids: on</SPAN></DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV><SPAN id=xbc8eca36ab9140d6abdd836d39e48bc8>
<DIV>[root@<STRONG>gfs01a </STRONG>glusterfs]# tail -f cli.log <BR>[2015-10-17 16:05:44.299933] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now<BR>[2015-10-17 16:05:44.331233] I [input.c:36:cli_batch] 0-: Exiting with: 0<BR>[2015-10-17 16:06:33.397631] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now<BR>[2015-10-17 16:06:33.432970] I [input.c:36:cli_batch] 0-: Exiting with: 0<BR>[2015-10-17 16:11:22.441290] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now<BR>[2015-10-17 16:11:22.472227] I [input.c:36:cli_batch] 0-: Exiting with: 0<BR>[2015-10-17 16:15:44.176391] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now<BR>[2015-10-17 16:15:44.205064] I [input.c:36:cli_batch] 0-: Exiting with: 0<BR>[2015-10-17 16:16:33.366424] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now<BR>[2015-10-17 16:16:33.377160] I [input.c:36:cli_batch] 0-: Exiting with: 0<BR></DIV>
<DIV> </DIV>
<DIV>[root@<STRONG>gfs01a</STRONG> glusterfs]# tail etc-glusterfs-glusterd.vol.log <BR>[2015-10-17 15:56:33.177207] I [glusterd-handler.c:3836:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Source<BR>[2015-10-17 16:01:22.303635] I [glusterd-handler.c:3836:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Software<BR>[2015-10-17 16:05:44.320555] I [glusterd-handler.c:3836:__glusterd_handle_status_volume] 0-management: Received status volume req for volume homegfs<BR>[2015-10-17 16:06:17.204783] W [rpcsvc.c:254:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330)<BR>[2015-10-17 16:06:17.204811] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully<BR>[2015-10-17 16:06:33.408695] I [glusterd-handler.c:3836:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Source<BR>[2015-10-17 16:11:22.462374] I [glusterd-handler.c:3836:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Software<BR>[2015-10-17 16:12:30.608092] E [glusterd-op-sm.c:207:glusterd_get_txn_opinfo] 0-: Unable to get transaction opinfo for transaction ID : d143b66b-2ac9-4fd9-8635-fe1eed41d56b<BR>[2015-10-17 16:15:44.198292] I [glusterd-handler.c:3836:__glusterd_handle_status_volume] 0-management: Received status volume req for volume homegfs<BR>[2015-10-17 16:16:33.368170] I [glusterd-handler.c:3836:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Source<BR></DIV></SPAN></DIV>
<DIV> </DIV>
<DIV>[root@<STRONG>gfs01b</STRONG> glusterfs]# tail -f glustershd.log <BR>[2015-10-17 16:11:45.996447] I [afr-self-heal-metadata.c:54:__afr_selfheal_metadata_do] 0-homegfs-replicate-1: performing metadata selfheal on 0a65d73a-a416-418e-92f0-5cec7d240433<BR>[2015-10-17 16:11:46.030947] I [afr-self-heal-common.c:476:afr_log_selfheal] 0-homegfs-replicate-1: Completed metadata selfheal on 0a65d73a-a416-418e-92f0-5cec7d240433. source=1 sinks=0 <BR>[2015-10-17 16:11:46.031241] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-homegfs-client-3: remote operation failed: No such file or directory. Path: <gfid:d2714957-0c83-4ab2-8cfc-1931c8e9d0bf> (d2714957-0c83-4ab2-8cfc-1931c8e9d0bf)<BR>[2015-10-17 16:11:46.031633] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-homegfs-client-3: remote operation failed: No such file or directory. Path: <gfid:87c5f875-c3e7-4b14-807a-4e6d940750fc> (87c5f875-c3e7-4b14-807a-4e6d940750fc)<BR>[2015-10-17 16:11:47.043367] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-homegfs-client-3: remote operation failed: No such file or directory. Path: <gfid:d2714957-0c83-4ab2-8cfc-1931c8e9d0bf> (d2714957-0c83-4ab2-8cfc-1931c8e9d0bf)<BR>[2015-10-17 16:11:47.054199] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-homegfs-client-3: remote operation failed: No such file or directory. Path: <gfid:87c5f875-c3e7-4b14-807a-4e6d940750fc> (87c5f875-c3e7-4b14-807a-4e6d940750fc)<BR>[2015-10-17 16:12:48.001869] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-homegfs-client-3: remote operation failed: No such file or directory. Path: <gfid:d2714957-0c83-4ab2-8cfc-1931c8e9d0bf> (d2714957-0c83-4ab2-8cfc-1931c8e9d0bf)<BR>[2015-10-17 16:12:48.012671] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-homegfs-client-3: remote operation failed: No such file or directory. Path: <gfid:87c5f875-c3e7-4b14-807a-4e6d940750fc> (87c5f875-c3e7-4b14-807a-4e6d940750fc)<BR>[2015-10-17 16:13:49.011591] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-homegfs-client-3: remote operation failed: No such file or directory. Path: <gfid:d2714957-0c83-4ab2-8cfc-1931c8e9d0bf> (d2714957-0c83-4ab2-8cfc-1931c8e9d0bf)<BR>[2015-10-17 16:13:49.018600] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-homegfs-client-3: remote operation failed: No such file or directory. Path: <gfid:87c5f875-c3e7-4b14-807a-4e6d940750fc> (87c5f875-c3e7-4b14-807a-4e6d940750fc)<BR></DIV>
<DIV>[root@<STRONG>gfs01b </STRONG>glusterfs]# tail cli.log <BR>[2015-10-16 10:52:16.002922] I [input.c:36:cli_batch] 0-: Exiting with: 0<BR>[2015-10-16 10:52:16.167432] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now<BR>[2015-10-16 10:52:18.248024] I [input.c:36:cli_batch] 0-: Exiting with: 0<BR>[2015-10-17 16:12:30.607603] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now<BR>[2015-10-17 16:12:30.628810] I [input.c:36:cli_batch] 0-: Exiting with: 0<BR>[2015-10-17 16:12:33.992818] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now<BR>[2015-10-17 16:12:33.998944] I [input.c:36:cli_batch] 0-: Exiting with: 0<BR>[2015-10-17 16:12:38.604461] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now<BR>[2015-10-17 16:12:38.605532] I [cli-rpc-ops.c:588:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0<BR>[2015-10-17 16:12:38.605659] I [input.c:36:cli_batch] 0-: Exiting with: 0</DIV>
<DIV> </DIV>
<DIV>[root@<STRONG>gfs01b</STRONG> glusterfs]# tail etc-glusterfs-glusterd.vol.log <BR></DIV>
<DIV>[2015-10-16 14:29:56.495120] E [rpcsvc.c:617:rpcsvc_handle_rpc_call] 0-rpc-service: Request received from non-privileged port. Failing request<BR>[2015-10-16 14:29:59.369109] E [rpcsvc.c:617:rpcsvc_handle_rpc_call] 0-rpc-service: Request received from non-privileged port. Failing request<BR>[2015-10-16 14:29:59.512093] E [rpcsvc.c:617:rpcsvc_handle_rpc_call] 0-rpc-service: Request received from non-privileged port. Failing request<BR>[2015-10-16 14:30:02.383574] E [rpcsvc.c:617:rpcsvc_handle_rpc_call] 0-rpc-service: Request received from non-privileged port. Failing request<BR>[2015-10-16 14:30:02.529206] E [rpcsvc.c:617:rpcsvc_handle_rpc_call] 0-rpc-service: Request received from non-privileged port. Failing request<BR>[2015-10-16 16:01:20.389100] E [rpcsvc.c:617:rpcsvc_handle_rpc_call] 0-rpc-service: Request received from non-privileged port. Failing request<BR>[2015-10-17 16:12:30.611161] W [glusterd-op-sm.c:4066:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<BR>[2015-10-17 16:12:30.612433] I [glusterd-handler.c:3836:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Software<BR>[2015-10-17 16:12:30.618444] I [glusterd-handler.c:3836:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Source<BR>[2015-10-17 16:12:30.624005] I [glusterd-handler.c:3836:__glusterd_handle_status_volume] 0-management: Received status volume req for volume homegfs<BR>[2015-10-17 16:12:33.993869] I [glusterd-handler.c:3836:__glusterd_handle_status_volume] 0-management: Received status volume req for volume homegfs<BR>[2015-10-17 16:12:38.605389] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<BR></DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>[root@gfs01b glusterfs]# gluster volume status homegfs<BR>Status of volume: homegfs<BR>Gluster process Port Online Pid<BR>------------------------------------------------------------------------------<BR>Brick gfsib01a.corvidtec.com:/data/brick01a/homegfs 49152 Y 3820<BR>Brick gfsib01b.corvidtec.com:/data/brick01b/homegfs 49152 Y 3808<BR>Brick gfsib01a.corvidtec.com:/data/brick02a/homegfs 49153 Y 3825<BR>Brick gfsib01b.corvidtec.com:/data/brick02b/homegfs 49153 Y 3813<BR>Brick gfsib02a.corvidtec.com:/data/brick01a/homegfs 49152 Y 3967<BR>Brick gfsib02b.corvidtec.com:/data/brick01b/homegfs 49152 Y 3952<BR>Brick gfsib02a.corvidtec.com:/data/brick02a/homegfs 49153 Y 3972<BR>Brick gfsib02b.corvidtec.com:/data/brick02b/homegfs 49153 Y 3957<BR>NFS Server on localhost 2049 Y 3822<BR>Self-heal Daemon on localhost N/A Y 3827<BR>NFS Server on 10.200.70.1 2049 Y 3834<BR>Self-heal Daemon on 10.200.70.1 N/A Y 3839<BR>NFS Server on gfsib02a.corvidtec.com 2049 Y 3981<BR>Self-heal Daemon on gfsib02a.corvidtec.com N/A Y 3986<BR>NFS Server on gfsib02b.corvidtec.com 2049 Y 3966<BR>Self-heal Daemon on gfsib02b.corvidtec.com N/A Y 3971<BR> <BR>Task Status of Volume homegfs<BR>------------------------------------------------------------------------------<BR>Task : Rebalance <BR>ID : 58b6cc76-c29c-4695-93fe-c42b1112e171<BR>Status : completed </DIV>
<DIV> </DIV>
<DIV><BR> </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV id=signature_old>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: Times New Roman"><SPAN id=xc493ce70c1c64653841aa9f5ac6ad16e><SPAN id=x834d59d53b4a45d8a1f25809459ef478>
<DIV class=WordSection1><SPAN id=x99d0c83487814dbda3bd1696eb29eaa0>
<P class=MsoNormal>======================== </P>
<P class=MsoNormal> </P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Times New Roman","serif"'>David F. Robinson, Ph.D.</SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Times New Roman","serif"'>President - Corvid Technologies</SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Times New Roman","serif"'>145 Overhill Drive</SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Times New Roman","serif"'>Mooresville, NC 28117</SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Times New Roman","serif"'>704.799.6944 x101 [Office]</SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Times New Roman","serif"'>704.252.1310 [Cell]</SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Times New Roman","serif"'>704.799.7974 [Fax]</SPAN></P>
<P class=MsoNormal>d<A href="mailto:David.Robinson@corvidtec.com">avid.robinson@corvidtec.com</A></P>
<P class=MsoNormal><A href="http://www.corvidtec.com/"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Times New Roman","serif"; COLOR: blue'>http://www.corvidtec.com</SPAN></A></P></SPAN></DIV></SPAN></SPAN></DIV></DIV>
<DIV> </DIV></BODY></HTML>