<html><head></head><body>You cannot join a trusted pool, you must be invited. <br>
<br>
In other words, a server inside the trusted pool must probe the new server, and that new server must not be part of another pool. <br><br><div class="gmail_quote">On June 4, 2015 7:24:28 PM PDT, "vyyy杨雨阳" <yuyangyang@Ctrip.com> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<pre class="k9mail">Can anyone help me?<br /><br />New nodes still can't join the cluster.<br />The new nodes can peer each other, just not exist cluster nodes.<br />I can not find any network problem, there were no firewall, port 24007 is connected, there were all at the same subnet...<br /><br />Thanks <br /><br /><br /><br />[root@SVR6993HW2285 glusterfs]# more etc-glusterfs-glusterd.vol.log<br />[2015-06-05 01:59:02.276107] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glust<br />erd version 3.6.3 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)<br />[2015-06-05 01:59:02.318477] I [glusterd.c:1214:init] 0-management: Maximum allowed open file descriptors set to 65536<br />[2015-06-05 01:59:02.318524] I [glusterd.c:1259:init] 0-management: Using /var/lib/glusterd as working directory<br />[2015-06-05 01:59:02.329603] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-trans<br />port/<a
href="http://rdma.so">rdma.so</a>: cannot open shared object file: No such file or directory<br />[2015-06-05 01:59:02.329625] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-<br />type 'rdma' is not valid or not found on this machine<br />[2015-06-05 01:59:02.329637] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the tra<br />nsport failed<br />[2015-06-05 01:59:02.343346] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed<br /> in the system<br />[2015-06-05 01:59:02.351861] I [glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2<br />[2015-06-05 01:59:02.363138] I [glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snap<br />s list.<br />Final graph:<br />+<hr />+<br /> 1: volume management<br /> 2: type mgmt/glusterd<br /> 3: option rpc-auth.auth-glusterfs on<br
/> 4: option rpc-auth.auth-unix on<br /> 5: option rpc-auth.auth-null on<br /> 6: option transport.socket.listen-backlog 128<br /> 7: option ping-timeout 30<br /> 8: option transport.socket.read-fail-log off<br /> 9: option transport.socket.keepalive-interval 2<br /> 10: option transport.socket.keepalive-time 10<br /> 11: option transport-type rdma<br /> 12: option working-directory /var/lib/glusterd<br /> 13: end-volume<br /> 14: <br />+<hr />+<br />[2015-06-05 02:00:43.025378] I [glusterd.c:146:glusterd_uuid_init] 0-management: retrieved UUID: 1b98af34-08f9-492f-b54e-7608<br />2588d63b<br />[2015-06-05 02:00:43.025462] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br />[2015-06-05 02:00:44.025069] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br />[2015-06-05 02:00:44.599862] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req<br />[2015-06-05 02:00:46.029823] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br />[2015-06-05 02:00:46.701845] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<br />[2015-06-05 02:01:43.119612] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br />[2015-06-05 02:01:44.165527] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br />[2015-06-05 02:01:44.851903] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br />[2015-06-05 02:01:46.190401] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br />[2015-06-05 02:01:46.722394] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<br />[2015-06-05 02:01:58.719968] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br />[2015-06-05 02:02:32.584999] I [glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glust<br /><a href="http://erfs05.sh2.ctripcorp.com">erfs05.sh2.ctripcorp.com</a> 24007<br />[2015-06-05 02:02:32.586281] I [glusterd-handler.c:3165:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: g<br /><a href="http://lusterfs05.sh2.ctripcorp.com">lusterfs05.sh2.ctripcorp.com</a> (24007)<br />[2015-06-05 02:02:32.602192] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br />[2015-06-05 02:02:32.606053] I [glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect returned 0<br />[2015-06-05 02:02:32.620958] E [glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk] 0-management: failed to get the <br />'versions' from peer (<a href="10.8.230.209:24007">10.8.230.209:24007</a>)<br />[2015-06-05
02:02:32.621020] I [MSGID: 106004] [glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management: Peer 000000<br />00-0000-0000-0000-000000000000, in Establishing Connection state, has disconnected from glusterd.<br />[2015-06-05 02:02:43.264141] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br />[2015-06-05 02:02:44.312918] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br />[2015-06-05 02:02:44.781434] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br />[2015-06-05 02:02:46.176888] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br />[2015-06-05 02:02:46.799916] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<br />[2015-06-05 02:03:42.720994] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br
/>[2015-06-05 02:03:43.771987] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br />[2015-06-05 02:03:44.659775] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br />[2015-06-05 02:03:45.956807] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br />[2015-06-05 02:03:46.688787] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<br />[2015-06-05 02:04:42.891131] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br />[2015-06-05 02:04:43.957423] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br />[2015-06-05 02:04:44.995638] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br />[2015-06-05 02:04:46.062171] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req<br />[2015-06-05 02:04:46.957327] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<br />[2015-06-05 02:05:42.966651] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br />[2015-06-05 02:05:44.008498] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br />[2015-06-05 02:05:45.049936] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br />[2015-06-05 02:05:46.087007] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br />[2015-06-05 02:05:47.029325] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<br />[2015-06-05 02:06:42.874089] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br />[2015-06-05 02:06:43.910998] W
[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br />[2015-06-05 02:06:44.932802] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br />[2015-06-05 02:06:46.053731] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br />[2015-06-05 02:06:47.055859] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<br />[2015-06-05 02:07:43.099647] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br />[2015-06-05 02:07:44.146454] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br />[2015-06-05 02:07:45.218982] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br />[2015-06-05 02:07:46.246303] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list
req<br />[2015-06-05 02:07:47.106954] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<br />[2015-06-05 02:08:43.226090] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br />[2015-06-05 02:08:44.292041] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br />[2015-06-05 02:08:45.351830] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br />[2015-06-05 02:08:45.688349] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br />[2015-06-05 02:08:46.651768] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<br />[2015-06-05 02:09:42.748628] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<br /><br /><br /><br />Best Regards <br /><br /><br />-----邮件原件-----<br />发件人:
vyyy杨雨阳 <br />发送时间: Thursday, June 04, 2015 11:00 AM<br />收件人: 'Atin Mukherjee'; Gluster-users@gluster.org<br />主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start<br /><br /> <br />The problem 2 has resolved.<br /><br />From glusterd log file, I find Cksums error as following:<br />[2015-06-04 01:25:03.137798] E [MSGID: 106010] [glusterd-utils.c:3297:glusterd_compare_friend_volume] 0-management: Version of Cksums JQStore2 differ. local cksum = 2617028277, remote cksum = 2210669223 on peer <a href="http://glusterfs09.sh2.ctripcorp.com">glusterfs09.sh2.ctripcorp.com</a><br /><br />The solution for this problem was to do the following, on the BAD node:<br />/etc/init.d/glusterd stop<br />rsync -havP --delete <a href="http://172.26.177.115">172.26.177.115</a>:/var/lib/glusterd/vols/ /var/lib/glusterd/vols/<br />/etc/init.d/glusterd start<br /><br /><br /> <br />The problem 1 is still under inverstigate:<br /><br />New nodes
can not join the cluster, log file shown: failed to get the 'versions' from peer<br />I have checked the file /var/lib/glusterd/ <a href="http://glusterd.info">glusterd.info</a>, all nodes are " operating-version=2"<br /><br /> <br /> [2015-06-04 02:13:49.199416] E [glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk] 0-management: failed to get the <br /> 'versions' from peer (<a href="10.8.230.211:24007">10.8.230.211:24007</a>)<br /> [2015-06-04 02:14:07.112577] I [glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glust<br /><a href="http:// erfs05.sh2.ctripcorp.com"> erfs05.sh2.ctripcorp.com</a> 24007<br /> [2015-06-04 02:14:07.113778] I [glusterd-handler.c:3165:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: g<br /><a href="http:// lusterfs05.sh2.ctripcorp.com"> lusterfs05.sh2.ctripcorp.com</a> (24007)<br /> [2015-06-04 02:14:07.131023] I
[rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br /> [2015-06-04 02:14:07.134829] I [glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect returned 0<br /> [<br /><br /><br /><br /><br />Best Regards <br />杨雨阳<br />Site Operations Center<br />Ext.15554 Seat.1#3F049<br />We are the F.I.R.S.T. <br />Fast, Innovation, Reliability , Secure, Team<br /><br /><br />-----邮件原件-----<br />发件人: vyyy杨雨阳 <br />发送时间: Wednesday, June 03, 2015 5:37 PM<br />收件人: 'Atin Mukherjee'; Gluster-users@gluster.org<br />主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start<br /><br />Hi Atin and everybody,<br /><br /> I have forced peer detach the news nodes ,The peer status seems even strange, glusterfs05 and glusterfs10 are peer in cluster, Can you help me? Any advice is appreciated.<br /><br />Glusterfs05 <br /> <br /> [root@sh02svr5956 ~]# gluster peer
status<br /> Number of Peers: 5<br /> <br /> Hostname: <a href="http://glusterfs06.sh2.ctripcorp.com">glusterfs06.sh2.ctripcorp.com</a><br /> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f<br /> State: Peer Rejected (Connected)<br /> <br /> Hostname: <a href="http://glusterfs07.sh2.ctripcorp.com">glusterfs07.sh2.ctripcorp.com</a><br /> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23<br /> State: Peer Rejected (Connected)<br /> <br /> Hostname: <a href="http://glusterfs08.sh2.ctripcorp.com">glusterfs08.sh2.ctripcorp.com</a><br /> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207<br /> State: Peer Rejected (Connected)<br /> <br /> Hostname: <a href="http://glusterfs10.sh2.ctripcorp.com">glusterfs10.sh2.ctripcorp.com</a><br /> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c<br /> State: Peer in Cluster (Connected)<br /> <br /> Hostname: <a
href="http://glusterfs09.sh2.ctripcorp.com">glusterfs09.sh2.ctripcorp.com</a><br /> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d<br /> State: Peer Rejected (Connected)<br /> [root@sh02svr5956 ~]#<br /> <br /> <br /> Glusterfs06 <br /> <br /> [root@SH02SVR5955 ~]# gluster peer status<br /> Number of Peers: 5<br /> <br /> Hostname: <a href="http://glusterfs10.sh2.ctripcorp.com">glusterfs10.sh2.ctripcorp.com</a><br /> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c<br /> State: Peer Rejected (Connected)<br /> <br /> Hostname: <a href="http://glusterfs08.sh2.ctripcorp.com">glusterfs08.sh2.ctripcorp.com</a><br /> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207<br /> State: Peer in Cluster (Connected)<br /> <br /> Hostname: <a href="http://glusterfs07.sh2.ctripcorp.com">glusterfs07.sh2.ctripcorp.com</a><br /> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23<br /> State: Peer in Cluster (Connected)<br
/> <br /> Hostname: <a href="http://10.8.230.209">10.8.230.209</a><br /> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6<br /> State: Peer Rejected (Connected)<br /> <br /> Hostname: <a href="http://glusterfs09.sh2.ctripcorp.com">glusterfs09.sh2.ctripcorp.com</a><br /> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d<br /> State: Peer in Cluster (Connected)<br /> [root@SH02SVR5955 ~]#<br /> <br /><br />Glusterfs07<br /><br />[root@SH02SVR5954 ~]# gluster peer status<br />Number of Peers: 5<br /><br />Hostname: <a href="http://glusterfs10.sh2.ctripcorp.com">glusterfs10.sh2.ctripcorp.com</a><br />Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c<br />State: Peer Rejected (Connected)<br /><br />Hostname: <a href="http://10.8.230.209">10.8.230.209</a><br />Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6<br />State: Peer Rejected (Connected)<br /><br />Hostname: <a href="http://glusterfs06.sh2.ctripcorp.com">glusterfs06.sh2.ctripcorp.com</a><br />Uuid:
2cb15023-28b0-4d0d-8a43-b8c6e570776f<br />State: Peer in Cluster (Connected)<br /><br />Hostname: <a href="http://glusterfs08.sh2.ctripcorp.com">glusterfs08.sh2.ctripcorp.com</a><br />Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207<br />State: Peer in Cluster (Connected)<br /><br />Hostname: <a href="http://glusterfs09.sh2.ctripcorp.com">glusterfs09.sh2.ctripcorp.com</a><br />Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d<br />State: Peer in Cluster (Connected)<br />[root@SH02SVR5954 ~]#<br /><br /> <br /> <br />Glusterfs08<br /><br />[root@SH02SVR5953 ~]# gluster peer status<br />Number of Peers: 5<br /><br />Hostname: <a href="http://glusterfs07.sh2.ctripcorp.com">glusterfs07.sh2.ctripcorp.com</a><br />Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23<br />State: Peer in Cluster (Connected)<br /><br />Hostname: <a href="http://glusterfs10.sh2.ctripcorp.com">glusterfs10.sh2.ctripcorp.com</a><br />Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c<br />State: Peer Rejected (Connected)<br /><br />Hostname:
<a href="http://glusterfs06.sh2.ctripcorp.com">glusterfs06.sh2.ctripcorp.com</a><br />Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f<br />State: Peer in Cluster (Connected)<br /><br />Hostname: <a href="http://10.8.230.209">10.8.230.209</a><br />Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6<br />State: Peer Rejected (Connected)<br /><br />Hostname: <a href="http://glusterfs09.sh2.ctripcorp.com">glusterfs09.sh2.ctripcorp.com</a><br />Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d<br />State: Peer in Cluster (Connected)<br />[root@SH02SVR5953 ~]#<br /><br /><br />[root@SH02SVR5952 ~]# gluster peer status<br />Number of Peers: 5<br /><br />Hostname: <a href="http://glusterfs06.sh2.ctripcorp.com">glusterfs06.sh2.ctripcorp.com</a><br />Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f<br />State: Peer in Cluster (Connected)<br /><br />Hostname: <a href="http://glusterfs07.sh2.ctripcorp.com">glusterfs07.sh2.ctripcorp.com</a><br />Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23<br />State: Peer in Cluster
(Connected)<br /><br />Hostname: <a href="http://glusterfs08.sh2.ctripcorp.com">glusterfs08.sh2.ctripcorp.com</a><br />Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207<br />State: Peer in Cluster (Connected)<br /><br />Hostname: <a href="http://10.8.230.209">10.8.230.209</a><br />Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6<br />State: Peer Rejected (Connected)<br /><br />Hostname: <a href="http://glusterfs10.sh2.ctripcorp.com">glusterfs10.sh2.ctripcorp.com</a><br />Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c<br />State: Peer Rejected (Connected)<br />[root@SH02SVR5952 ~]#<br /><br /><br />Glusterfs10<br /><br />[root@SH02SVR5951 ~]# gluster peer status<br />Number of Peers: 6<br /><br />Hostname: <a href="http://glusterfs07.sh2.ctripcorp.com">glusterfs07.sh2.ctripcorp.com</a><br />Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23<br />State: Peer Rejected (Connected)<br /><br />Hostname: <a href="http://glusterfs11.sh2.ctripcorp.com">glusterfs11.sh2.ctripcorp.com</a><br />Uuid:
2d703550-92b5-4f5e-af90-ff2fbf3366f0<br />State: Peer Rejected (Connected)<br /><br />Hostname: <a href="http://glusterfs09.sh2.ctripcorp.com">glusterfs09.sh2.ctripcorp.com</a><br />Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d<br />State: Peer Rejected (Connected)<br /><br />Hostname: <a href="http://glusterfs06.sh2.ctripcorp.com">glusterfs06.sh2.ctripcorp.com</a><br />Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f<br />State: Peer Rejected (Connected)<br /><br />Hostname: <a href="http://10.8.230.209">10.8.230.209</a><br />Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6<br />State: Peer in Cluster (Connected)<br /><br />Hostname: <a href="http://glusterfs08.sh2.ctripcorp.com">glusterfs08.sh2.ctripcorp.com</a><br />Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207<br />State: Peer Rejected (Connected)<br />[root@SH02SVR5951 ~]#<br /><br /><br /><br /><br /><br /><br /><br />Best Regards <br />杨雨阳<br />Site Operations Center<br />Ext.15554 Seat.1#3F049<br />We are the F.I.R.S.T. <br />Fast,
Innovation, Reliability , Secure, Team<br /><br /><br /><br />-----邮件原件-----<br />发件人: vyyy杨雨阳 <br />发送时间: Wednesday, June 03, 2015 2:55 PM<br />收件人: 'Atin Mukherjee'<br />主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start<br /><br />Can I delete the files in /var/lib/glusterd/peers which state is not =3 on all nodes, then restart glusterd?<br /><br /><br />[root@sh02svr5956 peers]# more 0475e11a-3c43-4b76-99dd-1d2e814c5f15 <br />uuid=0475e11a-3c43-4b76-99dd-1d2e814c5f15<br />state=6<br />hostname1=<a href="http://glusterfs12.sh2.ctripcorp.com">glusterfs12.sh2.ctripcorp.com</a><br /><br /><br /><br /><br />Best Regards <br />Yang<br /><br /><br />-----邮件原件-----<br />发件人: vyyy杨雨阳 <br />发送时间: Wednesday, June 03, 2015 2:39 PM<br />收件人: 'Atin Mukherjee'<br />主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start<br /><br />Since it's intranet, You
can only control my desktop, and then access these node.<br />Should I apply a webex room?<br /><br /><br />Best Regards <br />杨雨阳<br />Site Operations Center<br />Ext.15554 Seat.1#3F049<br />We are the F.I.R.S.T. <br />Fast, Innovation, Reliability , Secure, Team<br /><br /><br /><br />-----邮件原件-----<br />发件人: Atin Mukherjee [mailto:amukherj@redhat.com] <br />发送时间: Wednesday, June 03, 2015 2:36 PM<br />收件人: vyyy杨雨阳<br />主题: Re: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start<br /><br /><br />Is it possible to share your ip/user/passwd for few of these nodes?<br /><br />On 06/03/2015 11:50 AM, vyyy杨雨阳 wrote:<br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #729fcf; padding-left: 1ex;"> Can you remote control my desktop (like webex)?<br /> The nodes host bricks. And on the new nodes, I did "clean /var/lib/glusterd and restart" several times, but have no
effect.<br /> Is there any other work around?<br /> <br /> <br /> Best Regards <br /> Yang<br /> <br /> -----邮件原件-----<br /> 发件人: Atin Mukherjee [mailto:amukherj@redhat.com] <br /> 发送时间: Wednesday, June 03, 2015 12:33 PM<br /> 收件人: vyyy杨雨阳; Gluster-users@gluster.org<br /> 主题: Re: 答复: [Gluster-users] Gluster peer rejected and failed to start<br /> <br /> I can't comment on what went wrong till I can have visibility on the<br /> entire cluster. Does the node which fails to start host any brick? If<br /> not then work around could be to clean /var/lib/glusterd and restart<br /> glusterd, that should sync up data and bring this node in consistency.<br /> <br /> HTH,<br /> Atin<br /> <br /> On 06/03/2015 08:34 AM, vyyy杨雨阳 wrote:<br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #ad7fa8; padding-left: 1ex;"> I found that /var/lib/glusterd/peers can't include node's own peer, I delete this file.
Glusterd started finally.<br /> But the peer status still not normal.<br /> Seems that glusterfs10 leave the cluster and join the new nodes alone.<br /><br /> Peer status on glusterfs10<br /> [root@SH02SVR5951 ~]# gluster peer status<br /> Number of Peers: 7<br /><br /> Hostname: <a href="http://glusterfs07.sh2.ctripcorp.com">glusterfs07.sh2.ctripcorp.com</a><br /> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23<br /> State: Peer Rejected (Connected)<br /><br /> Hostname: <a href="http://glusterfs11.sh2.ctripcorp.com">glusterfs11.sh2.ctripcorp.com</a><br /> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0<br /> State: Accepted peer request (Connected)<br /><br /> Hostname: <a href="http://glusterfs09.sh2.ctripcorp.com">glusterfs09.sh2.ctripcorp.com</a><br /> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d<br /> State: Peer Rejected (Connected)<br /><br /> Hostname: <a href="http://glusterfs06.sh2.ctripcorp.com">glusterfs06.sh2.ctripcorp.com</a><br /> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f<br />
State: Peer Rejected (Connected)<br /><br /> Hostname: <a href="http://10.8.230.209">10.8.230.209</a><br /> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6<br /> State: Peer Rejected (Connected)<br /><br /> Hostname: <a href="http://glusterfs08.sh2.ctripcorp.com">glusterfs08.sh2.ctripcorp.com</a><br /> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207<br /> State: Peer Rejected (Connected)<br /><br /> Hostname: <a href="http://glusterfs12.sh2.ctripcorp.com">glusterfs12.sh2.ctripcorp.com</a><br /> Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15<br /> State: Peer in Cluster (Connected)<br /><br /><br /> <br /> Peer status on glusterfs10<br /><br /> [root@SVR6993HW2285 ~]# gluster peer status<br /> Number of Peers: 6<br /><br /> Hostname: <a href="http://glusterfs05.sh2.ctripcorp.com">glusterfs05.sh2.ctripcorp.com</a><br /> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6<br /> State: Peer Rejected (Connected)<br /><br /> Hostname: <a
href="http://glusterfs09.sh2.ctripcorp.com">glusterfs09.sh2.ctripcorp.com</a><br /> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d<br /> State: Peer Rejected (Connected)<br /><br /> Hostname: <a href="http://glusterfs07.sh2.ctripcorp.com">glusterfs07.sh2.ctripcorp.com</a><br /> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23<br /> State: Peer Rejected (Connected)<br /><br /> Hostname: <a href="http://glusterfs08.sh2.ctripcorp.com">glusterfs08.sh2.ctripcorp.com</a><br /> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207<br /> State: Peer Rejected (Connected)<br /><br /> Hostname: <a href="http://glusterfs06.sh2.ctripcorp.com">glusterfs06.sh2.ctripcorp.com</a><br /> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f<br /> State: Peer Rejected (Connected)<br /><br /> Hostname: <a href="http://glusterfs10.sh2.ctripcorp.com">glusterfs10.sh2.ctripcorp.com</a><br /> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c<br /> State: Peer in Cluster (Connected)<br /><br /><br /> Peer status on glusterfs05, this is the node
that I tried to peer probe new nodes (glusterfs11 to glusterfs14)<br /><br /> [root@sh02svr5956 ~]# gluster peer status<br /> Number of Peers: 9<br /><br /> Hostname: <a href="http://glusterfs06.sh2.ctripcorp.com">glusterfs06.sh2.ctripcorp.com</a><br /> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f<br /> State: Peer in Cluster (Connected)<br /><br /> Hostname: <a href="http://glusterfs07.sh2.ctripcorp.com">glusterfs07.sh2.ctripcorp.com</a><br /> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23<br /> State: Peer in Cluster (Connected)<br /><br /> Hostname: <a href="http://glusterfs08.sh2.ctripcorp.com">glusterfs08.sh2.ctripcorp.com</a><br /> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207<br /> State: Peer in Cluster (Connected)<br /><br /> Hostname: <a href="http://glusterfs10.sh2.ctripcorp.com">glusterfs10.sh2.ctripcorp.com</a><br /> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c<br /> State: Peer Rejected (Connected)<br /><br /> Hostname: <a
href="http://glusterfs09.sh2.ctripcorp.com">glusterfs09.sh2.ctripcorp.com</a><br /> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d<br /> State: Peer in Cluster (Connected)<br /><br /> Hostname: <a href="http://glusterfs11.sh2.ctripcorp.com">glusterfs11.sh2.ctripcorp.com</a><br /> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0<br /> State: Peer Rejected (Connected)<br /><br /> Hostname: <a href="http://glusterfs12.sh2.ctripcorp.com">glusterfs12.sh2.ctripcorp.com</a><br /> Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15<br /> State: Peer in Cluster (Connected)<br /><br /> Hostname: <a href="http://glusterfs13.sh2.ctripcorp.com">glusterfs13.sh2.ctripcorp.com</a><br /> Uuid: 43e96d31-6c6d-4096-832a-440e544554ca<br /> State: Peer Rejected (Connected)<br /><br /> Hostname: <a href="http://glusterfs14.sh2.ctripcorp.com">glusterfs14.sh2.ctripcorp.com</a><br /> Uuid: 45c11af9-7850-4598-ad62-4bb343b62524<br /> State: Peer Rejected (Connected)<br /> [root@sh02svr5956 ~]#<br /><br /><br /> <br /><br />
Best Regards <br /> Yang <br /><br /><br /> -----邮件原件-----<br /> 发件人: vyyy杨雨阳 <br /> 发送时间: Wednesday, June 03, 2015 10:04 AM<br /> 收件人: 'Atin Mukherjee'; 'Gluster-users@gluster.org'<br /> 主题: [Gluster-users] Gluster peer rejected and failed to start<br /><br /> I am still messed, can you help me figure out problem 2 (A node of cluster failed to start glusterd) priority<br /><br /> The glusterd log on the node gluster10 shown ' resolve brick failed in restore'<br /> I checked /var/lib/glusterd/vols, seems ok<br /><br /> Glusterd log file Error message:<br /><br /> [2015-06-03 01:31:32.564010] E [glusterd-store.c:4244:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore<br /> [2015-06-03 01:31:32.564036] E [xlator.c:425:xlator_init] 0-management: Initialization of volume 'management' failed, review <br /> your volfile again<br /><br /><br /><br /><br /> Best Regards <br /> Yang<br /><br /> -----邮件原件-----<br />
发件人: vyyy杨雨阳 <br /> 发送时间: Tuesday, June 02, 2015 6:27 PM<br /> 收件人: 'Atin Mukherjee'; Gluster-users@gluster.org<br /> 主题: 答复: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start<br /><br /> Gluster11 to gluster14 are new nodes I tried to join exit cluster but failed <br /> <br /> <br /> On gluster12:<br /> [root@SVR6994HW2285 ~]# gluster peer status<br /> Number of Peers: 1<br /><br /> Hostname: <a href="http://glusterfs05.sh2.ctripcorp.com">glusterfs05.sh2.ctripcorp.com</a><br /> Uuid: 04f22ee8-8e0</blockquote></blockquote></pre></blockquote></div><br>
-- <br>
Sent from my Android device with K-9 Mail. Please excuse my brevity.</body></html>