<p dir="ltr"></p>
<p dir="ltr">-Atin<br>
Sent from one plus one<br>
On 07-Apr-2016 9:32 pm, "Ernie Dunbar" <<a href="mailto:maillist@lightspeed.ca">maillist@lightspeed.ca</a>> wrote:<br>
><br>
> On 2016-04-06 21:20, Atin Mukherjee wrote:<br>
>><br>
>> On 04/07/2016 04:04 AM, Ernie Dunbar wrote:<br>
>>><br>
>>> On 2016-04-06 11:42, Ernie Dunbar wrote:<br>
>>>><br>
>>>> I've already successfully created a Gluster cluster, but when I try to<br>
>>>> add a new node, gluster on the new node claims it can't find the<br>
>>>> hostname of the first node in the cluster.<br>
>>>><br>
>>>> I've added the hostname <a href="http://nfs1.lightspeed.ca">nfs1.lightspeed.ca</a> to /etc/hosts like this:<br>
>>>><br>
>>>> root@nfs3:/home/ernied# cat /etc/hosts<br>
>>>> 127.0.0.1 localhost<br>
>>>> 192.168.1.31 <a href="http://nfs1.lightspeed.ca">nfs1.lightspeed.ca</a> nfs1<br>
>>>> 192.168.1.32 <a href="http://nfs2.lightspeed.ca">nfs2.lightspeed.ca</a> nfs2<br>
>>>> 127.0.1.1 <a href="http://nfs3.lightspeed.ca">nfs3.lightspeed.ca</a> nfs3<br>
>>>><br>
>>>><br>
>>>> # The following lines are desirable for IPv6 capable hosts<br>
>>>> ::1 localhost ip6-localhost ip6-loopback<br>
>>>> ff02::1 ip6-allnodes<br>
>>>> ff02::2 ip6-allrouters<br>
>>>><br>
>>>> I can ping the hostname:<br>
>>>><br>
>>>> root@nfs3:/home/ernied# ping -c 3 nfs1<br>
>>>> PING <a href="http://nfs1.lightspeed.ca">nfs1.lightspeed.ca</a> (192.168.1.31) 56(84) bytes of data.<br>
>>>> 64 bytes from <a href="http://nfs1.lightspeed.ca">nfs1.lightspeed.ca</a> (192.168.1.31): icmp_seq=1 ttl=64<br>
>>>> time=0.148 ms<br>
>>>> 64 bytes from <a href="http://nfs1.lightspeed.ca">nfs1.lightspeed.ca</a> (192.168.1.31): icmp_seq=2 ttl=64<br>
>>>> time=0.126 ms<br>
>>>> 64 bytes from <a href="http://nfs1.lightspeed.ca">nfs1.lightspeed.ca</a> (192.168.1.31): icmp_seq=3 ttl=64<br>
>>>> time=0.133 ms<br>
>>>><br>
>>>> --- <a href="http://nfs1.lightspeed.ca">nfs1.lightspeed.ca</a> ping statistics ---<br>
>>>> 3 packets transmitted, 3 received, 0% packet loss, time 1998ms<br>
>>>> rtt min/avg/max/mdev = 0.126/0.135/0.148/0.016 ms<br>
>>>><br>
>>>> I can get gluster to probe the hostname:<br>
>>>><br>
>>>> root@nfs3:/home/ernied# gluster peer probe nfs1<br>
>>>> peer probe: success. Host nfs1 port 24007 already in peer list<br>
>>>><br>
>>>> But if I try to create the brick on the new node, it says that the<br>
>>>> host can't be found? Um...<br>
>>>><br>
>>>> root@nfs3:/home/ernied# gluster volume create gv2 replica 3<br>
>>>> nfs1.lightspeed.ca:/brick1/gv2/ nfs2.lightspeed.ca:/brick1/gv2/<br>
>>>> nfs3.lightspeed.ca:/brick1/gv2<br>
>>>> volume create: gv2: failed: Failed to find host <a href="http://nfs1.lightspeed.ca">nfs1.lightspeed.ca</a><br>
>>>><br>
>>>> Our logs from /var/log/glusterfs/etc-glusterfs-glusterd.vol.log:<br>
>>>><br>
>>>> [2016-04-06 18:19:18.107459] E [MSGID: 106452]<br>
>>>> [glusterd-utils.c:5825:glusterd_new_brick_validate] 0-management:<br>
>>>> Failed to find host <a href="http://nfs1.lightspeed.ca">nfs1.lightspeed.ca</a><br>
>>>> [2016-04-06 18:19:18.107496] E [MSGID: 106536]<br>
>>>> [glusterd-volume-ops.c:1364:glusterd_op_stage_create_volume]<br>
>>>> 0-management: Failed to find host <a href="http://nfs1.lightspeed.ca">nfs1.lightspeed.ca</a><br>
>>>> [2016-04-06 18:19:18.107516] E [MSGID: 106301]<br>
>>>> [glusterd-syncop.c:1281:gd_stage_op_phase] 0-management: Staging of<br>
>>>> operation 'Volume Create' failed on localhost : Failed to find host<br>
>>>> <a href="http://nfs1.lightspeed.ca">nfs1.lightspeed.ca</a><br>
>>>> [2016-04-06 18:19:18.231864] E [MSGID: 106170]<br>
>>>> [glusterd-handshake.c:1051:gd_validate_mgmt_hndsk_req] 0-management:<br>
>>>> Request from peer <a href="http://192.168.1.31:65530">192.168.1.31:65530</a> has an entry in peerinfo, but<br>
>>>> uuid does not match<br>
>><br>
>> We have introduced a new check to reject a peer if the request is coming<br>
>> from a node where the hostname matches but UUID is different. This can<br>
>> happen if a node goes through a re-installation and its<br>
>> /var/lib/glusterd/* content is wiped off. Look at [1] for more details.<br>
>><br>
>> [1] <a href="http://review.gluster.org/13519">http://review.gluster.org/13519</a><br>
>><br>
>> Do confirm if that's the case.<br>
><br>
><br>
><br>
> I couldn't say if that's *exactly* the case, but it's pretty close. I don't recall ever removing /var/lib/glusterd/* or any of its contents, but the operating system isn't exactly the way it was when I first tried to add this node to the cluster.<br>
><br>
> What should I do to *fix* the problem though, so I can add this node to the cluster? This bug report doesn't appear to provide a solution. I've tried removing the node from the cluster, and that failed too. Things seem to be in a very screwey state right now.<br></p>
<p dir="ltr">I should have given the work around earlier. Find the peer file for the faulty node in /var/lib/glusterd/peers/ and delete the same from all the nodes but the faulty node. Restart glusterd instance on all those nodes. Ensure /var/lib/glusterd/ content is empty, restart glusterd and then peer probe this node from any of the node in the existing cluster. You should also bump up the op-version once cluster is stable.</p>
<p dir="ltr">><br>
><br>
>><br>
>>>> [2016-04-06 18:19:18.231919] E [MSGID: 106170]<br>
>>>> [glusterd-handshake.c:1060:gd_validate_mgmt_hndsk_req] 0-management:<br>
>>>> Rejecting management handshake request from unknown peer<br>
>>>> <a href="http://192.168.1.31:65530">192.168.1.31:65530</a><br>
>>>><br>
>>>> That error about the entry in peerinfo doesn't match anything in<br>
>>>> Google besides the source code for Gluster. My guess is that my<br>
>>>> earlier unsuccessful attempts to add this node before v3.7.10 have<br>
>>>> created a conflict that needs to be cleared.<br>
>>><br>
>>><br>
>>><br>
>>> More interesting, is what happens when I try to add the third server to<br>
>>> the brick from the first gluster server:<br>
>>><br>
>>> root@nfs1:/home/ernied# gluster volume add-brick gv2 replica 3<br>
>>> nfs3:/brick1/gv2<br>
>>> volume add-brick: failed: One or more nodes do not support the required<br>
>>> op-version. Cluster op-version must atleast be 30600.<br>
>>><br>
>>> Yet, when I view the operating version in /var/lib/glusterd/<a href="http://glusterd.info">glusterd.info</a>:<br>
>>><br>
>>> root@nfs1:/home/ernied# cat /var/lib/glusterd/<a href="http://glusterd.info">glusterd.info</a><br>
>>> UUID=1207917a-23bc-4bae-8238-cd691b7082c7<br>
>>> operating-version=30501<br>
>>><br>
>>> root@nfs2:/home/ernied# cat /var/lib/glusterd/<a href="http://glusterd.info">glusterd.info</a><br>
>>> UUID=e394fcec-41da-482a-9b30-089f717c5c06<br>
>>> operating-version=30501<br>
>>><br>
>>> root@nfs3:/home/ernied# cat /var/lib/glusterd/<a href="http://glusterd.info">glusterd.info</a><br>
>>> UUID=ae191e96-9cd6-4e2b-acae-18f2cc45e6ed<br>
>>> operating-version=30501<br>
>>><br>
>>> I see that the operating version is the same on all nodes!<br>
>><br>
>> Here cluster op-version is pretty old. You need to make sure that you<br>
>> bump up the op-version by 'gluster volume set all cluster.op-version<br>
>> 30710'. add-brick code path has a check that your cluster op-version has<br>
>> to be at least 30600 if you are with gluster version >=3.6 which is the<br>
>> case here.<br>
>>><br>
>>> _______________________________________________<br>
>>> Gluster-users mailing list<br>
>>> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
>>> <a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
>>><br>
> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</p>