<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Atin,
<div class=""><br class="">
</div>
<div class="">I performed a <i class="">gluster volume set <volumename> performance.flush-behind</i> <i class="">off/on</i> toggle on both volumes and after that the probe was successful.</div>
<div class=""><br class="">
</div>
<div class="">So many thanks for your support.</div>
<div class=""><br class="">
</div>
<div class="">Some additional info, in our lab I did some tests starting with gluster version 3.6.4 and was not able to reproduce the problem. After that I went looking for some differences with our production cluster and found out that we started there with
version 3.5.x which we upgraded to version 3.6.4. So maybe the bug/incompatibility is introduced somewhere after an upgrade procedure?</div>
<div class=""><br class="">
</div>
<div class="">Greetings</div>
<div class="">Davy</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 14 Sep 2015, at 07:43, Atin Mukherjee <<a href="mailto:amukherj@redhat.com" class="">amukherj@redhat.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">Davy,<br class="">
<br class="">
This seems to be an issue which we also faced couple of months back<br class="">
during upgrade testing and a bugzilla [1] was raised for the same. At<br class="">
the time we didn't have the work around to make peer probe work, but<br class="">
somehow I managed to get the workaround today.<br class="">
<br class="">
Could you do an explicit volume set on the existing cluster and then do<br class="">
a peer probe? Let me know if that works.<br class="">
<br class="">
[1] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1248895" class="">https://bugzilla.redhat.com/show_bug.cgi?id=1248895</a><br class="">
<br class="">
Thanks,<br class="">
Atin<br class="">
<br class="">
On 09/11/2015 05:41 PM, Davy Croonen wrote:<br class="">
<blockquote type="cite" class="">Atin<br class="">
<br class="">
Please see the requested attachments.<br class="">
<br class="">
KR<br class="">
Davy<br class="">
<br class="">
<blockquote type="cite" class="">On 11 Sep 2015, at 14:03, Atin Mukherjee <<a href="mailto:amukherj@redhat.com" class="">amukherj@redhat.com</a>> wrote:<br class="">
<br class="">
Could you attach the contents of /var/lib/glusterd/vol/<volname>/info<br class="">
file from both the nodes?<br class="">
<br class="">
~Atin<br class="">
<br class="">
On 09/11/2015 04:50 PM, Davy Croonen wrote:<br class="">
<blockquote type="cite" class="">Thanks for your quick respons.<br class="">
<br class="">
As reported in the log the checksums are indeed not the same. On<br class="">
gfs01a-dcg it is 'info=1266454712’ and on gfs02a-dcg it is<br class="">
'info=2613085848’. Of course my next question is how can I fix this?<br class="">
<br class="">
I already tried by stopping the gluster daemon on gfs02a-dcg, deleting<br class="">
the entire vols directory and starting the gluster daemon again. On the<br class="">
gfs01a-dcg host I now did a gluster peer status which shows:<br class="">
<br class="">
Hostname: <a href="http://gfs02a-dcg.intnet.be" class="">gfs02a-dcg.intnet.be</a> <<a href="http://gfs02a-dcg.intnet.be" class="">http://gfs02a-dcg.intnet.be</a>><br class="">
Uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f<br class="">
State: Peer in Cluster (Connected)<br class="">
<br class="">
But, the checksum of the public volume is still not the same on<br class="">
gfs01a-dcg and gfs02a-dcg and also running a gluster peer status on<br class="">
gfs01b-dcg (the replica of gfs01a-dcg) gives me:<br class="">
<br class="">
Hostname: <a href="http://gfs02a-dcg.intnet.be" class="">gfs02a-dcg.intnet.be</a> <<a href="http://gfs02a-dcg.intnet.be" class="">http://gfs02a-dcg.intnet.be</a>><br class="">
Uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f<br class="">
State: Peer Rejected (Connected)<br class="">
<br class="">
So my question remains any way to fix this?<br class="">
<br class="">
Kind regards<br class="">
<br class="">
Davy<br class="">
<br class="">
<blockquote type="cite" class="">On 11 Sep 2015, at 12:39, Mohammed Rafi K C <<a href="mailto:rkavunga@redhat.com" class="">rkavunga@redhat.com</a><br class="">
<<a href="mailto:rkavunga@redhat.com" class="">mailto:rkavunga@redhat.com</a>>> wrote:<br class="">
<br class="">
Can you check the checksum of the volume "public" in both of the<br class="">
current nodes. Checksums are located in<br class="">
(/var/lib/glusterd/vols/public/cksum).<br class="">
<br class="">
Regards<br class="">
Rafi KC<br class="">
<br class="">
On 09/11/2015 03:24 PM, Davy Croonen wrote:<br class="">
<blockquote type="cite" class="">Hi all<br class="">
<br class="">
We have a production cluster with 2 nodes (gfs01a and gfs01b) in a<br class="">
distributed replicate setup with glusterfs 3.6.4. We want to expand<br class="">
the volume with 2 extra nodes (gfs02a and gfs02b) because we are<br class="">
running out of diskspace. Therefor we deployed 2 extra nodes with<br class="">
glusterfs 3.6.4.<br class="">
<br class="">
Now, while probing the 2 new nodes from a node in the existing<br class="">
cluster we got the following error:<br class="">
<br class="">
root@gfs01a-dcg:~# gluster peer probe <a href="http://gfs02a-dcg.intnet.be" class="">
gfs02a-dcg.intnet.be</a><br class="">
<<a href="http://gfs02a-dcg.intnet.be/" class="">http://gfs02a-dcg.intnet.be/</a>><br class="">
peer probe: success.<br class="">
root@gfs01a-dcg:~# gluster peer status<br class="">
Number of Peers: 2<br class="">
<br class="">
Hostname: <a href="http://gfs01b-dcg.intnet.be" class="">gfs01b-dcg.intnet.be</a> <<a href="http://gfs01b-dcg.intnet.be/" class="">http://gfs01b-dcg.intnet.be/</a>><br class="">
Uuid: cfc83cf2-b719-40c7-afea-b23accc714c3<br class="">
State: Peer in Cluster (Connected)<br class="">
<br class="">
Hostname: <a href="http://gfs02a-dcg.intnet.be" class="">gfs02a-dcg.intnet.be</a> <<a href="http://gfs02a-dcg.intnet.be/" class="">http://gfs02a-dcg.intnet.be/</a>><br class="">
Uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f<br class="">
*State: Peer Rejected (Connected)*<br class="">
<br class="">
In the log file /var/log/glusterfs/etc-glusterfs-glusterd.vol.log the<br class="">
following entries are written:<br class="">
<br class="">
[2015-09-11 09:37:49.405906] I<br class="">
[glusterd-handler.c:1031:__glusterd_handle_cli_probe] 0-glusterd:<br class="">
Received CLI probe req <a href="http://gfs02a-dcg.intnet.be" class="">gfs02a-dcg.intnet.be</a><br class="">
<<a href="http://gfs02a-dcg.intnet.be/" class="">http://gfs02a-dcg.intnet.be/</a>> 24007<br class="">
[2015-09-11 09:37:49.428630] I<br class="">
[glusterd-handler.c:3198:glusterd_probe_begin] 0-glusterd: Unable to<br class="">
find peerinfo for host: <a href="http://gfs02a-dcg.intnet.be" class="">gfs02a-dcg.intnet.be</a><br class="">
<<a href="http://gfs02a-dcg.intnet.be/" class="">http://gfs02a-dcg.intnet.be/</a>> (24007)<br class="">
[2015-09-11 09:37:49.438636] I<br class="">
[rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting<br class="">
frame-timeout to 600<br class="">
[2015-09-11 09:37:49.440513] I<br class="">
[glusterd-handler.c:3131:glusterd_friend_add] 0-management: connect<br class="">
returned 0<br class="">
[2015-09-11 09:37:49.474316] I<br class="">
[glusterd-rpc-ops.c:245:__glusterd_probe_cbk] 0-management: Received<br class="">
probe resp from uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f, host:<br class="">
<a href="http://gfs02a-dcg.intnet.be" class="">gfs02a-dcg.intnet.be</a> <<a href="http://gfs02a-dcg.intnet.be/" class="">http://gfs02a-dcg.intnet.be/</a>><br class="">
[2015-09-11 09:37:49.481801] I<br class="">
[glusterd-rpc-ops.c:387:__glusterd_probe_cbk] 0-glusterd: Received<br class="">
resp to probe req<br class="">
[2015-09-11 09:37:51.650265] I<br class="">
[glusterd-rpc-ops.c:437:__glusterd_friend_add_cbk] 0-glusterd:<br class="">
Received ACC from uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f, host:<br class="">
<a href="http://gfs02a-dcg.intnet.be" class="">gfs02a-dcg.intnet.be</a> <<a href="http://gfs02a-dcg.intnet.be/" class="">http://gfs02a-dcg.intnet.be/</a>>, port: 0<br class="">
[2015-09-11 09:37:51.665861] I<br class="">
[glusterd-handshake.c:1119:__glusterd_mgmt_hndsk_versions_ack]<br class="">
0-management: using the op-version 30603<br class="">
[2015-09-11 09:37:51.690170] I<br class="">
[glusterd-handler.c:2543:__glusterd_handle_probe_query] 0-glusterd:<br class="">
Received probe from uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f<br class="">
[2015-09-11 09:37:51.692652] I<br class="">
[glusterd-handler.c:2595:__glusterd_handle_probe_query] 0-glusterd:<br class="">
Responded to <a href="http://gfs02a-dcg.intnet.be" class="">gfs02a-dcg.intnet.be</a> <<a href="http://gfs02a-dcg.intnet.be/" class="">http://gfs02a-dcg.intnet.be/</a>>,<br class="">
op_ret: 0, op_errno: 0, ret: 0<br class="">
[2015-09-11 09:37:51.706203] I<br class="">
[glusterd-handler.c:2232:__glusterd_handle_incoming_friend_req]<br class="">
0-glusterd: Received probe from uuid:<br class="">
29592d5b-242b-43b5-afc5-5f9a1496d59f<br class="">
*[2015-09-11 09:37:51.708909] E [MSGID: 106010]<br class="">
[glusterd-utils.c:3297:glusterd_compare_friend_volume] 0-management:<br class="">
Version of Cksums public differ. local cksum = 1932535021, remote<br class="">
cksum = 2474653383 on peer <a href="http://gfs02a-dcg.intnet.be" class="">gfs02a-dcg.intnet.be</a><br class="">
<<a href="http://gfs02a-dcg.intnet.be/" class="">http://gfs02a-dcg.intnet.be/</a>>*<br class="">
[2015-09-11 09:37:51.709026] I<br class="">
[glusterd-handler.c:3367:glusterd_xfer_friend_add_resp] 0-glusterd:<br class="">
Responded to <a href="http://gfs02a-dcg.intnet.be" class="">gfs02a-dcg.intnet.be</a> <<a href="http://gfs02a-dcg.intnet.be/" class="">http://gfs02a-dcg.intnet.be/</a>> (0),<br class="">
ret: 0<br class="">
[2015-09-11 09:37:55.537231] I<br class="">
[glusterd-handler.c:1241:__glusterd_handle_cli_list_friends]<br class="">
0-glusterd: Received cli list req<br class="">
<br class="">
The exact same error appears while probing the second node (gfs02b).<br class="">
<br class="">
Anyone any idea how to solve this?<br class="">
<br class="">
Thanks in advance.<br class="">
<br class="">
Kind regards<br class="">
Davy<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
Gluster-users mailing list<br class="">
<a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">
http://www.gluster.org/mailman/listinfo/gluster-users<br class="">
</blockquote>
<br class="">
</blockquote>
<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
Gluster-users mailing list<br class="">
<a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">
http://www.gluster.org/mailman/listinfo/gluster-users<br class="">
<br class="">
</blockquote>
</blockquote>
</blockquote>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>