<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<br>
<div class="moz-cite-prefix">On 02/26/2016 01:53 AM, Mohammed Rafi K
C wrote:<br>
</div>
<blockquote cite="mid:56CF62A8.2080004@redhat.com" type="cite">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<br>
<br>
<div class="moz-cite-prefix">On 02/26/2016 01:32 AM, Steve Dainard
wrote:<br>
</div>
<blockquote
cite="mid:CAEMJtDvEbpz0y+BibB8QcXiFFA3OPmOZT_d9zCJTh4q0UxxjCw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>I haven't done anything more than peer thus far, so I'm a
bit confused as to how the volume info fits in, can you
expand on this a bit?<br>
</div>
<div><br>
</div>
<div>Failed commits? Is this split brain on the replica
volumes? I don't get any return from 'gluster volume heal
<volname> info' on all the replica volumes, but if I
try a gluster volume heal <volname> full I get:
'Launching heal operation to perform full self heal on
volume <volname> has been unsuccessful'.</div>
</div>
</blockquote>
<br>
forget about this. it is not for metadata selfheal .<br>
<br>
<blockquote
cite="mid:CAEMJtDvEbpz0y+BibB8QcXiFFA3OPmOZT_d9zCJTh4q0UxxjCw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>I have 5 volumes total.</div>
<div><br>
</div>
<div>'Replica 3' volumes running on gluster01/02/03:</div>
<div>vm-storage</div>
<div>iso-storage</div>
<div>export-domain-storage</div>
<div>env-modules</div>
<div><br>
</div>
<div>And one distributed only volume 'storage' info shown
below:<br>
</div>
<div>
<div><br>
</div>
<div><b>From existing host gluster01/02:</b></div>
<div>
<div>type=0</div>
<div>count=4</div>
<div>status=1</div>
<div>sub_count=0</div>
<div>stripe_count=1</div>
<div>replica_count=1</div>
<div>disperse_count=0</div>
<div>redundancy_count=0</div>
<div>version=25</div>
<div>transport-type=0</div>
<div>volume-id=26d355cb-c486-481f-ac16-e25390e73775</div>
<div>username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c</div>
<div>password=</div>
<div>op-version=3</div>
<div>client-op-version=3</div>
<div>quota-version=1</div>
<div>parent_volname=N/A</div>
<div>restored_from_snap=00000000-0000-0000-0000-000000000000</div>
<div>snap-max-hard-limit=256</div>
<div>features.quota-deem-statfs=on</div>
<div>features.inode-quota=on</div>
<div>diagnostics.brick-log-level=WARNING</div>
<div>features.quota=on</div>
<div>performance.readdir-ahead=on</div>
<div>performance.cache-size=1GB</div>
<div>performance.stat-prefetch=on</div>
<div>brick-0=10.0.231.50:-mnt-raid6-storage-storage</div>
<div>brick-1=10.0.231.51:-mnt-raid6-storage-storage</div>
<div>brick-2=10.0.231.52:-mnt-raid6-storage-storage</div>
<div>brick-3=10.0.231.53:-mnt-raid6-storage-storage</div>
</div>
<div><br>
</div>
<div>
<div><b>From existing host gluster03/04:</b><br>
</div>
<div>
<div>type=0</div>
<div>count=4</div>
<div>status=1</div>
<div>sub_count=0</div>
<div>stripe_count=1</div>
<div>replica_count=1</div>
<div>disperse_count=0</div>
<div>redundancy_count=0</div>
<div>version=25</div>
<div>transport-type=0</div>
<div>volume-id=26d355cb-c486-481f-ac16-e25390e73775</div>
<div>username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c</div>
<div>password=</div>
<div>op-version=3</div>
<div>client-op-version=3</div>
<div>quota-version=1</div>
<div>parent_volname=N/A</div>
<div>restored_from_snap=00000000-0000-0000-0000-000000000000</div>
<div>snap-max-hard-limit=256</div>
<div>features.quota-deem-statfs=on</div>
<div>features.inode-quota=on</div>
<div>performance.stat-prefetch=on</div>
<div>performance.cache-size=1GB</div>
<div>performance.readdir-ahead=on</div>
<div>features.quota=on</div>
<div>diagnostics.brick-log-level=WARNING</div>
<div>brick-0=10.0.231.50:-mnt-raid6-storage-storage</div>
<div>brick-1=10.0.231.51:-mnt-raid6-storage-storage</div>
<div>brick-2=10.0.231.52:-mnt-raid6-storage-storage</div>
<div>brick-3=10.0.231.53:-mnt-raid6-storage-storage</div>
</div>
<div><br>
</div>
<div>So far between gluster01/02 and gluster03/04 the
configs are the same, although the ordering is different
for some of the features.</div>
<div><br>
</div>
<div>On gluster05/06 the ordering is different again, and
the quota-version=0 instead of 1.</div>
</div>
</div>
</div>
</blockquote>
<br>
This is why the peer shows as rejected. Can you check the
op-version of all the glusterd including the one which is in
reject state. you can find out the op-version here in
/var/lib/glusterd/glusterd.info <br>
</blockquote>
<br>
If all the op-version are same and 3.7.6, then to work-around the
issue, you can manually make it quota-version=1, and restarting the
glusterd will solve the problem, But I would strongly recommend you
to figure out the RCA. May be you can file a bug for this.<br>
<br>
Rafi<br>
<br>
<blockquote cite="mid:56CF62A8.2080004@redhat.com" type="cite"> <br>
Rafi KC<br>
<br>
<blockquote
cite="mid:CAEMJtDvEbpz0y+BibB8QcXiFFA3OPmOZT_d9zCJTh4q0UxxjCw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div><br>
</div>
<div><b>From new hosts gluster05/gluster06:</b></div>
<div>type=0</div>
<div>count=4</div>
<div>status=1</div>
<div>sub_count=0</div>
<div>stripe_count=1</div>
<div>replica_count=1</div>
<div>disperse_count=0</div>
<div>redundancy_count=0</div>
<div>version=25</div>
<div>transport-type=0</div>
<div>volume-id=26d355cb-c486-481f-ac16-e25390e73775</div>
<div>username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c</div>
<div>password=</div>
<div>op-version=3</div>
<div>client-op-version=3</div>
<div>quota-version=0</div>
<div>parent_volname=N/A</div>
<div>restored_from_snap=00000000-0000-0000-0000-000000000000</div>
<div>snap-max-hard-limit=256</div>
<div>performance.stat-prefetch=on</div>
<div>performance.cache-size=1GB</div>
<div>performance.readdir-ahead=on</div>
<div>features.quota=on</div>
<div>diagnostics.brick-log-level=WARNING</div>
<div>features.inode-quota=on</div>
<div>features.quota-deem-statfs=on</div>
<div>brick-0=10.0.231.50:-mnt-raid6-storage-storage</div>
<div>brick-1=10.0.231.51:-mnt-raid6-storage-storage</div>
<div>brick-2=10.0.231.52:-mnt-raid6-storage-storage</div>
<div>brick-3=10.0.231.53:-mnt-raid6-storage-storage</div>
</div>
<div><br>
</div>
</div>
<div>Also, I forgot to mention that when I initially peer'd
the two new hosts, glusterd crashed on gluster03 and had to
be restarted (log attached) but has been fine since.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Steve</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Feb 25, 2016 at 11:27 AM,
Mohammed Rafi K C <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:rkavunga@redhat.com"
target="_blank">rkavunga@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><span class=""> <br>
<br>
<div>On 02/25/2016 11:45 PM, Steve Dainard wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hello,<br>
<br>
I upgraded from 3.6.6 to 3.7.6 a couple weeks ago.
I just peered 2 new nodes to a 4 node cluster and
gluster peer status is:<br>
<br>
# gluster peer status <b><-- from node
gluster01</b><br>
Number of Peers: 5<br>
<br>
Hostname: 10.0.231.51<br>
Uuid: b01de59a-4428-486b-af49-cb486ab44a07<br>
State: Peer in Cluster (Connected)<br>
<br>
Hostname: 10.0.231.52<br>
Uuid: 75143760-52a3-4583-82bb-a9920b283dac<br>
State: Peer in Cluster (Connected)<br>
<br>
Hostname: 10.0.231.53<br>
Uuid: 2c0b8bb6-825a-4ddd-9958-d8b46e9a2411<br>
State: Peer in Cluster (Connected)<br>
<br>
Hostname: 10.0.231.54 <b><-- new node
gluster05</b><br>
Uuid: 408d88d6-0448-41e8-94a3-bf9f98255d9c<br>
<b>State: Peer Rejected (Connected)</b><br>
<br>
Hostname: 10.0.231.55 <b><-- new node gluster06</b><br>
Uuid: 9c155c8e-2cd1-4cfc-83af-47129b582fd3<br>
<b>State: Peer Rejected (Connected)</b><br>
</div>
</blockquote>
<br>
</span> Looks like your configuration files are
mismatching, ie the checksum calculation differs on this
two node than the others,<br>
<br>
Did you had any failed commit ?<br>
<br>
Compare your /var/lib/glusterd/<volname>/info of
the failed node against good one, mostly you could see
some difference.<br>
<br>
can you paste the /var/lib/glusterd/<volname>/info
?<br>
<br>
Regards<br>
Rafi KC<br>
<br>
<br>
<blockquote type="cite"><span class="">
<div dir="ltr">
<div><b><br>
</b></div>
<div>I followed the write-up here: <a
moz-do-not-send="true"
href="http://www.gluster.org/community/documentation/index.php/Resolving_Peer_Rejected"
target="_blank">http://www.gluster.org/community/documentation/index.php/Resolving_Peer_Rejected</a>
and the two new nodes peer'd properly but after
a reboot of the two new nodes I'm seeing the
same Peer Rejected (Connected) State.</div>
<div><br>
</div>
<div>I've attached logs from an existing node, and
the two new nodes.</div>
<div><br>
</div>
<div>Thanks for any suggestions,</div>
<div>Steve</div>
<div><br>
</div>
<div>
<div><br>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</span>
<pre>_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>