<div dir="ltr"><div>FYI Gluster storage node hostnames are gluster0[1-6].</div><div><br></div>Full dump attached. I see a few clients not on 30706. Most notably the two debian 7 servers (using packages from <a href="http://gluster.org">gluster.org</a>) seem to be running lower op versions than the centos7 machines (every other client in <a href="http://10.0.231.0/24">10.0.231.0/24</a> subnet).<div><br></div><div><div>glusterd.client1.identifier=<a href="http://10.0.231.10:1023">10.0.231.10:1023</a> <-- <b>debian 7</b>, glusterfs 3.7.6 built on Feb 4 2016 06:25:19</div><div>glusterd.client1.volname=storage</div><div>glusterd.client1.max-op-version=30603</div><div>glusterd.client1.min-op-version=1</div><div><br></div><div>glusterd.client2.identifier=<a href="http://10.0.231.51:65515">10.0.231.51:65515</a> <-- gluster02 (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)</div><div>glusterd.client2.volname=</div><div>glusterd.client2.max-op-version=0</div><div>glusterd.client2.min-op-version=0</div><div><br></div><div>glusterd.client3.identifier=<a href="http://10.0.231.54:65521">10.0.231.54:65521</a> <-- gluster05 (no actual mounts)</div><div>glusterd.client3.volname=</div><div>glusterd.client3.max-op-version=0</div><div>glusterd.client3.min-op-version=0</div><div><br></div><div>glusterd.client4.identifier=<a href="http://10.0.231.11:1022">10.0.231.11:1022</a> <---<b> debian 7</b>, glusterfs 3.7.6 built on Feb 4 2016 06:25:19<br></div><div>glusterd.client4.volname=storage</div><div>glusterd.client4.max-op-version=30603</div><div>glusterd.client4.min-op-version=1</div><div><br></div><div>glusterd.client5.identifier=<a href="http://10.0.231.55:65530">10.0.231.55:65530</a> <-- gluster06 (no actual mounts)</div><div>glusterd.client5.volname=</div><div>glusterd.client5.max-op-version=0</div><div>glusterd.client5.min-op-version=0</div><div><br></div><div>glusterd.client6.identifier=<a href="http://10.0.231.53:65516">10.0.231.53:65516</a> <-- gluster04 (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)</div><div>glusterd.client6.volname=</div><div>glusterd.client6.max-op-version=0</div><div>glusterd.client6.min-op-version=0</div><div><br></div><div>glusterd.client7.identifier=<a href="http://10.0.231.50:65529">10.0.231.50:65529</a></div><div>glusterd.client7.volname=export-domain-storage.10.0.231.50.mnt-lv-export-domain-storage-export-domain-storage</div><div>glusterd.client7.max-op-version=30706</div><div>glusterd.client7.min-op-version=1</div></div><div><br></div><div>...</div><div><br></div><div>Debian package info:</div><div><div>apt-cache policy glusterfs-client</div><div>glusterfs-client:</div><div> Installed: 3.7.6-2</div><div> Candidate: 3.7.6-2</div><div> Version table:</div><div> *** 3.7.6-2 0</div><div> 500 <a href="http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/">http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/</a> wheezy/main amd64 Packages</div></div><div><br></div><div><br></div><div>Thanks</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 2, 2016 at 10:29 PM, Gaurav Garg <span dir="ltr"><<a href="mailto:ggarg@redhat.com" target="_blank">ggarg@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Steve,<br>
<br>
As atin pointed out to take statedump by running #kill -SIGUSR1 $(pidof glusterd) command. it will create .dump file in /var/run/gluster/ directory. client-op-version information will be present in dump file.<br>
<br>
Thanks,<br>
~Gaurav<br>
<span class="im HOEnZb"><br>
----- Original Message -----<br>
From: "Steve Dainard" <<a href="mailto:sdainard@spd1.com">sdainard@spd1.com</a>><br>
</span><div class="HOEnZb"><div class="h5">To: "Gaurav Garg" <<a href="mailto:ggarg@redhat.com">ggarg@redhat.com</a>><br>
Cc: "<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a> List" <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
Sent: Thursday, March 3, 2016 12:07:25 AM<br>
Subject: Re: [Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set<br>
<br>
>From the the client side logs I can see version info on mount:<br>
<br>
Final graph:<br>
+------------------------------------------------------------------------------+<br>
1: volume storage-client-0<br>
2: type protocol/client<br>
3: option clnt-lk-version 1<br>
4: option volfile-checksum 0<br>
5: option volfile-key /storage<br>
6: option client-version 3.7.6<br>
7: option process-uuid<br>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0<br>
8: option fops-version 1298437<br>
9: option ping-timeout 42<br>
10: option remote-host 10.0.231.50<br>
11: option remote-subvolume /mnt/raid6-storage/storage<br>
12: option transport-type socket<br>
13: option send-gids true<br>
14: end-volume<br>
15:<br>
16: volume storage-client-1<br>
17: type protocol/client<br>
18: option clnt-lk-version 1<br>
19: option volfile-checksum 0<br>
20: option volfile-key /storage<br>
21: option client-version 3.7.6<br>
22: option process-uuid<br>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0<br>
23: option fops-version 1298437<br>
24: option ping-timeout 42<br>
25: option remote-host 10.0.231.51<br>
26: option remote-subvolume /mnt/raid6-storage/storage<br>
27: option transport-type socket<br>
28: option send-gids true<br>
29: end-volume<br>
30:<br>
31: volume storage-client-2<br>
32: type protocol/client<br>
33: option clnt-lk-version 1<br>
34: option volfile-checksum 0<br>
35: option volfile-key /storage<br>
36: option client-version 3.7.6<br>
37: option process-uuid<br>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0<br>
38: option fops-version 1298437<br>
39: option ping-timeout 42<br>
40: option remote-host 10.0.231.52<br>
41: option remote-subvolume /mnt/raid6-storage/storage<br>
42: option transport-type socket<br>
43: option send-gids true<br>
44: end-volume<br>
45:<br>
46: volume storage-client-3<br>
47: type protocol/client<br>
48: option clnt-lk-version 1<br>
49: option volfile-checksum 0<br>
50: option volfile-key /storage<br>
51: option client-version 3.7.6<br>
52: option process-uuid<br>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0<br>
53: option fops-version 1298437<br>
54: option ping-timeout 42<br>
55: option remote-host 10.0.231.53<br>
56: option remote-subvolume /mnt/raid6-storage/storage<br>
57: option transport-type socket<br>
58: option send-gids true<br>
59: end-volume<br>
60:<br>
61: volume storage-client-4<br>
62: type protocol/client<br>
63: option ping-timeout 42<br>
64: option remote-host 10.0.231.54<br>
65: option remote-subvolume /mnt/raid6-storage/storage<br>
66: option transport-type socket<br>
67: option send-gids true<br>
68: end-volume<br>
69:<br>
70: volume storage-client-5<br>
71: type protocol/client<br>
72: option ping-timeout 42<br>
73: option remote-host 10.0.231.55<br>
74: option remote-subvolume /mnt/raid6-storage/storage<br>
75: option transport-type socket<br>
76: option send-gids true<br>
77: end-volume<br>
78:<br>
79: volume storage-dht<br>
80: type cluster/distribute<br>
81: subvolumes storage-client-0 storage-client-1 storage-client-2<br>
storage-client-3 storage-client-4 storage-client-5<br>
82: end-volume<br>
<br>
<br>
But not the client op-version, how can I retrieve this info?<br>
<br>
Thanks<br>
<br>
On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg <<a href="mailto:ggarg@redhat.com">ggarg@redhat.com</a>> wrote:<br>
<br>
> Hi Steve,<br>
><br>
> Which version you have upgraded client, could you tell us client<br>
> op-version after upgrade ?<br>
><br>
><br>
> have you upgraded all of your clients ?<br>
><br>
><br>
> Thanks,<br>
> Gaurav<br>
><br>
><br>
> ----- Original Message -----<br>
> From: "Steve Dainard" <<a href="mailto:sdainard@spd1.com">sdainard@spd1.com</a>><br>
> To: "<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a> List" <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
> Sent: Wednesday, March 2, 2016 1:10:27 AM<br>
> Subject: [Gluster-users] gluster 3.7.6 volume set: failed: One or more<br>
> connected clients cannot support the feature being set<br>
><br>
> Gluster 3.7.6<br>
> 'storage' is a distributed volume<br>
><br>
> # gluster volume set storage rebal-throttle lazy<br>
> volume set: failed: One or more connected clients cannot support the<br>
> feature being set. These clients need to be upgraded or disconnected before<br>
> running this command again<br>
><br>
> I found a client connected using version 3.6.7 so I upgraded &<br>
> umount/mount the gluster volume on the client but I'm still getting this<br>
> error.<br>
><br>
> I've run grep "accepted client from" /var/log/glusterfs/bricks/* | grep -v<br>
> 3.7.6 and I get a few returns from the client above, all dated last week.<br>
><br>
> I've run 'gluster volume status storage clients' and checked the connected<br>
> clients manually, they're all running 3.7.6.<br>
><br>
> /var/log/gluster/etc-glusterfs-glusterd.vol.log:<br>
> [2016-03-01 19:23:20.180821] E [MSGID: 106022]<br>
> [glusterd-utils.c:10154:glusterd_check_client_op_version_support]<br>
> 0-management: One or more c<br>
> lients don't support the required op-version<br>
> [2016-03-01 19:23:20.180853] E [MSGID: 106301]<br>
> [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of<br>
> operation 'Volume Set' fa<br>
> iled on localhost : One or more connected clients cannot support the<br>
> feature being set. These clients need to be upgraded or disconnected be<br>
> fore running this command again<br>
><br>
> Also tried setting the diagnostics.brick-log-level logging level and got<br>
> the same error.<br>
><br>
> /var/lib/glusterd/vols/storage/info:<br>
> type=0<br>
> count=6<br>
> status=1<br>
> sub_count=0<br>
> stripe_count=1<br>
> replica_count=1<br>
> disperse_count=0<br>
> redundancy_count=0<br>
> version=26<br>
> transport-type=0<br>
> volume-id=26d355cb-c486-481f-ac16-e25390e73775<br>
> username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c<br>
> password=<br>
> op-version=3<br>
> client-op-version=3<br>
> quota-version=1<br>
> parent_volname=N/A<br>
> restored_from_snap=00000000-0000-0000-0000-000000000000<br>
> snap-max-hard-limit=256<br>
> features.quota-deem-statfs=on<br>
> features.inode-quota=on<br>
> diagnostics.brick-log-level=WARNING<br>
> features.quota=on<br>
> performance.readdir-ahead=on<br>
> performance.cache-size=1GB<br>
> performance.stat-prefetch=on<br>
> brick-0=10.0.231.50:-mnt-raid6-storage-storage<br>
> brick-1=10.0.231.51:-mnt-raid6-storage-storage<br>
> brick-2=10.0.231.52:-mnt-raid6-storage-storage<br>
> brick-3=10.0.231.53:-mnt-raid6-storage-storage<br>
> brick-4=10.0.231.54:-mnt-raid6-storage-storage<br>
> brick-5=10.0.231.55:-mnt-raid6-storage-storage<br>
><br>
><br>
> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
><br>
</div></div></blockquote></div><br></div>