<p dir="ltr"></p>
<p dir="ltr">-Atin<br>
Sent from one plus one<br>
On 04-Mar-2016 3:35 am, "Steve Dainard" <<a href="mailto:sdainard@spd1.com">sdainard@spd1.com</a>> wrote:<br>
><br>
> FYI Gluster storage node hostnames are gluster0[1-6].<br>
><br>
> Full dump attached. I see a few clients not on 30706. Most notably the two debian 7 servers (using packages from <a href="http://gluster.org">gluster.org</a>) seem to be running lower op versions than the centos7 machines (every other client in <a href="http://10.0.231.0/24">10.0.231.0/24</a> subnet).<br>
><br>
> glusterd.client1.identifier=<a href="http://10.0.231.10:1023">10.0.231.10:1023</a> <-- debian 7, glusterfs 3.7.6 built on Feb 4 2016 06:25:19<br>
> glusterd.client1.volname=storage<br>
> glusterd.client1.max-op-version=30603<br>
This is running with 3.6.3 and that's why volume set fails. Unmount this client and upgrade and remount it back.<br>
> glusterd.client1.min-op-version=1<br>
><br>
> glusterd.client2.identifier=<a href="http://10.0.231.51:65515">10.0.231.51:65515</a> <-- gluster02 (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)<br>
> glusterd.client2.volname=<br>
> glusterd.client2.max-op-version=0<br>
> glusterd.client2.min-op-version=0<br>
><br>
> glusterd.client3.identifier=<a href="http://10.0.231.54:65521">10.0.231.54:65521</a> <-- gluster05 (no actual mounts)<br>
> glusterd.client3.volname=<br>
> glusterd.client3.max-op-version=0<br>
> glusterd.client3.min-op-version=0<br>
><br>
> glusterd.client4.identifier=<a href="http://10.0.231.11:1022">10.0.231.11:1022</a> <--- debian 7, glusterfs 3.7.6 built on Feb 4 2016 06:25:19<br>
> glusterd.client4.volname=storage<br>
> glusterd.client4.max-op-version=30603<br>
> glusterd.client4.min-op-version=1<br>
><br>
> glusterd.client5.identifier=<a href="http://10.0.231.55:65530">10.0.231.55:65530</a> <-- gluster06 (no actual mounts)<br>
> glusterd.client5.volname=<br>
> glusterd.client5.max-op-version=0<br>
> glusterd.client5.min-op-version=0<br>
><br>
> glusterd.client6.identifier=<a href="http://10.0.231.53:65516">10.0.231.53:65516</a> <-- gluster04 (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)<br>
> glusterd.client6.volname=<br>
> glusterd.client6.max-op-version=0<br>
> glusterd.client6.min-op-version=0<br>
><br>
> glusterd.client7.identifier=<a href="http://10.0.231.50:65529">10.0.231.50:65529</a><br>
> glusterd.client7.volname=export-domain-storage.10.0.231.50.mnt-lv-export-domain-storage-export-domain-storage<br>
> glusterd.client7.max-op-version=30706<br>
> glusterd.client7.min-op-version=1<br>
><br>
> ...<br>
><br>
> Debian package info:<br>
> apt-cache policy glusterfs-client<br>
> glusterfs-client:<br>
> Installed: 3.7.6-2<br>
> Candidate: 3.7.6-2<br>
> Version table:<br>
> *** 3.7.6-2 0<br>
> 500 <a href="http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/">http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/</a> wheezy/main amd64 Packages<br>
><br>
><br>
> Thanks<br>
><br>
> On Wed, Mar 2, 2016 at 10:29 PM, Gaurav Garg <<a href="mailto:ggarg@redhat.com">ggarg@redhat.com</a>> wrote:<br>
>><br>
>> Hi Steve,<br>
>><br>
>> As atin pointed out to take statedump by running #kill -SIGUSR1 $(pidof glusterd) command. it will create .dump file in /var/run/gluster/ directory. client-op-version information will be present in dump file.<br>
>><br>
>> Thanks,<br>
>> ~Gaurav<br>
>><br>
>> ----- Original Message -----<br>
>> From: "Steve Dainard" <<a href="mailto:sdainard@spd1.com">sdainard@spd1.com</a>><br>
>> To: "Gaurav Garg" <<a href="mailto:ggarg@redhat.com">ggarg@redhat.com</a>><br>
>> Cc: "<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a> List" <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
>> Sent: Thursday, March 3, 2016 12:07:25 AM<br>
>> Subject: Re: [Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set<br>
>><br>
>> From the the client side logs I can see version info on mount:<br>
>><br>
>> Final graph:<br>
>> +------------------------------------------------------------------------------+<br>
>> 1: volume storage-client-0<br>
>> 2: type protocol/client<br>
>> 3: option clnt-lk-version 1<br>
>> 4: option volfile-checksum 0<br>
>> 5: option volfile-key /storage<br>
>> 6: option client-version 3.7.6<br>
>> 7: option process-uuid<br>
>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0<br>
>> 8: option fops-version 1298437<br>
>> 9: option ping-timeout 42<br>
>> 10: option remote-host 10.0.231.50<br>
>> 11: option remote-subvolume /mnt/raid6-storage/storage<br>
>> 12: option transport-type socket<br>
>> 13: option send-gids true<br>
>> 14: end-volume<br>
>> 15:<br>
>> 16: volume storage-client-1<br>
>> 17: type protocol/client<br>
>> 18: option clnt-lk-version 1<br>
>> 19: option volfile-checksum 0<br>
>> 20: option volfile-key /storage<br>
>> 21: option client-version 3.7.6<br>
>> 22: option process-uuid<br>
>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0<br>
>> 23: option fops-version 1298437<br>
>> 24: option ping-timeout 42<br>
>> 25: option remote-host 10.0.231.51<br>
>> 26: option remote-subvolume /mnt/raid6-storage/storage<br>
>> 27: option transport-type socket<br>
>> 28: option send-gids true<br>
>> 29: end-volume<br>
>> 30:<br>
>> 31: volume storage-client-2<br>
>> 32: type protocol/client<br>
>> 33: option clnt-lk-version 1<br>
>> 34: option volfile-checksum 0<br>
>> 35: option volfile-key /storage<br>
>> 36: option client-version 3.7.6<br>
>> 37: option process-uuid<br>
>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0<br>
>> 38: option fops-version 1298437<br>
>> 39: option ping-timeout 42<br>
>> 40: option remote-host 10.0.231.52<br>
>> 41: option remote-subvolume /mnt/raid6-storage/storage<br>
>> 42: option transport-type socket<br>
>> 43: option send-gids true<br>
>> 44: end-volume<br>
>> 45:<br>
>> 46: volume storage-client-3<br>
>> 47: type protocol/client<br>
>> 48: option clnt-lk-version 1<br>
>> 49: option volfile-checksum 0<br>
>> 50: option volfile-key /storage<br>
>> 51: option client-version 3.7.6<br>
>> 52: option process-uuid<br>
>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0<br>
>> 53: option fops-version 1298437<br>
>> 54: option ping-timeout 42<br>
>> 55: option remote-host 10.0.231.53<br>
>> 56: option remote-subvolume /mnt/raid6-storage/storage<br>
>> 57: option transport-type socket<br>
>> 58: option send-gids true<br>
>> 59: end-volume<br>
>> 60:<br>
>> 61: volume storage-client-4<br>
>> 62: type protocol/client<br>
>> 63: option ping-timeout 42<br>
>> 64: option remote-host 10.0.231.54<br>
>> 65: option remote-subvolume /mnt/raid6-storage/storage<br>
>> 66: option transport-type socket<br>
>> 67: option send-gids true<br>
>> 68: end-volume<br>
>> 69:<br>
>> 70: volume storage-client-5<br>
>> 71: type protocol/client<br>
>> 72: option ping-timeout 42<br>
>> 73: option remote-host 10.0.231.55<br>
>> 74: option remote-subvolume /mnt/raid6-storage/storage<br>
>> 75: option transport-type socket<br>
>> 76: option send-gids true<br>
>> 77: end-volume<br>
>> 78:<br>
>> 79: volume storage-dht<br>
>> 80: type cluster/distribute<br>
>> 81: subvolumes storage-client-0 storage-client-1 storage-client-2<br>
>> storage-client-3 storage-client-4 storage-client-5<br>
>> 82: end-volume<br>
>><br>
>><br>
>> But not the client op-version, how can I retrieve this info?<br>
>><br>
>> Thanks<br>
>><br>
>> On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg <<a href="mailto:ggarg@redhat.com">ggarg@redhat.com</a>> wrote:<br>
>><br>
>> > Hi Steve,<br>
>> ><br>
>> > Which version you have upgraded client, could you tell us client<br>
>> > op-version after upgrade ?<br>
>> ><br>
>> ><br>
>> > have you upgraded all of your clients ?<br>
>> ><br>
>> ><br>
>> > Thanks,<br>
>> > Gaurav<br>
>> ><br>
>> ><br>
>> > ----- Original Message -----<br>
>> > From: "Steve Dainard" <<a href="mailto:sdainard@spd1.com">sdainard@spd1.com</a>><br>
>> > To: "<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a> List" <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
>> > Sent: Wednesday, March 2, 2016 1:10:27 AM<br>
>> > Subject: [Gluster-users] gluster 3.7.6 volume set: failed: One or more<br>
>> > connected clients cannot support the feature being set<br>
>> ><br>
>> > Gluster 3.7.6<br>
>> > 'storage' is a distributed volume<br>
>> ><br>
>> > # gluster volume set storage rebal-throttle lazy<br>
>> > volume set: failed: One or more connected clients cannot support the<br>
>> > feature being set. These clients need to be upgraded or disconnected before<br>
>> > running this command again<br>
>> ><br>
>> > I found a client connected using version 3.6.7 so I upgraded &<br>
>> > umount/mount the gluster volume on the client but I'm still getting this<br>
>> > error.<br>
>> ><br>
>> > I've run grep "accepted client from" /var/log/glusterfs/bricks/* | grep -v<br>
>> > 3.7.6 and I get a few returns from the client above, all dated last week.<br>
>> ><br>
>> > I've run 'gluster volume status storage clients' and checked the connected<br>
>> > clients manually, they're all running 3.7.6.<br>
>> ><br>
>> > /var/log/gluster/etc-glusterfs-glusterd.vol.log:<br>
>> > [2016-03-01 19:23:20.180821] E [MSGID: 106022]<br>
>> > [glusterd-utils.c:10154:glusterd_check_client_op_version_support]<br>
>> > 0-management: One or more c<br>
>> > lients don't support the required op-version<br>
>> > [2016-03-01 19:23:20.180853] E [MSGID: 106301]<br>
>> > [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of<br>
>> > operation 'Volume Set' fa<br>
>> > iled on localhost : One or more connected clients cannot support the<br>
>> > feature being set. These clients need to be upgraded or disconnected be<br>
>> > fore running this command again<br>
>> ><br>
>> > Also tried setting the diagnostics.brick-log-level logging level and got<br>
>> > the same error.<br>
>> ><br>
>> > /var/lib/glusterd/vols/storage/info:<br>
>> > type=0<br>
>> > count=6<br>
>> > status=1<br>
>> > sub_count=0<br>
>> > stripe_count=1<br>
>> > replica_count=1<br>
>> > disperse_count=0<br>
>> > redundancy_count=0<br>
>> > version=26<br>
>> > transport-type=0<br>
>> > volume-id=26d355cb-c486-481f-ac16-e25390e73775<br>
>> > username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c<br>
>> > password=<br>
>> > op-version=3<br>
>> > client-op-version=3<br>
>> > quota-version=1<br>
>> > parent_volname=N/A<br>
>> > restored_from_snap=00000000-0000-0000-0000-000000000000<br>
>> > snap-max-hard-limit=256<br>
>> > features.quota-deem-statfs=on<br>
>> > features.inode-quota=on<br>
>> > diagnostics.brick-log-level=WARNING<br>
>> > features.quota=on<br>
>> > performance.readdir-ahead=on<br>
>> > performance.cache-size=1GB<br>
>> > performance.stat-prefetch=on<br>
>> > brick-0=10.0.231.50:-mnt-raid6-storage-storage<br>
>> > brick-1=10.0.231.51:-mnt-raid6-storage-storage<br>
>> > brick-2=10.0.231.52:-mnt-raid6-storage-storage<br>
>> > brick-3=10.0.231.53:-mnt-raid6-storage-storage<br>
>> > brick-4=10.0.231.54:-mnt-raid6-storage-storage<br>
>> > brick-5=10.0.231.55:-mnt-raid6-storage-storage<br>
>> ><br>
>> ><br>
>> > _______________________________________________<br>
>> > Gluster-users mailing list<br>
>> > <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
>> > <a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
>> ><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</p>