<div dir="ltr">And I'll mention again, those Debian clients are not running 3.6. As shown in previous email, they are both running:<div><br></div><div><div style=""><div style="font-size:12.8px"># apt-cache policy glusterfs-client</div><div style="font-size:12.8px">glusterfs-client:</div><div style="font-size:12.8px"> Installed: 3.7.6-2</div><div style="font-size:12.8px"> Candidate: 3.7.6-2</div><div style="font-size:12.8px"> Version table:</div><div style="font-size:12.8px"> *** 3.7.6-2 0</div><div style="font-size:12.8px">    500 <a href="http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/" target="_blank">http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/</a> wheezy/main amd64 Packages</div><div style="font-size:12.8px"><br></div><div style=""><div style=""><span style="font-size:12.8px"># glusterfs --version</span></div><div style=""><span style="font-size:12.8px">glusterfs 3.7.6 built on Feb  4 2016 06:25:19</span></div><div style=""><span style="font-size:12.8px">Repository revision: git://<a href="http://git.gluster.com/glusterfs.git">git.gluster.com/glusterfs.git</a></span></div><div style=""><span style="font-size:12.8px">Copyright (c) 2006-2013 Red Hat, Inc. <<a href="http://www.redhat.com/">http://www.redhat.com/</a>></span></div><div style=""><span style="font-size:12.8px">GlusterFS comes with ABSOLUTELY NO WARRANTY.</span></div><div style=""><span style="font-size:12.8px">It is licensed to you under your choice of the GNU Lesser</span></div><div style=""><span style="font-size:12.8px">General Public License, version 3 or any later version (LGPLv3</span></div><div style=""><span style="font-size:12.8px">or later), or the GNU General Public License, version 2 (GPLv2),</span></div><div style=""><span style="font-size:12.8px">in all cases as published by the Free Software Foundation.</span></div></div><div style="font-size:12.8px"><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 4, 2016 at 8:53 AM, Steve Dainard <span dir="ltr"><<a href="mailto:sdainard@spd1.com" target="_blank">sdainard@spd1.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Except that even when I unmount gluster from both Debian clients, I'm still getting the same error, and a dump of those clients after unmounting still shows:<div><br></div><div><div>glusterd.client1.identifier=<a href="http://10.0.231.10:1023" target="_blank">10.0.231.10:1023</a></div><div>glusterd.client1.volname=storage</div><div>glusterd.client1.max-op-version=30603</div><div>glusterd.client1.min-op-version=1</div><div><br></div><div>glusterd.client4.identifier=<a href="http://10.0.231.11:1022" target="_blank">10.0.231.11:1022</a></div><div>glusterd.client4.volname=storage</div><div>glusterd.client4.max-op-version=30603</div><div>glusterd.client4.min-op-version=1</div><div><br></div><div><br></div><div><br></div><div><br></div></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 3, 2016 at 5:28 PM, Atin Mukherjee <span dir="ltr"><<a href="mailto:atin.mukherjee83@gmail.com" target="_blank">atin.mukherjee83@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr"></p>
<p dir="ltr"><span>-Atin<br>
Sent from one plus one<br></span><span>
On 04-Mar-2016 3:35 am, "Steve Dainard" <<a href="mailto:sdainard@spd1.com" target="_blank">sdainard@spd1.com</a>> wrote:<br>
><br>
> FYI Gluster storage node hostnames are gluster0[1-6].<br>
><br>
> Full dump attached. I see a few clients not on 30706. Most notably the two debian 7 servers (using packages from <a href="http://gluster.org" target="_blank">gluster.org</a>) seem to be running lower op versions than the centos7 machines (every other client in <a href="http://10.0.231.0/24" target="_blank">10.0.231.0/24</a> subnet).<br>
><br>
> glusterd.client1.identifier=<a href="http://10.0.231.10:1023" target="_blank">10.0.231.10:1023</a> <-- debian 7, glusterfs 3.7.6 built on Feb  4 2016 06:25:19<br>
> glusterd.client1.volname=storage<br>
> glusterd.client1.max-op-version=30603<br></span>
This is running with 3.6.3 and that's why volume set fails. Unmount this client and upgrade and remount it back.</p><div><div><br>
> glusterd.client1.min-op-version=1<br>
><br>
> glusterd.client2.identifier=<a href="http://10.0.231.51:65515" target="_blank">10.0.231.51:65515</a> <-- gluster02 (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)<br>
> glusterd.client2.volname=<br>
> glusterd.client2.max-op-version=0<br>
> glusterd.client2.min-op-version=0<br>
><br>
> glusterd.client3.identifier=<a href="http://10.0.231.54:65521" target="_blank">10.0.231.54:65521</a> <-- gluster05 (no actual mounts)<br>
> glusterd.client3.volname=<br>
> glusterd.client3.max-op-version=0<br>
> glusterd.client3.min-op-version=0<br>
><br>
> glusterd.client4.identifier=<a href="http://10.0.231.11:1022" target="_blank">10.0.231.11:1022</a> <--- debian 7, glusterfs 3.7.6 built on Feb  4 2016 06:25:19<br>
> glusterd.client4.volname=storage<br>
> glusterd.client4.max-op-version=30603<br>
> glusterd.client4.min-op-version=1<br>
><br>
> glusterd.client5.identifier=<a href="http://10.0.231.55:65530" target="_blank">10.0.231.55:65530</a> <-- gluster06 (no actual mounts)<br>
> glusterd.client5.volname=<br>
> glusterd.client5.max-op-version=0<br>
> glusterd.client5.min-op-version=0<br>
><br>
> glusterd.client6.identifier=<a href="http://10.0.231.53:65516" target="_blank">10.0.231.53:65516</a> <-- gluster04 (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)<br>
> glusterd.client6.volname=<br>
> glusterd.client6.max-op-version=0<br>
> glusterd.client6.min-op-version=0<br>
><br>
> glusterd.client7.identifier=<a href="http://10.0.231.50:65529" target="_blank">10.0.231.50:65529</a><br>
> glusterd.client7.volname=export-domain-storage.10.0.231.50.mnt-lv-export-domain-storage-export-domain-storage<br>
> glusterd.client7.max-op-version=30706<br>
> glusterd.client7.min-op-version=1<br>
><br>
> ...<br>
><br>
> Debian package info:<br>
> apt-cache policy glusterfs-client<br>
> glusterfs-client:<br>
> Â Installed: 3.7.6-2<br>
> Â Candidate: 3.7.6-2<br>
> Â Version table:<br>
> Â *** 3.7.6-2 0<br>
> Â Â Â Â 500 <a href="http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/" target="_blank">http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/</a> wheezy/main amd64 Packages<br>
><br>
><br>
> Thanks<br>
><br>
> On Wed, Mar 2, 2016 at 10:29 PM, Gaurav Garg <<a href="mailto:ggarg@redhat.com" target="_blank">ggarg@redhat.com</a>> wrote:<br>
>><br>
>> Hi Steve,<br>
>><br>
>> As atin pointed out to take statedump by running #kill -SIGUSR1 $(pidof glusterd)Â command. it will create .dump file in /var/run/gluster/ directory. client-op-version information will be present in dump file.<br>
>><br>
>> Thanks,<br>
>> ~Gaurav<br>
>><br>
>> ----- Original Message -----<br>
>> From: "Steve Dainard" <<a href="mailto:sdainard@spd1.com" target="_blank">sdainard@spd1.com</a>><br>
>> To: "Gaurav Garg" <<a href="mailto:ggarg@redhat.com" target="_blank">ggarg@redhat.com</a>><br>
>> Cc: "<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a> List" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br>
>> Sent: Thursday, March 3, 2016 12:07:25 AM<br>
>> Subject: Re: [Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set<br>
>><br>
>> From the the client side logs I can see version info on mount:<br>
>><br>
>> Final graph:<br>
>> +------------------------------------------------------------------------------+<br>
>> Â 1: volume storage-client-0<br>
>> Â 2:Â Â Â type protocol/client<br>
>> Â 3:Â Â Â option clnt-lk-version 1<br>
>> Â 4:Â Â Â option volfile-checksum 0<br>
>> Â 5:Â Â Â option volfile-key /storage<br>
>> Â 6:Â Â Â option client-version 3.7.6<br>
>> Â 7:Â Â Â option process-uuid<br>
>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0<br>
>> Â 8:Â Â Â option fops-version 1298437<br>
>> Â 9:Â Â Â option ping-timeout 42<br>
>> Â 10:Â Â Â option remote-host 10.0.231.50<br>
>> Â 11:Â Â Â option remote-subvolume /mnt/raid6-storage/storage<br>
>> Â 12:Â Â Â option transport-type socket<br>
>> Â 13:Â Â Â option send-gids true<br>
>> Â 14: end-volume<br>
>> Â 15:<br>
>> Â 16: volume storage-client-1<br>
>> Â 17:Â Â Â type protocol/client<br>
>> Â 18:Â Â Â option clnt-lk-version 1<br>
>> Â 19:Â Â Â option volfile-checksum 0<br>
>> Â 20:Â Â Â option volfile-key /storage<br>
>> Â 21:Â Â Â option client-version 3.7.6<br>
>> Â 22:Â Â Â option process-uuid<br>
>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0<br>
>> Â 23:Â Â Â option fops-version 1298437<br>
>> Â 24:Â Â Â option ping-timeout 42<br>
>> Â 25:Â Â Â option remote-host 10.0.231.51<br>
>> Â 26:Â Â Â option remote-subvolume /mnt/raid6-storage/storage<br>
>> Â 27:Â Â Â option transport-type socket<br>
>> Â 28:Â Â Â option send-gids true<br>
>> Â 29: end-volume<br>
>> Â 30:<br>
>> Â 31: volume storage-client-2<br>
>> Â 32:Â Â Â type protocol/client<br>
>> Â 33:Â Â Â option clnt-lk-version 1<br>
>> Â 34:Â Â Â option volfile-checksum 0<br>
>> Â 35:Â Â Â option volfile-key /storage<br>
>> Â 36:Â Â Â option client-version 3.7.6<br>
>> Â 37:Â Â Â option process-uuid<br>
>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0<br>
>> Â 38:Â Â Â option fops-version 1298437<br>
>> Â 39:Â Â Â option ping-timeout 42<br>
>> Â 40:Â Â Â option remote-host 10.0.231.52<br>
>> Â 41:Â Â Â option remote-subvolume /mnt/raid6-storage/storage<br>
>> Â 42:Â Â Â option transport-type socket<br>
>> Â 43:Â Â Â option send-gids true<br>
>> Â 44: end-volume<br>
>> Â 45:<br>
>> Â 46: volume storage-client-3<br>
>> Â 47:Â Â Â type protocol/client<br>
>> Â 48:Â Â Â option clnt-lk-version 1<br>
>> Â 49:Â Â Â option volfile-checksum 0<br>
>> Â 50:Â Â Â option volfile-key /storage<br>
>> Â 51:Â Â Â option client-version 3.7.6<br>
>> Â 52:Â Â Â option process-uuid<br>
>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0<br>
>> Â 53:Â Â Â option fops-version 1298437<br>
>> Â 54:Â Â Â option ping-timeout 42<br>
>> Â 55:Â Â Â option remote-host 10.0.231.53<br>
>> Â 56:Â Â Â option remote-subvolume /mnt/raid6-storage/storage<br>
>> Â 57:Â Â Â option transport-type socket<br>
>> Â 58:Â Â Â option send-gids true<br>
>> Â 59: end-volume<br>
>> Â 60:<br>
>> Â 61: volume storage-client-4<br>
>> Â 62:Â Â Â type protocol/client<br>
>> Â 63:Â Â Â option ping-timeout 42<br>
>> Â 64:Â Â Â option remote-host 10.0.231.54<br>
>> Â 65:Â Â Â option remote-subvolume /mnt/raid6-storage/storage<br>
>> Â 66:Â Â Â option transport-type socket<br>
>> Â 67:Â Â Â option send-gids true<br>
>> Â 68: end-volume<br>
>> Â 69:<br>
>> Â 70: volume storage-client-5<br>
>> Â 71:Â Â Â type protocol/client<br>
>> Â 72:Â Â Â option ping-timeout 42<br>
>> Â 73:Â Â Â option remote-host 10.0.231.55<br>
>> Â 74:Â Â Â option remote-subvolume /mnt/raid6-storage/storage<br>
>> Â 75:Â Â Â option transport-type socket<br>
>> Â 76:Â Â Â option send-gids true<br>
>> Â 77: end-volume<br>
>> Â 78:<br>
>> Â 79: volume storage-dht<br>
>> Â 80:Â Â Â type cluster/distribute<br>
>> Â 81:Â Â Â subvolumes storage-client-0 storage-client-1 storage-client-2<br>
>> storage-client-3 storage-client-4 storage-client-5<br>
>> Â 82: end-volume<br>
>><br>
>><br>
>> But not the client op-version, how can I retrieve this info?<br>
>><br>
>> Thanks<br>
>><br>
>> On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg <<a href="mailto:ggarg@redhat.com" target="_blank">ggarg@redhat.com</a>> wrote:<br>
>><br>
>> > Hi Steve,<br>
>> ><br>
>> > Which version you have upgraded client, could you tell us client<br>
>> > op-version after upgrade ?<br>
>> ><br>
>> ><br>
>> > have you upgraded all of your clients ?<br>
>> ><br>
>> ><br>
>> > Thanks,<br>
>> > Gaurav<br>
>> ><br>
>> ><br>
>> > ----- Original Message -----<br>
>> > From: "Steve Dainard" <<a href="mailto:sdainard@spd1.com" target="_blank">sdainard@spd1.com</a>><br>
>> > To: "<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a> List" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br>
>> > Sent: Wednesday, March 2, 2016 1:10:27 AM<br>
>> > Subject: [Gluster-users] gluster 3.7.6 volume set: failed: One or more<br>
>> > connected clients cannot support the feature being set<br>
>> ><br>
>> > Gluster 3.7.6<br>
>> > 'storage' is a distributed volume<br>
>> ><br>
>> > # gluster volume set storage rebal-throttle lazy<br>
>> > volume set: failed: One or more connected clients cannot support the<br>
>> > feature being set. These clients need to be upgraded or disconnected before<br>
>> > running this command again<br>
>> ><br>
>> > I found a client connected using version 3.6.7 so I upgraded &<br>
>> > umount/mount the gluster volume on the client but I'm still getting this<br>
>> > error.<br>
>> ><br>
>> > I've run grep "accepted client from" /var/log/glusterfs/bricks/* | grep -v<br>
>> > 3.7.6 and I get a few returns from the client above, all dated last week.<br>
>> ><br>
>> > I've run 'gluster volume status storage clients' and checked the connected<br>
>> > clients manually, they're all running 3.7.6.<br>
>> ><br>
>> > /var/log/gluster/etc-glusterfs-glusterd.vol.log:<br>
>> > [2016-03-01 19:23:20.180821] E [MSGID: 106022]<br>
>> > [glusterd-utils.c:10154:glusterd_check_client_op_version_support]<br>
>> > 0-management: One or more c<br>
>> > lients don't support the required op-version<br>
>> > [2016-03-01 19:23:20.180853] E [MSGID: 106301]<br>
>> > [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of<br>
>> > operation 'Volume Set' fa<br>
>> > iled on localhost : One or more connected clients cannot support the<br>
>> > feature being set. These clients need to be upgraded or disconnected be<br>
>> > fore running this command again<br>
>> ><br>
>> > Also tried setting the diagnostics.brick-log-level logging level and got<br>
>> > the same error.<br>
>> ><br>
>> > /var/lib/glusterd/vols/storage/info:<br>
>> > type=0<br>
>> > count=6<br>
>> > status=1<br>
>> > sub_count=0<br>
>> > stripe_count=1<br>
>> > replica_count=1<br>
>> > disperse_count=0<br>
>> > redundancy_count=0<br>
>> > version=26<br>
>> > transport-type=0<br>
>> > volume-id=26d355cb-c486-481f-ac16-e25390e73775<br>
>> > username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c<br>
>> > password=<br>
>> > op-version=3<br>
>> > client-op-version=3<br>
>> > quota-version=1<br>
>> > parent_volname=N/A<br>
>> > restored_from_snap=00000000-0000-0000-0000-000000000000<br>
>> > snap-max-hard-limit=256<br>
>> > features.quota-deem-statfs=on<br>
>> > features.inode-quota=on<br>
>> > diagnostics.brick-log-level=WARNING<br>
>> > features.quota=on<br>
>> > performance.readdir-ahead=on<br>
>> > performance.cache-size=1GB<br>
>> > performance.stat-prefetch=on<br>
>> > brick-0=10.0.231.50:-mnt-raid6-storage-storage<br>
>> > brick-1=10.0.231.51:-mnt-raid6-storage-storage<br>
>> > brick-2=10.0.231.52:-mnt-raid6-storage-storage<br>
>> > brick-3=10.0.231.53:-mnt-raid6-storage-storage<br>
>> > brick-4=10.0.231.54:-mnt-raid6-storage-storage<br>
>> > brick-5=10.0.231.55:-mnt-raid6-storage-storage<br>
>> ><br>
>> ><br>
>> > _______________________________________________<br>
>> > Gluster-users mailing list<br>
>> > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
>> > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
>> ><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</div></div><p></p>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>