<div dir="ltr">Yep, unmount was successful. I didn&#39;t see any useful logging to say otherwise, but I have reduced logging on my clients because the logs grow way too quickly otherwise.<div><br></div><div>I had a maintenance window this morning, and worked around this by unmounting and rebooting the two debian hosts. I was then able to set the lazy rebalance without restarting any gluster services so I&#39;m ticking along now.</div><div><br></div><div>Maybe the client state info is cached for some period of time after the client disconnects.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 4, 2016 at 11:20 PM, Atin Mukherjee <span dir="ltr">&lt;<a href="mailto:atin.mukherjee83@gmail.com" target="_blank">atin.mukherjee83@gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">This is really weird, if you unmount the clients then statedump shouldn&#39;t be showing the same. Was unmount successful? Do you see any related error log entry in mount &amp; glusterd log?</p><span class="im HOEnZb">
<p dir="ltr">-Atin<br>
Sent from one plus one</p>
</span><div class="HOEnZb"><div class="h5"><div class="gmail_quote">On 04-Mar-2016 10:23 pm, &quot;Steve Dainard&quot; &lt;<a href="mailto:sdainard@spd1.com" target="_blank">sdainard@spd1.com</a>&gt; wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Except that even when I unmount gluster from both Debian clients, I&#39;m still getting the same error, and a dump of those clients after unmounting still shows:<div><br></div><div><div>glusterd.client1.identifier=<a href="http://10.0.231.10:1023" target="_blank">10.0.231.10:1023</a></div><div>glusterd.client1.volname=storage</div><div>glusterd.client1.max-op-version=30603</div><div>glusterd.client1.min-op-version=1</div><div><br></div><div>glusterd.client4.identifier=<a href="http://10.0.231.11:1022" target="_blank">10.0.231.11:1022</a></div><div>glusterd.client4.volname=storage</div><div>glusterd.client4.max-op-version=30603</div><div>glusterd.client4.min-op-version=1</div><div><br></div><div><br></div><div><br></div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 3, 2016 at 5:28 PM, Atin Mukherjee <span dir="ltr">&lt;<a href="mailto:atin.mukherjee83@gmail.com" target="_blank">atin.mukherjee83@gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr"></p>
<p dir="ltr"><span>-Atin<br>
Sent from one plus one<br></span><span>
On 04-Mar-2016 3:35 am, &quot;Steve Dainard&quot; &lt;<a href="mailto:sdainard@spd1.com" target="_blank">sdainard@spd1.com</a>&gt; wrote:<br>
&gt;<br>
&gt; FYI Gluster storage node hostnames are gluster0[1-6].<br>
&gt;<br>
&gt; Full dump attached. I see a few clients not on 30706. Most notably the two debian 7 servers (using packages from <a href="http://gluster.org" target="_blank">gluster.org</a>) seem to be running lower op versions than the centos7 machines (every other client in <a href="http://10.0.231.0/24" target="_blank">10.0.231.0/24</a> subnet).<br>
&gt;<br>
&gt; glusterd.client1.identifier=<a href="http://10.0.231.10:1023" target="_blank">10.0.231.10:1023</a> &lt;-- debian 7, glusterfs 3.7.6 built on Feb  4 2016 06:25:19<br>
&gt; glusterd.client1.volname=storage<br>
&gt; glusterd.client1.max-op-version=30603<br></span>
This is running with 3.6.3 and that&#39;s why volume set fails. Unmount this client and upgrade and remount it back.</p><div><div><br>
&gt; glusterd.client1.min-op-version=1<br>
&gt;<br>
&gt; glusterd.client2.identifier=<a href="http://10.0.231.51:65515" target="_blank">10.0.231.51:65515</a> &lt;-- gluster02 (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)<br>
&gt; glusterd.client2.volname=<br>
&gt; glusterd.client2.max-op-version=0<br>
&gt; glusterd.client2.min-op-version=0<br>
&gt;<br>
&gt; glusterd.client3.identifier=<a href="http://10.0.231.54:65521" target="_blank">10.0.231.54:65521</a> &lt;-- gluster05 (no actual mounts)<br>
&gt; glusterd.client3.volname=<br>
&gt; glusterd.client3.max-op-version=0<br>
&gt; glusterd.client3.min-op-version=0<br>
&gt;<br>
&gt; glusterd.client4.identifier=<a href="http://10.0.231.11:1022" target="_blank">10.0.231.11:1022</a> &lt;--- debian 7, glusterfs 3.7.6 built on Feb  4 2016 06:25:19<br>
&gt; glusterd.client4.volname=storage<br>
&gt; glusterd.client4.max-op-version=30603<br>
&gt; glusterd.client4.min-op-version=1<br>
&gt;<br>
&gt; glusterd.client5.identifier=<a href="http://10.0.231.55:65530" target="_blank">10.0.231.55:65530</a> &lt;-- gluster06 (no actual mounts)<br>
&gt; glusterd.client5.volname=<br>
&gt; glusterd.client5.max-op-version=0<br>
&gt; glusterd.client5.min-op-version=0<br>
&gt;<br>
&gt; glusterd.client6.identifier=<a href="http://10.0.231.53:65516" target="_blank">10.0.231.53:65516</a> &lt;-- gluster04 (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)<br>
&gt; glusterd.client6.volname=<br>
&gt; glusterd.client6.max-op-version=0<br>
&gt; glusterd.client6.min-op-version=0<br>
&gt;<br>
&gt; glusterd.client7.identifier=<a href="http://10.0.231.50:65529" target="_blank">10.0.231.50:65529</a><br>
&gt; glusterd.client7.volname=export-domain-storage.10.0.231.50.mnt-lv-export-domain-storage-export-domain-storage<br>
&gt; glusterd.client7.max-op-version=30706<br>
&gt; glusterd.client7.min-op-version=1<br>
&gt;<br>
&gt; ...<br>
&gt;<br>
&gt; Debian package info:<br>
&gt; apt-cache policy glusterfs-client<br>
&gt; glusterfs-client:<br>
&gt;   Installed: 3.7.6-2<br>
&gt;   Candidate: 3.7.6-2<br>
&gt;   Version table:<br>
&gt;  *** 3.7.6-2 0<br>
&gt;         500 <a href="http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/" target="_blank">http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/</a> wheezy/main amd64 Packages<br>
&gt;<br>
&gt;<br>
&gt; Thanks<br>
&gt;<br>
&gt; On Wed, Mar 2, 2016 at 10:29 PM, Gaurav Garg &lt;<a href="mailto:ggarg@redhat.com" target="_blank">ggarg@redhat.com</a>&gt; wrote:<br>
&gt;&gt;<br>
&gt;&gt; Hi Steve,<br>
&gt;&gt;<br>
&gt;&gt; As atin pointed out to take statedump by running #kill -SIGUSR1 $(pidof glusterd)  command. it will create .dump file in /var/run/gluster/ directory. client-op-version information will be present in dump file.<br>
&gt;&gt;<br>
&gt;&gt; Thanks,<br>
&gt;&gt; ~Gaurav<br>
&gt;&gt;<br>
&gt;&gt; ----- Original Message -----<br>
&gt;&gt; From: &quot;Steve Dainard&quot; &lt;<a href="mailto:sdainard@spd1.com" target="_blank">sdainard@spd1.com</a>&gt;<br>
&gt;&gt; To: &quot;Gaurav Garg&quot; &lt;<a href="mailto:ggarg@redhat.com" target="_blank">ggarg@redhat.com</a>&gt;<br>
&gt;&gt; Cc: &quot;<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a> List&quot; &lt;<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>&gt;<br>
&gt;&gt; Sent: Thursday, March 3, 2016 12:07:25 AM<br>
&gt;&gt; Subject: Re: [Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set<br>
&gt;&gt;<br>
&gt;&gt; From the the client side logs I can see version info on mount:<br>
&gt;&gt;<br>
&gt;&gt; Final graph:<br>
&gt;&gt; +------------------------------------------------------------------------------+<br>
&gt;&gt;   1: volume storage-client-0<br>
&gt;&gt;   2:     type protocol/client<br>
&gt;&gt;   3:     option clnt-lk-version 1<br>
&gt;&gt;   4:     option volfile-checksum 0<br>
&gt;&gt;   5:     option volfile-key /storage<br>
&gt;&gt;   6:     option client-version 3.7.6<br>
&gt;&gt;   7:     option process-uuid<br>
&gt;&gt; template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0<br>
&gt;&gt;   8:     option fops-version 1298437<br>
&gt;&gt;   9:     option ping-timeout 42<br>
&gt;&gt;  10:     option remote-host 10.0.231.50<br>
&gt;&gt;  11:     option remote-subvolume /mnt/raid6-storage/storage<br>
&gt;&gt;  12:     option transport-type socket<br>
&gt;&gt;  13:     option send-gids true<br>
&gt;&gt;  14: end-volume<br>
&gt;&gt;  15:<br>
&gt;&gt;  16: volume storage-client-1<br>
&gt;&gt;  17:     type protocol/client<br>
&gt;&gt;  18:     option clnt-lk-version 1<br>
&gt;&gt;  19:     option volfile-checksum 0<br>
&gt;&gt;  20:     option volfile-key /storage<br>
&gt;&gt;  21:     option client-version 3.7.6<br>
&gt;&gt;  22:     option process-uuid<br>
&gt;&gt; template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0<br>
&gt;&gt;  23:     option fops-version 1298437<br>
&gt;&gt;  24:     option ping-timeout 42<br>
&gt;&gt;  25:     option remote-host 10.0.231.51<br>
&gt;&gt;  26:     option remote-subvolume /mnt/raid6-storage/storage<br>
&gt;&gt;  27:     option transport-type socket<br>
&gt;&gt;  28:     option send-gids true<br>
&gt;&gt;  29: end-volume<br>
&gt;&gt;  30:<br>
&gt;&gt;  31: volume storage-client-2<br>
&gt;&gt;  32:     type protocol/client<br>
&gt;&gt;  33:     option clnt-lk-version 1<br>
&gt;&gt;  34:     option volfile-checksum 0<br>
&gt;&gt;  35:     option volfile-key /storage<br>
&gt;&gt;  36:     option client-version 3.7.6<br>
&gt;&gt;  37:     option process-uuid<br>
&gt;&gt; template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0<br>
&gt;&gt;  38:     option fops-version 1298437<br>
&gt;&gt;  39:     option ping-timeout 42<br>
&gt;&gt;  40:     option remote-host 10.0.231.52<br>
&gt;&gt;  41:     option remote-subvolume /mnt/raid6-storage/storage<br>
&gt;&gt;  42:     option transport-type socket<br>
&gt;&gt;  43:     option send-gids true<br>
&gt;&gt;  44: end-volume<br>
&gt;&gt;  45:<br>
&gt;&gt;  46: volume storage-client-3<br>
&gt;&gt;  47:     type protocol/client<br>
&gt;&gt;  48:     option clnt-lk-version 1<br>
&gt;&gt;  49:     option volfile-checksum 0<br>
&gt;&gt;  50:     option volfile-key /storage<br>
&gt;&gt;  51:     option client-version 3.7.6<br>
&gt;&gt;  52:     option process-uuid<br>
&gt;&gt; template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0<br>
&gt;&gt;  53:     option fops-version 1298437<br>
&gt;&gt;  54:     option ping-timeout 42<br>
&gt;&gt;  55:     option remote-host 10.0.231.53<br>
&gt;&gt;  56:     option remote-subvolume /mnt/raid6-storage/storage<br>
&gt;&gt;  57:     option transport-type socket<br>
&gt;&gt;  58:     option send-gids true<br>
&gt;&gt;  59: end-volume<br>
&gt;&gt;  60:<br>
&gt;&gt;  61: volume storage-client-4<br>
&gt;&gt;  62:     type protocol/client<br>
&gt;&gt;  63:     option ping-timeout 42<br>
&gt;&gt;  64:     option remote-host 10.0.231.54<br>
&gt;&gt;  65:     option remote-subvolume /mnt/raid6-storage/storage<br>
&gt;&gt;  66:     option transport-type socket<br>
&gt;&gt;  67:     option send-gids true<br>
&gt;&gt;  68: end-volume<br>
&gt;&gt;  69:<br>
&gt;&gt;  70: volume storage-client-5<br>
&gt;&gt;  71:     type protocol/client<br>
&gt;&gt;  72:     option ping-timeout 42<br>
&gt;&gt;  73:     option remote-host 10.0.231.55<br>
&gt;&gt;  74:     option remote-subvolume /mnt/raid6-storage/storage<br>
&gt;&gt;  75:     option transport-type socket<br>
&gt;&gt;  76:     option send-gids true<br>
&gt;&gt;  77: end-volume<br>
&gt;&gt;  78:<br>
&gt;&gt;  79: volume storage-dht<br>
&gt;&gt;  80:     type cluster/distribute<br>
&gt;&gt;  81:     subvolumes storage-client-0 storage-client-1 storage-client-2<br>
&gt;&gt; storage-client-3 storage-client-4 storage-client-5<br>
&gt;&gt;  82: end-volume<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt; But not the client op-version, how can I retrieve this info?<br>
&gt;&gt;<br>
&gt;&gt; Thanks<br>
&gt;&gt;<br>
&gt;&gt; On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg &lt;<a href="mailto:ggarg@redhat.com" target="_blank">ggarg@redhat.com</a>&gt; wrote:<br>
&gt;&gt;<br>
&gt;&gt; &gt; Hi Steve,<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; Which version you have upgraded client, could you tell us client<br>
&gt;&gt; &gt; op-version after upgrade ?<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; have you upgraded all of your clients ?<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; Thanks,<br>
&gt;&gt; &gt; Gaurav<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; ----- Original Message -----<br>
&gt;&gt; &gt; From: &quot;Steve Dainard&quot; &lt;<a href="mailto:sdainard@spd1.com" target="_blank">sdainard@spd1.com</a>&gt;<br>
&gt;&gt; &gt; To: &quot;<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a> List&quot; &lt;<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>&gt;<br>
&gt;&gt; &gt; Sent: Wednesday, March 2, 2016 1:10:27 AM<br>
&gt;&gt; &gt; Subject: [Gluster-users] gluster 3.7.6 volume set: failed: One or more<br>
&gt;&gt; &gt; connected clients cannot support the feature being set<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; Gluster 3.7.6<br>
&gt;&gt; &gt; &#39;storage&#39; is a distributed volume<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; # gluster volume set storage rebal-throttle lazy<br>
&gt;&gt; &gt; volume set: failed: One or more connected clients cannot support the<br>
&gt;&gt; &gt; feature being set. These clients need to be upgraded or disconnected before<br>
&gt;&gt; &gt; running this command again<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; I found a client connected using version 3.6.7 so I upgraded &amp;<br>
&gt;&gt; &gt; umount/mount the gluster volume on the client but I&#39;m still getting this<br>
&gt;&gt; &gt; error.<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; I&#39;ve run grep &quot;accepted client from&quot; /var/log/glusterfs/bricks/* | grep -v<br>
&gt;&gt; &gt; 3.7.6 and I get a few returns from the client above, all dated last week.<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; I&#39;ve run &#39;gluster volume status storage clients&#39; and checked the connected<br>
&gt;&gt; &gt; clients manually, they&#39;re all running 3.7.6.<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; /var/log/gluster/etc-glusterfs-glusterd.vol.log:<br>
&gt;&gt; &gt; [2016-03-01 19:23:20.180821] E [MSGID: 106022]<br>
&gt;&gt; &gt; [glusterd-utils.c:10154:glusterd_check_client_op_version_support]<br>
&gt;&gt; &gt; 0-management: One or more c<br>
&gt;&gt; &gt; lients don&#39;t support the required op-version<br>
&gt;&gt; &gt; [2016-03-01 19:23:20.180853] E [MSGID: 106301]<br>
&gt;&gt; &gt; [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of<br>
&gt;&gt; &gt; operation &#39;Volume Set&#39; fa<br>
&gt;&gt; &gt; iled on localhost : One or more connected clients cannot support the<br>
&gt;&gt; &gt; feature being set. These clients need to be upgraded or disconnected be<br>
&gt;&gt; &gt; fore running this command again<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; Also tried setting the diagnostics.brick-log-level logging level and got<br>
&gt;&gt; &gt; the same error.<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; /var/lib/glusterd/vols/storage/info:<br>
&gt;&gt; &gt; type=0<br>
&gt;&gt; &gt; count=6<br>
&gt;&gt; &gt; status=1<br>
&gt;&gt; &gt; sub_count=0<br>
&gt;&gt; &gt; stripe_count=1<br>
&gt;&gt; &gt; replica_count=1<br>
&gt;&gt; &gt; disperse_count=0<br>
&gt;&gt; &gt; redundancy_count=0<br>
&gt;&gt; &gt; version=26<br>
&gt;&gt; &gt; transport-type=0<br>
&gt;&gt; &gt; volume-id=26d355cb-c486-481f-ac16-e25390e73775<br>
&gt;&gt; &gt; username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c<br>
&gt;&gt; &gt; password=<br>
&gt;&gt; &gt; op-version=3<br>
&gt;&gt; &gt; client-op-version=3<br>
&gt;&gt; &gt; quota-version=1<br>
&gt;&gt; &gt; parent_volname=N/A<br>
&gt;&gt; &gt; restored_from_snap=00000000-0000-0000-0000-000000000000<br>
&gt;&gt; &gt; snap-max-hard-limit=256<br>
&gt;&gt; &gt; features.quota-deem-statfs=on<br>
&gt;&gt; &gt; features.inode-quota=on<br>
&gt;&gt; &gt; diagnostics.brick-log-level=WARNING<br>
&gt;&gt; &gt; features.quota=on<br>
&gt;&gt; &gt; performance.readdir-ahead=on<br>
&gt;&gt; &gt; performance.cache-size=1GB<br>
&gt;&gt; &gt; performance.stat-prefetch=on<br>
&gt;&gt; &gt; brick-0=10.0.231.50:-mnt-raid6-storage-storage<br>
&gt;&gt; &gt; brick-1=10.0.231.51:-mnt-raid6-storage-storage<br>
&gt;&gt; &gt; brick-2=10.0.231.52:-mnt-raid6-storage-storage<br>
&gt;&gt; &gt; brick-3=10.0.231.53:-mnt-raid6-storage-storage<br>
&gt;&gt; &gt; brick-4=10.0.231.54:-mnt-raid6-storage-storage<br>
&gt;&gt; &gt; brick-5=10.0.231.55:-mnt-raid6-storage-storage<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; _______________________________________________<br>
&gt;&gt; &gt; Gluster-users mailing list<br>
&gt;&gt; &gt; <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
&gt;&gt; &gt; <a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
&gt;&gt; &gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; _______________________________________________<br>
&gt; Gluster-users mailing list<br>
&gt; <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
&gt; <a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</div></div><p></p>
</blockquote></div><br></div>
</blockquote></div>
</div></div></blockquote></div><br></div>