<div dir="ltr"><div><div><div><div><div><div><div>I upgraded my fedora 23 system to f24 a couple of days ago, now I am unable to mount my gluster cluster.<br><br></div>The update installed:<br><br>glusterfs-3.8.0-1.fc24.x86_64<br>glusterfs-libs-3.8.0-1.fc24.x86_64<br>glusterfs-fuse-3.8.0-1.fc24.x86_64<br>glusterfs-client-xlators-3.8.0-1.fc24.x86_64<br><br></div>the gluster is running 3.7.11<br><br></div>The volume is replica 3 <br><br></div>I see these errors in the mount log:<br><br><blockquote>[2016-06-24 17:55:34.016462] I [MSGID: 100030] [glusterfsd.c:2408:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.0 (args: /usr/sbin/glusterfs --volfile-server=gluster1 --volfile-id=homes /mnt/homes)<br>[2016-06-24 17:55:34.094345] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1<br>[2016-06-24 17:55:34.240135] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2<br>[2016-06-24 17:55:34.240130] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 4<br>[2016-06-24 17:55:34.240130] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3<br>[2016-06-24 17:55:34.241499] I [MSGID: 114020] [client.c:2356:notify] 0-homes-client-2: parent translators are ready, attempting connect on transport<br>[2016-06-24 17:55:34.249172] I [MSGID: 114020] [client.c:2356:notify] 0-homes-client-5: parent translators are ready, attempting connect on transport<br>[2016-06-24 17:55:34.250186] I [rpc-clnt.c:1855:rpc_clnt_reconfig] 0-homes-client-2: changing port to 49171 (from 0)<br>[2016-06-24 17:55:34.253347] I [MSGID: 114020] [client.c:2356:notify] 0-homes-client-6: parent translators are ready, attempting connect on transport<br>[2016-06-24 17:55:34.254213] I [rpc-clnt.c:1855:rpc_clnt_reconfig] 0-homes-client-5: changing port to 49154 (from 0)<br>[2016-06-24 17:55:34.255115] I [MSGID: 114057] [client-handshake.c:1441:select_server_supported_programs] 0-homes-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)<br>[2016-06-24 17:55:34.255861] W [MSGID: 114007] [client-handshake.c:1176:client_setvolume_cbk] 0-homes-client-2: failed to find key &#39;child_up&#39; in the options<br>[2016-06-24 17:55:34.259097] I [MSGID: 114057] [client-handshake.c:1441:select_server_supported_programs] 0-homes-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)<br>Final graph:<br>+------------------------------------------------------------------------------+<br>  1: volume homes-client-2<br>  2:     type protocol/client<br>  3:     option clnt-lk-version 1<br>  4:     option volfile-checksum 0<br>  5:     option volfile-key homes<br>  6:     option client-version 3.8.0<br>  7:     option process-uuid Island-29185-2016/06/24-17:55:34:10054-homes-client-2-0-0<br>  8:     option fops-version 1298437<br>  9:     option ping-timeout 20<br> 10:     option remote-host gluster-2<br> 11:     option remote-subvolume /export/brick2/home<br> 12:     option transport-type socket<br> 13:     option event-threads 4<br> 14:     option send-gids true<br> 15: end-volume<br> 16:  <br> 17: volume homes-client-5<br> 18:     type protocol/client<br> 19:     option clnt-lk-version 1<br> 20:     option volfile-checksum 0<br> 21:     option volfile-key homes<br> 22:     option client-version 3.8.0<br> 23:     option process-uuid Island-29185-2016/06/24-17:55:34:10054-homes-client-5-0-0<br> 24:     option fops-version 1298437<br> 25:     option ping-timeout 20<br> 26:     option remote-host <a href="http://gluster1.vsnet.gmu.edu">gluster1.vsnet.gmu.edu</a><br> 27:     option remote-subvolume /export/brick2/home<br> 28:     option transport-type socket<br> 29:     option event-threads 4<br> 30:     option send-gids true<br> 31: end-volume<br> 32:  <br> 33: volume homes-client-6<br> 34:     type protocol/client<br> 35:     option ping-timeout 20<br> 36:     option remote-host gluster0<br> 37:     option remote-subvolume /export/brick2/home<br> 38:     option transport-type socket<br> 39:     option event-threads 4<br> 40:     option send-gids true<br> 41: end-volume<br> 42:  <br> 43: volume homes-replicate-0<br> 44:     type cluster/replicate<br> 45:     option background-self-heal-count 20<br> 46:     option metadata-self-heal on<br> 47:     option data-self-heal off<br> 48:     option entry-self-heal on<br> 49:     option data-self-heal-window-size 8<br> 50:     option data-self-heal-algorithm diff<br> 51:     option eager-lock on<br> 52:     option quorum-type auto<br> 53:     option self-heal-readdir-size 64KB<br> 54:     subvolumes homes-client-2 homes-client-5 homes-client-6<br> 55: end-volume<br> 56:  <br> 57: volume homes-dht<br> 58:     type cluster/distribute<br> 59:     option min-free-disk 5%<br> 60:     option rebalance-stats on<br> 61:     option readdir-optimize on<br> 62:     subvolumes homes-replicate-0<br> 63: end-volume<br> 64:  <br> 65: volume homes-read-ahead<br> 66:     type performance/read-ahead<br> 67:     subvolumes homes-dht<br> 68: end-volume<br> 69:  <br> 70: volume homes-io-cache<br> 71:     type performance/io-cache<br> 72:     subvolumes homes-read-ahead<br> 73: end-volume<br> 74:  <br> 75: volume homes-quick-read<br> 76:     type performance/quick-read<br> 77:     subvolumes homes-io-cache<br> 78: end-volume<br> 79:  <br> 80: volume homes-open-behind<br> 81:     type performance/open-behind<br> 82:     subvolumes homes-quick-read<br> 83: end-volume<br> 84:  <br> 85: volume homes-md-cache<br> 86:     type performance/md-cache<br> 87:     subvolumes homes-open-behind<br> 88: end-volume<br> 89:  <br> 90: volume homes<br> 91:     type debug/io-stats<br> 92:     option log-level INFO<br> 93:     option latency-measurement off<br> 94:     option count-fop-hits on<br> 95:     subvolumes homes-md-cache<br> 96: end-volume<br> 97:  <br> 98: volume meta-autoload<br> 99:     type meta<br>100:     subvolumes homes<br>101: end-volume<br>102:  <br>+------------------------------------------------------------------------------+<br>[2016-06-24 17:55:34.261219] I [rpc-clnt.c:1855:rpc_clnt_reconfig] 0-homes-client-6: changing port to 49153 (from 0)<br>[2016-06-24 17:55:34.266096] I [MSGID: 114057] [client-handshake.c:1441:select_server_supported_programs] 0-homes-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)<br>[2016-06-24 17:55:34.266905] W [MSGID: 114007] [client-handshake.c:1176:client_setvolume_cbk] 0-homes-client-6: failed to find key &#39;child_up&#39; in the options<br>[2016-06-24 17:55:34.273618] W [MSGID: 114007] [client-handshake.c:1176:client_setvolume_cbk] 0-homes-client-5: failed to find key &#39;child_up&#39; in the options<br>[2016-06-24 17:55:45.071822] I [fuse-bridge.c:5241:fuse_graph_setup] 0-fuse: switched to graph 0<br>[2016-06-24 17:55:45.072098] I [fuse-bridge.c:4153:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.24<br>[2016-06-24 17:55:45.072229] I [MSGID: 108006] [afr-common.c:4268:afr_local_init] 0-homes-replicate-0: no subvolumes up<br>[2016-06-24 17:55:45.072537] E [dht-helper.c:1600:dht_inode_ctx_time_update] (--&gt;/usr/lib64/glusterfs/3.8.0/xlator/cluster/replicate.so(+0x49b4a) [0x7fd0e9e10b4a] --&gt;/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x32746) [0x7fd0e9b74746] --&gt;/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x99e4) [0x7fd0e9b4b9e4] ) 0-homes-dht: invalid argument: inode [Invalid argument]<br>[2016-06-24 17:55:45.072773] E [dht-helper.c:1600:dht_inode_ctx_time_update] (--&gt;/usr/lib64/glusterfs/3.8.0/xlator/cluster/replicate.so(+0x49b4a) [0x7fd0e9e10b4a] --&gt;/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x32746) [0x7fd0e9b74746] --&gt;/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x99e4) [0x7fd0e9b4b9e4] ) 0-homes-dht: invalid argument: inode [Invalid argument]<br>[2016-06-24 17:55:45.072817] W [fuse-bridge.c:767:fuse_attr_cbk] 0-glusterfs-fuse: 2: LOOKUP() / =&gt; -1 (Transport endpoint is not connected)<br>[2016-06-24 17:55:45.072968] E [dht-helper.c:1600:dht_inode_ctx_time_update] (--&gt;/usr/lib64/glusterfs/3.8.0/xlator/cluster/replicate.so(+0x49b4a) [0x7fd0e9e10b4a] --&gt;/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x32746) [0x7fd0e9b74746] --&gt;/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x99e4) [0x7fd0e9b4b9e4] ) 0-homes-dht: invalid argument: inode [Invalid argument]<br>[2016-06-24 17:55:45.076081] E [dht-helper.c:1600:dht_inode_ctx_time_update] (--&gt;/usr/lib64/glusterfs/3.8.0/xlator/cluster/replicate.so(+0x49b4a) [0x7fd0e9e10b4a] --&gt;/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x32746) [0x7fd0e9b74746] --&gt;/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x99e4) [0x7fd0e9b4b9e4] ) 0-homes-dht: invalid argument: inode [Invalid argument]<br>[2016-06-24 17:55:45.076285] E [dht-helper.c:1600:dht_inode_ctx_time_update] (--&gt;/usr/lib64/glusterfs/3.8.0/xlator/cluster/replicate.so(+0x49b4a) [0x7fd0e9e10b4a] --&gt;/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x32746) [0x7fd0e9b74746] --&gt;/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x99e4) [0x7fd0e9b4b9e4] ) 0-homes-dht: invalid argument: inode [Invalid argument]<br>[2016-06-24 17:55:45.090890] I [fuse-bridge.c:5082:fuse_thread_proc] 0-fuse: unmounting /mnt/homes<br>The message &quot;I [MSGID: 108006] [afr-common.c:4268:afr_local_init] 0-homes-replicate-0: no subvolumes up&quot; repeated 4 times between [2016-06-24 17:55:45.072229] and [2016-06-24 17:55:45.076259]<br>[2016-06-24 17:55:45.091051] W [glusterfsd.c:1286:cleanup_and_exit] (--&gt;/lib64/libpthread.so.0(+0x75ca) [0x7fd0f77495ca] --&gt;/usr/sbin/glusterfs(glusterfs_sigwaiter+0xf5) [0x5647b219d565] --&gt;/usr/sbin/glusterfs(cleanup_and_exit+0x54) [0x5647b219d3c4] ) 0-: received signum (15), shutting down<br>[2016-06-24 17:55:45.091070] I [fuse-bridge.c:5793:fini] 0-fuse: Unmounting &#39;/mnt/homes&#39;.<br></blockquote><br><br></div>I checked the release notes for 3.8.0 but I did not see any caveats or compatibility warnings.<br><br></div>Anyone else seeing issues with 3.8 clients mounting 3.7 volumes?<br><br><br></div>-Alastair<br><br><div><div><div><br></div></div></div></div>