<div dir="ltr"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="font-size:14px">One more thing, when I do this on server1, which has been in the pool for a long time:</div><div style="font-size:14px">server1:~$ mount server1:/vol1 mountpoint</div><div style="font-size:14px">It also fails.</div><div style="font-size:14px">The log gave me:</div></blockquote><div><br></div><div>My fault, I used localhost as endpoint.</div><div><br></div><div>I re-issued "mount -t glusterfs server01:/speech0 qqq"<br></div><div>and the log shows a lot of things like:</div><div><br></div><div><div>[2015-08-31 12:08:44.801169] W [socket.c:923:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT 0 on socket 57, Protocol not available</div><div>[2015-08-31 12:08:44.801187] E [socket.c:3019:socket_connect] 0-speech0-client-43: Failed to set keep-alive: Protocol not available</div><div>[2015-08-31 12:08:44.801305] W [socket.c:642:__socket_rwv] 0-speech0-client-43: readv on <a href="http://10.88.153.25:24007">10.88.153.25:24007</a> failed (Connection reset by peer)</div><div>[2015-08-31 12:08:44.801404] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1eb)[0x7fcf540db65b] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7fcf53ea71b7] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fcf53ea72ce] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7fcf53ea739b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7fcf53ea795f] ))))) 0-speech0-client-43: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-08-31 12:08:44.801294 (xid=0x17)</div><div>[2015-08-31 12:08:44.801423] W [MSGID: 114032] [client-handshake.c:1623:client_dump_version_cbk] 0-speech0-client-43: received RPC status error [Transport endpoint is not connected]</div><div>[2015-08-31 12:08:44.801440] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-speech0-client-43: disconnected from speech0-client-43. Client process will keep trying to connect to glusterd until brick's port is available</div><div>[2015-08-31 12:08:44.804488] W [socket.c:923:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT 0 on socket 57, Protocol not available</div><div>[2015-08-31 12:08:44.804505] E [socket.c:3019:socket_connect] 0-speech0-client-51: Failed to set keep-alive: Protocol not available</div><div>[2015-08-31 12:08:44.804775] W [socket.c:642:__socket_rwv] 0-speech0-client-51: readv on <a href="http://10.88.146.19:24007">10.88.146.19:24007</a> failed (Connection reset by peer)</div><div>[2015-08-31 12:08:44.804878] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1eb)[0x7fcf540db65b] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7fcf53ea71b7] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fcf53ea72ce] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7fcf53ea739b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7fcf53ea795f] ))))) 0-speech0-client-51: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-08-31 12:08:44.804693 (xid=0x18)</div><div>[2015-08-31 12:08:44.804898] W [MSGID: 114032] [client-handshake.c:1623:client_dump_version_cbk] 0-speech0-client-51: received RPC status error [Transport endpoint is not connected]</div><div>[2015-08-31 12:08:44.804917] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-speech0-client-51: disconnected from speech0-client-51. Client process will keep trying to connect to glusterd until brick's port is available</div></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2015-08-31 20:06 GMT+08:00 Yiping Peng <span dir="ltr"><<a href="mailto:barius.cn@gmail.com" target="_blank">barius.cn@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span class=""><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="font-size:14px">I believe the following events have happened in the cluster resulting</span><br style="font-size:14px"><span style="font-size:14px">into this situation:</span><br style="font-size:14px"><span style="font-size:14px">1. GlusterD & brick process on node 2 was brought down</span><br style="font-size:14px"><span style="font-size:14px">2. Node 1 was rebooted.</span><br></blockquote></span><div>Strangely enough, glusterfs, glusterd and glusterfsd are running on my server. Is glusterfsd the brick process? Also server01 has not been rebooted during the whole process.</div><div><br></div><div>glusterfsd has the following arguments:<br></div><div>/usr/sbin/glusterfsd -s <a href="http://server01.local.net" target="_blank">server01.local.net</a> --volfile-id speech0.server01.local.net.home-glusterfs-speech0-brick0 -p /var/lib/glusterd/vols/speech0/run/server01.local.net-home-glusterfs-speech0-brick0.pid -S /var/run/gluster/6bf40a98deade9dde8b615226bc57567.socket --brick-name /home/glusterfs/speech0/brick0 -l /var/log/glusterfs/bricks/home-glusterfs-speech0-brick0.log --xlator-option *-posix.glusterd-uuid=1c33ff18-2a6a-44cf-9a04-727fc96e92be --brick-port 49159 --xlator-option speech0-server.listen-port=49159<br><div><br></div><div>One more thing, when I do this on server1, which has been in the pool for a long time:</div><div><span style="font-size:14px">server1:~$ mount server1:/vol1 mountpoint</span></div><div><span style="font-size:14px">It also fails.</span></div><div><span style="font-size:14px">The log gave me:</span></div><div><br></div><div><div><span style="font-size:14px">[2015-08-31 11:56:57.123307] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.3 (args: /usr/sbin/glusterfs --volfile-server=localhost --volfile-id=/speech0 qqq)</span></div><div><span style="font-size:14px">[2015-08-31 11:56:57.134642] W [socket.c:923:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT 0 on socket 9, Protocol not available</span></div><div><span style="font-size:14px">[2015-08-31 11:56:57.134688] E [socket.c:3019:socket_connect] 0-glusterfs: Failed to set keep-alive: Protocol not available</span></div><div><span style="font-size:14px">[2015-08-31 11:56:57.135063] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1</span></div><div><span style="font-size:14px">[2015-08-31 11:56:57.135113] E [socket.c:2332:socket_connect_finish] 0-glusterfs: connection to <a href="http://127.0.0.1:24007" target="_blank">127.0.0.1:24007</a> failed (Connection reset by peer)</span></div><div><span style="font-size:14px">[2015-08-31 11:56:57.135149] E [glusterfsd-mgmt.c:1819:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: localhost (Transport endpoint is not connected)</span></div><div><span style="font-size:14px">[2015-08-31 11:56:57.135158] I [glusterfsd-mgmt.c:1825:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers</span></div><div><span style="font-size:14px">[2015-08-31 11:56:57.135333] W [glusterfsd.c:1219:cleanup_and_exit] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3) [0x7fb5e1be39a3] -->/usr/sbin/glusterfs() [0x4099c8] -->/usr/sbin/glusterfs(cleanup_and_exit+0x65) [0x4059b5] ) 0-: received signum (1), shutting down</span></div><div><span style="font-size:14px">[2015-08-31 11:56:57.135371] I [fuse-bridge.c:5595:fini] 0-fuse: Unmounting '/home/speech/pengyiping/qqq'.</span></div><div><span style="font-size:14px">[2015-08-31 11:56:57.140640] W [glusterfsd.c:1219:cleanup_and_exit] (-->/lib64/libpthread.so.0() [0x318b207851] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x405e4d] -->/usr/sbin/glusterfs(cleanup_and_exit+0x65) [0x4059b5] ) 0-: received signum (15), shutting down</span></div><div><br></div></div></div><div><br></div><div>Any help is much appreciated.</div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">2015-08-31 19:15 GMT+08:00 Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I believe the following events have happened in the cluster resulting<br>
into this situation:<br>
1. GlusterD & brick process on node 2 was brought down<br>
2. Node 1 was rebooted.<br>
<br>
In the above case the mount will definitely fail since the brick process<br>
was not started as in a 2 node set up glusterd waits its peers to come<br>
up before it starts the bricks. Could you check whether the brick<br>
process is running or not?<br>
<br>
Thanks,<br>
Atin<br>
<div><div><br>
On 08/31/2015 04:17 PM, Yiping Peng wrote:<br>
> I've tried both: assuming server1 is already in pool, server2 is undergoing<br>
> peer-probing<br>
><br>
> server2:~$ mount server1:/vol1 mountpoint, fail;<br>
> server2:~$ mount server2:/vol1 mountpoint, fail.<br>
><br>
> Strange enough. I *should* be able to mount server1:/vol1 on server2. But<br>
> this is not the case :(<br>
> Maybe something is broken in the server pool, as I'm seeing disconnected<br>
> nodes?<br>
><br>
><br>
> 2015-08-31 18:02 GMT+08:00 Ravishankar N <<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>>:<br>
><br>
>><br>
>><br>
>> On 08/31/2015 12:53 PM, Merlin Morgenstern wrote:<br>
>><br>
>> Trying to mount the brick on the same physical server with deamon running<br>
>> on this server but not on the other server:<br>
>><br>
>> @node2:~$ sudo mount -t glusterfs gs2:/volume1 /data/nfs<br>
>> Mount failed. Please check the log file for more details.<br>
>><br>
>> For mount to succeed the glusterd must be up on the node that you specify<br>
>> as the volfile-server; gs2 in this case. You can use -o<br>
>> backupvolfile-server=gs1 as a fallback.<br>
>> -Ravi<br>
>><br>
>> _______________________________________________<br>
>> Gluster-users mailing list<br>
>> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
>> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
>><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
><br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>