<div dir="ltr">David, <br><br>can you stop glusterfs on affected machine and remove gluster related socket extension files from /var/run ? Start glusterfs service again and try once more ?<br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 26, 2015 at 5:57 PM, David F. Robinson <span dir="ltr"><<a href="mailto:david.robinson@corvidtec.com" target="_blank">david.robinson@corvidtec.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div>Tried shutting down glusterd and glusterfsd and restarting.</div>
<div> </div>
<div>[2015-01-26 14:52:53.548330] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2015-01-26 14:52:53.549763] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2015-01-26 14:52:53.551245] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2015-01-26 14:52:53.552819] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2015-01-26 14:52:53.554289] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2015-01-26 14:52:53.555769] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2015-01-26 14:52:53.564429] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2015-01-26 14:52:53.565578] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/0cdef7faa934cfe52676689ff8c0110f.socket failed (Invalid argument)<br>[2015-01-26 14:52:53.566488] I [MSGID: 106005] [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick gfsib01bkp.corvidtec.com:/data/brick01bkp/Software_bkp has disconnected from glusterd.<br>[2015-01-26 14:52:53.567453] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/09e734d5e8d52bb796896c7a33d0a3ff.socket failed (Invalid argument)<br>[2015-01-26 14:52:53.568248] I [MSGID: 106005] [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick gfsib01bkp.corvidtec.com:/data/brick02bkp/Software_bkp has disconnected from glusterd.<br>[2015-01-26 14:52:53.569009] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/3f6844c74682f39fa7457082119628c5.socket failed (Invalid argument)<br>[2015-01-26 14:52:53.569851] I [MSGID: 106005] [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick gfsib01bkp.corvidtec.com:/data/brick01bkp/Source_bkp has disconnected from glusterd.<br>[2015-01-26 14:52:53.570818] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/34d5cc70aba63082bbb467ab450bd08b.socket failed (Invalid argument)<br>[2015-01-26 14:52:53.571777] I [MSGID: 106005] [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick gfsib01bkp.corvidtec.com:/data/brick02bkp/Source_bkp has disconnected from glusterd.<br>[2015-01-26 14:52:53.572681] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/0cd747876dca36cb21ecc7a36f7f897c.socket failed (Invalid argument)<br>[2015-01-26 14:52:53.573533] I [MSGID: 106005] [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick gfsib01bkp.corvidtec.com:/data/brick01bkp/homegfs_bkp has disconnected from glusterd.<br>[2015-01-26 14:52:53.574433] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/88744e1365b414d41e720e480700716a.socket failed (Invalid argument)<br>[2015-01-26 14:52:53.575399] I [MSGID: 106005] [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick gfsib01bkp.corvidtec.com:/data/brick02bkp/homegfs_bkp has disconnected from glusterd.<br>[2015-01-26 14:52:53.575434] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed (Invalid argument)<br>[2015-01-26 14:52:53.575447] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: nfs has disconnected from glusterd.<br>[2015-01-26 14:52:53.579663] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /data/brick01bkp/homegfs_bkp on port 49152<br>[2015-01-26 14:52:53.581943] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /data/brick02bkp/Source_bkp on port 49156<br>[2015-01-26 14:52:53.583487] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /data/brick01bkp/Source_bkp on port 49153<br>[2015-01-26 14:52:53.584921] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /data/brick02bkp/Software_bkp on port 49157<br>[2015-01-26 14:52:53.585719] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /data/brick01bkp/Software_bkp on port 49154<br>[2015-01-26 14:52:53.586281] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /data/brick02bkp/homegfs_bkp on port 49155<br></div><div><div class="h5">
<div> </div>
<div> </div>
<div> </div>
<div>------ Original Message ------</div>
<div>From: "David F. Robinson" <<a href="mailto:david.robinson@corvidtec.com" target="_blank">david.robinson@corvidtec.com</a>></div>
<div>To: "<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>>; "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a>></div>
<div>Sent: 1/26/2015 9:50:09 AM</div>
<div>Subject: v3.6.2</div>
<div> </div>
<div>
<blockquote cite="http://em11912ec4-8747-437b-be14-89e6ba6eba6f@dfrobins-vaio" type="cite">
<div>I have a server with v3.6.2 from which I cannot mount using NFS. The FUSE mount works, however, I cannot get the NFS mount to work. From /var/log/message:</div>
<div> </div>
<div>Jan 26 09:27:28 gfs01bkp mount[2810]: mount to NFS server '<a href="http://gfsib01bkp.corvidtec.com" target="_blank">gfsib01bkp.corvidtec.com</a>' failed: Connection refused, retrying<br>Jan 26 09:27:53 gfs01bkp mount[4456]: mount to NFS server '<a href="http://gfsib01bkp.corvidtec.com" target="_blank">gfsib01bkp.corvidtec.com</a>' failed: Connection refused, retrying<br>Jan 26 09:29:28 gfs01bkp mount[2810]: mount to NFS server '<a href="http://gfsib01bkp.corvidtec.com" target="_blank">gfsib01bkp.corvidtec.com</a>' failed: Connection refused, retrying<br>Jan 26 09:29:53 gfs01bkp mount[4456]: mount to NFS server '<a href="http://gfsib01bkp.corvidtec.com" target="_blank">gfsib01bkp.corvidtec.com</a>' failed: Connection refused, retrying<br>Jan 26 09:31:28 gfs01bkp mount[2810]: mount to NFS server '<a href="http://gfsib01bkp.corvidtec.com" target="_blank">gfsib01bkp.corvidtec.com</a>' failed: Connection refused, retrying<br>Jan 26 09:31:53 gfs01bkp mount[4456]: mount to NFS server '<a href="http://gfsib01bkp.corvidtec.com" target="_blank">gfsib01bkp.corvidtec.com</a>' failed: Connection refused, retrying<br>Jan 26 09:33:28 gfs01bkp mount[2810]: mount to NFS server '<a href="http://gfsib01bkp.corvidtec.com" target="_blank">gfsib01bkp.corvidtec.com</a>' failed: Connection refused, retrying<br>Jan 26 09:33:53 gfs01bkp mount[4456]: mount to NFS server '<a href="http://gfsib01bkp.corvidtec.com" target="_blank">gfsib01bkp.corvidtec.com</a>' failed: Connection refused, retrying<br>Jan 26 09:35:28 gfs01bkp mount[2810]: mount to NFS server '<a href="http://gfsib01bkp.corvidtec.com" target="_blank">gfsib01bkp.corvidtec.com</a>' failed: Connection refused, retrying<br>Jan 26 09:35:53 gfs01bkp mount[4456]: mount to NFS server '<a href="http://gfsib01bkp.corvidtec.com" target="_blank">gfsib01bkp.corvidtec.com</a>' failed: Connection refused, retrying<br></div>
<div> </div>
<div> </div>
<div>I also am continually getting the following errors in /var/log/glusterfs:</div>
<div> </div>
<div>[root@gfs01bkp glusterfs]# tail -f etc-glusterfs-glusterd.vol.log <br>[2015-01-26 14:41:51.260827] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed (Invalid argument)<br>[2015-01-26 14:41:54.261240] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed (Invalid argument)<br>[2015-01-26 14:41:57.261642] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed (Invalid argument)<br>[2015-01-26 14:42:00.262073] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed (Invalid argument)<br>[2015-01-26 14:42:03.262504] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed (Invalid argument)<br>[2015-01-26 14:42:06.262935] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed (Invalid argument)<br>[2015-01-26 14:42:09.263334] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed (Invalid argument)<br>[2015-01-26 14:42:12.263761] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed (Invalid argument)<br>[2015-01-26 14:42:15.264177] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed (Invalid argument)<br>[2015-01-26 14:42:18.264623] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed (Invalid argument)<br>[2015-01-26 14:42:21.265053] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed (Invalid argument)<br>[2015-01-26 14:42:24.265504] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed (Invalid argument)<br>^C<br></div>
<div> </div>
<div>Also, when I try to NFS mount my gluster volume, I am getting </div>
<div> </div>
<div>
<div style="FONT-SIZE:11pt;FONT-FAMILY:Calibri"><span>
</span><br></div></div></blockquote></div></div></div></div></blockquote></div><br></div></div>