<div dir="ltr">Hi Jiffin,<div><br></div><div>Thanks! I have 3.7.11-ubuntu1~trusty1 installed and using NFSv4 mount protocols. </div><div><br></div><div>Doing a forced lookup lists the root directories but shows 0 files in each.</div><div><br></div><div>Thanks!</div><div>Alan</div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Jun 3, 2016 at 3:09 AM Jiffin Tony Thottan <<a href="mailto:jthottan@redhat.com">jthottan@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Hi Alan,</p>
<p>I try to reproduce issue with my set up and get back to u.</p>
<p>can u please mention mount protocol and gluster package
version(3.7-?)<br>
</p>
Incase if u can't find /var/log/ganesha.log(it is default location
for fedora and centos),<br>
Just the system log messages and grep for ganesha.<br>
<br>
Also can try to perform force lookup on directory using "ls
<dirname>/* -ltr"<br>
<br>
--<br>
Jiffin</div><div bgcolor="#FFFFFF" text="#000000"><br>
<br>
<div>On 02/06/16 00:16, Alan Hartless wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Yes, I had a brick that I restored and so it had
existing files. After the crash, it wouldn't let me re-add it
because it said the files were already part of a gluster. So I
followed <a href="https://joejulian.name/blog/glusterfs-path-or-a-prefix-of-it-is-already-part-of-a-volume/" target="_blank">https://joejulian.name/blog/glusterfs-path-or-a-prefix-of-it-is-already-part-of-a-volume/</a> to
reset it.
<div><br>
</div>
<div>Also correct that I can access all files through fuse but
only the root directory via ganesha NFS4 or any
directories/files that have since been created. </div>
<div><br>
</div>
<div>Using a forced lookup on a specific file, I found that I
can reach it and even edit it. But a ls or dir will not list
it or any of it's parent directories. Even after editing the
file, it does not list with ls. </div>
<div><br>
</div>
<div>I'm using gluster 3.7 and ganesha 2.3 from Gluster's Ubuntu
repositories. </div>
<div><br>
</div>
<div>I don't have a <span>/var/log/ganesha.log
but I do </span><span>/var/log/ganesha-gfapi.log.
I tailed it while restarting ganesha and got this for the
specific volume:</span></div>
<div><span><br>
</span></div>
<div>
<div><font color="#212121" face="helvetica neue, helvetica,
arial, sans-serif">[2016-06-01 18:44:44.876385] I [MSGID:
114020] [client.c:2106:notify] 0-letsencrypt-client-0:
parent translators are ready, attempting connect on
transport</font></div>
<div><font color="#212121" face="helvetica neue, helvetica,
arial, sans-serif">[2016-06-01 18:44:44.876903] I [MSGID:
114020] [client.c:2106:notify] 0-letsencrypt-client-1:
parent translators are ready, attempting connect on
transport</font></div>
<div><font color="#212121" face="helvetica neue, helvetica,
arial, sans-serif">[2016-06-01 18:44:44.877193] I
[rpc-clnt.c:1868:rpc_clnt_reconfig]
0-letsencrypt-client-0: changing port to 49154 (from 0)</font></div>
<div><font color="#212121" face="helvetica neue, helvetica,
arial, sans-serif">[2016-06-01 18:44:44.877837] I [MSGID:
114057]
[client-handshake.c:1437:select_server_supported_programs]
0-letsencrypt-client-0: Using Program GlusterFS 3.3, Num
(1298437), Version (330)</font></div>
<div><font color="#212121" face="helvetica neue, helvetica,
arial, sans-serif">[2016-06-01 18:44:44.878234] I [MSGID:
114046] [client-handshake.c:1213:client_setvolume_cbk]
0-letsencrypt-client-0: Connected to letsencrypt-client-0,
attached to remote volume '/gluster_volume/letsencrypt'.</font></div>
<div><font color="#212121" face="helvetica neue, helvetica,
arial, sans-serif">[2016-06-01 18:44:44.878253] I [MSGID:
114047] [client-handshake.c:1224:client_setvolume_cbk]
0-letsencrypt-client-0: Server and Client lk-version
numbers are not same, reopening the fds</font></div>
<div><font color="#212121" face="helvetica neue, helvetica,
arial, sans-serif">[2016-06-01 18:44:44.878338] I [MSGID:
108005] [afr-common.c:4007:afr_notify]
0-letsencrypt-replicate-0: Subvolume
'letsencrypt-client-0' came back up; going online.</font></div>
<div><font color="#212121" face="helvetica neue, helvetica,
arial, sans-serif">[2016-06-01 18:44:44.878390] I [MSGID:
114035] [client-handshake.c:193:client_set_lk_version_cbk]
0-letsencrypt-client-0: Server lk version = 1</font></div>
<div><font color="#212121" face="helvetica neue, helvetica,
arial, sans-serif">[2016-06-01 18:44:44.878505] I
[rpc-clnt.c:1868:rpc_clnt_reconfig]
0-letsencrypt-client-1: changing port to 49154 (from 0)</font></div>
<div><font color="#212121" face="helvetica neue, helvetica,
arial, sans-serif">[2016-06-01 18:44:44.879568] I [MSGID:
114057]
[client-handshake.c:1437:select_server_supported_programs]
0-letsencrypt-client-1: Using Program GlusterFS 3.3, Num
(1298437), Version (330)</font></div>
<div><font color="#212121" face="helvetica neue, helvetica,
arial, sans-serif">[2016-06-01 18:44:44.880155] I [MSGID:
114046] [client-handshake.c:1213:client_setvolume_cbk]
0-letsencrypt-client-1: Connected to letsencrypt-client-1,
attached to remote volume '/gluster_volume/letsencrypt'.</font></div>
<div><font color="#212121" face="helvetica neue, helvetica,
arial, sans-serif">[2016-06-01 18:44:44.880175] I [MSGID:
114047] [client-handshake.c:1224:client_setvolume_cbk]
0-letsencrypt-client-1: Server and Client lk-version
numbers are not same, reopening the fds</font></div>
<div><font color="#212121" face="helvetica neue, helvetica,
arial, sans-serif">[2016-06-01 18:44:44.896801] I [MSGID:
114035] [client-handshake.c:193:client_set_lk_version_cbk]
0-letsencrypt-client-1: Server lk version = 1</font></div>
<div><font color="#212121" face="helvetica neue, helvetica,
arial, sans-serif">[2016-06-01 18:44:44.898290] I [MSGID:
108031] [afr-common.c:1900:afr_local_discovery_cbk]
0-letsencrypt-replicate-0: selecting local read_child
letsencrypt-client-0</font></div>
<div><font color="#212121" face="helvetica neue, helvetica,
arial, sans-serif">[2016-06-01 18:44:44.898798] I [MSGID:
104041] [glfs-resolve.c:869:__glfs_active_subvol]
0-letsencrypt: switched to graph
676c7573-7465-7266-732d-6e6f64652d63 (0)</font></div>
<div><font color="#212121" face="helvetica neue, helvetica,
arial, sans-serif">[2016-06-01 18:44:45.913545] I [MSGID:
104045] [glfs-master.c:95:notify] 0-gfapi: New graph
676c7573-7465-7266-732d-6e6f64652d63 (0) coming up</font></div>
</div>
<div><font color="#212121" face="helvetica neue, helvetica,
arial, sans-serif"><br>
</font></div>
<div><font color="#212121" face="helvetica neue, helvetica,
arial, sans-serif">I also tailed it while accessing files
through a mount point but nothing was logged.</font></div>
<div><br>
</div>
<div>This is the ganesha config for the specific volume I'm
testing with. I have others but they are the same except for
export ID and the paths.</div>
<div><br>
</div>
<div>
<div>EXPORT</div>
<div>{</div>
<div> Export_Id = 3;</div>
<div> Path = "/letsencrypt";</div>
<div> Pseudo = "/letsencrypt";</div>
<div> FSAL {</div>
<div> name = GLUSTER;</div>
<div> hostname = "localhost";</div>
<div> volume = "letsencrypt";</div>
<div> }</div>
<div> Access_type = RW;</div>
<div> Squash = No_root_squash;</div>
<div> Disable_ACL = TRUE;</div>
<div>}</div>
</div>
<div><br>
</div>
<div>Many thanks!</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Sun, May 29, 2016 at 12:46 PM Jiffin Tony
Thottan <<a href="mailto:jthottan@redhat.com" target="_blank">jthottan@redhat.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p><br>
</p>
<br>
<div>On 28/05/16 08:07, Alan Hartless wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">I had everything working well when I had a
complete melt down :-) Well got all that sorted and
everything back up and running or so I thought. Now NFS
ganesha is not showing any existing files but the root
level of the brick. It's empty for all subdirectories.
New files or directories added show up as well.
Everything shows up when using the fuse client.
<div><br>
</div>
</div>
</blockquote>
<br>
</div>
<div bgcolor="#FFFFFF" text="#000000"> If I understand your
issue correctly<br>
* You have created a volume using brick which contains pre
existing file and directories<br>
* When you tried to access the files via ganesha, it does
not show up. But with fuse it is visible.<br>
<br>
Can please try to perform force lookup on the
directories/files(ls <path to directory/file>) from
the ganesha mount?<br>
Also check the ganesha logs (/var/log/ganesha.log and
/var/log/ganesha-gfapi.log) for clues.<br>
IMO there was similar issue exists for older version of
ganesha(v2.1 I guess). if possible can you also share <br>
the ganesha configuration for that volume<br>
<br>
</div>
<div bgcolor="#FFFFFF" text="#000000">
<blockquote type="cite">
<div dir="ltr">
<div>I've tried self healing, editing files, etc but the
issue persists. If I move the folders and back, they
show up. But I have a live setup and can't afford the
time to move GBs of data to a new location and back.
Is there anything I can do to trigger something for
the files to show up in NFS again without having to
move directories?</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Alan</div>
</div>
<br>
<fieldset></fieldset>
<br>
</blockquote>
</div>
<div bgcolor="#FFFFFF" text="#000000">
<blockquote type="cite">
<pre>_______________________________________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</div>
</blockquote>
</div>
</blockquote>
<br>
</div></blockquote></div>