<div dir="ltr">Hi RafiKC, <div><br></div><div>Thanks for replying I've attached a tar.gz file with code and repro.txt to help reproduce the issue </div><div><div><br></div><div>One point to note as well was our original mount options was as follow<br></div><div><font face="monospace, monospace">- mount -t glusterfs -o acl <host>:/volume1 /mnt/...</font></div><div><br></div><div>Introducing the entry_timeout option <font face="monospace, monospace"> mount -t glusterfs -o acl,<b>entry-timeout=0</b></font> (entry_timeout from the ubuntu fuse documentation <span style="color:rgb(51,51,51);font-family:inherit;font-size:12px;line-height:16px">The timeout in seconds for which name lookups will be cached.) does away with the stale file handle error. I suppose this comes with a performance hit, any insight on this is from gluster's point is appreciated as well. </span></div><div><br></div><div><div>Log snippet from brick/data-glusterfs-volume1-brick1-brick.log (please let me know if there was another log file that I can get information)</div></div><div><br></div><div><br></div><div><div><font face="monospace, monospace">[2016-03-24 00:23:53.718403] W [server-resolve.c:437:resolve_anonfd_simple] 0-server: inode for the gfid (2af5d8c7-2321-4d77-bda9-ad883ae8d230) is not found. anonymous fd creation failed</font></div><div><font face="monospace, monospace">[2016-03-24 00:23:53.718543] W [server-resolve.c:437:resolve_anonfd_simple] 0-server: inode for the gfid (2af5d8c7-2321-4d77-bda9-ad883ae8d230) is not found. anonymous fd creation failed</font></div><div><font face="monospace, monospace">[2016-03-24 00:23:53.718573] I [server-rpc-fops.c:1235:server_fstat_cbk] 0-volume1-server: 15309: FSTAT -2 (2af5d8c7-2321-4d77-bda9-ad883ae8d230) ==> (No such file or directory)</font></div><div><font face="monospace, monospace">[2016-03-24 00:24:09.679523] W [server-resolve.c:437:resolve_anonfd_simple] 0-server: inode for the gfid (02b2e084-261b-45c5-8283-d7babd219a4d) is not found. anonymous fd creation failed</font></div><div><font face="monospace, monospace">[2016-03-24 00:24:09.679602] W [server-resolve.c:437:resolve_anonfd_simple] 0-server: inode for the gfid (02b2e084-261b-45c5-8283-d7babd219a4d) is not found. anonymous fd creation failed</font></div><div><font face="monospace, monospace">[2016-03-24 00:24:09.679618] I [server-rpc-fops.c:1235:server_fstat_cbk] 0-volume1-server: 23548: FSTAT -2 (02b2e084-261b-45c5-8283-d7babd219a4d) ==> (No such file or directory)</font></div><div><font face="monospace, monospace">[2016-03-24 00:24:53.789620] W [server-resolve.c:437:resolve_anonfd_simple] 0-server: inode for the gfid (74b2fd0e-966a-40fd-9554-bf30e2c0b7ea) is not found. anonymous fd creation failed</font></div><div><font face="monospace, monospace">[2016-03-24 00:24:53.789694] W [server-resolve.c:437:resolve_anonfd_simple] 0-server: inode for the gfid (74b2fd0e-966a-40fd-9554-bf30e2c0b7ea) is not found. anonymous fd creation failed</font></div><div><font face="monospace, monospace">[2016-03-24 00:24:53.789709] I [server-rpc-fops.c:1235:server_fstat_cbk] 0-volume1-server: 38694: FSTAT -2 (74b2fd0e-966a-40fd-9554-bf30e2c0b7ea) ==> (No such file or directory)</font></div></div><div><br></div><div><font face="monospace, monospace">===</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="arial, helvetica, sans-serif">Log snippet from /var/log/glusterfs/mnt-repovolume1.log (on client side)</font></div><div><font face="monospace, monospace"><br></font></div><div><div style="font-family:monospace,monospace">[2016-03-24 00:24:53.747930] I [dht-rename.c:1344:dht_rename] 0-volume1-dht: renaming /hhh/master.lock (hash=volume1-replicate-0/cache=volume1-replicate-0) => /hhh/master (hash=volume1-replicate-0/cache=volume1-replicate-0)</div><div style="font-family:monospace,monospace">[2016-03-24 00:24:53.760365] I [dht-rename.c:1344:dht_rename] 0-volume1-dht: renaming /hhh/master.lock (hash=volume1-replicate-0/cache=volume1-replicate-0) => /hhh/master (hash=volume1-replicate-0/cache=volume1-replicate-0)</div><div style="font-family:monospace,monospace">[2016-03-24 00:24:53.778746] W [client-rpc-fops.c:1472:client3_3_fstat_cbk] 0-volume1-client-1: remote operation failed: No such file or directory</div><div style="font-family:monospace,monospace">[2016-03-24 00:24:53.779381] W [MSGID: 108008] [afr-read-txn.c:225:afr_read_txn] 0-volume1-replicate-0: Unreadable subvolume -1 found with event generation 2. (Possible split-brain)</div><div style="font-family:monospace,monospace">[2016-03-24 00:24:53.779816] E [dht-helper.c:940:dht_migration_complete_check_task] 0-volume1-dht: (null): failed to get the 'linkto' xattr Stale file handle</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">Log snippet on my program</div><div style="font-family:monospace,monospace"><div>[master] Size: 64 bytes. nlink 1, Inode: 8be6731fd9f9fbe4 File Permissions:-rw-r--r--commit() filename = master, lock_filename = master.lock</div><div>read(master) :: Stale file handle</div></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">Thanks</font></div><div><font face="arial, helvetica, sans-serif">Rama</font></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 22, 2016 at 11:25 PM, Mohammed Rafi K C <span dir="ltr"><<a href="mailto:rkavunga@redhat.com" target="_blank">rkavunga@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><span class="">
<br>
<br>
<div>On 03/23/2016 04:10 AM, Rama Shenai
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi, We had some questions with respect to expectations of
atomicity of rename in gluster.</div>
<div><br>
</div>
<div>To elaborate : </div>
<div><br>
</div>
<div>We have setup with two machines (lets call them M1 and M2)
which essentially access a file (F) on a gluster volume
(mounted by M1 and M2)</div>
<div>A program does the following steps sequentially on each of
the two machines (M1 & M2) in an infinite loop</div>
<div> 1) Opens the file F in O_RDWR|O_EXCL mode, reads some data
and closes (F)</div>
<div> 2) Renames some other file F' => F</div>
<div> </div>
<div>Periodically either M1 or M2 sees a "Stalefile handle
error" when it tries to read the file (step (1)) after opening
the file in O_RDWR|O_EXCL (the open is successful)</div>
<div><br>
</div>
<div>The specific error reported the client volume logs
(/var/log/glusterfs/mnt-repos-volume1.log)</div>
<div>[2016-03-21 16:53:17.897902] I
[dht-rename.c:1344:dht_rename] 0-volume1-dht: renaming
master.lock
(hash=volume1-replicate-0/cache=volume1-replicate-0) =>
master (hash=volume1-replicate-0/cache=<nul>)</div>
<div>[2016-03-21 16:53:18.735090] W
[client-rpc-fops.c:504:client3_3_stat_cbk] 0-volume1-client-0:
remote operation failed: Stale file handle</div>
</div>
</blockquote>
<br></span>
Hi Rama,<br>
<br>
ESTALE error in rename normally generated when either the source
file is not resolvable (deleted or inaccessible) or when the parent
of destination is not resolvable. It can happen when let's say file
F' was present when your application did a lookup before rename, but
if it is got renamed by Node M1 before M2 could rename it. Basically
a race between two rename on the same file can result in ESTALE for
either of one.<br>
<br>
To confirm this, Can you please paste the log message from brick
"0-volume1-client-0". You can find out the brick name from the
graph.<br>
<br>
Also if you can share the program or snippet that used to reproduce
this issue, that would be great.<br>
<br>
Rafi KC<br>
<br>
<br>
<br>
<blockquote type="cite"><span class="">
<div dir="ltr">
<div><br>
</div>
<div>We see no error when: have two processes of the above
program running on the same machine (say on M1) accessing the
file F on the gluster volume, for which we want to understand
the expectations of atomicity in gluster specifically
specifically for rename, and if the above is a bug.</div>
<div><br>
</div>
<div>Also glusterfs --version => glusterfs 3.6.9 built on
Mar 2 2016 18:21:14</div>
<div><br>
</div>
<div>We also would like to know if there any parameter in the
one translators that we can tweak to prevent this problem</div>
<div><br>
</div>
<div>Any help or insights here is appreciated</div>
<div><br>
</div>
<div>Thanks</div>
<div>Rama</div>
<div><br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</span><pre>_______________________________________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</div>
</blockquote></div><br></div>