<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from rtf -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<font face="Calibri" size="2"><span style="font-size:10.5pt;">
<div align="left" style="text-align:justify;"><font color="#1F497D">For Life cycle of inode in glusterfs which showed in <a href="https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Developer-guide/datastructure-inode/"><font color="#0563C1"><u>https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Developer-guide/datastructure-inode/</u></font></a></font></div>
<div align="left" style="text-align:justify;"><font color="#1F497D">It shows that “<font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">A inode is removed from the inode table and eventually destroyed when unlink or rmdir operation is
performed on a file/directory, or the the lru limit of the inode table has been exceeded</span></font><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">.</span></font><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">”</span></font></font></div>
<div align="left" style="text-align:justify;"><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">Now the default value for inode lru limit is 32k for glusterfs, </span></font></div>
<div align="left" style="text-align:justify;"><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">When we </span><span style="background-color:#FCFCFC;">“</span><span style="background-color:#FCFCFC;">du</span><span style="background-color:#FCFCFC;">”</span><span style="background-color:#FCFCFC;">
or </span><span style="background-color:#FCFCFC;">“</span><span style="background-color:#FCFCFC;">ls</span><span style="background-color:#FCFCFC;"> –R” </span><span style="background-color:#FCFCFC;"> large a</span><span style="background-color:#FCFCFC;">mount
files in directory which bigger than 32K,</span><span style="background-color:#FCFCFC;"> it could easy lead to the limit of lru.</span></font></div>
<div align="left" style="text-align:justify;"><font face="Arial" color="#404040"> </font></div>
<div align="left" style="text-align:justify;"><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">@gluster-expert, when glusterfs </span><span style="background-color:#FCFCFC;">destroy</span><span style="background-color:#FCFCFC;"> </span><span style="background-color:#FCFCFC;">the
inode due to the LRU limit, does glusterfs notify to the kernel? (from my study</span><span style="background-color:#FCFCFC;"> now</span><span style="background-color:#FCFCFC;">, it seems not)</span></font></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman" color="#404040"> </font></div>
<div align="left" style="text-align:justify;"><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">@linux-fsdevel-expert, could you please clarify the mechanism of </span><span style="background-color:#FCFCFC;">inode recycle mechanism
or </span><span style="background-color:#FCFCFC;">fuse-forget FOP for inode</span><span style="background-color:#FCFCFC;"> for us</span><span style="background-color:#FCFCFC;">?</span></font></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman" color="#404040"> </font></div>
<div align="left" style="text-align:justify;"><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">Is it possible that kernel free the inode</span><span style="background-color:#FCFCFC;">(which will trigger fuse-forget to glusterfs) </span><span style="background-color:#FCFCFC;">
later than </span><span style="background-color:#FCFCFC;">the destroy in glusterfs due to lru limit?</span></font></div>
<div align="left" style="text-align:justify;"><font face="Arial" color="#404040"> </font></div>
<div align="left" style="text-align:justify;"><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">If it is possible , then the nodeid (which is conver from the memory address in glusterfs) maybe </span><span style="background-color:#FCFCFC;">stale,
and when it pass to the glusterfs userspace, the glusterfs just conver the u64 nodeid to memory address, and try to access the address, it will lead to </span><span style="background-color:#FCFCFC;">invalid access and </span><span style="background-color:#FCFCFC;">coredump</span><span style="background-color:#FCFCFC;">
</span><span style="background-color:#FCFCFC;">finally</span><span style="background-color:#FCFCFC;"> </span><span style="background-color:#FCFCFC;">!</span></font></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman" color="#404040"> </font></div>
<div align="left" style="text-align:justify;"><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">Thanks & Best Regards,</span></font></div>
<div align="left" style="text-align:justify;"><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">George</span></font></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman" color="#1F497D"> </font></div>
<div><font face="宋体" size="2"><span style="font-size:11pt;">_____________________________________________<br>
<b>From:</b> Lian, George (Nokia - CN/Hangzhou) <br>
<b>Sent:</b> Friday, December 09, 2016 9:49 AM<br>
<b>To:</b> 'Gluster-devel@gluster.org' <Gluster-devel@gluster.org><br>
<b>Cc:</b> Zhou, Cynthia (Nokia - CN/Hangzhou) <cynthia.zhou@nokia.com>; Bao, Xiaohui (Nokia - CN/Hangzhou) <xiaohui.bao@nokia.com>; Zhang, Bingxuan (Nokia - CN/Hangzhou) <bingxuan.zhang@nokia.com>; Li, Deqian (Nokia - CN/Hangzhou) <deqian.li@nokia.com><br>
<b>Subject:</b> "du" a large count files in a directory casue mounted glusterfs filesystem coredump</span></font></div>
<div><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;">Hi, GlusterFS Expert,</div>
<div align="left" style="text-align:justify;"> </div>
<div align="left" style="text-align:justify;">Now we have an issue when run “du” command for a large count files/directory in a directory, in our environment there are more than 150k files in the directory.</div>
<div align="left" style="text-align:justify;"><span style="background-color:silver;"># df -i .</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:silver;">Filesystem Inodes IUsed IFree IUse% Mounted on</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:silver;">169.254.0.23:/home 261888</span><span style="background-color:yellow;"> 154146</span><span style="background-color:silver;"> 107742 59% /home</span></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;">Now we run “du” command in this directory, it is so easy to cause glusterfs process coredump, and the coredump backtrace shows it always caused by do_forget API, but last call some time difference. Please see the
detail backtrace as the end of this mail.</div>
<div align="left" style="text-align:justify;">From my investigation, the issue maybe caused by the unsafe call of API “fuse_ino_to_inode”, </div>
<div align="left" style="text-align:justify;">I JUST GUESS in some un-expect case, when call “fuse_ino_to_inode” with nodeid which came from forget FOP, just call “fuse_ino_to_inode” to get the address from simply mapping of uint64 to memory address, </div>
<div align="left" style="text-align:justify;">the inode address maybe just destroyed by “<font face="Arial" color="#404040"><span style="background-color:#FCFCFC;"> </span></font><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">the
lru limit of the inode table has been exceeded” in our large file case, so this operation maybe not safe, and the coredump backtrace also show there are more difference case when core occurred.</span></font></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;">Could you please share your comments on my investigation?</div>
<div align="left" style="text-align:justify;">And BTW I have some questions, </div>
<ol style="text-align:justify;margin:0;padding-left:18pt;">
<li>How the inode number in “stat” command mapping to the inode in glusterfs?</li></ol>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
stat log</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
File: ‘log’</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
Size: 4096 Blocks: 8 IO Block: 4096 directory</div>
<div align="left" style="text-align:justify;padding-left:18pt;">Device: fd10h/64784d I<span style="background-color:yellow;">node: 14861593</span> Links: 3</div>
<div align="left" style="text-align:justify;padding-left:18pt;"> </div>
<ol start="2" style="text-align:justify;margin:0;padding-left:18pt;">
<li>When will system call the forget FOP and where the nodeid parameter came from in system?</li></ol>
<div align="left" style="text-align:justify;padding-left:18pt;"><font face="Times New Roman"> </font></div>
<ol start="3" style="text-align:justify;margin:0;padding-left:18pt;">
<li>When the inode is <font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">eventually destroyed due to lru limit, and when the same file is FOPed next time, does the address of this inode is same address in next lookup? If not same, is
there exist an case the FOP of forget give out an old nodeid than glusterfs has?</span></font></li></ol>
<div align="left" style="text-align:justify;"><font face="Times New Roman" color="#404040"> </font></div>
<div align="left" style="text-align:justify;"><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">Thanks & Best Regards,</span></font></div>
<div align="left" style="text-align:justify;"><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">George</span></font></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<ol style="text-align:justify;margin:0;padding-left:18pt;">
<li>Coredump backtrace 1</li></ol>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#0 0x00007fcd610a69e7 in __list_splice (list=0x26c350c, head=0x7fcd56c23db0) at list.h:121</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#1 0x00007fcd610a6a51 in list_splice_init (list=0x26c350c, head=0x7fcd56c23db0) at list.h:146</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#2 0x00007fcd610a95c8 in inode_table_prune (table=0x26c347c) at inode.c:1330</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#3 0x00007fcd610a8a02 in inode_forget (inode=0x7fcd5001147c, nlookup=1) at inode.c:977</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#4 0x00007fcd5f151e24 in do_forget (this=0xc43590, unique=437787, nodeid=140519787271292, nlookup=1) at fuse-bridge.c:637</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#5 0x00007fcd5f151fd3 in fuse_batch_forget (this=0xc43590, finh=0x7fcd50c266c0, msg=0x7fcd50c266e8) at fuse-bridge.c:676</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#6 0x00007fcd5f168aff in fuse_thread_proc (data=0xc43590) at fuse-bridge.c:4909</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#7 0x00007fcd6080b414 in start_thread (arg=0x7fcd56c24700) at pthread_create.c:333</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#8 0x00007fcd600f7b9f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
(gdb) print head</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
$6 = (struct list_head *) 0x7fcd56c23db0</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
(gdb) print *head</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
$7 = {next = 0x7fcd56c23db0, prev = 0x7fcd56c23db0}</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
(gdb) print head->next</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
$8 = (struct list_head *) 0x7fcd56c23db0</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
(gdb) print list->prev</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
$9 = (struct list_head *) 0x5100000000</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
(gdb) print (list->prev)->next</div>
<div align="left" style="text-align:justify;padding-left:18pt;">Cannot access memory at address 0x5100000000</div>
<ol start="2" style="text-align:justify;margin:0;padding-left:18pt;">
<li>Coredump backtrace 2</li></ol>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#1 0x00007f612ab4a43a in __GI_abort () at abort.c:89</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#2 0x00007f612ab41ccd in __assert_fail_base (fmt=0x7f612ac76618 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
assertion=assertion@entry=0x7f612bc01ec1 "inode->nlookup >= nlookup", file=file@entry=0x7f612bc01d9b "inode.c", line=line@entry=607,</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
function=function@entry=0x7f612bc02339 <__PRETTY_FUNCTION__.10128> "__inode_forget") at assert.c:92</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#3 0x00007f612ab41d82 in __GI___assert_fail (assertion=0x7f612bc01ec1 "inode->nlookup >= nlookup", file=0x7f612bc01d9b "inode.c", line=607,</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
function=0x7f612bc02339 <__PRETTY_FUNCTION__.10128> "__inode_forget") at assert.c:101</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#4 0x00007f612bbade56 in __inode_forget (inode=0x7f611801d68c, nlookup=4) at inode.c:607</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#5 0x00007f612bbae9ea in inode_forget (inode=0x7f611801d68c, nlookup=4) at inode.c:973</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#6 0x00007f6129defdd5 in do_forget (this=0x1a895c0, unique=436589, nodeid=140054991328908, nlookup=4) at fuse-bridge.c:633</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#7 0x00007f6129defe94 in fuse_forget (this=0x1a895c0, finh=0x7f6118c28be0, msg=0x7f6118c28c08) at fuse-bridge.c:652</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#8 0x00007f6129e06ab0 in fuse_thread_proc (data=0x1a895c0) at fuse-bridge.c:4905</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#9 0x00007f612b311414 in start_thread (arg=0x7f61220d0700) at pthread_create.c:333</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#10 0x00007f612abfdb9f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
(gdb) f 5</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#5 0x00007f612bbae9ea in inode_forget (inode=0x7f611801d68c, nlookup=4) at inode.c:973</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
973 inode.c: No such file or directory.</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
(gdb) f 4</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#4 0x00007f612bbade56 in __inode_forget (inode=0x7f611801d68c, nlookup=4) at inode.c:607</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
607 in inode.c</div>
<div align="left" style="text-align:justify;padding-left:18pt;">(gdb) print inode->nlookup</div>
<ol start="3" style="text-align:justify;margin:0;padding-left:18pt;">
<li>Coredump backtrace 3</li></ol>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#1 0x00007f86b0b0f43a in __GI_abort () at abort.c:89</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#2 0x00007f86b0b06ccd in __assert_fail_base (fmt=0x7f86b0c3b618 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
assertion=assertion@entry=0x7f86b12e1f38 "INTERNAL_SYSCALL_ERRNO (e, __err) != ESRCH || !robust",</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
file=file@entry=0x7f86b12e1e7c "../nptl/pthread_mutex_lock.c", line=line@entry=352,</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
function=function@entry=0x7f86b12e1fe0 <__PRETTY_FUNCTION__.8666> "__pthread_mutex_lock_full") at assert.c:92</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#3 0x00007f86b0b06d82 in __GI___assert_fail (assertion=assertion@entry=0x7f86b12e1f38 "INTERNAL_SYSCALL_ERRNO (e, __err) != ESRCH || !robust",</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
file=file@entry=0x7f86b12e1e7c "../nptl/pthread_mutex_lock.c", line=line@entry=352,</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
function=function@entry=0x7f86b12e1fe0 <__PRETTY_FUNCTION__.8666> "__pthread_mutex_lock_full") at assert.c:101</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#4 0x00007f86b12d89da in __pthread_mutex_lock_full (mutex=0x7f86a12ffcac) at ../nptl/pthread_mutex_lock.c:352</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#5 0x00007f86b1b729f1 in inode_ref (inode=0x7f86a03cefec) at inode.c:476</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#6 0x00007f86afdafb04 in fuse_ino_to_inode (ino=140216190693356, fuse=0x1a541f0) at fuse-helpers.c:390</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#7 0x00007f86afdb4d6b in do_forget (this=0x1a541f0, unique=96369, nodeid=140216190693356, nlookup=1) at fuse-bridge.c:629</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#8 0x00007f86afdb4f84 in fuse_batch_forget (this=0x1a541f0, finh=0x7f86a03b4f90, msg=0x7f86a03b4fb8) at fuse-bridge.c:674</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#9 0x00007f86afdcbab0 in fuse_thread_proc (data=0x1a541f0) at fuse-bridge.c:4905</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#10 0x00007f86b12d6414 in start_thread (arg=0x7f86a7ac8700) at pthread_create.c:333</div>
<div align="left" style="text-align:justify;padding-left:18pt;">#11 0x00007f86b0bc2b9f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105</div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
</span></font>
</body>
</html>