<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from rtf -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<font face="Calibri" size="2"><span style="font-size:10.5pt;">
<div align="left" style="text-align:justify;">Hi, GlusterFS Expert,</div>
<div align="left" style="text-align:justify;"> </div>
<div align="left" style="text-align:justify;">Now we have an issue when run “du” command for a large count files/directory in a directory, in our environment there are more than 150k files in the directory.</div>
<div align="left" style="text-align:justify;"><span style="background-color:silver;"># df -i .</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:silver;">Filesystem Inodes IUsed IFree IUse% Mounted on</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:silver;">169.254.0.23:/home 261888</span><span style="background-color:yellow;"> 154146</span><span style="background-color:silver;"> 107742 59% /home</span></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;">Now we run “du” command in this directory, it is so easy to cause glusterfs process coredump, and the coredump backtrace shows it always caused by do_forget API, but last call some time difference. Please see the
detail backtrace as the end of this mail.</div>
<div align="left" style="text-align:justify;">From my investigation, the issue maybe caused by the unsafe call of API “fuse_ino_to_inode”, </div>
<div align="left" style="text-align:justify;">I JUST GUESS in some un-expect case, when call “fuse_ino_to_inode” with nodeid which came from forget FOP, just call “fuse_ino_to_inode” to get the address from simply mapping of uint64 to memory address, </div>
<div align="left" style="text-align:justify;">the inode address maybe just destroyed by “<font face="Arial" color="#404040"><span style="background-color:#FCFCFC;"> </span></font><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">the
lru limit of the inode table has been exceede</span></font><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">d</span></font><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">” in our large file case, so this
op</span></font><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">eration maybe not safe, and the coredump backtrace also show there are more difference case when core occurred.</span></font></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;">Could you please share your comments on my investigation?</div>
<div align="left" style="text-align:justify;">And BTW I have some questions, </div>
<ol style="text-align:justify;margin:0;padding-left:18pt;">
<li>How the inode number in “stat” command mapping to the inode in glusterfs?</li></ol>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
stat log</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
File: ‘log’</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
Size: 4096 Blocks: 8 IO Block: 4096 directory</div>
<div align="left" style="text-align:justify;padding-left:18pt;">Device: fd10h/64784d I<span style="background-color:yellow;">node: 14861593</span> Links: 3</div>
<div align="left" style="text-align:justify;padding-left:18pt;"><font face="Times New Roman"> </font></div>
<ol start="2" style="text-align:justify;margin:0;padding-left:18pt;">
<li>When will system call the forget FOP and where the nodeid parameter came from in system?</li></ol>
<div align="left" style="text-align:justify;padding-left:18pt;"><font face="Times New Roman"> </font></div>
<ol start="3" style="text-align:justify;margin:0;padding-left:18pt;">
<li>When the inode is <font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">eventually destroye</span></font><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">d</span></font><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">
due to lru limit, and when the same file is FOP</span></font><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">ed</span></font><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;"> next time, does the address of
</span></font><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">this </span></font><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">inode is same</span></font><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">
address</span></font><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;"> in next lookup?</span></font><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;"> If not same, is there exist an case</span></font><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">
the FOP of forget give out an old nodeid than glusterfs has?</span></font></li></ol>
<div align="left" style="text-align:justify;"><font face="Times New Roman" color="#404040"> </font></div>
<div align="left" style="text-align:justify;"><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">Thanks & </span><span style="background-color:#FCFCFC;">Best Regards,</span></font></div>
<div align="left" style="text-align:justify;"><font face="Arial" color="#404040"><span style="background-color:#FCFCFC;">George</span></font></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<ol style="text-align:justify;margin:0;padding-left:18pt;">
<li>Coredump backtrace 1</li></ol>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#0 0x00007fcd610a69e7 in __list_splice (list=0x26c350c, head=0x7fcd56c23db0) at list.h:121</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#1 0x00007fcd610a6a51 in list_splice_init (list=0x26c350c, head=0x7fcd56c23db0) at list.h:146</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#2 0x00007fcd610a95c8 in inode_table_prune (table=0x26c347c) at inode.c:1330</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#3 0x00007fcd610a8a02 in inode_forget (inode=0x7fcd5001147c, nlookup=1) at inode.c:977</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#4 0x00007fcd5f151e24 in do_forget (this=0xc43590, unique=437787, nodeid=140519787271292, nlookup=1) at fuse-bridge.c:637</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#5 0x00007fcd5f151fd3 in fuse_batch_forget (this=0xc43590, finh=0x7fcd50c266c0, msg=0x7fcd50c266e8) at fuse-bridge.c:676</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#6 0x00007fcd5f168aff in fuse_thread_proc (data=0xc43590) at fuse-bridge.c:4909</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#7 0x00007fcd6080b414 in start_thread (arg=0x7fcd56c24700) at pthread_create.c:333</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#8 0x00007fcd600f7b9f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
(gdb) print head</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
$6 = (struct list_head *) 0x7fcd56c23db0</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
(gdb) print *head</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
$7 = {next = 0x7fcd56c23db0, prev = 0x7fcd56c23db0}</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
(gdb) print head->next</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
$8 = (struct list_head *) 0x7fcd56c23db0</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
(gdb) print list->prev</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
$9 = (struct list_head *) 0x5100000000</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
(gdb) print (list->prev)->next</div>
<div align="left" style="text-align:justify;padding-left:18pt;">Cannot access memory at address 0x5100000000</div>
<ol start="2" style="text-align:justify;margin:0;padding-left:18pt;">
<li>Coredump backtrace 2</li></ol>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#1 0x00007f612ab4a43a in __GI_abort () at abort.c:89</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#2 0x00007f612ab41ccd in __assert_fail_base (fmt=0x7f612ac76618 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
assertion=assertion@entry=0x7f612bc01ec1 "inode->nlookup >= nlookup", file=file@entry=0x7f612bc01d9b "inode.c", line=line@entry=607,</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
function=function@entry=0x7f612bc02339 <__PRETTY_FUNCTION__.10128> "__inode_forget") at assert.c:92</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#3 0x00007f612ab41d82 in __GI___assert_fail (assertion=0x7f612bc01ec1 "inode->nlookup >= nlookup", file=0x7f612bc01d9b "inode.c", line=607,</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
function=0x7f612bc02339 <__PRETTY_FUNCTION__.10128> "__inode_forget") at assert.c:101</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#4 0x00007f612bbade56 in __inode_forget (inode=0x7f611801d68c, nlookup=4) at inode.c:607</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#5 0x00007f612bbae9ea in inode_forget (inode=0x7f611801d68c, nlookup=4) at inode.c:973</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#6 0x00007f6129defdd5 in do_forget (this=0x1a895c0, unique=436589, nodeid=140054991328908, nlookup=4) at fuse-bridge.c:633</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#7 0x00007f6129defe94 in fuse_forget (this=0x1a895c0, finh=0x7f6118c28be0, msg=0x7f6118c28c08) at fuse-bridge.c:652</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#8 0x00007f6129e06ab0 in fuse_thread_proc (data=0x1a895c0) at fuse-bridge.c:4905</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#9 0x00007f612b311414 in start_thread (arg=0x7f61220d0700) at pthread_create.c:333</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#10 0x00007f612abfdb9f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
(gdb) f 5</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#5 0x00007f612bbae9ea in inode_forget (inode=0x7f611801d68c, nlookup=4) at inode.c:973</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
973 inode.c: No such file or directory.</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
(gdb) f 4</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#4 0x00007f612bbade56 in __inode_forget (inode=0x7f611801d68c, nlookup=4) at inode.c:607</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
607 in inode.c</div>
<div align="left" style="text-align:justify;padding-left:18pt;">(gdb) print inode->nlookup</div>
<ol start="3" style="text-align:justify;margin:0;padding-left:18pt;">
<li>Coredump backtrace 3</li></ol>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#1 0x00007f86b0b0f43a in __GI_abort () at abort.c:89</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#2 0x00007f86b0b06ccd in __assert_fail_base (fmt=0x7f86b0c3b618 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
assertion=assertion@entry=0x7f86b12e1f38 "INTERNAL_SYSCALL_ERRNO (e, __err) != ESRCH || !robust",</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
file=file@entry=0x7f86b12e1e7c "../nptl/pthread_mutex_lock.c", line=line@entry=352,</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
function=function@entry=0x7f86b12e1fe0 <__PRETTY_FUNCTION__.8666> "__pthread_mutex_lock_full") at assert.c:92</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#3 0x00007f86b0b06d82 in __GI___assert_fail (assertion=assertion@entry=0x7f86b12e1f38 "INTERNAL_SYSCALL_ERRNO (e, __err) != ESRCH || !robust",</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
file=file@entry=0x7f86b12e1e7c "../nptl/pthread_mutex_lock.c", line=line@entry=352,</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
function=function@entry=0x7f86b12e1fe0 <__PRETTY_FUNCTION__.8666> "__pthread_mutex_lock_full") at assert.c:101</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#4 0x00007f86b12d89da in __pthread_mutex_lock_full (mutex=0x7f86a12ffcac) at ../nptl/pthread_mutex_lock.c:352</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#5 0x00007f86b1b729f1 in inode_ref (inode=0x7f86a03cefec) at inode.c:476</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#6 0x00007f86afdafb04 in fuse_ino_to_inode (ino=140216190693356, fuse=0x1a541f0) at fuse-helpers.c:390</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#7 0x00007f86afdb4d6b in do_forget (this=0x1a541f0, unique=96369, nodeid=140216190693356, nlookup=1) at fuse-bridge.c:629</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#8 0x00007f86afdb4f84 in fuse_batch_forget (this=0x1a541f0, finh=0x7f86a03b4f90, msg=0x7f86a03b4fb8) at fuse-bridge.c:674</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#9 0x00007f86afdcbab0 in fuse_thread_proc (data=0x1a541f0) at fuse-bridge.c:4905</div>
<div align="left" style="text-indent:21pt;text-align:justify;padding-left:18pt;">
#10 0x00007f86b12d6414 in start_thread (arg=0x7f86a7ac8700) at pthread_create.c:333</div>
<div align="left" style="text-align:justify;padding-left:18pt;">#11 0x00007f86b0bc2b9f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105</div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
</span></font>
</body>
</html>