<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:宋体;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.im
{mso-style-name:im;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">I suppose there seems a defect on mdc_writev_cbk and mdc_fstat<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">Let’s assume in 2 timestamp which called write and fstat operation in application:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">T0: write (process a)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">T1: read (process b) with the data of T0 of process a.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">T2: fstat (process c)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">In my view, mdc_write is non-block operation and have some lock to protect in afr xlator, because mdc_fstat not check the lock in AFR xaltor, so
mdc_writev_cbk which called “mdc_inode_iatt_set_validate” maybe later than mdc_fstat.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">Such like<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">T3: fstat result of T2 without the “mdc_inode_iatt_set_validate” of T0 when stat-prefetch options is on.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">T4: “mdc_inode_iatt_set_validate” is called of T0 in mdc_writev_cbk.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">Lets’ assume T0<T1<T2<T3<T4, is the above assumption is reasonable case when in multi-process environment and the load of CPU is high?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">If it is reasonable, then issue of “tail issue” will be happened.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">So maybe a fix suggestion is on mdc_fstat operation , we should add an operation to check whether the writev operation is ongoing or not, if write-operation
is ongoing, should goto uncached label in mdc_fstat function.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">Could you please confirm the above assumption and suggestion?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks & Best Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">George<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Lian, George (Nokia - CN/Hangzhou)
<br>
<b>Sent:</b> Monday, October 31, 2016 4:25 PM<br>
<b>To:</b> Pranith Kumar Karampuri <pkarampu@redhat.com>; Raghavendra Gowdappa <rgowdapp@redhat.com><br>
<b>Cc:</b> I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX_GMS <I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com>; Zhang, Bingxuan (Nokia - CN/Hangzhou) <bingxuan.zhang@nokia.com>; Gluster-devel@gluster.org; Zizka, Jan (Nokia - CZ/Prague) <jan.zizka@nokia.com><br>
<b>Subject:</b> RE: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">How can we enable debug.trace so that we can inspect the debug data on different xlator?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">I just set “debug.trace on” and “debug.log-file yes” seems not work now.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">And one more update for this issue, if we set performance.stat-prefetch to off, the issue will not be occurred. (our previous test maybe not correct</span><span lang="EN-US" style="font-size:10.5pt;font-family:Wingdings;color:#1F497D">J</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">
)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks & Best Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">George<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Pranith Kumar Karampuri [<a href="mailto:pkarampu@redhat.com">mailto:pkarampu@redhat.com</a>]
<br>
<b>Sent:</b> Friday, October 28, 2016 2:39 PM<br>
<b>To:</b> Lian, George (Nokia - CN/Hangzhou) <<a href="mailto:george.lian@nokia.com">george.lian@nokia.com</a>><br>
<b>Cc:</b> Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com">rgowdapp@redhat.com</a>>; I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX_GMS <<a href="mailto:I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com">I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com</a>>; Zhang,
Bingxuan (Nokia - CN/Hangzhou) <<a href="mailto:bingxuan.zhang@nokia.com">bingxuan.zhang@nokia.com</a>>;
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>; Zizka, Jan (Nokia - CZ/Prague) <<a href="mailto:jan.zizka@nokia.com">jan.zizka@nokia.com</a>><br>
<b>Subject:</b> Re: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">hi George,<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"> It would help if we can identify the bare minimum xlators which are contributing to the issue like Raghavendra was mentioning earlier. We were wondering if it is possible for you to help us in identifying the issue
by running the workload on a modified setup? We can suggest testing out using custom volfiles so that we can slowly build the graph which could be causing this issue. We would like you guys to try out this problem with just posix-xlator and fuse and nothing
else.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Thu, Oct 27, 2016 at 1:40 PM, Lian, George (Nokia - CN/Hangzhou) <<a href="mailto:george.lian@nokia.com" target="_blank">george.lian@nokia.com</a>> wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<p class="MsoNormal"><span lang="EN-US">Hi, Raghavendra,<br>
<br>
Could you please give some suggestion for this issue? we try to find the clue for this issue for a long time, but it has no progress:(<br>
<br>
<span class="im">Thanks & Best Regards,</span><br>
<span class="im">George</span><br>
<br>
<span class="im">-----Original Message-----</span><br>
<span class="im">From: Lian, George (Nokia - CN/Hangzhou)</span><br>
<span class="im">Sent: Wednesday, October 19, 2016 4:40 PM</span><br>
<span class="im">To: 'Raghavendra Gowdappa' <<a href="mailto:rgowdapp@redhat.com">rgowdapp@redhat.com</a>></span><br>
<span class="im">Cc: <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>; I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX_GMS <<a href="mailto:I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com">I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com</a>>; Zhang,
Bingxuan (Nokia - CN/Hangzhou) <<a href="mailto:bingxuan.zhang@nokia.com">bingxuan.zhang@nokia.com</a>>; Zizka, Jan (Nokia - CZ/Prague) <<a href="mailto:jan.zizka@nokia.com">jan.zizka@nokia.com</a>></span><br>
<span class="im">Subject: RE: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file</span><br>
<br>
<span class="im">Hi, Raghavendra</span><br>
<br>
<span class="im">Just now, we test it with glusterfs log with debug-level "TRACE", and let some application trigger "glusterfs" produce large log, in that case, when we set write-behind and stat-prefetch both OFF,</span><br>
<span class="im">Tail the glusterfs log such like mnt-{VOLUME-NAME}.log, it still failed with "file truncated",</span><br>
<br>
<span class="im">So that means if file's IO in huge amount, the issue will still be there even write-behind and stat-prefetch both OFF.</span><br>
<br>
<span class="im">Best Regards,</span><br>
<span class="im">George</span><br>
<br>
<span class="im">-----Original Message-----</span><br>
<span class="im">From: Raghavendra Gowdappa [mailto:<a href="mailto:rgowdapp@redhat.com">rgowdapp@redhat.com</a>]</span><o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Sent: Wednesday, October 19, 2016 2:54 PM<br>
To: Lian, George (Nokia - CN/Hangzhou) <<a href="mailto:george.lian@nokia.com">george.lian@nokia.com</a>><br>
Cc: <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>; I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX_GMS <<a href="mailto:I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com">I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com</a>>; Zhang, Bingxuan (Nokia
- CN/Hangzhou) <<a href="mailto:bingxuan.zhang@nokia.com">bingxuan.zhang@nokia.com</a>>; Zizka, Jan (Nokia - CZ/Prague) <<a href="mailto:jan.zizka@nokia.com">jan.zizka@nokia.com</a>><br>
Subject: Re: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file<br>
<br>
<br>
<br>
----- Original Message -----<br>
> From: "George Lian (Nokia - CN/Hangzhou)" <<a href="mailto:george.lian@nokia.com">george.lian@nokia.com</a>><br>
> To: "Raghavendra Gowdappa" <<a href="mailto:rgowdapp@redhat.com">rgowdapp@redhat.com</a>><br>
> Cc: <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>, "I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX_GMS"<br>
> <<a href="mailto:I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com">I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com</a>>, "Bingxuan Zhang (Nokia - CN/Hangzhou)"<br>
> <<a href="mailto:bingxuan.zhang@nokia.com">bingxuan.zhang@nokia.com</a>>, "Jan Zizka (Nokia - CZ/Prague)" <<a href="mailto:jan.zizka@nokia.com">jan.zizka@nokia.com</a>><br>
> Sent: Wednesday, October 19, 2016 12:05:01 PM<br>
> Subject: RE: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file<br>
><br>
> Hi, Raghavendra,<br>
><br>
> Thanks a lots for your quickly update!<br>
> In my case, there are so many process(write) is writing to the syslog file,<br>
> it do involve the writer is in the same host and writing in same mount point<br>
> while the tail(reader) is reading it.<br>
><br>
> The bug I just guess is:<br>
> When a writer write the data with write-behind, it call the call-back<br>
> function " mdc_writev_cbk" and called "mdc_inode_iatt_set_validate" to<br>
> validate the "iatt" data, but with the code I mentioned last mail, it do<br>
> nothing.<br>
<br>
mdc_inode_iatt_set_validate has following code<br>
<br>
<snippet><br>
if (!iatt || !iatt->ia_ctime) {<br>
mdc->ia_time = 0;<br>
goto unlock;<br>
}<br>
</snippet><br>
<br>
Which means a NULL iatt sets mdc->ia_time to 0. This results in subsequent lookup/stat calls to be NOT served from md-cache. Instead, the stat is served from backend bricks. So, I don't see an issue here.<br>
<br>
However, one case where a NULL iatt is different from a valid iatt (which differs from the value stored in md-cache) is that the latter results in a call to inode_invalidate. This invalidation propagates to kernel and all dentry and page cache corresponding
to file is purged. So, I am suspecting whether the stale stat you saw was served from kernel cache (not from glusterfs). If this is the case, having mount options "attribute-timeout=0" and "entry-timeout=0" should've helped.<br>
<br>
I am still at loss to point out the RCA for this issue.<br>
<br>
<br>
> And in same time, the reader(tail) read the "iatt" data, but in case of the<br>
> cache-time is not timeout, it will return the "iatt" data without the last<br>
> change.<br>
><br>
> Do your think it is a possible bug?<br>
><br>
> Thanks & Best Regards,<br>
> George<br>
><br>
> -----Original Message-----<br>
> From: Raghavendra Gowdappa [mailto:<a href="mailto:rgowdapp@redhat.com">rgowdapp@redhat.com</a>]<br>
> Sent: Wednesday, October 19, 2016 2:06 PM<br>
> To: Lian, George (Nokia - CN/Hangzhou) <<a href="mailto:george.lian@nokia.com">george.lian@nokia.com</a>><br>
> Cc: <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>; I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX_GMS<br>
> <<a href="mailto:I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com">I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com</a>>; Zhang, Bingxuan (Nokia -<br>
> CN/Hangzhou) <<a href="mailto:bingxuan.zhang@nokia.com">bingxuan.zhang@nokia.com</a>>; Zizka, Jan (Nokia - CZ/Prague)<br>
> <<a href="mailto:jan.zizka@nokia.com">jan.zizka@nokia.com</a>><br>
> Subject: Re: [Gluster-devel] Issue about the size of fstat is less than the<br>
> really size of the syslog file<br>
><br>
><br>
><br>
> ----- Original Message -----<br>
> > From: "George Lian (Nokia - CN/Hangzhou)" <<a href="mailto:george.lian@nokia.com">george.lian@nokia.com</a>><br>
> > To: "Raghavendra Gowdappa" <<a href="mailto:rgowdapp@redhat.com">rgowdapp@redhat.com</a>><br>
> > Cc: <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>, "I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX_GMS"<br>
> > <<a href="mailto:I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com">I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com</a>>, "Bingxuan Zhang (Nokia<br>
> > - CN/Hangzhou)"<br>
> > <<a href="mailto:bingxuan.zhang@nokia.com">bingxuan.zhang@nokia.com</a>>, "Jan Zizka (Nokia - CZ/Prague)"<br>
> > <<a href="mailto:jan.zizka@nokia.com">jan.zizka@nokia.com</a>><br>
> > Sent: Wednesday, October 19, 2016 10:51:24 AM<br>
> > Subject: RE: [Gluster-devel] Issue about the size of fstat is less than the<br>
> > really size of the syslog file<br>
> ><br>
> > Hi, Raghavendra,<br>
> ><br>
> > When we disable md-cache(gluster volume set log<br>
> > performance.md-cache-timeout<br>
> > 0), the issue seems gone.<br>
> > (we can't disable with " gluster volume set log performance.md-cache off"<br>
> > why?)<br>
><br>
> Please use<br>
> #gluster volume set log performance.stat-prefetch off<br>
><br>
> ><br>
> > So I double confuse that the code I abstract in last mail maybe have some<br>
> > issue for this case.<br>
> > Could you please share your comments?<br>
><br>
> Please find my comments below.<br>
><br>
> ><br>
> > Thanks & Best Regards,<br>
> > George<br>
> ><br>
> > -----Original Message-----<br>
> > From: Lian, George (Nokia - CN/Hangzhou)<br>
> > Sent: Friday, October 14, 2016 1:44 PM<br>
> > To: 'Raghavendra Gowdappa' <<a href="mailto:rgowdapp@redhat.com">rgowdapp@redhat.com</a>><br>
> > Cc: <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>; I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX_GMS<br>
> > <<a href="mailto:I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com">I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com</a>>; Zhang, Bingxuan (Nokia<br>
> > -<br>
> > CN/Hangzhou) <<a href="mailto:bingxuan.zhang@nokia.com">bingxuan.zhang@nokia.com</a>>; Zizka, Jan (Nokia - CZ/Prague)<br>
> > <<a href="mailto:jan.zizka@nokia.com">jan.zizka@nokia.com</a>><br>
> > Subject: RE: [Gluster-devel] Issue about the size of fstat is less than the<br>
> > really size of the syslog file<br>
> ><br>
> > Hi, Raghavendra,<br>
> ><br>
> > Our version of GlusterFS is 3.6.9, and I also check the newest code of main<br>
> > branch, the function of " mdc_inode_iatt_set_validate" is almost same, from<br>
> > the following code of this function,<br>
> > We could see a "TODO" comments inline, does it mean if we enhance<br>
> > write-behind feature, the "iatt" field in callback will be NULL, so that<br>
> > inode_invalidate will not be called? So the size of file will not update<br>
> > since "write behind" enabled ?<br>
> > Is it the root cause for "tail" application failed with "file truncated"<br>
> > issue ?<br>
> ><br>
> > LOCK (&mdc->lock);<br>
> > {<br>
> > if (!iatt || !iatt->ia_ctime) {<br>
> > mdc->ia_time = 0;<br>
> > goto unlock;<br>
> > }<br>
> ><br>
> > /*<br>
> > * Invalidate the inode if the mtime or ctime has changed<br>
> > * and the prebuf doesn't match the value we have cached.<br>
> > * TODO: writev returns with a NULL iatt due to<br>
> > * performance/write-behind, causing invalidation on writes.<br>
> > */<br>
><br>
> The issue explained in this comment is hit only when writes are done. But, in<br>
> your use-case only "tail" is the application running on the mount (If I am<br>
> not wrong, the writer is running on a different mountpoint). So, I doubt<br>
> you are hitting this issue. But, you are saying that the issue goes away<br>
> when write-behind/md-cache is turned off pointing to some interaction<br>
> between md-cache and write-behind causing the issue. I need more time to<br>
> look into this issue. Can you file a bug on this?<br>
><br>
> > if (IA_ISREG(inode->ia_type) &&<br>
> > ((iatt->ia_mtime != mdc->md_mtime) ||<br>
> > (iatt->ia_ctime != mdc->md_ctime)))<br>
> > if (!prebuf || (prebuf->ia_ctime != mdc->md_ctime) ||<br>
> > (prebuf->ia_mtime != mdc->md_mtime))<br>
> > inode_invalidate(inode);<br>
> ><br>
> > mdc_from_iatt (mdc, iatt);<br>
> ><br>
> > time (&mdc->ia_time);<br>
> > }<br>
> ><br>
> > Best Regards,<br>
> > George<br>
> > -----Original Message-----<br>
> > From: Raghavendra Gowdappa [mailto:<a href="mailto:rgowdapp@redhat.com">rgowdapp@redhat.com</a>]<br>
> > Sent: Thursday, October 13, 2016 8:58 PM<br>
> > To: Lian, George (Nokia - CN/Hangzhou) <<a href="mailto:george.lian@nokia.com">george.lian@nokia.com</a>><br>
> > Cc: <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>; I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX_GMS<br>
> > <<a href="mailto:I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com">I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com</a>>; Zhang, Bingxuan (Nokia<br>
> > -<br>
> > CN/Hangzhou) <<a href="mailto:bingxuan.zhang@nokia.com">bingxuan.zhang@nokia.com</a>>; Zizka, Jan (Nokia - CZ/Prague)<br>
> > <<a href="mailto:jan.zizka@nokia.com">jan.zizka@nokia.com</a>><br>
> > Subject: Re: [Gluster-devel] Issue about the size of fstat is less than the<br>
> > really size of the syslog file<br>
> ><br>
> ><br>
> ><br>
> > ----- Original Message -----<br>
> > > From: "George Lian (Nokia - CN/Hangzhou)" <<a href="mailto:george.lian@nokia.com">george.lian@nokia.com</a>><br>
> > > To: <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> > > Cc: "I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX_GMS"<br>
> > > <<a href="mailto:I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com">I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com</a>>, "Bingxuan Zhang<br>
> > > (Nokia<br>
> > > - CN/Hangzhou)" <<a href="mailto:bingxuan.zhang@nokia.com">bingxuan.zhang@nokia.com</a>>, "Jan Zizka (Nokia -<br>
> > > CZ/Prague)"<br>
> > > <<a href="mailto:jan.zizka@nokia.com">jan.zizka@nokia.com</a>><br>
> > > Sent: Thursday, October 13, 2016 2:33:53 PM<br>
> > > Subject: [Gluster-devel] Issue about the size of fstat is less than the<br>
> > > really size of the syslog file<br>
> > ><br>
> > > Hi, Dear Expert,<br>
> > > We have use glusterfs as a network filesystem, and syslog store in there,<br>
> > > some clients on different host may write the syslog file via </span>“<span lang="EN-US">glusterfs</span>”<span lang="EN-US"><br>
> > > mount point.<br>
> > > Now we encounter an issue when we </span>“<span lang="EN-US">tail</span>”<span lang="EN-US"> the syslog file, it will<br>
> > > occasional<br>
> > > failed with error </span>“<span lang="EN-US"> file truncated </span>”<span lang="EN-US"><br>
> > > As we study and trace with the </span>“<span lang="EN-US">tail</span>”<span lang="EN-US"> source code, it failed with the<br>
> > > following code:<br>
> > > if ( S_ISREG (mode) && stats.st_size < f[i].size )<br>
> > > {<br>
> > > error (0, 0, _("%s: file truncated"), quotef (name));<br>
> > > /* Assume the file was truncated to 0,<br>
> > > and therefore output all "new" data. */<br>
> > > xlseek (fd, 0, SEEK_SET, name);<br>
> > > f[i].size = 0;<br>
> > > }<br>
> > > When stats.st_size < f[i].size, what mean the size report by fstat is<br>
> > > less<br>
> > > than </span>“<span lang="EN-US">tail</span>”<span lang="EN-US"> had read, it lead to
</span>“<span lang="EN-US">file truncated</span>”<span lang="EN-US">, we also use
</span>“<span lang="EN-US">strace</span>”<span lang="EN-US"><br>
> > > tools to trace the tail application, the related tail strace log as the<br>
> > > below:<br>
> > > nanosleep({1, 0}, NULL) = 0<br>
> > > fstat(3, {st_mode=S_IFREG|0644, st_size=192543105, ...}) = 0<br>
> > > nanosleep({1, 0}, NULL) = 0<br>
> > > fstat(3, {st_mode=S_IFREG|0644, st_size=192543105, ...}) = 0<br>
> > > nanosleep({1, 0}, NULL) = 0<br>
> > > fstat(3, {st_mode=S_IFREG|0644, st_size=192543105, ...}) = 0<br>
> > > nanosleep({1, 0}, NULL) = 0<br>
> > > fstat(3, {st_mode=S_IFREG|0644, st_size=192544549, ...}) = 0<br>
> > > read(3, " Data </span>…<span lang="EN-US"> -"..., 8192) = 1444<br>
> > > read(3, " Data.. "..., 8192) = 720<br>
> > > read(3, "", 8192) = 0<br>
> > > fstat(3, {st_mode=S_IFREG|0644, st_size=192544789, ...}) = 0<br>
> > > write(1, </span>“<span lang="EN-US">DATA</span>…<span lang="EN-US">..</span>”<span lang="EN-US"> ) = 2164<br>
> > > write(2, "tail: ", 6tail: ) = 6<br>
> > > write(2, "/mnt/log/master/syslog: file tru"..., 38/mnt/log/master/syslog:<br>
> > > file truncated) = 38<br>
> > > as the above strace log, tail has read 1444+720=2164 bytes,<br>
> > > but fstat tell </span>“<span lang="EN-US">tail</span>”<span lang="EN-US"> 192544789
</span>–<span lang="EN-US"> 192543105 = 1664 which less than 2164,<br>
> > > so<br>
> > > it lead to </span>“<span lang="EN-US">tail</span>”<span lang="EN-US"> application
</span>“<span lang="EN-US">file truncated</span>”<span lang="EN-US">.<br>
> > > And if we turn off </span>“<span lang="EN-US">write-behind</span>”<span lang="EN-US"> feature, the issue will not be<br>
> > > reproduced<br>
> > > any more.<br>
> ><br>
> > That seems strange. There are no writes happening on the fd/inode through<br>
> > which tail is reading/stating from. So, it seems strange that write-behind<br>
> > is involved here. I suspect whether any of md-cache/read-ahead/io-cache is<br>
> > causing the issue. Can you,<br>
> ><br>
> > 1. Turn off md-cache, read-ahead, io-cache xlators<br>
> > 2. mount glusterfs with --attribute-timeout=0<br>
> > 3. set write-behind on<br>
> ><br>
> > and rerun the tests? If you don't hit the issue, you can experiment by<br>
> > turning on/off of md-cache, read-ahead and io-cache translators and see<br>
> > what<br>
> > are the minimal number of xlators that need to be turned off to not hit the<br>
> > issue (with write-behind on)?<br>
> ><br>
> > regards,<br>
> > Raghavendra<br>
> ><br>
> > > So we think it may be related to cache consistence issue due to<br>
> > > performance<br>
> > > consider, but we still have concern that:<br>
> > > The syslog file is used only with </span>“<span lang="EN-US">Append</span>”<span lang="EN-US"> mode, so the size of file<br>
> > > shouldn</span>’<span lang="EN-US">t be reduced, when a client read the file, why
</span>“<span lang="EN-US">fstat</span>”<span lang="EN-US"> can</span>’<span lang="EN-US">t<br>
> > > return<br>
> > > the really size match to the cache?<br>
> > > From current investigation, we doubt that the current implement of<br>
> > > </span>“<span lang="EN-US">glusterfs</span>”<span lang="EN-US"> has a bug on
</span>“<span lang="EN-US">fstat</span>”<span lang="EN-US"> when cache is on.<br>
> > > Your comments is our highly appreciated!<br>
> > > Thanks & Best Regards<br>
> > > George<br>
> > ><br>
> > > _______________________________________________<br>
> > > Gluster-devel mailing list<br>
> > > <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> > > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" target="_blank">
http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
> ><br>
><br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><o:p></o:p></span></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US"><br>
<br clear="all">
<br>
-- <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Pranith<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</body>
</html>