<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 3, 2016 at 10:27 AM, Raghavendra G <span dir="ltr"><<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="gmail-">On Thu, Nov 3, 2016 at 7:16 AM, Lian, George (Nokia - CN/Hangzhou) <span dir="ltr"><<a href="mailto:george.lian@nokia.com" target="_blank">george.lian@nokia.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span>>Yes. I was assuming that the previous results were tested with:<br>
>1. write-behind on with the fix<br>
>2. quick-read and readdir-ahead off<br>
</span><span># gluster volume info log<br>
<br>
</span>performance.quick-read: off<br>
performance.readdir-ahead: off<br>
performance.stat-prefetch: on<br>
performance.write-behind: on<br>
<br>
<br>
with the above configuration and write-behind.so with patch 2, the "tail truncated" issue still be there.<br>
<br>
# tail -f syslog >/dev/null<br>
<span>tail: syslog: file truncated<br>
tail: syslog: file truncated<br>
<br>
</span>FYI,<br></blockquote><div><br></div></span><div>Thanks George. I'll take a look.<br></div></div></div></div></blockquote><div><br></div><div>Can you please test with following configuration?</div><div><br></div><div>1. write-behind on with my fix</div><div>2. readdir-ahead and quick-read off</div><div>3. performance.stat-prefetch on</div><div>4. performance.force-readdirp off</div><div>5. dht.force-readdirp off</div><div>6. Also mount glusterfs with option "use-readdirp=no"</div><div><br></div><div><div>[root@booradley glusterfs]# mount -t glusterfs -o use-readdirp=no booradley:/newptop /mnt</div><div><br></div><div>[root@booradley glusterfs]# ps ax | grep -i mnt</div><div>14418 ? Ssl 0:00 /usr/local/sbin/glusterfs --use-readdirp=no --volfile-server=booradley --volfile-id=/newptop /mnt</div><div><br></div><div>[root@booradley glusterfs]# gluster volume set newptop performance.write-behind on</div><div>volume set: success</div><div><br></div><div>[root@booradley glusterfs]# gluster volume set newptop performance.quick-read off</div><div>volume set: success</div><div><br></div><div>[root@booradley glusterfs]# gluster volume set newptop performance.stat-prefetch on</div><div>volume set: success</div><div><br></div><div>[root@booradley glusterfs]# gluster volume set newptop performance.force-readdirp off</div><div>volume set: success</div><div><br></div><div>[root@booradley glusterfs]# gluster volume set newptop dht.force-readdirp off</div><div>volume set: success</div><div><br></div><div>[root@booradley glusterfs]# gluster volume set newptop performance.readdir-ahead off<br></div><div>volume set: success</div><div><br></div><div>[root@booradley glusterfs]# gluster volume info newptop</div><div> </div><div>Volume Name: newptop</div><div>Type: Distribute</div><div>Volume ID: 092756e1-e095-4e05-9f14-3e9a6aed908c</div><div>Status: Started</div><div>Snapshot Count: 0</div><div>Number of Bricks: 1</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: booradley:/home/export/newptop</div><div>Options Reconfigured:</div><div>dht.force-readdirp: off</div><div>performance.force-readdirp: off</div><div>performance.stat-prefetch: on</div><div>performance.write-behind: on</div><div>performance.quick-read: off</div><div>transport.address-family: inet</div><div>performance.readdir-ahead: off</div><div>nfs.disable: on</div><div><br></div></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> <br></div><div><div class="gmail-h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<span class="gmail-m_8097377878677447059im gmail-m_8097377878677447059HOEnZb"><br>
Best Regards,<br>
George<br>
<br>
<br>
-----Original Message-----<br>
From: Raghavendra Gowdappa [mailto:<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>]<br>
</span><div class="gmail-m_8097377878677447059HOEnZb"><div class="gmail-m_8097377878677447059h5">Sent: Wednesday, November 02, 2016 5:41 PM<br>
To: Lian, George (Nokia - CN/Hangzhou) <<a href="mailto:george.lian@nokia.com" target="_blank">george.lian@nokia.com</a>><br>
Cc: Raghavendra G <<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>>; <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>; Zizka, Jan (Nokia - CZ/Prague) <<a href="mailto:jan.zizka@nokia.com" target="_blank">jan.zizka@nokia.com</a>>; Zhang, Bingxuan (Nokia - CN/Hangzhou) <<a href="mailto:bingxuan.zhang@nokia.com" target="_blank">bingxuan.zhang@nokia.com</a>><br>
Subject: Re: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file<br>
<br>
<br>
<br>
----- Original Message -----<br>
> From: "George Lian (Nokia - CN/Hangzhou)" <<a href="mailto:george.lian@nokia.com" target="_blank">george.lian@nokia.com</a>><br>
> To: "Raghavendra Gowdappa" <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>><br>
> Cc: "Raghavendra G" <<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>>, <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>, "Jan Zizka (Nokia - CZ/Prague)"<br>
> <<a href="mailto:jan.zizka@nokia.com" target="_blank">jan.zizka@nokia.com</a>>, "Bingxuan Zhang (Nokia - CN/Hangzhou)" <<a href="mailto:bingxuan.zhang@nokia.com" target="_blank">bingxuan.zhang@nokia.com</a>><br>
> Sent: Wednesday, November 2, 2016 1:38:44 PM<br>
> Subject: RE: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file<br>
><br>
> Yes, I confirm use the Patch 2.<br>
><br>
> One update: the issue is occurred when readdir-ahead off and write-behind on.<br>
> Seems gone when write-behind and readdir-ahead and quick-read all off.<br>
> Not verified with readdir-ahead and quick-read both off and write-behind on<br>
> till now.<br>
><br>
> Need I test it with write-behind on and readdir-ahead and quick-read both<br>
> off?<br>
<br>
Yes. I was assuming that the previous results were tested with:<br>
1. write-behind on with the fix<br>
2. quick-read and readdir-ahead off<br>
<br>
If not, test results with this configuration will help.<br>
<br>
><br>
> Best Regards,<br>
> George<br>
><br>
> -----Original Message-----<br>
> From: Raghavendra Gowdappa [mailto:<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>]<br>
> Sent: Wednesday, November 02, 2016 4:04 PM<br>
> To: Lian, George (Nokia - CN/Hangzhou) <<a href="mailto:george.lian@nokia.com" target="_blank">george.lian@nokia.com</a>><br>
> Cc: Raghavendra G <<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>>; <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>;<br>
> Zizka, Jan (Nokia - CZ/Prague) <<a href="mailto:jan.zizka@nokia.com" target="_blank">jan.zizka@nokia.com</a>>; Zhang, Bingxuan (Nokia<br>
> - CN/Hangzhou) <<a href="mailto:bingxuan.zhang@nokia.com" target="_blank">bingxuan.zhang@nokia.com</a>><br>
> Subject: Re: [Gluster-devel] Issue about the size of fstat is less than the<br>
> really size of the syslog file<br>
><br>
><br>
><br>
> ----- Original Message -----<br>
> > From: "George Lian (Nokia - CN/Hangzhou)" <<a href="mailto:george.lian@nokia.com" target="_blank">george.lian@nokia.com</a>><br>
> > To: "Raghavendra Gowdappa" <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>><br>
> > Cc: "Raghavendra G" <<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>>, <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>,<br>
> > "Jan Zizka (Nokia - CZ/Prague)"<br>
> > <<a href="mailto:jan.zizka@nokia.com" target="_blank">jan.zizka@nokia.com</a>>, "Bingxuan Zhang (Nokia - CN/Hangzhou)"<br>
> > <<a href="mailto:bingxuan.zhang@nokia.com" target="_blank">bingxuan.zhang@nokia.com</a>><br>
> > Sent: Wednesday, November 2, 2016 1:29:13 PM<br>
> > Subject: RE: [Gluster-devel] Issue about the size of fstat is less than the<br>
> > really size of the syslog file<br>
> ><br>
> > Hi,<br>
> ><br>
> > When those 3 options turn off, the issue seems gone in about 3 hours,<br>
> > otherwise, the issue will be occurred about every 10 minutes.<br>
><br>
> That's a good news. IIRC, you mentioned that you saw the issue with just<br>
> write-behind on, with fix applied (readdir-ahead and quick-read off). Can<br>
> you please confirm you had patcset 2 of <a href="http://review.gluster.org/15757" rel="noreferrer" target="_blank">http://review.gluster.org/1575<wbr>7</a>?<br>
> patchset 1 had some issues that I corrected in 2.<br>
><br>
> ><br>
> > Best Regards,<br>
> > George<br>
> ><br>
> > -----Original Message-----<br>
> > From: Raghavendra Gowdappa [mailto:<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>]<br>
> > Sent: Wednesday, November 02, 2016 1:07 PM<br>
> > To: Lian, George (Nokia - CN/Hangzhou) <<a href="mailto:george.lian@nokia.com" target="_blank">george.lian@nokia.com</a>><br>
> > Cc: Raghavendra G <<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>>; <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>;<br>
> > Zizka, Jan (Nokia - CZ/Prague) <<a href="mailto:jan.zizka@nokia.com" target="_blank">jan.zizka@nokia.com</a>>; Zhang, Bingxuan<br>
> > (Nokia<br>
> > - CN/Hangzhou) <<a href="mailto:bingxuan.zhang@nokia.com" target="_blank">bingxuan.zhang@nokia.com</a>><br>
> > Subject: Re: [Gluster-devel] Issue about the size of fstat is less than the<br>
> > really size of the syslog file<br>
> ><br>
> > Can you try with following xlators turned off?<br>
> ><br>
> > 1. write-behind<br>
> > 2. readdir-ahead<br>
> > 3. quick-read<br>
> ><br>
> > regards,<br>
> > Raghavendra<br>
> ><br>
> > ----- Original Message -----<br>
> > > From: "George Lian (Nokia - CN/Hangzhou)" <<a href="mailto:george.lian@nokia.com" target="_blank">george.lian@nokia.com</a>><br>
> > > To: "Raghavendra Gowdappa" <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>>, "Raghavendra G"<br>
> > > <<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>><br>
> > > Cc: <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>, "Jan Zizka (Nokia - CZ/Prague)"<br>
> > > <<a href="mailto:jan.zizka@nokia.com" target="_blank">jan.zizka@nokia.com</a>>, "Bingxuan Zhang (Nokia -<br>
> > > CN/Hangzhou)" <<a href="mailto:bingxuan.zhang@nokia.com" target="_blank">bingxuan.zhang@nokia.com</a>><br>
> > > Sent: Wednesday, November 2, 2016 10:26:08 AM<br>
> > > Subject: RE: [Gluster-devel] Issue about the size of fstat is less than<br>
> > > the<br>
> > > really size of the syslog file<br>
> > ><br>
> > > Hi,<br>
> > ><br>
> > > Sorry, it seems the issue still there with readdir-ahead off.<br>
> > ><br>
> > > # tail -f syslog >syslog.2.log<br>
> > > tail: syslog: file truncated<br>
> > > tail: syslog: file truncated<br>
> > > tail: syslog: file truncated<br>
> > > tail: syslog: file truncated<br>
> > > tail: syslog: file truncated<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > tail: syslog: file truncated<br>
> > ><br>
> > > The last line is new truncated error message after the following set with<br>
> > > option readdir-ahead.<br>
> > ><br>
> > ><br>
> > > # gluster volume set log performance.readdir-ahead off<br>
> > > volume set: success<br>
> > > [root@SN-0(RNC-124) /root]<br>
> > > # gluster volume info log<br>
> > ><br>
> > > Volume Name: log<br>
> > > Type: Replicate<br>
> > > Volume ID: 9d987a23-1d64-4857-980a-548f1d<wbr>d0d1e6<br>
> > > Status: Started<br>
> > > Number of Bricks: 1 x 2 = 2<br>
> > > Transport-type: tcp<br>
> > > Bricks:<br>
> > > Brick1: 169.254.0.12:/mnt/bricks/log/b<wbr>rick<br>
> > > Brick2: 169.254.0.13:/mnt/bricks/log/b<wbr>rick<br>
> > > Options Reconfigured:<br>
> > > performance.readdir-ahead: off<br>
> > ><br>
> > > BRs<br>
> > > George<br>
> > ><br>
> > > -----Original Message-----<br>
> > > From: <a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@gluster.<wbr>org</a><br>
> > > [mailto:<a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@<wbr>gluster.org</a>] On Behalf Of Raghavendra<br>
> > > Gowdappa<br>
> > > Sent: Tuesday, November 01, 2016 9:29 PM<br>
> > > To: Raghavendra G <<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>><br>
> > > Cc: <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>; Zizka, Jan (Nokia - CZ/Prague)<br>
> > > <<a href="mailto:jan.zizka@nokia.com" target="_blank">jan.zizka@nokia.com</a>>; Zhang, Bingxuan (Nokia - CN/Hangzhou)<br>
> > > <<a href="mailto:bingxuan.zhang@nokia.com" target="_blank">bingxuan.zhang@nokia.com</a>><br>
> > > Subject: Re: [Gluster-devel] Issue about the size of fstat is less than<br>
> > > the<br>
> > > really size of the syslog file<br>
> > ><br>
> > ><br>
> > ><br>
> > > ----- Original Message -----<br>
> > > > From: "Raghavendra G" <<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>><br>
> > > > To: "Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>><br>
> > > > Cc: <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>, "Jan Zizka (Nokia - CZ/Prague)"<br>
> > > > <<a href="mailto:jan.zizka@nokia.com" target="_blank">jan.zizka@nokia.com</a>>, "Bingxuan Zhang (Nokia -<br>
> > > > CN/Hangzhou)" <<a href="mailto:bingxuan.zhang@nokia.com" target="_blank">bingxuan.zhang@nokia.com</a>><br>
> > > > Sent: Tuesday, November 1, 2016 6:55:31 PM<br>
> > > > Subject: Re: [Gluster-devel] Issue about the size of fstat is less than<br>
> > > > the<br>
> > > > really size of the syslog file<br>
> > > ><br>
> > > > Hi George,<br>
> > > ><br>
> > > > You can try your tests in two scenarios:<br>
> > > ><br>
> > > > 1. Switch off following performance translators:<br>
> > > > a. write-behind<br>
> > > > b. quick-read<br>
> > > > c. readdir-ahead<br>
> > > ><br>
> > > > Once you see that the issue goes away in the above scenario, you can<br>
> > > > apply<br>
> > > > write-behind fix [1] and turn write-behind on.<br>
> > > ><br>
> > > > We have a fix in quick-read at [2].<br>
> > > ><br>
> > > > You can also try to test with the quick-read fix and turning quick-read<br>
> > > > on.<br>
> > > > However, since your issue is stale stat I doubt quick-read is not the<br>
> > ><br>
> > > s/quick-read is not the culprit/quick-read is the culprit/<br>
> > ><br>
> > > > culprit here (The issue with quick-read is stale data).<br>
> > > ><br>
> > > > We see that readdir-ahead is not updating/flushing the stats stored<br>
> > > > along<br>
> > > > with dentries when write or file modifications happen. That seem to be<br>
> > > > the<br>
> > > > issue with readdir-ahead. For now, during both of the above two<br>
> > > > scenarios,<br>
> > > > please keep readdir-ahead off.<br>
> > > ><br>
> > > > [1] <a href="http://review.gluster.org/15757/2" rel="noreferrer" target="_blank">http://review.gluster.org/1575<wbr>7/2</a><br>
> > > > [2] <a href="http://review.gluster.org/#/c/15759/2" rel="noreferrer" target="_blank">http://review.gluster.org/#/c/<wbr>15759/2</a><br>
> > > ><br>
> > > > regards,<br>
> > > > Raghavendra<br>
> > > ><br>
> > > > On Tue, Nov 1, 2016 at 7:55 AM, Pranith Kumar Karampuri <<br>
> > > > <a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a><br>
> > > > > wrote:<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > On Tue, Nov 1, 2016 at 7:32 AM, Lian, George (Nokia - CN/Hangzhou) <<br>
> > > > <a href="mailto:george.lian@nokia.com" target="_blank">george.lian@nokia.com</a> > wrote:<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > Hi,<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > I will test it with your patches and update to you when I have result.<br>
> > > ><br>
> > > > hi George,<br>
> > > > Please use <a href="http://review.gluster.org/#/c/15757/2" rel="noreferrer" target="_blank">http://review.gluster.org/#/c/<wbr>15757/2</a> i.e. second version of<br>
> > > > Raghavendra's patch. I tested it and it worked fine. We are still<br>
> > > > trying<br>
> > > > to<br>
> > > > figure out quick-read and readdir-ahead as I type this mail.<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > Thanks a lots<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > Best Regards,<br>
> > > ><br>
> > > > George<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > From: Pranith Kumar Karampuri [mailto: <a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a> ]<br>
> > > > Sent: Monday, October 31, 2016 11:23 AM<br>
> > > > To: Lian, George (Nokia - CN/Hangzhou) < <a href="mailto:george.lian@nokia.com" target="_blank">george.lian@nokia.com</a> ><br>
> > > > Cc: Raghavendra Gowdappa < <a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a> >; Zhang, Bingxuan<br>
> > > > (Nokia<br>
> > > > -<br>
> > > > CN/Hangzhou) < <a href="mailto:bingxuan.zhang@nokia.com" target="_blank">bingxuan.zhang@nokia.com</a> >; <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a> ;<br>
> > > > Zizka, Jan (Nokia - CZ/Prague) < <a href="mailto:jan.zizka@nokia.com" target="_blank">jan.zizka@nokia.com</a> ><br>
> > > ><br>
> > > > Subject: Re: [Gluster-devel] Issue about the size of fstat is less than<br>
> > > > the<br>
> > > > really size of the syslog file<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > Removing <a href="mailto:I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com" target="_blank">I_EXT_MBB_WCDMA_SWD3_DA1_MATRI<wbr>X@internal.nsn.com</a> , it is<br>
> > > > causing<br>
> > > > mail delivery problems for me.<br>
> > > ><br>
> > > ><br>
> > > > George,<br>
> > > ><br>
> > > ><br>
> > > > Raghavendra and I made some progress on this issue. We were in parallel<br>
> > > > working on another issue which is similar where elastic search indices<br>
> > > > are<br>
> > > > getting corrupted because of wrong stat sizes in our opinion. So I have<br>
> > > > been<br>
> > > > running different translator stacks in identifying the problematic<br>
> > > > xlators<br>
> > > > which are leading to indices corruption.<br>
> > > ><br>
> > > ><br>
> > > > We found the list to be 1) Write-behind, 2) Quick-read, 3)<br>
> > > > Readdir-ahead.<br>
> > > > Raghavendra and I just had a chat and we are suspecting that lack of<br>
> > > > lookup/readdirp implementation in write-behind could be the reason for<br>
> > > > this<br>
> > > > problem. Similar problems may exist in other two xlators too. But we<br>
> > > > are<br>
> > > > working on write-behind with priority.<br>
> > > ><br>
> > > ><br>
> > > > Our theory is this:<br>
> > > ><br>
> > > ><br>
> > > > If we do a 4KB write for example and it is cached in write-behind and<br>
> > > > we<br>
> > > > do<br>
> > > > a<br>
> > > > lookup on the file/do a readdirp on the directory with this file we<br>
> > > > send<br>
> > > > out<br>
> > > > wrong stat value to the kernel. There are different caches between<br>
> > > > kernel<br>
> > > > and gluster which may lead to fstat never coming till write-behind. So<br>
> > > > we<br>
> > > > need to make sure that we don't get into this situation.<br>
> > > ><br>
> > > ><br>
> > > > Action items:<br>
> > > ><br>
> > > ><br>
> > > > At the moment Raghavendra is working on a patch to implement<br>
> > > > lookup/readdirp<br>
> > > > in write-behind. I am going to test the same for elastic search. Will<br>
> > > > it<br>
> > > > be<br>
> > > > possible for you to test your application against the same patch and<br>
> > > > confirm<br>
> > > > that the patch fixes the problem?<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > On Fri, Oct 28, 2016 at 12:08 PM, Pranith Kumar Karampuri <<br>
> > > > <a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a> > wrote:<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > hi George,<br>
> > > ><br>
> > > ><br>
> > > > It would help if we can identify the bare minimum xlators which are<br>
> > > > contributing to the issue like Raghavendra was mentioning earlier. We<br>
> > > > were<br>
> > > > wondering if it is possible for you to help us in identifying the issue<br>
> > > > by<br>
> > > > running the workload on a modified setup? We can suggest testing out<br>
> > > > using<br>
> > > > custom volfiles so that we can slowly build the graph which could be<br>
> > > > causing<br>
> > > > this issue. We would like you guys to try out this problem with just<br>
> > > > posix-xlator and fuse and nothing else.<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > On Thu, Oct 27, 2016 at 1:40 PM, Lian, George (Nokia - CN/Hangzhou) <<br>
> > > > <a href="mailto:george.lian@nokia.com" target="_blank">george.lian@nokia.com</a> > wrote:<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > Hi, Raghavendra,<br>
> > > ><br>
> > > > Could you please give some suggestion for this issue? we try to find<br>
> > > > the<br>
> > > > clue<br>
> > > > for this issue for a long time, but it has no progress:(<br>
> > > ><br>
> > > > Thanks & Best Regards,<br>
> > > > George<br>
> > > ><br>
> > > > -----Original Message-----<br>
> > > > From: Lian, George (Nokia - CN/Hangzhou)<br>
> > > > Sent: Wednesday, October 19, 2016 4:40 PM<br>
> > > > To: 'Raghavendra Gowdappa' < <a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a> ><br>
> > > > Cc: <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a> ; I_EXT_MBB_WCDMA_SWD3_DA1_MATRI<wbr>X_GMS <<br>
> > > > <a href="mailto:I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com" target="_blank">I_EXT_MBB_WCDMA_SWD3_DA1_MATRI<wbr>X@internal.nsn.com</a> >; Zhang, Bingxuan<br>
> > > > (Nokia<br>
> > > > -<br>
> > > > CN/Hangzhou) < <a href="mailto:bingxuan.zhang@nokia.com" target="_blank">bingxuan.zhang@nokia.com</a> >; Zizka, Jan (Nokia -<br>
> > > > CZ/Prague)<br>
> > > > <<br>
> > > > <a href="mailto:jan.zizka@nokia.com" target="_blank">jan.zizka@nokia.com</a> ><br>
> > > > Subject: RE: [Gluster-devel] Issue about the size of fstat is less than<br>
> > > > the<br>
> > > > really size of the syslog file<br>
> > > ><br>
> > > > Hi, Raghavendra<br>
> > > ><br>
> > > > Just now, we test it with glusterfs log with debug-level "TRACE", and<br>
> > > > let<br>
> > > > some application trigger "glusterfs" produce large log, in that case,<br>
> > > > when<br>
> > > > we set write-behind and stat-prefetch both OFF,<br>
> > > > Tail the glusterfs log such like mnt-{VOLUME-NAME}.log, it still failed<br>
> > > > with<br>
> > > > "file truncated",<br>
> > > ><br>
> > > > So that means if file's IO in huge amount, the issue will still be<br>
> > > > there<br>
> > > > even<br>
> > > > write-behind and stat-prefetch both OFF.<br>
> > > ><br>
> > > > Best Regards,<br>
> > > > George<br>
> > > ><br>
> > > > -----Original Message-----<br>
> > > > From: Raghavendra Gowdappa [mailto: <a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a> ]<br>
> > > ><br>
> > > ><br>
> > > > Sent: Wednesday, October 19, 2016 2:54 PM<br>
> > > > To: Lian, George (Nokia - CN/Hangzhou) < <a href="mailto:george.lian@nokia.com" target="_blank">george.lian@nokia.com</a> ><br>
> > > > Cc: <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a> ; I_EXT_MBB_WCDMA_SWD3_DA1_MATRI<wbr>X_GMS <<br>
> > > > <a href="mailto:I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com" target="_blank">I_EXT_MBB_WCDMA_SWD3_DA1_MATRI<wbr>X@internal.nsn.com</a> >; Zhang, Bingxuan<br>
> > > > (Nokia<br>
> > > > -<br>
> > > > CN/Hangzhou) < <a href="mailto:bingxuan.zhang@nokia.com" target="_blank">bingxuan.zhang@nokia.com</a> >; Zizka, Jan (Nokia -<br>
> > > > CZ/Prague)<br>
> > > > <<br>
> > > > <a href="mailto:jan.zizka@nokia.com" target="_blank">jan.zizka@nokia.com</a> ><br>
> > > > Subject: Re: [Gluster-devel] Issue about the size of fstat is less than<br>
> > > > the<br>
> > > > really size of the syslog file<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > ----- Original Message -----<br>
> > > > > From: "George Lian (Nokia - CN/Hangzhou)" < <a href="mailto:george.lian@nokia.com" target="_blank">george.lian@nokia.com</a> ><br>
> > > > > To: "Raghavendra Gowdappa" < <a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a> ><br>
> > > > > Cc: <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a> , "I_EXT_MBB_WCDMA_SWD3_DA1_MATR<wbr>IX_GMS"<br>
> > > > > < <a href="mailto:I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com" target="_blank">I_EXT_MBB_WCDMA_SWD3_DA1_MATRI<wbr>X@internal.nsn.com</a> >, "Bingxuan Zhang<br>
> > > > > (Nokia - CN/Hangzhou)"<br>
> > > > > < <a href="mailto:bingxuan.zhang@nokia.com" target="_blank">bingxuan.zhang@nokia.com</a> >, "Jan Zizka (Nokia - CZ/Prague)" <<br>
> > > > > <a href="mailto:jan.zizka@nokia.com" target="_blank">jan.zizka@nokia.com</a> ><br>
> > > > > Sent: Wednesday, October 19, 2016 12:05:01 PM<br>
> > > > > Subject: RE: [Gluster-devel] Issue about the size of fstat is less<br>
> > > > > than<br>
> > > > > the<br>
> > > > > really size of the syslog file<br>
> > > > ><br>
> > > > > Hi, Raghavendra,<br>
> > > > ><br>
> > > > > Thanks a lots for your quickly update!<br>
> > > > > In my case, there are so many process(write) is writing to the syslog<br>
> > > > > file,<br>
> > > > > it do involve the writer is in the same host and writing in same<br>
> > > > > mount<br>
> > > > > point<br>
> > > > > while the tail(reader) is reading it.<br>
> > > > ><br>
> > > > > The bug I just guess is:<br>
> > > > > When a writer write the data with write-behind, it call the call-back<br>
> > > > > function " mdc_writev_cbk" and called "mdc_inode_iatt_set_validate"<br>
> > > > > to<br>
> > > > > validate the "iatt" data, but with the code I mentioned last mail, it<br>
> > > > > do<br>
> > > > > nothing.<br>
> > > ><br>
> > > > mdc_inode_iatt_set_validate has following code<br>
> > > ><br>
> > > > <snippet><br>
> > > > if (!iatt || !iatt->ia_ctime) {<br>
> > > > mdc->ia_time = 0;<br>
> > > > goto unlock;<br>
> > > > }<br>
> > > > </snippet><br>
> > > ><br>
> > > > Which means a NULL iatt sets mdc->ia_time to 0. This results in<br>
> > > > subsequent<br>
> > > > lookup/stat calls to be NOT served from md-cache. Instead, the stat is<br>
> > > > served from backend bricks. So, I don't see an issue here.<br>
> > > ><br>
> > > > However, one case where a NULL iatt is different from a valid iatt<br>
> > > > (which<br>
> > > > differs from the value stored in md-cache) is that the latter results<br>
> > > > in<br>
> > > > a<br>
> > > > call to inode_invalidate. This invalidation propagates to kernel and<br>
> > > > all<br>
> > > > dentry and page cache corresponding to file is purged. So, I am<br>
> > > > suspecting<br>
> > > > whether the stale stat you saw was served from kernel cache (not from<br>
> > > > glusterfs). If this is the case, having mount options<br>
> > > > "attribute-timeout=0"<br>
> > > > and "entry-timeout=0" should've helped.<br>
> > > ><br>
> > > > I am still at loss to point out the RCA for this issue.<br>
> > > ><br>
> > > ><br>
> > > > > And in same time, the reader(tail) read the "iatt" data, but in case<br>
> > > > > of<br>
> > > > > the<br>
> > > > > cache-time is not timeout, it will return the "iatt" data without the<br>
> > > > > last<br>
> > > > > change.<br>
> > > > ><br>
> > > > > Do your think it is a possible bug?<br>
> > > > ><br>
> > > > > Thanks & Best Regards,<br>
> > > > > George<br>
> > > > ><br>
> > > > > -----Original Message-----<br>
> > > > > From: Raghavendra Gowdappa [mailto: <a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a> ]<br>
> > > > > Sent: Wednesday, October 19, 2016 2:06 PM<br>
> > > > > To: Lian, George (Nokia - CN/Hangzhou) < <a href="mailto:george.lian@nokia.com" target="_blank">george.lian@nokia.com</a> ><br>
> > > > > Cc: <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a> ; I_EXT_MBB_WCDMA_SWD3_DA1_MATRI<wbr>X_GMS<br>
> > > > > < <a href="mailto:I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com" target="_blank">I_EXT_MBB_WCDMA_SWD3_DA1_MATRI<wbr>X@internal.nsn.com</a> >; Zhang, Bingxuan<br>
> > > > > (Nokia -<br>
> > > > > CN/Hangzhou) < <a href="mailto:bingxuan.zhang@nokia.com" target="_blank">bingxuan.zhang@nokia.com</a> >; Zizka, Jan (Nokia -<br>
> > > > > CZ/Prague)<br>
> > > > > < <a href="mailto:jan.zizka@nokia.com" target="_blank">jan.zizka@nokia.com</a> ><br>
> > > > > Subject: Re: [Gluster-devel] Issue about the size of fstat is less<br>
> > > > > than<br>
> > > > > the<br>
> > > > > really size of the syslog file<br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > > > ----- Original Message -----<br>
> > > > > > From: "George Lian (Nokia - CN/Hangzhou)" < <a href="mailto:george.lian@nokia.com" target="_blank">george.lian@nokia.com</a> ><br>
> > > > > > To: "Raghavendra Gowdappa" < <a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a> ><br>
> > > > > > Cc: <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a> ,<br>
> > > > > > "I_EXT_MBB_WCDMA_SWD3_DA1_MATR<wbr>IX_GMS"<br>
> > > > > > < <a href="mailto:I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com" target="_blank">I_EXT_MBB_WCDMA_SWD3_DA1_MATRI<wbr>X@internal.nsn.com</a> >, "Bingxuan<br>
> > > > > > Zhang<br>
> > > > > > (Nokia<br>
> > > > > > - CN/Hangzhou)"<br>
> > > > > > < <a href="mailto:bingxuan.zhang@nokia.com" target="_blank">bingxuan.zhang@nokia.com</a> >, "Jan Zizka (Nokia - CZ/Prague)"<br>
> > > > > > < <a href="mailto:jan.zizka@nokia.com" target="_blank">jan.zizka@nokia.com</a> ><br>
> > > > > > Sent: Wednesday, October 19, 2016 10:51:24 AM<br>
> > > > > > Subject: RE: [Gluster-devel] Issue about the size of fstat is less<br>
> > > > > > than<br>
> > > > > > the<br>
> > > > > > really size of the syslog file<br>
> > > > > ><br>
> > > > > > Hi, Raghavendra,<br>
> > > > > ><br>
> > > > > > When we disable md-cache(gluster volume set log<br>
> > > > > > performance.md-cache-timeout<br>
> > > > > > 0), the issue seems gone.<br>
> > > > > > (we can't disable with " gluster volume set log<br>
> > > > > > performance.md-cache<br>
> > > > > > off"<br>
> > > > > > why?)<br>
> > > > ><br>
> > > > > Please use<br>
> > > > > #gluster volume set log performance.stat-prefetch off<br>
> > > > ><br>
> > > > > ><br>
> > > > > > So I double confuse that the code I abstract in last mail maybe<br>
> > > > > > have<br>
> > > > > > some<br>
> > > > > > issue for this case.<br>
> > > > > > Could you please share your comments?<br>
> > > > ><br>
> > > > > Please find my comments below.<br>
> > > > ><br>
> > > > > ><br>
> > > > > > Thanks & Best Regards,<br>
> > > > > > George<br>
> > > > > ><br>
> > > > > > -----Original Message-----<br>
> > > > > > From: Lian, George (Nokia - CN/Hangzhou)<br>
> > > > > > Sent: Friday, October 14, 2016 1:44 PM<br>
> > > > > > To: 'Raghavendra Gowdappa' < <a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a> ><br>
> > > > > > Cc: <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a> ; I_EXT_MBB_WCDMA_SWD3_DA1_MATRI<wbr>X_GMS<br>
> > > > > > < <a href="mailto:I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com" target="_blank">I_EXT_MBB_WCDMA_SWD3_DA1_MATRI<wbr>X@internal.nsn.com</a> >; Zhang,<br>
> > > > > > Bingxuan<br>
> > > > > > (Nokia<br>
> > > > > > -<br>
> > > > > > CN/Hangzhou) < <a href="mailto:bingxuan.zhang@nokia.com" target="_blank">bingxuan.zhang@nokia.com</a> >; Zizka, Jan (Nokia -<br>
> > > > > > CZ/Prague)<br>
> > > > > > < <a href="mailto:jan.zizka@nokia.com" target="_blank">jan.zizka@nokia.com</a> ><br>
> > > > > > Subject: RE: [Gluster-devel] Issue about the size of fstat is less<br>
> > > > > > than<br>
> > > > > > the<br>
> > > > > > really size of the syslog file<br>
> > > > > ><br>
> > > > > > Hi, Raghavendra,<br>
> > > > > ><br>
> > > > > > Our version of GlusterFS is 3.6.9, and I also check the newest code<br>
> > > > > > of<br>
> > > > > > main<br>
> > > > > > branch, the function of " mdc_inode_iatt_set_validate" is almost<br>
> > > > > > same,<br>
> > > > > > from<br>
> > > > > > the following code of this function,<br>
> > > > > > We could see a "TODO" comments inline, does it mean if we enhance<br>
> > > > > > write-behind feature, the "iatt" field in callback will be NULL, so<br>
> > > > > > that<br>
> > > > > > inode_invalidate will not be called? So the size of file will not<br>
> > > > > > update<br>
> > > > > > since "write behind" enabled ?<br>
> > > > > > Is it the root cause for "tail" application failed with "file<br>
> > > > > > truncated"<br>
> > > > > > issue ?<br>
> > > > > ><br>
> > > > > > LOCK (&mdc->lock);<br>
> > > > > > {<br>
> > > > > > if (!iatt || !iatt->ia_ctime) {<br>
> > > > > > mdc->ia_time = 0;<br>
> > > > > > goto unlock;<br>
> > > > > > }<br>
> > > > > ><br>
> > > > > > /*<br>
> > > > > > * Invalidate the inode if the mtime or ctime has changed<br>
> > > > > > * and the prebuf doesn't match the value we have cached.<br>
> > > > > > * TODO: writev returns with a NULL iatt due to<br>
> > > > > > * performance/write-behind, causing invalidation on writes.<br>
> > > > > > */<br>
> > > > ><br>
> > > > > The issue explained in this comment is hit only when writes are done.<br>
> > > > > But,<br>
> > > > > in<br>
> > > > > your use-case only "tail" is the application running on the mount (If<br>
> > > > > I<br>
> > > > > am<br>
> > > > > not wrong, the writer is running on a different mountpoint). So, I<br>
> > > > > doubt<br>
> > > > > you are hitting this issue. But, you are saying that the issue goes<br>
> > > > > away<br>
> > > > > when write-behind/md-cache is turned off pointing to some interaction<br>
> > > > > between md-cache and write-behind causing the issue. I need more time<br>
> > > > > to<br>
> > > > > look into this issue. Can you file a bug on this?<br>
> > > > ><br>
> > > > > > if (IA_ISREG(inode->ia_type) &&<br>
> > > > > > ((iatt->ia_mtime != mdc->md_mtime) ||<br>
> > > > > > (iatt->ia_ctime != mdc->md_ctime)))<br>
> > > > > > if (!prebuf || (prebuf->ia_ctime != mdc->md_ctime) ||<br>
> > > > > > (prebuf->ia_mtime != mdc->md_mtime))<br>
> > > > > > inode_invalidate(inode);<br>
> > > > > ><br>
> > > > > > mdc_from_iatt (mdc, iatt);<br>
> > > > > ><br>
> > > > > > time (&mdc->ia_time);<br>
> > > > > > }<br>
> > > > > ><br>
> > > > > > Best Regards,<br>
> > > > > > George<br>
> > > > > > -----Original Message-----<br>
> > > > > > From: Raghavendra Gowdappa [mailto: <a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a> ]<br>
> > > > > > Sent: Thursday, October 13, 2016 8:58 PM<br>
> > > > > > To: Lian, George (Nokia - CN/Hangzhou) < <a href="mailto:george.lian@nokia.com" target="_blank">george.lian@nokia.com</a> ><br>
> > > > > > Cc: <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a> ; I_EXT_MBB_WCDMA_SWD3_DA1_MATRI<wbr>X_GMS<br>
> > > > > > < <a href="mailto:I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com" target="_blank">I_EXT_MBB_WCDMA_SWD3_DA1_MATRI<wbr>X@internal.nsn.com</a> >; Zhang,<br>
> > > > > > Bingxuan<br>
> > > > > > (Nokia<br>
> > > > > > -<br>
> > > > > > CN/Hangzhou) < <a href="mailto:bingxuan.zhang@nokia.com" target="_blank">bingxuan.zhang@nokia.com</a> >; Zizka, Jan (Nokia -<br>
> > > > > > CZ/Prague)<br>
> > > > > > < <a href="mailto:jan.zizka@nokia.com" target="_blank">jan.zizka@nokia.com</a> ><br>
> > > > > > Subject: Re: [Gluster-devel] Issue about the size of fstat is less<br>
> > > > > > than<br>
> > > > > > the<br>
> > > > > > really size of the syslog file<br>
> > > > > ><br>
> > > > > ><br>
> > > > > ><br>
> > > > > > ----- Original Message -----<br>
> > > > > > > From: "George Lian (Nokia - CN/Hangzhou)" < <a href="mailto:george.lian@nokia.com" target="_blank">george.lian@nokia.com</a><br>
> > > > > > > ><br>
> > > > > > > To: <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> > > > > > > Cc: "I_EXT_MBB_WCDMA_SWD3_DA1_MATR<wbr>IX_GMS"<br>
> > > > > > > < <a href="mailto:I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@internal.nsn.com" target="_blank">I_EXT_MBB_WCDMA_SWD3_DA1_MATRI<wbr>X@internal.nsn.com</a> >, "Bingxuan<br>
> > > > > > > Zhang<br>
> > > > > > > (Nokia<br>
> > > > > > > - CN/Hangzhou)" < <a href="mailto:bingxuan.zhang@nokia.com" target="_blank">bingxuan.zhang@nokia.com</a> >, "Jan Zizka (Nokia -<br>
> > > > > > > CZ/Prague)"<br>
> > > > > > > < <a href="mailto:jan.zizka@nokia.com" target="_blank">jan.zizka@nokia.com</a> ><br>
> > > > > > > Sent: Thursday, October 13, 2016 2:33:53 PM<br>
> > > > > > > Subject: [Gluster-devel] Issue about the size of fstat is less<br>
> > > > > > > than<br>
> > > > > > > the<br>
> > > > > > > really size of the syslog file<br>
> > > > > > ><br>
> > > > > > > Hi, Dear Expert,<br>
> > > > > > > We have use glusterfs as a network filesystem, and syslog store<br>
> > > > > > > in<br>
> > > > > > > there,<br>
> > > > > > > some clients on different host may write the syslog file via<br>
> > > > > > > “glusterfs”<br>
> > > > > > > mount point.<br>
> > > > > > > Now we encounter an issue when we “tail” the syslog file, it will<br>
> > > > > > > occasional<br>
> > > > > > > failed with error “ file truncated ”<br>
> > > > > > > As we study and trace with the “tail” source code, it failed with<br>
> > > > > > > the<br>
> > > > > > > following code:<br>
> > > > > > > if ( S_ISREG (mode) && stats.st_size < f[i].size )<br>
> > > > > > > {<br>
> > > > > > > error (0, 0, _("%s: file truncated"), quotef (name));<br>
> > > > > > > /* Assume the file was truncated to 0,<br>
> > > > > > > and therefore output all "new" data. */<br>
> > > > > > > xlseek (fd, 0, SEEK_SET, name);<br>
> > > > > > > f[i].size = 0;<br>
> > > > > > > }<br>
> > > > > > > When stats.st_size < f[i].size, what mean the size report by<br>
> > > > > > > fstat<br>
> > > > > > > is<br>
> > > > > > > less<br>
> > > > > > > than “tail” had read, it lead to “file truncated”, we also use<br>
> > > > > > > “strace”<br>
> > > > > > > tools to trace the tail application, the related tail strace log<br>
> > > > > > > as<br>
> > > > > > > the<br>
> > > > > > > below:<br>
> > > > > > > nanosleep({1, 0}, NULL) = 0<br>
> > > > > > > fstat(3, {st_mode=S_IFREG|0644, st_size=192543105, ...}) = 0<br>
> > > > > > > nanosleep({1, 0}, NULL) = 0<br>
> > > > > > > fstat(3, {st_mode=S_IFREG|0644, st_size=192543105, ...}) = 0<br>
> > > > > > > nanosleep({1, 0}, NULL) = 0<br>
> > > > > > > fstat(3, {st_mode=S_IFREG|0644, st_size=192543105, ...}) = 0<br>
> > > > > > > nanosleep({1, 0}, NULL) = 0<br>
> > > > > > > fstat(3, {st_mode=S_IFREG|0644, st_size=192544549, ...}) = 0<br>
> > > > > > > read(3, " Data … -"..., 8192) = 1444<br>
> > > > > > > read(3, " Data.. "..., 8192) = 720<br>
> > > > > > > read(3, "", 8192) = 0<br>
> > > > > > > fstat(3, {st_mode=S_IFREG|0644, st_size=192544789, ...}) = 0<br>
> > > > > > > write(1, “DATA…..” ) = 2164<br>
> > > > > > > write(2, "tail: ", 6tail: ) = 6<br>
> > > > > > > write(2, "/mnt/log/master/syslog: file tru"...,<br>
> > > > > > > 38/mnt/log/master/syslog:<br>
> > > > > > > file truncated) = 38<br>
> > > > > > > as the above strace log, tail has read 1444+720=2164 bytes,<br>
> > > > > > > but fstat tell “tail” 192544789 – 192543105 = 1664 which less<br>
> > > > > > > than<br>
> > > > > > > 2164,<br>
> > > > > > > so<br>
> > > > > > > it lead to “tail” application “file truncated”.<br>
> > > > > > > And if we turn off “write-behind” feature, the issue will not be<br>
> > > > > > > reproduced<br>
> > > > > > > any more.<br>
> > > > > ><br>
> > > > > > That seems strange. There are no writes happening on the fd/inode<br>
> > > > > > through<br>
> > > > > > which tail is reading/stating from. So, it seems strange that<br>
> > > > > > write-behind<br>
> > > > > > is involved here. I suspect whether any of<br>
> > > > > > md-cache/read-ahead/io-cache<br>
> > > > > > is<br>
> > > > > > causing the issue. Can you,<br>
> > > > > ><br>
> > > > > > 1. Turn off md-cache, read-ahead, io-cache xlators<br>
> > > > > > 2. mount glusterfs with --attribute-timeout=0<br>
> > > > > > 3. set write-behind on<br>
> > > > > ><br>
> > > > > > and rerun the tests? If you don't hit the issue, you can experiment<br>
> > > > > > by<br>
> > > > > > turning on/off of md-cache, read-ahead and io-cache translators and<br>
> > > > > > see<br>
> > > > > > what<br>
> > > > > > are the minimal number of xlators that need to be turned off to not<br>
> > > > > > hit<br>
> > > > > > the<br>
> > > > > > issue (with write-behind on)?<br>
> > > > > ><br>
> > > > > > regards,<br>
> > > > > > Raghavendra<br>
> > > > > ><br>
> > > > > > > So we think it may be related to cache consistence issue due to<br>
> > > > > > > performance<br>
> > > > > > > consider, but we still have concern that:<br>
> > > > > > > The syslog file is used only with “Append” mode, so the size of<br>
> > > > > > > file<br>
> > > > > > > shouldn’t be reduced, when a client read the file, why “fstat”<br>
> > > > > > > can’t<br>
> > > > > > > return<br>
> > > > > > > the really size match to the cache?<br>
> > > > > > > From current investigation, we doubt that the current implement<br>
> > > > > > > of<br>
> > > > > > > “glusterfs” has a bug on “fstat” when cache is on.<br>
> > > > > > > Your comments is our highly appreciated!<br>
> > > > > > > Thanks & Best Regards<br>
> > > > > > > George<br>
> > > > > > ><br>
> > > > > > > ______________________________<wbr>_________________<br>
> > > > > > > Gluster-devel mailing list<br>
> > > > > > > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> > > > > > > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-devel</a><br>
> > > > > ><br>
> > > > ><br>
> > > > ______________________________<wbr>_________________<br>
> > > > Gluster-devel mailing list<br>
> > > > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> > > > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-devel</a><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > --<br>
> > > ><br>
> > > ><br>
> > > > Pranith<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > --<br>
> > > ><br>
> > > ><br>
> > > > Pranith<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > --<br>
> > > > Pranith<br>
> > > ><br>
> > > > ______________________________<wbr>_________________<br>
> > > > Gluster-devel mailing list<br>
> > > > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> > > > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-devel</a><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > --<br>
> > > > Raghavendra G<br>
> > > ><br>
> > > > ______________________________<wbr>_________________<br>
> > > > Gluster-devel mailing list<br>
> > > > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> > > > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-devel</a><br>
> > > ______________________________<wbr>_________________<br>
> > > Gluster-devel mailing list<br>
> > > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> > > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-devel</a><br>
> > ><br>
> ><br>
><br>
______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-devel</a></div></div></blockquote></div></div></div><span class="gmail-HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="gmail-m_8097377878677447059gmail_signature">Raghavendra G<br></div>
</font></span></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Raghavendra G<br></div>
</div></div>