<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from rtf -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<font face="Calibri" size="2"><span style="font-size:10.5pt;">
<div align="left" style="text-align:justify;">Hi, Dear Expert,</div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;">We have use glusterfs as a network filesystem, and syslog store in there, some clients on different host may write the syslog file via “glusterfs” mount point.</div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;">Now we encounter an issue when we “tail” the syslog file, it will occasional failed with error “<font color="red">file truncated</font>”</div>
<div align="left" style="text-align:justify;">As we study and trace with the “tail” source code, it failed with the following code:</div>
<div align="left" style="text-indent:84pt;text-align:justify;"><span style="background-color:silver;">if (</span><span style="background-color:yellow;">S_ISREG (mode) && stats.st_size < f[i].size</span><span style="background-color:silver;">)</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:silver;"> {</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:silver;"> error (0, 0, _("%s: file truncated"), quotef (name));</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:silver;"> /* Assume the file was truncated to 0,</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:silver;"> and therefore output all "new" data. */</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:silver;"> xlseek (fd, 0, SEEK_SET, name);</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:silver;"> f[i].size = 0;</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:silver;"> }</span></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;">When stats.st_size < f[i].size, what mean the size report by fstat is less than “tail” had read, it lead to “file truncated”, we also use “strace” tools to trace the tail application, the related tail strace log
as the below:</div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;"><span style="background-color:yellow;">nanosleep({1, 0}, NULL) = 0</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:yellow;">fstat(3, {st_mode=S_IFREG|0644, st_size=192543105, ...}) = 0</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:yellow;">nanosleep({1, 0}, NULL) = 0</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:yellow;">fstat(3, {st_mode=S_IFREG|0644, st_size=192543105, ...}) = 0</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:yellow;">nanosleep({1, 0}, NULL) = 0</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:yellow;">fstat(3, {st_mode=S_IFREG|0644, st_size=192543105, ...}) = 0</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:yellow;">nanosleep({1, 0}, NULL) = 0</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:yellow;">fstat(3, {st_mode=S_IFREG|0644, st_size=192544549, ...}) = 0</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:yellow;">read(3, "</span><span style="background-color:yellow;">Data …</span><span style="background-color:yellow;">-"..., 8192) = 1444</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:yellow;">read(3, "</span><span style="background-color:yellow;">Data..</span><span style="background-color:yellow;">"..., 8192) = 720</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:yellow;">read(3, "", 8192) = 0</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:yellow;">fstat(3, {st_mode=S_IFREG|0644, st_size=192544789, ...}) = 0</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:yellow;">write(1, </span><span style="background-color:yellow;">“DATA…..”</span><span style="background-color:yellow;">) = 2164</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:yellow;">write(2, "tail: ", 6tail: ) = 6</span></div>
<div align="left" style="text-align:justify;"><span style="background-color:yellow;">write(2, "/mnt/log/master/syslog: file tru"..., 38/mnt/log/master/syslog: file truncated) = 38</span></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;">as the above strace log, tail has read 1444+720=2164 bytes,</div>
<div align="left" style="text-align:justify;">but fstat tell “tail” <span style="background-color:yellow;">192544789</span> – <span style="background-color:yellow;">192543105</span> = 1664 which less than 2164, so it lead to “tail” application “file truncated”.</div>
<div align="left" style="text-align:justify;"> </div>
<div align="left" style="text-align:justify;">And if we turn off “write-behind” feature, the issue will not be reproduced any more.</div>
<div align="left" style="text-align:justify;">So we think it may be related to cache consistence issue due to performance consider, but we still have concern that:</div>
<div align="left" style="text-align:justify;"> </div>
<div align="left" style="text-align:justify;">The syslog file is used only with “Append” mode, so the size of file shouldn’t be reduced, when a client read the file, why “fstat” can’t return the really size match to the cache?</div>
<div align="left" style="text-align:justify;">From current investigation, we doubt that the current implement of “glusterfs” has a bug on “fstat” when cache is on.</div>
<div align="left" style="text-align:justify;"> </div>
<div align="left" style="text-align:justify;">Your comments is our highly appreciated!</div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;">Thanks & Best Regards</div>
<div align="left" style="text-align:justify;"> </div>
<div align="left" style="text-align:justify;">George</div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
<div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div>
</span></font>
</body>
</html>