<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 09/02/2015 12:45 PM, Raghavendra
Bhat wrote:<br>
</div>
<blockquote cite="mid:55E6A219.6010306@redhat.com" type="cite">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
<div class="moz-cite-prefix"><br>
Hi Christian,<br>
<br>
I have been working on it since couple of days. I have not been
able to recreate the issue. I will continue to recreate and get
back to you in a day or two.<br>
<br>
Regards,<br>
Raghavendra Bhat<br>
<br>
</div>
</blockquote>
<br>
Hi Christian,<br>
<br>
As per our tests (me and Raghavendra G in CC) we found that the data
was being served from cache. In fact in our tests data was being
served from the kernel cache itself. So we tried with dropping the
data cache (i.e. echo 1 > /proc/sys/vm/drop_caches) to see if
the read requests coming to glusterfs (since the data cache in the
kernel is dropped) . We found that the read calls were coming to
glusterfs and glusterfs is serving the requests from the cache (i.e.
io-cache xlator). If the memory pressure on the system is huge and
kernel is sending forgets to the glusterfs client, then there is a
possibility that the inodes are forgotten along with the data cached
within them.<br>
<br>
Can you please enable trace log level for the client and run your
tests? (gluster volume set <volname> client-log-level TRACE)
Once your tests are done please give the logs.<br>
<br>
NOTE: Enabling trace log level will increase the log file size
faster due to more logging.<br>
<br>
Regards,<br>
Raghavendra Bhat <br>
<blockquote cite="mid:55E6A219.6010306@redhat.com" type="cite">
<div class="moz-cite-prefix"> <br>
On 09/02/2015 12:45 AM, Christian Rice wrote:<br>
</div>
<blockquote cite="mid:D20B470D.116F5B%25crice@pandora.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<div>This is still an issue for me, I don’t need anyone to tear
the code apart, but I’d be grateful if someone would even
chime in and say “yeah, we’ve seen that too."</div>
<div><br>
</div>
</blockquote>
<blockquote cite="mid:D20B470D.116F5B%25crice@pandora.com"
type="cite">
<div> </div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt;
text-align:left; color:black; BORDER-BOTTOM: medium none;
BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT:
0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid;
BORDER-RIGHT: medium none; PADDING-TOP: 3pt"> <span
style="font-weight:bold">From: </span>Christian Rice <<a
moz-do-not-send="true" href="mailto:crice@pandora.com">crice@pandora.com</a>><br>
<span style="font-weight:bold">Date: </span>Sunday, August
30, 2015 at 11:18 PM<br>
<span style="font-weight:bold">To: </span>"<a
moz-do-not-send="true"
href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>"
<<a moz-do-not-send="true"
href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
<span style="font-weight:bold">Subject: </span>[Gluster-users]
Gluster 3.6.3 performance.cache-size not working as expected
in some cases<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;">
<div style="color: rgb(0, 0, 0); font-family: Calibri,
sans-serif; font-size: 14px;"> I am confused about my
caching problem. I’ll try to keep this as
straightforward as possible and include the basic
details...</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri,
sans-serif; font-size: 14px;"> <br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri,
sans-serif; font-size: 14px;"> I have a sixteen node
distributed volume, one brick per node, XFS isize=512,
Debian 7/Wheezy, 32GB RAM minimally. Every brick node
is also a gluster client, and also importantly an HTTP
server. We use a back-end 1GbE network for gluster
traffic (eth1). There are a couple dozen gluster
client-only systems accessing this volume, as well.</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri,
sans-serif; font-size: 14px;"> <br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri,
sans-serif; font-size: 14px;"> We had a really hot spot
on one brick due to an oft-requested file, and every
time any httpd process on any gluster client was asked
to deliver the file, it was physically fetching it (we
could see this traffic using, say, ‘iftop -i eth1’,) so
we thought to increase the volume cache timeout and
cache size. We set the following values for testing:</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri,
sans-serif; font-size: 14px;"> <br>
</div>
<div><font face="Calibri,sans-serif">performance.cache-size
16GB</font></div>
<div><font face="Calibri,sans-serif">performance.cache-refresh-timeout:
30</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div>This test was run from a node that didn’t have the
requested file on the local brick:</div>
<div><br>
</div>
<div>while(true); do cat /path/to/file > /dev/null;
done</div>
<div><br>
</div>
<div>and what had been very high traffic on the gluster
backend network, delivering the data repeatedly to my
requesting node, dropped to nothing visible.</div>
<div><br>
</div>
<div>I thought good, problem fixed. Caching works. My
colleague had run a test early on to show this perf
issue, so he ran it again to sign off.</div>
<div><br>
</div>
<div>His testing used curl, because all the real front end
traffic is HTTP, and all the gluster nodes are web
servers, which are of course using the fuse mount to
access the document root. Even with our performance
tuning, the traffic on the gluster backend subnet was
continuous and undiminished. I saw no evidence of cache
(again using ‘iftop -i eth1’, which showed a steady 75+%
of line rate on a 1GbE link.</div>
<div><br>
</div>
<div>Does that make sense at all? We had theorized that
we wouldn’t get to use VFS/kernel page cache on any node
except maybe the one which held the data in the local
brick. That’s what drove us to setting the gluster
performance cache. But it doesn’t seem to come into
play with http access.</div>
<div><br>
</div>
<div><br>
</div>
<div>Volume info:</div>
<div>
<div>Volume Name: DOCROOT</div>
<div>Type: Distribute</div>
<div>Volume ID: 3aecd277-4d26-44cd-879d-cffbb1fec6ba</div>
<div>Status: Started</div>
<div>Number of Bricks: 16</div>
<div>Transport-type: tcp</div>
<div>Bricks:</div>
<div><snipped list of bricks></div>
<div>Options Reconfigured:</div>
<div>performance.cache-refresh-timeout: 30</div>
<div>performance.cache-size: 16GB</div>
</div>
<div><br>
</div>
<div>The net result of being overwhelmed by a hot spot is
all the gluster client nodes lose access to the gluster
volume—it becomes so busy it hangs. When the traffic
goes away (failing health checks by load balancers
causes requests to be redirected elsewhere), the volume
eventually unfreezes and life goes on.</div>
<div><br>
</div>
<div>I wish I could type ALL that into a google query and
get a lucid answer :)</div>
<div><br>
</div>
<div>Regards,</div>
<div>Christian</div>
</div>
</div>
</span> <br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>