<br>I’ve been trying for weeks to reproduce the performance problems in our preproduction environments but can’t. As a result, selling that just upgrading to 3.6.x and hoping it goes away might be tricky. 3.6 is perceived as a little too bleeding edge, and we’ve actually had some other not fully explained issues with this cluster recently that make us hesitate. I don’t think they’re related.<div><br>On Tue, Feb 3, 2015 at 4:58 AM, Justin Clift <justin@gluster.org> wrote:<br>
<blockquote type="cite"><div class="plaintext" style="white-space: pre-wrap;">----- Original Message -----
<blockquote> Hello List,
So I've been frustraded by intermittent performance problems throughout
January. The problem occurs on a two node setup running 3.4.5, 16 gigs
of ram with a bunch of local disk. For sometimes an hour for sometimes
weeks at a time (I have extensive graphs in OpenNMS) our Gluster boxes
will get their CPUs pegged, and in vmstat they'll show extremely high
numbers of context switches and interrupts. Eventually things calm
down. During this time, memory usage actually drops. Overall usage on
the box goes from between 6-10 gigs to right around 4 gigs, and stays
there. That's what really puzzles me.
When performance is problematic, sar shows one device, the device
corresponding to the glusterfsd problem using all the CPU doing lots of
little reads, Sometimes 70k/second, very small avg rq size, say 10-12.
Afraid I don't have any saved output handy, but I can try to capture
some next time it happens. I have tons of information frankly, but am
trying to keep this reasonably brief.
There are more than a dozen volumes on this two node setup. The CPU
usage is pretty much entirely contained to one volume, a 1.5 TB volume
that is just shy of 70% full. It stores uploaded files for a web app.
What I hate about this app and so am always suspicious of, is that it
stores a directory for every user in one level, so under the /data
directory in the volume, there are 450,000 sub directories at this
point.
The only real mitigation step that's been taken so far was to turn off
the self-heal daemon on the volume, as I thought maybe crawling that
large directory was getting expensive. This doesn't seem to have done
anything as the problem still occurs.
At this point I figure there are one of two things sorts of things
happening really broadly: one we're running into some sort of bug or
performance problem with gluster we should either fix perhaps by
upgrading or tuning around, or two, some process we're running but not
aware of is hammering the file system causing problems.
If it's the latter option, can anyone give me any tips on figuring out
what might be hammering the system? I can use volume top to see what a
brick is doing, but I can't figure out how to tell what clients are
doing what.
Apologies for the somewhat broad nature of the question, any input
thoughts would be much appreciated. I can certainly provide more info
about some things if it would help, but I've tried not to write a novel
here.
</blockquote>
Out of curiosity, are you able to test using GlusterFS 3.6.2? We've
had a bunch of pretty in-depth upstream testing at decent scale (100+
nodes) from 3.5.x onwards, with lots of performance issues identified
and fixed on the way through.
So, I'm kinda hopeful the problem you're describing is fixed in newer
releases. :D
Regards and best wishes,
Justin Clift
<div>--
</div>GlusterFS - <a href="http://www.gluster.org">http://www.gluster.org</a>
An open source, distributed file system scaling to several
petabytes, and handling thousands of clients.
My personal twitter: twitter.com/realjustinclift
</div></blockquote></div>