Efficiency Still Matters

One of the most common knocks against GlusterFS is that it eats too many CPU cycles. For example:

Really not liking GlusterFS now. Performance is quite poor and CPU usage way too high for what it does.

I’ve talked about performance issues and expectations many times. With regard to CPU usage, I can’t resist saying that most people who say “too high for what it does” don’t actually understand what it does, but Frank still has a good point. I’ll be the first to admit that the GlusterFS code is pretty inefficient. Over on my other blog I obliquely mention one example, which I discuss as a correctness issue bit which also affects performance. Every time you write a variable you actually don’t need, you don’t just use up a few CPU cycles. You might also generate more cache-coherency traffic or push something else out of cache/TLB. The effects one time are minor, but for code that executes millions of times a second these things add up. If an idiom that drives more memory access is repeated across thousands of functions, then it has a measurable effect on overall cache efficiency. You owe it to yourself, your colleagues, and your users to avoid sins like the following (all pet peeves in the GlusterFS code).

• Overuse of macros and inlines that hide their true cost from profilers etc. I particularly despise macro abuse, because macros are also not type-safe and are annoying to debug through.
• Too much locking and unlocking, often hidden (see above) and/or forced by bad calling conventions. Lock cycles are expensive.
• Too many memory-allocator trips per request for stack frames, translator-local structures, dictionaries, etc. Also, custom allocators that don’t implement thread affinity.
• Linear searches in dictionaries and fd/inode context lists.
• Repeating arguments through every stage of a deep translator chain, instead of using a request-block idiom that keeps arguments in one place.
• Calling functions/macros on every iteration of a loop instead of just once (compilers can catch these some of the time but not always).
• Functions that are only called from one place, but defined in a separate module to maximize linkage difficulty and cache effects.

As I said, these things add up. For some of these I can point to one specific place where an improvement could be made, but most of these issues don’t live in a particular piece of code. They’re aspects/idioms that appear everywhere, in every piece of code, not causing a distinct pause but just slowing down the “clock rate” at which every line executes. In addition to all of the other things I’m doing with specific code, one of my long-term goals on the project is to start changing habits away from these inefficient calling conventions and coding style to something a bit “tighter” as befits system code. Let’s see how that goes.