<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 27, 2016 at 12:42 PM, Pranith Kumar Karampuri <span dir="ltr"><<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Mon, Jun 27, 2016 at 11:52 AM, Xavier Hernandez <span dir="ltr"><<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Manoj,<br>
<br>
I always enable client-io-threads option for disperse volumes. It improves performance sensibly, most probably because of the problem you have detected.<br>
<br>
I don't see any other way to solve that problem.<br></blockquote><div><br></div></span><div>I agree. Updated the bug with same info.<br></div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I think it would be a lot better to have a true thread pool (and maybe an I/O thread pool shared by fuse, client and server xlators) in libglusterfs instead of the io-threads xlator. This would allow each xlator to decide when and what should be parallelized in a more intelligent way, since basing the decision solely on the fop type seems too simplistic to me.<br>
<br>
In the specific case of EC, there are a lot of operations to perform for a single high level fop, and not all of them require the same priority. Also some of them could be executed in parallel instead of sequentially.<br></blockquote><div><br></div></span><div>I think it is high time we actually schedule(for which release) to get this in gluster. May be you should send out a doc where we can work out details? I will be happy to explore options to integrate io-threads, syncop/barrier with this infra based on the design may be.<br></div></div></div></div></blockquote><div><br></div><div>I was just thinking why we can't reuse synctask framework. It already scales up/down based on the tasks. At max it uses 16 threads. Whatever we want to be executed in parallel we can create a synctask around it and run it. Would that be good enough?<br> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Xavi<div><div><br>
<br>
On 25/06/16 19:42, Manoj Pillai wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
----- Original Message -----<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
From: "Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>><br>
To: "Xavier Hernandez" <<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><br>
Cc: "Manoj Pillai" <<a href="mailto:mpillai@redhat.com" target="_blank">mpillai@redhat.com</a>>, "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a>><br>
Sent: Thursday, June 23, 2016 8:50:44 PM<br>
Subject: performance issues Manoj found in EC testing<br>
<br>
hi Xavi,<br>
Meet Manoj from performance team Redhat. He has been testing EC<br>
performance in his stretch clusters. He found some interesting things we<br>
would like to share with you.<br>
<br>
1) When we perform multiple streams of big file writes(12 parallel dds I<br>
think) he found one thread to be always hot (99%CPU always). He was asking<br>
me if fuse_reader thread does any extra processing in EC compared to<br>
replicate. Initially I thought it would just lock and epoll threads will<br>
perform the encoding but later realized that once we have the lock and<br>
version details, next writes on the file would be encoded in the same<br>
thread that comes to EC. write-behind could play a role and make the writes<br>
come to EC in an epoll thread but we saw consistently there was just one<br>
thread that is hot. Not multiple threads. We will be able to confirm this<br>
in tomorrow's testing.<br>
<br>
2) This is one more thing Raghavendra G found, that our current<br>
implementation of epoll doesn't let other epoll threads pick messages from<br>
a socket while one thread is processing one message from that socket. In<br>
EC's case that can be encoding of the write/decoding read. This will not<br>
let replies of operations on different files to be processed in parallel.<br>
He thinks this can be fixed for 3.9.<br>
<br>
Manoj will be raising a bug to gather all his findings. I just wanted to<br>
introduce him and let you know the interesting things he is finding before<br>
you see the bug :-).<br>
--<br>
Pranith<br>
</blockquote>
<br>
Thanks, Pranith :).<br>
<br>
Here's the bug: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1349953" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1349953</a><br>
<br>
Comparing EC and replica-2 runs, the hot thread is seen in both cases, so<br>
I have not opened this as an EC bug. But initial impression is that<br>
performance impact for EC is particularly bad (details in the bug).<br>
<br>
-- Manoj<br>
<br>
</blockquote>
</div></div></blockquote></div></div></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</font></span></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>