<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 27, 2016 at 2:38 PM, Manoj Pillai <span dir="ltr"><<a href="mailto:mpillai@redhat.com" target="_blank">mpillai@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><br>
<br>
----- Original Message -----<br>
> From: "Raghavendra Gowdappa" <<a href="mailto:rgowdapp@redhat.com">rgowdapp@redhat.com</a>><br>
> To: "Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>><br>
> Cc: "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a>><br>
</span><div><div class="h5">> Sent: Monday, June 27, 2016 12:48:49 PM<br>
> Subject: Re: [Gluster-devel] performance issues Manoj found in EC testing<br>
><br>
><br>
><br>
> ----- Original Message -----<br>
> > From: "Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>><br>
> > To: "Xavier Hernandez" <<a href="mailto:xhernandez@datalab.es">xhernandez@datalab.es</a>><br>
> > Cc: "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a>><br>
> > Sent: Monday, June 27, 2016 12:42:35 PM<br>
> > Subject: Re: [Gluster-devel] performance issues Manoj found in EC testing<br>
> ><br>
> ><br>
> ><br>
> > On Mon, Jun 27, 2016 at 11:52 AM, Xavier Hernandez < <a href="mailto:xhernandez@datalab.es">xhernandez@datalab.es</a><br>
> > ><br>
> > wrote:<br>
> ><br>
> ><br>
> > Hi Manoj,<br>
> ><br>
> > I always enable client-io-threads option for disperse volumes. It improves<br>
> > performance sensibly, most probably because of the problem you have<br>
> > detected.<br>
> ><br>
> > I don't see any other way to solve that problem.<br>
> ><br>
> > I agree. Updated the bug with same info.<br>
> ><br>
> ><br>
> ><br>
> > I think it would be a lot better to have a true thread pool (and maybe an<br>
> > I/O<br>
> > thread pool shared by fuse, client and server xlators) in libglusterfs<br>
> > instead of the io-threads xlator. This would allow each xlator to decide<br>
> > when and what should be parallelized in a more intelligent way, since<br>
> > basing<br>
> > the decision solely on the fop type seems too simplistic to me.<br>
> ><br>
> > In the specific case of EC, there are a lot of operations to perform for a<br>
> > single high level fop, and not all of them require the same priority. Also<br>
> > some of them could be executed in parallel instead of sequentially.<br>
> ><br>
> > I think it is high time we actually schedule(for which release) to get this<br>
> > in gluster. May be you should send out a doc where we can work out details?<br>
> > I will be happy to explore options to integrate io-threads, syncop/barrier<br>
> > with this infra based on the design may be.<br>
><br>
> +1. I can volunteer too.<br>
<br>
</div></div>Thanks, folks! As a quick update, throughput on a single client test jumped<br>
from ~180 MB/s to 700+MB/s after enabling client-io-threads. Throughput is<br>
now more in line with what is expected for this workload based on<br>
back-of-the-envelope calculations.<br>
<br>
Are there any reservations about recommending client-io-threads=on as<br>
"default" tuning, until the enhancement discussed above becomes reality?<br></blockquote><div><br></div><div>The only thing I can think of is possible races we may have to address after enabling this option. So I would let it bake on master for a while with this as default may be?<br><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span class="HOEnZb"><font color="#888888"><br>
-- Manoj<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
><br>
> ><br>
> ><br>
> ><br>
> > Xavi<br>
> ><br>
> ><br>
> > On 25/06/16 19:42, Manoj Pillai wrote:<br>
> ><br>
> ><br>
> ><br>
> > ----- Original Message -----<br>
> ><br>
> ><br>
> > From: "Pranith Kumar Karampuri" < <a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a> ><br>
> > To: "Xavier Hernandez" < <a href="mailto:xhernandez@datalab.es">xhernandez@datalab.es</a> ><br>
> > Cc: "Manoj Pillai" < <a href="mailto:mpillai@redhat.com">mpillai@redhat.com</a> >, "Gluster Devel" <<br>
> > <a href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a> ><br>
> > Sent: Thursday, June 23, 2016 8:50:44 PM<br>
> > Subject: performance issues Manoj found in EC testing<br>
> ><br>
> > hi Xavi,<br>
> > Meet Manoj from performance team Redhat. He has been testing EC<br>
> > performance in his stretch clusters. He found some interesting things we<br>
> > would like to share with you.<br>
> ><br>
> > 1) When we perform multiple streams of big file writes(12 parallel dds I<br>
> > think) he found one thread to be always hot (99%CPU always). He was asking<br>
> > me if fuse_reader thread does any extra processing in EC compared to<br>
> > replicate. Initially I thought it would just lock and epoll threads will<br>
> > perform the encoding but later realized that once we have the lock and<br>
> > version details, next writes on the file would be encoded in the same<br>
> > thread that comes to EC. write-behind could play a role and make the writes<br>
> > come to EC in an epoll thread but we saw consistently there was just one<br>
> > thread that is hot. Not multiple threads. We will be able to confirm this<br>
> > in tomorrow's testing.<br>
> ><br>
> > 2) This is one more thing Raghavendra G found, that our current<br>
> > implementation of epoll doesn't let other epoll threads pick messages from<br>
> > a socket while one thread is processing one message from that socket. In<br>
> > EC's case that can be encoding of the write/decoding read. This will not<br>
> > let replies of operations on different files to be processed in parallel.<br>
> > He thinks this can be fixed for 3.9.<br>
> ><br>
> > Manoj will be raising a bug to gather all his findings. I just wanted to<br>
> > introduce him and let you know the interesting things he is finding before<br>
> > you see the bug :-).<br>
> > --<br>
> > Pranith<br>
> ><br>
> > Thanks, Pranith :).<br>
> ><br>
> > Here's the bug: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1349953" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1349953</a><br>
> ><br>
> > Comparing EC and replica-2 runs, the hot thread is seen in both cases, so<br>
> > I have not opened this as an EC bug. But initial impression is that<br>
> > performance impact for EC is particularly bad (details in the bug).<br>
> ><br>
> > -- Manoj<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Pranith<br>
> ><br>
> > _______________________________________________<br>
> > Gluster-devel mailing list<br>
> > <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
> _______________________________________________<br>
> Gluster-devel mailing list<br>
> <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>