<div dir="ltr">Hi Pranith,<div><br></div><div>Will this patch improve the heal performance on distributed disperse volume?. Currently we are getting 10MB/s heal performance on 10G backed network. SHD daemon takes 5 days to complete the heal operation for single 4TB( 3.5 TB data) disk failure.</div><div><br></div><div>Regards,</div><div>Backer</div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Oct 14, 2015 at 9:08 PM, Ben Turner <span dir="ltr"><<a href="mailto:bturner@redhat.com" target="_blank">bturner@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">----- Original Message -----<br>
> From: "Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>><br>
> To: "Ben Turner" <<a href="mailto:bturner@redhat.com">bturner@redhat.com</a>>, "Humble Devassy Chirammal" <<a href="mailto:humble.devassy@gmail.com">humble.devassy@gmail.com</a>>, "Atin Mukherjee"<br>
> <<a href="mailto:atin.mukherjee83@gmail.com">atin.mukherjee83@gmail.com</a>><br>
> Cc: "gluster-users" <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
> Sent: Wednesday, October 14, 2015 1:39:14 AM<br>
> Subject: Re: [Gluster-users] Speed up heal performance<br>
><br>
><br>
><br>
> On 10/13/2015 07:11 PM, Ben Turner wrote:<br>
> > ----- Original Message -----<br>
> >> From: "Humble Devassy Chirammal" <<a href="mailto:humble.devassy@gmail.com">humble.devassy@gmail.com</a>><br>
> >> To: "Atin Mukherjee" <<a href="mailto:atin.mukherjee83@gmail.com">atin.mukherjee83@gmail.com</a>><br>
> >> Cc: "Ben Turner" <<a href="mailto:bturner@redhat.com">bturner@redhat.com</a>>, "gluster-users"<br>
> >> <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
> >> Sent: Tuesday, October 13, 2015 6:14:46 AM<br>
> >> Subject: Re: [Gluster-users] Speed up heal performance<br>
> >><br>
> >>> Good news is we already have a WIP patch <a href="http://review.glusterd.org/10851" rel="noreferrer" target="_blank">review.glusterd.org/10851</a> to<br>
> >> introduce multi threaded shd. Credits to Richard/Shreyas from facebook for<br>
> >> this. IIRC, we also have a BZ for the same<br>
> >> Isnt it the same bugzilla (<br>
> >> <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1221737" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1221737</a>) mentioned in the<br>
> >> commit log?<br>
> > @Lindsay - No need for a BZ, the above BZ should suffice.<br>
> ><br>
> > @Anyone - In the commit I see:<br>
> ><br>
> > { .key = "cluster.shd-max-threads",<br>
> > .voltype = "cluster/replicate",<br>
> > .option = "shd-max-threads",<br>
> > .op_version = 1,<br>
> > .flags = OPT_FLAG_CLIENT_OPT<br>
> > },<br>
> > { .key = "cluster.shd-thread-batch-size",<br>
> > .voltype = "cluster/replicate",<br>
> > .option = "shd-thread-batch-size",<br>
> > .op_version = 1,<br>
> > .flags = OPT_FLAG_CLIENT_OPT<br>
> > },<br>
> ><br>
> > So we can tune max threads and thread batch size? I understand max<br>
> > threads, but what is batch size? In my testing on 10G NICs with a backend<br>
> > that will service 10G throughput I see about 1.5 GB per minute of SH<br>
> > throughput. To Lindsay's other point, will this patch improve SH<br>
> > throughput? My systems can write at 1.5 GB / Sec and NICs can to 1.2 GB /<br>
> > sec but I only see ~1.5 GB per _minute_ of SH throughput. If we can not<br>
> > only make SH multi threaded, but improve the performance of a single<br>
> > thread that would be awesome. Super bonus points if we can have some sort<br>
> > of tunible that can limit the bandwidth each thread can consume. It would<br>
> > be great to be able to crank things up when the systems aren't busy and<br>
> > slow things down when load increases.<br>
> This patch is not merged because I thought we needed throttling feature<br>
> to go in before we can merge this for better control of the self-heal<br>
> speed. We are doing that for 3.8. So expect to see both of these for 3.8.<br>
<br>
</div></div>Great news! You da man Pranith, next time I am on your side of the world beers are on me :)<br>
<span class="HOEnZb"><font color="#888888"><br>
-b<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
><br>
> Pranith<br>
> ><br>
> > -b<br>
> ><br>
> ><br>
> >> --Humble<br>
> >><br>
> >><br>
> >> On Tue, Oct 13, 2015 at 7:26 AM, Atin Mukherjee<br>
> >> <<a href="mailto:atin.mukherjee83@gmail.com">atin.mukherjee83@gmail.com</a>><br>
> >> wrote:<br>
> >><br>
> >>> -Atin<br>
> >>> Sent from one plus one<br>
> >>> On Oct 13, 2015 3:16 AM, "Ben Turner" <<a href="mailto:bturner@redhat.com">bturner@redhat.com</a>> wrote:<br>
> >>>> ----- Original Message -----<br>
> >>>>> From: "Lindsay Mathieson" <<a href="mailto:lindsay.mathieson@gmail.com">lindsay.mathieson@gmail.com</a>><br>
> >>>>> To: "gluster-users" <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
> >>>>> Sent: Friday, October 9, 2015 9:18:11 AM<br>
> >>>>> Subject: [Gluster-users] Speed up heal performance<br>
> >>>>><br>
> >>>>> Is there any way to max out heal performance? My cluster is unused<br>
> >>> overnight,<br>
> >>>>> and lightly used at lunchtimes, it would be handy to speed up a heal.<br>
> >>>>><br>
> >>>>> The only tuneable I found was cluster.self-heal-window-size, which<br>
> >>> doesn't<br>
> >>>>> seem to make much difference.<br>
> >>>> I don't know of any way to speed this up, maybe someone else could chime<br>
> >>> in here that knows the heal daemon better than me. Maybe you could open<br>
> >>> an<br>
> >>> RFE on this? In my testing I only see 2 files getting healed at a time<br>
> >>> per<br>
> >>> replica pair. I would like to see this be multi threaded(if its not<br>
> >>> already) with the ability to tune it to control resource usage(similar to<br>
> >>> what we did in the rebalance refactoring done recently). If you let me<br>
> >>> know the BZ # I'll add my data + suggestions, I have been testing this<br>
> >>> pretty extensively in recent weeks and good data + some ideas on how to<br>
> >>> speed things up.<br>
> >>> Good news is we already have a WIP patch <a href="http://review.glusterd.org/10851" rel="noreferrer" target="_blank">review.glusterd.org/10851</a> to<br>
> >>> introduce multi threaded shd. Credits to Richard/Shreyas from facebook<br>
> >>> for<br>
> >>> this. IIRC, we also have a BZ for the same but the patch is in rfc as of<br>
> >>> now. AFAIK, this is a candidate to land in 3.8 as well, Vijay can correct<br>
> >>> me otherwise.<br>
> >>>> -b<br>
> >>>><br>
> >>>>> thanks,<br>
> >>>>> --<br>
> >>>>> Lindsay<br>
> >>>>><br>
> >>>>> _______________________________________________<br>
> >>>>> Gluster-users mailing list<br>
> >>>>> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> >>>>> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> >>>> _______________________________________________<br>
> >>>> Gluster-users mailing list<br>
> >>>> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> >>>> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> >>> _______________________________________________<br>
> >>> Gluster-users mailing list<br>
> >>> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> >>> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> >>><br>
> > _______________________________________________<br>
> > Gluster-users mailing list<br>
> > <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
><br>
><br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div><div class="gmail_signature"><br><br></div>
</div></div>