<html><body><div style="font-family: garamond,new york,times,serif; font-size: 12pt; color: #000000"><div><br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Lindsay Mathieson" <lindsay.mathieson@gmail.com><br><b>To: </b>"Krutika Dhananjay" <kdhananj@redhat.com><br><b>Cc: </b>"gluster-users" <gluster-users@gluster.org><br><b>Sent: </b>Tuesday, October 27, 2015 5:17:09 PM<br><b>Subject: </b>Re: [Gluster-users] Shard Volume testing (3.7.5)<br><div><br></div><div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On 26 October 2015 at 14:54, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank" data-mce-href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;"><div><br></div><div>Hi Lindsay,<br></div><div><br></div><div>Thank you for trying out sharding and for your feedback. :) Please find my comments inline.<br></div></blockquote><div><br></div><div>Hi Krutika, thanks for the feed back.<br><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;"><div>With block size as low as 4MB, to the replicate module, these individual shards appear as large number of small(er) files, effectively turning it into some form of a small-file workload.<br></div><div>There is an enhancement being worked on in AFR by Pranith, which attempts to improve write performance which will especially be useful when used with sharding. That should make this problem go away.<br></div></blockquote><div><br><div><br></div></div><div>Cool, also for my purposes (VM Image hosting), block sizes of 512MB are just as good and improve things considerably.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;"><div><br></div><div><br></div><blockquote style="border-left:2px solid rgb(16,16,255);margin-left:5px;padding-left:5px;color:rgb(0,0,0);font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt" data-mce-style="border-left: 2px solid #1010ff; margin-left: 5px; padding-left: 5px; color: #000000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><div dir="ltr"><div><div><div><div><div><div><div><br></div><div>One Bug:<br></div>After heals completed I shut down the VM's and run a MD5SUM on the VM image (via glusterfs) on each nodes. They all matched except for one time on gn3. Once I unmounted/remounted the datastore on gn3 the md5sum matched.</div></div></div></div></div></div></div></blockquote><div><br></div><div>This could possibly be the effect of a caching bug reported at <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1272986" target="_blank" data-mce-href="https://bugzilla.redhat.com/show_bug.cgi?id=1272986">https://bugzilla.redhat.com/show_bug.cgi?id=1272986</a>. The fix is out for review and I'm confident that it will make it into 3.7.6.<br></div></blockquote><div><br></div><div>Cool, I can replicate it fairly reliable at the moment.<br><div><br></div></div><div>Would it occur when using qemu/gfapi direct?<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;"><div><br></div><div><br></div><blockquote style="border-left:2px solid rgb(16,16,255);margin-left:5px;padding-left:5px;color:rgb(0,0,0);font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt" data-mce-style="border-left: 2px solid #1010ff; margin-left: 5px; padding-left: 5px; color: #000000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><div dir="ltr"><div><div><div><div><div><div><br><div><br></div></div>One Oddity:<br></div>gluster volume heals datastore info *always* shows a split brain on the directory, but it always heals without intervention. Dunno if this is normal on not.</div></div></div></div></div></blockquote><div><br></div><div>Which directory would this be?</div></blockquote><div><br></div><div>Oddly it was the .shard directory<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;"><div>Do you have the glustershd logs?<br></div></blockquote><div><br></div><div>Sorry no, and I haven't managed to replicate it again. Will keep trying.<br><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;"><div>Here is some documentation on sharding: <a>https://gluster.readthedocs.org/en/release-3.7.0/Features/shard/</a><a>. </a>Let me know if you have more questions, and I will be happy to answer them.<br></div><div>The problems we foresaw with too many 4MB shards is that</div><div>i. entry self-heal under /.shard could result in complete crawl of the /.shard directory during heal, or</div><div>ii. a disk replacement could involve lot many files needing to be created and healed to the sink brick,<br></div><div>both of which would result in slower "entry" heal and rather high resource consumption from self-heal daemon.<br></div></blockquote><div><br></div><div>Thanks, most interesting reading.<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;"><div><br></div><div>Fortunately, with the introduction of more granular changelogs in replicate module to identify exactly what files under a given directory need to be healed to the sink brick, these problems should go away.<br></div><div>In fact this enhancement is being worked upon as we speak and is targeted to be out by 3.8. Here is some doc: <a href="http://review.gluster.org/#/c/12257/1/in_progress/afr-self-heal-improvements.md" target="_blank" data-mce-href="http://review.gluster.org/#/c/12257/1/in_progress/afr-self-heal-improvements.md">http://review.gluster.org/#/c/12257/1/in_progress/afr-self-heal-improvements.md</a> (read section "Granular entry self-heals").<br></div></blockquote><div><br></div><div>That look very interesting - in fact from my point of view, it replaces the need for sharding altogether, that being the speed of heals. </div></div></div></div></blockquote><div><br></div><div>So sharding also helps with better disk utilization in distributed-replicated volumes for large files (like VM images).<br></div><div>So if you have a 2x3 volume with each brick having 10G space (say), even though the aggregated size of the volume (due to the presence of distribute) is 20G, without sharding you cannot create an image whose size is between 11G-20G on the volume.<br></div><div>With sharding, breaking large files into smaller pieces will ensure better utilisation of available space.<br></div><div>There are other long-term benefits one could reap from using sharding: for instance, for someone who might want to use tiering in VM store use-case, having sharding will be beneficial in terms of only migrating the shards between hot and cold tiers, as opposed to moving large files in full, even if only a small portion of the file is changed/accessed. :)<br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;"><div><br></div><div>Yes. So Paul Cuzner and Satheesaran who have been testing sharding here have reported better write performance with 512M shards. I'd be interested to know what you feel about performance with relatively larger shards (think 512M).</div></blockquote></div><br></div><div class="gmail_extra">Seq Read speeds basically tripled, and seq writes improved to the limit of the network connection.</div></div></blockquote><div><br></div><div>OK. And what about the data heal performance with 512M shards? Satisfactory?<br></div><div><br></div><div>-Krutika<br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><div dir="ltr"><div class="gmail_extra"><br><div><br></div></div><div class="gmail_extra">Cheers,<br></div><div class="gmail_extra"><br><br clear="all"><br>-- <br><div class="gmail_signature">Lindsay</div></div></div></blockquote><div><br></div></div></body></html>