<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Joe,</div><div class=""><br class=""></div><div class="">Thanks for that, that was educational. Gluster docs claim that since 3.7, DHT hash ranges are weighted based on brick sizes by default:</div><div class=""><br class=""></div><div class=""><div class=""><font face="Courier New" class="">$ gluster volume get <vol cluster.weighted-rebalance</font></div><div class=""><font face="Courier New" class="">Option Value </font></div><div class=""><font face="Courier New" class="">------ ----- </font></div><div class=""><font face="Courier New" class="">cluster.weighted-rebalance on </font></div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">When running rebalance with force, I see this in the rebalance log:</div><div class=""><br class=""></div><div class=""><div class=""><font face="Courier New" class="">...</font></div><div class=""><font face="Courier New" class="">[2016-10-11 16:38:37.655144] I [MSGID: 109045] [dht-selfheal.c:1751:dht_fix_layout_of_directory] 0-cronut-dht: subvolume 10 (cronut-replicate-10): <b class="">5721127</b> chunks</font></div><div class=""><font face="Courier New" class="">[2016-10-11 16:38:37.655154] I [MSGID: 109045] [dht-selfheal.c:1751:dht_fix_layout_of_directory] 0-cronut-dht: subvolume 11 (cronut-replicate-11): <b class="">7628846</b> chunks</font></div></div><div class=""><font face="Courier New" class="">…</font></div><div class=""><br class=""></div><div class="">subvolume >=11 are 8TB, subvolume <= 10 is are 6TB.</div><div class=""><br class=""></div><div class="">Do you think it is possible to even out usage on all bricks by % utilized now? This would be the case if gluster rebalanced simply by what the scaled DHT says, including all required data migrations?</div><div class=""><br class=""></div><div class="">It would be preferable for us to avoid having to depend on cluster.min-free-disk to manage overflow later on - as this introduces one extra read of the link followed by the actual IOP.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Jackie</div><br class=""><div><blockquote type="cite" class=""><div class="">On Oct 10, 2016, at 11:13 AM, Joe Julian <<a href="mailto:joe@julianfamily.org" class="">joe@julianfamily.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta content="text/html; charset=windows-1252" http-equiv="Content-Type" class="">
<div bgcolor="#FFFFFF" text="#000000" class=""><p class="">I've written an example of how gluster's dht works on my blog at
<a class="moz-txt-link-freetext" href="https://joejulian.name/blog/dht-misses-are-expensive/">https://joejulian.name/blog/dht-misses-are-expensive/</a> which might
make it clear why the end result is not what you expected.</p><p class="">By setting cluster.min-free-disk (defaults to 10%) you can, at
least, ensure that your new bricks are utilized as needed to
prevent over filling your smaller bricks.<br class="">
</p>
<div class="moz-cite-prefix">On 10/10/2016 10:13 AM, Jackie Tung
wrote:<br class="">
</div>
<blockquote cite="mid:9AA43039-99E9-42FE-A7A2-72422AD3F6B6@drive.ai" type="cite" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
Hi,
<div class=""><br class="">
</div>
<div class="">We have a 2 node, distributed replicated setup (11
bricks on each node). Each of these bricks are 6TB in size.</div>
<div class=""><br class="">
</div>
<div class="">node_A:/brick1 replicates node_B:/brick1</div>
<div class="">node_A:/brick2 replicates node_B:/brick2</div>
<div class="">node_A:/brick3 replicates node_B:/brick3</div>
<div class="">…</div>
<div class="">…</div>
<div class="">node_A:/brick11 replicates node_B:/brick11</div>
<div class=""><br class="">
</div>
<div class="">We recently added 5 more bricks to make it 16 bricks
on each node in total. Each of these new bricks are 8TB in
size.</div>
<div class=""><br class="">
</div>
<div class="">We completed a full rebalance operation (status says
“completed”).</div>
<div class=""><br class="">
</div>
<div class="">However the end result is somewhat unexpected:</div>
<div class="">
<div class=""><b class=""><font class="" face="Courier New">/dev/sdl1
7.3T 2.2T 5.2T 29%</font></b></div>
<div class=""><b class=""><font class="" face="Courier New">/dev/sdk1
7.3T 2.0T 5.3T 28%</font></b></div>
<div class=""><b class=""><font class="" face="Courier New">/dev/sdj1
7.3T 2.0T 5.3T 28%</font></b></div>
<div class=""><b class=""><font class="" face="Courier New">/dev/sdn1
7.3T 2.2T 5.2T 30%</font></b></div>
<div class=""><b class=""><font class="" face="Courier New">/dev/sdp1
7.3T 2.2T 5.2T 30%</font></b></div>
<div class=""><font class="" face="Courier New">/dev/sdc1 5.5T
2.3T 3.2T 42%</font></div>
<div class=""><font class="" face="Courier New">/dev/sdf1 5.5T
2.3T 3.2T 43%</font></div>
<div class=""><font class="" face="Courier New">/dev/sdo1 5.5T
2.3T 3.2T 42%</font></div>
<div class=""><font class="" face="Courier New">/dev/sda1 5.5T
2.3T 3.2T 43%</font></div>
<div class=""><font class="" face="Courier New">/dev/sdi1 5.5T
2.3T 3.2T 42%</font></div>
<div class=""><font class="" face="Courier New">/dev/sdh1 5.5T
2.3T 3.2T 43%</font></div>
<div class=""><font class="" face="Courier New">/dev/sde1 5.5T
2.3T 3.2T 42%</font></div>
<div class=""><font class="" face="Courier New">/dev/sdb1 5.5T
2.3T 3.2T 42%</font></div>
<div class=""><font class="" face="Courier New">/dev/sdm1 5.5T
2.3T 3.2T 42%</font></div>
<div class=""><font class="" face="Courier New">/dev/sdg1 5.5T
2.3T 3.2T 42%</font></div>
<div class=""><font class="" face="Courier New">/dev/sdd1 5.5T
2.3T 3.2T 42%</font></div>
</div>
<div class=""><br class="">
</div>
<div class="">The df output in <b class="">bold</b> are the new
8TB drives.</div>
<div class="">Was I wrong to expect the % usage to be roughly
equal? Is there some parameter I need to tweak to make
rebalance account for disk sizes properly?</div>
<div class=""><br class="">
</div>
<div class="">I’m using Gluster 3.8 on Ubuntu.</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Jackie</div>
<br class=""><p class=""><font color="#808080" face="Courier New" size="1" class="">The
information in this email is confidential and may be legally
privileged. It is intended solely for the addressee. Access to
this email by anyone else is unauthorized. If you are not the
intended recipient, any disclosure, copying, distribution or
any action taken or omitted to be taken in reliance on it, is
prohibited and may be unlawful.</font></p>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre wrap="" class="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br class="">
</div>
_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">http://www.gluster.org/mailman/listinfo/gluster-users</div></blockquote></div><br class=""></body></html>
<br>
<p><font color="#808080" face="Courier New" size="1">The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful.</font></p>