<div dir="ltr">Ok I have all my data created and I just started the rebalance. One thing to not in the client log I see the following spamming:<div><br></div><div><div>[root@gqac006 ~]# cat /var/log/glusterfs/gluster-mount-.log | wc -l</div><div>394042</div></div><div><br></div><div><div>[2015-05-01 00:47:55.591150] I [MSGID: 109036] [dht-common.c:6478:dht_log_new_layout_for_dir_selfheal] 0-testvol-dht: Setting layout of /file_dstdir/<a href="http://gqac006.sbu.lab.eng.bos.redhat.com/thrd_05/d_001/d_000/d_004/d_006">gqac006.sbu.lab.eng.bos.redhat.com/thrd_05/d_001/d_000/d_004/d_006</a> with [Subvol_name: testvol-replicate-0, Err: -1 , Start: 0 , Stop: 2141429669 ], [Subvol_name: testvol-replicate-1, Err: -1 , Start: 2141429670 , Stop: 4294967295 ], </div><div>[2015-05-01 00:47:55.596147] I [dht-selfheal.c:1587:dht_selfheal_layout_new_directory] 0-testvol-dht: chunk size = 0xffffffff / 19920276 = 0xd7</div><div>[2015-05-01 00:47:55.596177] I [dht-selfheal.c:1626:dht_selfheal_layout_new_directory] 0-testvol-dht: assigning range size 0x7fa39fa6 to testvol-replicate-1</div><div>[2015-05-01 00:47:55.596189] I [dht-selfheal.c:1626:dht_selfheal_layout_new_directory] 0-testvol-dht: assigning range size 0x7fa39fa6 to testvol-replicate-0</div><div>[2015-05-01 00:47:55.597081] I [MSGID: 109036] [dht-common.c:6478:dht_log_new_layout_for_dir_selfheal] 0-testvol-dht: Setting layout of /file_dstdir/<a href="http://gqac006.sbu.lab.eng.bos.redhat.com/thrd_05/d_001/d_000/d_004/d_005">gqac006.sbu.lab.eng.bos.redhat.com/thrd_05/d_001/d_000/d_004/d_005</a> with [Subvol_name: testvol-replicate-0, Err: -1 , Start: 2141429670 , Stop: 4294967295 ], [Subvol_name: testvol-replicate-1, Err: -1 , Start: 0 , Stop: 2141429669 ], </div><div>[2015-05-01 00:47:55.601853] I [dht-selfheal.c:1587:dht_selfheal_layout_new_directory] 0-testvol-dht: chunk size = 0xffffffff / 19920276 = 0xd7</div><div>[2015-05-01 00:47:55.601882] I [dht-selfheal.c:1626:dht_selfheal_layout_new_directory] 0-testvol-dht: assigning range size 0x7fa39fa6 to testvol-replicate-1</div><div>[2015-05-01 00:47:55.601895] I [dht-selfheal.c:1626:dht_selfheal_layout_new_directory] 0-testvol-dht: assigning range size 0x7fa39fa6 to testvol-replicate-0</div><div><br></div></div><div>Just to confirm the patch is in, glusterfs-3.8dev-0.71.gita7f8482.el6.x86_64. Correct?</div><div><br></div><div>Here is the info on the data set:</div><div><br></div><div><div>hosts in test : ['<a href="http://gqac006.sbu.lab.eng.bos.redhat.com">gqac006.sbu.lab.eng.bos.redhat.com</a>', '<a href="http://gqas003.sbu.lab.eng.bos.redhat.com">gqas003.sbu.lab.eng.bos.redhat.com</a>']</div><div>top test directory(s) : ['/gluster-mount']</div><div>peration : create</div><div>files/thread : 500000</div><div>threads : 8</div><div>record size (KB, 0 = maximum) : 0</div><div>file size (KB) : 64</div><div>file size distribution : fixed</div><div>files per dir : 100</div><div>dirs per dir : 10</div></div><div><div>total threads = 16</div><div>total files = 7222600</div><div>total data = 440.833 GB</div><div> 90.28% of requested files processed, minimum is 70.00</div><div>8107.852862 sec elapsed time</div><div>890.815377 files/sec</div><div>890.815377 IOPS</div><div>55.675961 MB/sec</div></div><div><br></div><div>Here is the rebalance run after about 5 or so minutes:</div><div><br></div><div>[root@gqas001 ~]# gluster v rebalance testvol status</div><div> Node Rebalanced-files size scanned failures skipped status run time in secs</div><div> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------</div><div> localhost 32203 2.0GB 120858 0 5184 in progress 1294.00</div><div> <a href="http://gqas011.sbu.lab.eng.bos.redhat.com">gqas011.sbu.lab.eng.bos.redhat.com</a> 0 0Bytes 0 0 0 failed 0.00</div><div> <a href="http://gqas016.sbu.lab.eng.bos.redhat.com">gqas016.sbu.lab.eng.bos.redhat.com</a> 9364 585.2MB 53121 0 0 in progress 1294.00</div><div> <a href="http://gqas013.sbu.lab.eng.bos.redhat.com">gqas013.sbu.lab.eng.bos.redhat.com</a> 0 0Bytes 14750 0 0 in progress 1294.00</div><div> <a href="http://gqas014.sbu.lab.eng.bos.redhat.com">gqas014.sbu.lab.eng.bos.redhat.com</a> 0 0Bytes 0 0 0 failed 0.00</div><div> <a href="http://gqas015.sbu.lab.eng.bos.redhat.com">gqas015.sbu.lab.eng.bos.redhat.com</a> 0 0Bytes 196382 0 0 in progress 1294.00</div><div>volume rebalance: testvol: success: </div><div><br></div><div>The hostnames are there if you want to poke around. I had a problem with one of the added systems being on a different version of glusterfs so I had to update everything to glusterfs-3.8dev-0.99.git7d7b80e.el6.x86_64, remove the bricks I just added, and add them back. Something may have went wrong in that process but I thought I did everything correctly. I'll start fresh tomorrow. I figured I'd let this run over night.<br></div><div><br></div><div>-b</div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 29, 2015 at 9:48 PM, Benjamin Turner <span dir="ltr"><<a href="mailto:bennyturns@gmail.com" target="_blank">bennyturns@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Sweet! Here is the baseline:<div><br></div><div><div>[root@gqas001 ~]# gluster v rebalance testvol status</div><span class=""><div> Node Rebalanced-files size scanned failures skipped status run time in secs</div><div> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------</div></span><div> localhost 1328575 81.1GB 9402953 0 0 completed 98500.00</div><div> <a href="http://gqas012.sbu.lab.eng.bos.redhat.com" target="_blank">gqas012.sbu.lab.eng.bos.redhat.com</a> 0 0Bytes 8000011 0 0 completed 51982.00</div><div> <a href="http://gqas003.sbu.lab.eng.bos.redhat.com" target="_blank">gqas003.sbu.lab.eng.bos.redhat.com</a> 0 0Bytes 8000011 0 0 completed 51982.00</div><div> <a href="http://gqas004.sbu.lab.eng.bos.redhat.com" target="_blank">gqas004.sbu.lab.eng.bos.redhat.com</a> 1326290 81.0GB 9708625 0 0 completed 98500.00</div><div> <a href="http://gqas013.sbu.lab.eng.bos.redhat.com" target="_blank">gqas013.sbu.lab.eng.bos.redhat.com</a> 0 0Bytes 8000011 0 0 completed 51982.00</div><div> <a href="http://gqas014.sbu.lab.eng.bos.redhat.com" target="_blank">gqas014.sbu.lab.eng.bos.redhat.com</a> 0 0Bytes 8000011 0 0 completed 51982.00</div><div>volume rebalance: testvol: success: </div></div><div><br></div><div>I'll have a run on the patch started tomorrow.</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>-b</div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 29, 2015 at 12:51 PM, Nithya Balachandran <span dir="ltr"><<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><br>
Doh my mistake, I thought it was merged. I was just running with the<br>
upstream 3.7 daily. Can I use this run as my baseline and then I can run<br>
next time on the patch to show the % improvement? I'll wipe everything and<br>
try on the patch, any idea when it will be merged?<br>
<br>
</span>Yes, it would be very useful to have this run as the baseline. The patch has just been merged in master. It should be backported to 3.7 in a day or so.<br>
<br>
Regards,<br>
Nithya<br>
<div><div><br>
<br>
> > > ><br>
> > > > ><br>
> > > > > On Wed, Apr 22, 2015 at 1:10 AM, Nithya Balachandran<br>
> > > > > <<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>><br>
> > > > > wrote:<br>
> > > > ><br>
> > > > > > That sounds great. Thanks.<br>
> > > > > ><br>
> > > > > > Regards,<br>
> > > > > > Nithya<br>
> > > > > ><br>
> > > > > > ----- Original Message -----<br>
> > > > > > From: "Benjamin Turner" <<a href="mailto:bennyturns@gmail.com" target="_blank">bennyturns@gmail.com</a>><br>
> > > > > > To: "Nithya Balachandran" <<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>><br>
> > > > > > Cc: "Susant Palai" <<a href="mailto:spalai@redhat.com" target="_blank">spalai@redhat.com</a>>, "Gluster Devel" <<br>
> > > > > > <a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a>><br>
> > > > > > Sent: Wednesday, 22 April, 2015 12:14:14 AM<br>
> > > > > > Subject: Re: [Gluster-devel] Rebalance improvement design<br>
> > > > > ><br>
> > > > > > I am setting up a test env now, I'll have some feedback for you<br>
> this<br>
> > > > > > week.<br>
> > > > > ><br>
> > > > > > -b<br>
> > > > > ><br>
> > > > > > On Tue, Apr 21, 2015 at 11:36 AM, Nithya Balachandran<br>
> > > > > > <<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a><br>
> > > > > > ><br>
> > > > > > wrote:<br>
> > > > > ><br>
> > > > > > > Hi Ben,<br>
> > > > > > ><br>
> > > > > > > Did you get a chance to try this out?<br>
> > > > > > ><br>
> > > > > > > Regards,<br>
> > > > > > > Nithya<br>
> > > > > > ><br>
> > > > > > > ----- Original Message -----<br>
> > > > > > > From: "Susant Palai" <<a href="mailto:spalai@redhat.com" target="_blank">spalai@redhat.com</a>><br>
> > > > > > > To: "Benjamin Turner" <<a href="mailto:bennyturns@gmail.com" target="_blank">bennyturns@gmail.com</a>><br>
> > > > > > > Cc: "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a>><br>
> > > > > > > Sent: Monday, April 13, 2015 9:55:07 AM<br>
> > > > > > > Subject: Re: [Gluster-devel] Rebalance improvement design<br>
> > > > > > ><br>
> > > > > > > Hi Ben,<br>
> > > > > > > Uploaded a new patch here:<br>
> <a href="http://review.gluster.org/#/c/9657/" target="_blank">http://review.gluster.org/#/c/9657/</a>.<br>
> > > > > > > We<br>
> > > > > > > can<br>
> > > > > > > start perf test on it. :)<br>
> > > > > > ><br>
> > > > > > > Susant<br>
> > > > > > ><br>
> > > > > > > ----- Original Message -----<br>
> > > > > > > From: "Susant Palai" <<a href="mailto:spalai@redhat.com" target="_blank">spalai@redhat.com</a>><br>
> > > > > > > To: "Benjamin Turner" <<a href="mailto:bennyturns@gmail.com" target="_blank">bennyturns@gmail.com</a>><br>
> > > > > > > Cc: "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a>><br>
> > > > > > > Sent: Thursday, 9 April, 2015 3:40:09 PM<br>
> > > > > > > Subject: Re: [Gluster-devel] Rebalance improvement design<br>
> > > > > > ><br>
> > > > > > > Thanks Ben. RPM is not available and I am planning to refresh<br>
> the<br>
> > > > > > > patch<br>
> > > > > > in<br>
> > > > > > > two days with some more regression fixes. I think we can run<br>
> the<br>
> > > > > > > tests<br>
> > > > > > post<br>
> > > > > > > that. Any larger data-set will be good(say 3 to 5 TB).<br>
> > > > > > ><br>
> > > > > > > Thanks,<br>
> > > > > > > Susant<br>
> > > > > > ><br>
> > > > > > > ----- Original Message -----<br>
> > > > > > > From: "Benjamin Turner" <<a href="mailto:bennyturns@gmail.com" target="_blank">bennyturns@gmail.com</a>><br>
> > > > > > > To: "Vijay Bellur" <<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>><br>
> > > > > > > Cc: "Susant Palai" <<a href="mailto:spalai@redhat.com" target="_blank">spalai@redhat.com</a>>, "Gluster Devel" <<br>
> > > > > > > <a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a>><br>
> > > > > > > Sent: Thursday, 9 April, 2015 2:10:30 AM<br>
> > > > > > > Subject: Re: [Gluster-devel] Rebalance improvement design<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > I have some rebalance perf regression stuff I have been<br>
> working on,<br>
> > > > > > > is<br>
> > > > > > > there an RPM with these patches anywhere so that I can try it<br>
> on my<br>
> > > > > > > systems? If not I'll just build from:<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > git fetch git:// <a href="http://review.gluster.org/glusterfs" target="_blank">review.gluster.org/glusterfs</a><br>
> > > > > > > refs/changes/57/9657/8<br>
> > > > > > > &&<br>
> > > > > > > git cherry-pick FETCH_HEAD<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > I will have _at_least_ 10TB of storage, how many TBs of data<br>
> should<br>
> > > > > > > I<br>
> > > > > > > run<br>
> > > > > > > with?<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > -b<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > On Tue, Apr 7, 2015 at 9:07 AM, Vijay Bellur <<br>
> <a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a> ><br>
> > > > > > wrote:<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > On 04/07/2015 03:08 PM, Susant Palai wrote:<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > Here is one test performed on a 300GB data set and around<br>
> 100%(1/2<br>
> > > > > > > the<br>
> > > > > > > time) improvement was seen.<br>
> > > > > > ><br>
> > > > > > > [root@gprfs031 ~]# gluster v i<br>
> > > > > > ><br>
> > > > > > > Volume Name: rbperf<br>
> > > > > > > Type: Distribute<br>
> > > > > > > Volume ID: 35562662-337e-4923-b862- d0bbb0748003<br>
> > > > > > > Status: Started<br>
> > > > > > > Number of Bricks: 4<br>
> > > > > > > Transport-type: tcp<br>
> > > > > > > Bricks:<br>
> > > > > > > Brick1: gprfs029-10ge:/bricks/ gprfs029/brick1<br>
> > > > > > > Brick2: gprfs030-10ge:/bricks/ gprfs030/brick1<br>
> > > > > > > Brick3: gprfs031-10ge:/bricks/ gprfs031/brick1<br>
> > > > > > > Brick4: gprfs032-10ge:/bricks/ gprfs032/brick1<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > Added server 32 and started rebalance force.<br>
> > > > > > ><br>
> > > > > > > Rebalance stat for new changes:<br>
> > > > > > > [root@gprfs031 ~]# gluster v rebalance rbperf status<br>
> > > > > > > Node Rebalanced-files size scanned failures skipped status run<br>
> time<br>
> > > > > > > in<br>
> > > > > > secs<br>
> > > > > > > --------- ----------- ----------- ----------- -----------<br>
> > > > > > > -----------<br>
> > > > > > > ------------ --------------<br>
> > > > > > > localhost 74639 36.1GB 297319 0 0 completed 1743.00<br>
> > > > > > > 172.17.40.30 67512 33.5GB 269187 0 0 completed 1395.00<br>
> > > > > > > gprfs029-10ge 79095 38.8GB 284105 0 0 completed 1559.00<br>
> > > > > > > gprfs032-10ge 0 0Bytes 0 0 0 completed 402.00<br>
> > > > > > > volume rebalance: rbperf: success:<br>
> > > > > > ><br>
> > > > > > > Rebalance stat for old model:<br>
> > > > > > > [root@gprfs031 ~]# gluster v rebalance rbperf status<br>
> > > > > > > Node Rebalanced-files size scanned failures skipped status run<br>
> time<br>
> > > > > > > in<br>
> > > > > > secs<br>
> > > > > > > --------- ----------- ----------- ----------- -----------<br>
> > > > > > > -----------<br>
> > > > > > > ------------ --------------<br>
> > > > > > > localhost 86493 42.0GB 634302 0 0 completed 3329.00<br>
> > > > > > > gprfs029-10ge 94115 46.2GB 687852 0 0 completed 3328.00<br>
> > > > > > > gprfs030-10ge 74314 35.9GB 651943 0 0 completed 3072.00<br>
> > > > > > > gprfs032-10ge 0 0Bytes 594166 0 0 completed 1943.00<br>
> > > > > > > volume rebalance: rbperf: success:<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > This is interesting. Thanks for sharing & well done! Maybe we<br>
> > > > > > > should<br>
> > > > > > > attempt a much larger data set and see how we fare there :).<br>
> > > > > > ><br>
> > > > > > > Regards,<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > Vijay<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > ______________________________ _________________<br>
> > > > > > > Gluster-devel mailing list<br>
> > > > > > > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> > > > > > > <a href="http://www.gluster.org/" target="_blank">http://www.gluster.org/</a> mailman/listinfo/gluster-devel<br>
> > > > > > ><br>
> > > > > > > _______________________________________________<br>
> > > > > > > Gluster-devel mailing list<br>
> > > > > > > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> > > > > > > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
> > > > > > > _______________________________________________<br>
> > > > > > > Gluster-devel mailing list<br>
> > > > > > > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> > > > > > > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
> > > > > > ><br>
> > > > > ><br>
> > > > ><br>
> > > > _______________________________________________<br>
> > > > Gluster-devel mailing list<br>
> > > > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> > > > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
> > > ><br>
> > > _______________________________________________<br>
> > > Gluster-devel mailing list<br>
> > > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> > > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
> > ><br>
> ><br>
><br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>