<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head><meta content="text/html;charset=UTF-8" http-equiv="Content-Type"></head><body ><div style='font-size:10pt;'>Hi Team,<br> <br> Any options to solve below issues.<br><br>Regards<br>Prabu<br> <br><div class="zmail_extra"><div id="1"><br>---- On Thu, 07 May 2015 12:23:02 +0530 <b><gjprabu@zohocorp.com></b> wrote ---- <br></div><blockquote style="border-left: 1px solid #0000FF; padding-left: 6px; margin:0 0 0 5px"><div><div style="font-size:10pt;">Hi Vijay,<br><br> Do we have any other options to increase the performance.<br><br><div><div><span style="color: rgb(0, 0, 0);">Regards</span></div><span style="color: rgb(0, 0, 0);">Prabu<br></span><div style="color: rgb(255, 0, 0);"><b><br clear="none"></b></div><font size="3"><span style="color: rgb(192, 192, 192); font-family: arial,helvetica,sans-serif;"><font style="font-family: 'courier new',courier,monospace;" size="5"><span style="color: rgb(0, 0, 255);"></span></font><span></span></span></font></div><br><div><div><br>---- On Wed, 06 May 2015 15:51:20 +0530 <b>gjprabu <<a subj="" mailid="gjprabu%40zohocorp.com" href="mailto:gjprabu@zohocorp.com" target="_blank">gjprabu@zohocorp.com</a>></b> wrote ---- <br></div><blockquote style="border-left: 1px solid #0000ff;padding-left: 6px; margin: 0 0 0 5px"><div><div style="font-size:10pt;">Hi Vijay,<br><br> We tired on physical machines but its doesn't improve speed.<br><br><blockquote style="border: 1px solid rgb(204, 204, 204); padding: 7px; background-color: rgb(245, 245, 245);"><div># gluster volume info<br> <br>Volume Name: integvoltest<br>Type: Replicate<br>Volume ID: 6c66afb9-d466-428e-b944-e15d7a1be5f2<br>Status: Started<br>Number of Bricks: 1 x 2 = 2<br>Transport-type: tcp<br>Bricks:<br>Brick1: integ-gluster3:/srv/sdb1/brick7<br>Brick2: integ-gluster4:/srv/sdb1/brick7<br>Options Reconfigured:<br>diagnostics.count-fop-hits: on<br>diagnostics.latency-measurement: on<br>cluster.ensure-durability: off<br>cluster.readdir-optimize: on<br>performance.readdir-ahead: on<br>server.event-threads: 30<br>client.event-threads: 30<br></div></blockquote> <br> <br><blockquote style="border: 1px solid rgb(204, 204, 204); padding: 7px; background-color: rgb(245, 245, 245);"><div> <br>gluster volume profile integvoltest info<br>Brick: integ-gluster3:/srv/sdb1/brick7<br>--------------------------------------<br>Cumulative Stats:<br> Block Size: 4b+ 8b+ 16b+ <br> No. of Reads: 0 0 1 <br>No. of Writes: 2 4 8 <br> <br> Block Size: 32b+ 64b+ 128b+ <br> No. of Reads: 2 2 2 <br>No. of Writes: 8 6 6 <br> <br> Block Size: 256b+ 512b+ 1024b+ <br> No. of Reads: 0 0 0 <br>No. of Writes: 4 2 6 <br> <br> Block Size: 2048b+ 4096b+ <br> No. of Reads: 0 0 <br>No. of Writes: 2 6507 <br> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br> --------- ----------- ----------- ----------- ------------ ----<br> 0.00 0.00 us 0.00 us 0.00 us 37 FORGET<br> 0.00 0.00 us 0.00 us 0.00 us 61 RELEASE<br> 0.00 0.00 us 0.00 us 0.00 us 202442 RELEASEDIR<br> 0.00 142.00 us 142.00 us 142.00 us 1 REMOVEXATTR<br> 0.00 101.00 us 79.00 us 142.00 us 3 STAT<br> 0.00 313.00 us 313.00 us 313.00 us 1 XATTROP<br> 0.00 120.00 us 96.00 us 145.00 us 3 READ<br> 0.00 101.75 us 69.00 us 158.00 us 4 STATFS<br> 0.00 131.25 us 112.00 us 147.00 us 4 GETXATTR<br> 0.00 256.00 us 216.00 us 309.00 us 3 UNLINK<br> 0.00 820.00 us 820.00 us 820.00 us 1 SYMLINK<br> 0.00 109.80 us 72.00 us 197.00 us 10 READDIR<br> 0.00 125.58 us 100.00 us 161.00 us 12 SETATTR<br> 0.00 138.36 us 102.00 us 196.00 us 11 OPEN<br> 0.00 55.38 us 24.00 us 240.00 us 29 FLUSH<br> 0.00 445.00 us 125.00 us 937.00 us 4 READDIRP<br> 0.01 306.43 us 165.00 us 394.00 us 7 RENAME<br> 0.01 199.55 us 153.00 us 294.00 us 11 SETXATTR<br> 0.01 72.64 us 28.00 us 227.00 us 47 FINODELK<br> 0.02 67.69 us 30.00 us 241.00 us 96 ENTRYLK<br> 0.03 1038.18 us 943.00 us 1252.00 us 11 MKDIR<br> 0.03 251.49 us 147.00 us 865.00 us 53 FXATTROP<br> 0.06 1115.60 us 808.00 us 1860.00 us 20 CREATE<br> 0.07 323.83 us 31.00 us 22132.00 us 88 INODELK<br> 1.41 170.57 us 79.00 us 2022.00 us 3262 WRITE<br> 26.35 103.15 us 4.00 us 260.00 us 100471 OPENDIR<br> 71.98 139.07 us 47.00 us 471.00 us 203591 LOOKUP<br> <br> Duration: 1349 seconds<br> Data Read: 624 bytes<br>Data Written: 26675732 bytes<br> <br>Interval 25 Stats:<br> Block Size: 4096b+ <br> No. of Reads: 0 <br>No. of Writes: 108 <br> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br> --------- ----------- ----------- ----------- ------------ ----<br> 0.00 0.00 us 0.00 us 0.00 us 4887 RELEASEDIR<br> 1.06 155.19 us 117.00 us 248.00 us 108 WRITE<br> 25.69 83.24 us 35.00 us 260.00 us 4888 OPENDIR<br> 73.25 117.83 us 51.00 us 406.00 us 9844 LOOKUP<br> <br> Duration: 17 seconds<br> Data Read: 0 bytes<br>Data Written: 442368 bytes<br> <br>Brick: integ-gluster4:/srv/sdb1/brick7<br>--------------------------------------<br>Cumulative Stats:<br> Block Size: 4b+ 8b+ 16b+ <br> No. of Reads: 0 0 1 <br>No. of Writes: 2 4 8 <br> <br> Block Size: 32b+ 64b+ 128b+ <br> No. of Reads: 2 2 2 <br>No. of Writes: 8 6 6 <br> <br> Block Size: 256b+ 512b+ 1024b+ <br> No. of Reads: 0 0 0 <br>No. of Writes: 4 2 6 <br> <br> Block Size: 2048b+ 4096b+ <br> No. of Reads: 0 0 <br>No. of Writes: 2 6507 <br> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br> --------- ----------- ----------- ----------- ------------ ----<br> 0.00 0.00 us 0.00 us 0.00 us 37 FORGET<br> 0.00 0.00 us 0.00 us 0.00 us 61 RELEASE<br> 0.00 0.00 us 0.00 us 0.00 us 202444 RELEASEDIR<br> 0.00 124.00 us 124.00 us 124.00 us 1 REMOVEXATTR<br> 0.00 186.00 us 186.00 us 186.00 us 1 XATTROP<br> 0.00 90.00 us 49.00 us 131.00 us 3 GETXATTR<br> 0.00 276.00 us 276.00 us 276.00 us 1 SYMLINK<br> 0.00 92.75 us 49.00 us 119.00 us 4 STATFS<br> 0.00 125.00 us 58.00 us 161.00 us 3 UNLINK<br> 0.00 135.75 us 90.00 us 173.00 us 4 READ<br> 0.00 64.73 us 54.00 us 131.00 us 11 SETXATTR<br> 0.00 93.80 us 40.00 us 129.00 us 10 READDIR<br> 0.00 95.00 us 75.00 us 159.00 us 12 SETATTR<br> 0.00 121.27 us 93.00 us 156.00 us 11 OPEN<br> 0.00 257.14 us 106.00 us 404.00 us 7 RENAME<br> 0.00 62.76 us 26.00 us 245.00 us 29 FLUSH<br> 0.00 313.73 us 256.00 us 397.00 us 11 MKDIR<br> 0.00 74.08 us 23.00 us 243.00 us 49 FINODELK<br> 0.01 62.12 us 20.00 us 145.00 us 88 INODELK<br> 0.01 58.81 us 18.00 us 120.00 us 96 ENTRYLK<br> 0.01 370.35 us 285.00 us 479.00 us 20 CREATE<br> 0.01 149.85 us 70.00 us 230.00 us 53 FXATTROP<br> 0.03 69.19 us 32.00 us 167.00 us 350 STAT<br> 0.46 130.36 us 59.00 us 1042.00 us 3262 WRITE<br> 10.07 91.73 us 5.00 us 759.00 us 100472 OPENDIR<br> 24.90 111.87 us 40.00 us 954.00 us 203595 LOOKUP<br> 64.49 293.59 us 28.00 us 2779.00 us 200938 READDIRP<br> <br> Duration: 1349 seconds<br> Data Read: 624 bytes<br>Data Written: 26675732 bytes<br> <br>Interval 25 Stats:<br> Block Size: 4096b+ <br> No. of Reads: 0 <br>No. of Writes: 108 <br> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br> --------- ----------- ----------- ----------- ------------ ----<br> 0.00 0.00 us 0.00 us 0.00 us 4888 RELEASEDIR<br> 0.03 92.18 us 55.00 us 161.00 us 17 STAT<br> 0.31 144.44 us 108.00 us 382.00 us 108 WRITE<br> 11.20 113.73 us 53.00 us 203.00 us 4887 OPENDIR<br> 26.70 134.65 us 79.00 us 265.00 us 9844 LOOKUP<br> 61.76 313.65 us 56.00 us 652.00 us 9774 READDIRP<br> <br> Duration: 17 seconds<br> Data Read: 0 bytes<br>Data Written: 442368 bytes</div></blockquote> <br><blockquote style="border: 1px solid rgb(204, 204, 204); padding: 7px; background-color: rgb(245, 245, 245);"><div>strace -Tcf git clone<br>% time seconds usecs/call calls errors syscall<br>------ ----------- ----------- --------- --------- ----------------<br>100.00 0.000026 2 13 mmap<br> 0.00 0.000000 0 3 read<br> 0.00 0.000000 0 77 write<br> 0.00 0.000000 0 4 open<br> 0.00 0.000000 0 4 close<br> 0.00 0.000000 0 4 fstat<br> 0.00 0.000000 0 7 mprotect<br> 0.00 0.000000 0 1 munmap<br> 0.00 0.000000 0 3 brk<br> 0.00 0.000000 0 2 rt_sigaction<br> 0.00 0.000000 0 1 rt_sigprocmask<br> 0.00 0.000000 0 1 1 access<br> 0.00 0.000000 0 1 execve<br> 0.00 0.000000 0 1 getrlimit<br> 0.00 0.000000 0 1 arch_prctl<br> 0.00 0.000000 0 2 1 futex<br> 0.00 0.000000 0 1 set_tid_address<br> 0.00 0.000000 0 1 set_robust_list<br>------ ----------- ----------- --------- --------- ----------------<br>100.00 0.000026 127 2 total</div></blockquote> <br><br><div><div><span style="color: rgb(0, 0, 0);">Regards</span></div><span style="color: rgb(0, 0, 0);">Prabu<br></span><div style="color: rgb(255, 0, 0);"><b><br clear="none"></b></div><font size="3"><span style="color: rgb(192, 192, 192); font-family: arial,helvetica,sans-serif;"><font style="font-family: 'courier new',courier,monospace;" size="5"><span style="color: rgb(0, 0, 255);"></span></font><span></span></span></font></div><br><div><div><br>---- On Tue, 05 May 2015 15:51:58 +0530 <b>Vijay Bellur<<a subj="" mailid="vbellur%40redhat.com" href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></b> wrote ---- <br></div><blockquote style="border-left: 1px solid #0000ff; padding-left: 6px; margin:0 0 0 5px"><div>On 05/05/2015 03:43 PM, Kamal wrote: <br>> Hi Vijay, <br>> <br>> We Tried the same. But its doesn't improve speed. <br>> <br>> For testing glusterfs, we are running storage in virtual machines. Is <br>> that will make any difference ? <br> <br>Performance testing on physical machines is a better bet as it takes <br>some variables away from the equation. <br> <br>Since you happen to use a replicated volume, can you try by enabling <br>this option? <br> <br>gluster volume set <volname> cluster.ensure-durability off <br> <br>Additionally you might want to try "strace -Tcf git clone .." and <br>"gluster volume profile ..." to figure out where the latency is stemming <br>from. <br> <br> <br>> But, copying the same folder between two storage machines is really <br>> fast. fyi. <br>> <br> <br>Does this copying involve gluster or not? <br> <br>Regards, <br>Vijay <br> <br>> Regards, <br>> Kamal <br>> <br>> <br>> <br>> ---- On Tue, 05 May 2015 15:08:27 +0530 *Vijay <br>> Bellur<<a subj="" mailid="vbellur%40redhat.com" href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>>* wrote ---- <br>> <br>> On 05/05/2015 12:59 PM, Kamal wrote: <br>> > Hi Amukher, <br>> > <br>> > Even after upgrade to 3.7 small files transfer rate is slow. <br>> > <br>> > Below is the volume info. <br>> > <br>> > Volume Name: integvol1 <br>> > Type: Replicate <br>> > Volume ID: 31793ba4-eeca-462a-a0cd-9adfb281225b <br>> > Status: Started <br>> > Number of Bricks: 1 x 2 = 2 <br>> > Transport-type: tcp <br>> > Bricks: <br>> > Brick1: integ-gluster1:/srv/sdb2/brick4 <br>> > Brick2: integ-gluster2:/srv/sdb2/brick4 <br>> > Options Reconfigured: <br>> > server.event-threads: 30 <br>> > client.event-threads: 30 <br>> > ---- <br>> > <br>> > I understand that for replication it would take some more time, but <br>> > here its taking more time. <br>> > <br>> > <br>> > Time taken for git clone in non gluster directory = 25 sec <br>> > <br>> > Time taken for git clone in gluster directory = 14 minutes <br>> > <br>> > Its a huge difference. Plz let me know any other tuning <br>> parameters need <br>> > to be done. <br>> > <br>> > <br>> <br>> I have seen this before and it primarily seems to be related to the <br>> readdir calls done by git clone. <br>> <br>> Turning on these options might help to some extent: <br>> <br>> gluster volume set <volname> performance.readdir-ahead on <br>> <br>> gluster volume set <volname> cluster.readdir-optimize on <br>> <br>> Please do let us know what you observe with these options enabled. <br>> <br>> Regards, <br>> Vijay <br>> <br>> <br>> <br> <br>_______________________________________________ <br>Gluster-users mailing list <br><a subj="" mailid="Gluster-users%40gluster.org" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <br><a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a> <br></div></blockquote><br></div><br></div></div></blockquote><br></div><br></div></div></blockquote><br></div><br></div></body></html>