<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head><meta content="text/html;charset=UTF-8" http-equiv="Content-Type" /></head><body>Looks like 17 seconds. That's not 14 minutes. <br><br><div class="gmail_quote">On May 7, 2015 10:55:05 PM PDT, gjprabu <gjprabu@zohocorp.com> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div style="font-size:10pt;">Hi Team,<br /> <br /> Any options to solve below issues.<br /><br />Regards<br />Prabu<br /> <br /><div class="zmail_extra"><div id="1"><br />---- On Thu, 07 May 2015 12:23:02 +0530 <b><gjprabu@zohocorp.com></b> wrote ---- <br /></div><blockquote style="border-left: 1px solid #0000FF; padding-left: 6px; margin:0 0 0 5px"><div><div style="font-size:10pt;">Hi Vijay,<br /><br /> Do we have any other options to increase the performance.<br /><br /><div><div><span style="color: rgb(0, 0, 0);">Regards</span></div><span style="color: rgb(0, 0, 0);">Prabu<br /></span><div style="color: rgb(255, 0, 0);"><b><br clear="none" /></b></div><font size="3"><span style="color: rgb(192, 192, 192); font-family: arial,helvetica,sans-serif;"><font style="font-family: 'courier new',courier,monospace;" size="5"><span style="color: rgb(0, 0,
255);"></span></font><span></span></span></font></div><br /><div><div><br />---- On Wed, 06 May 2015 15:51:20 +0530 <b>gjprabu <<a subj="" mailid="gjprabu%40zohocorp.com" href="mailto:gjprabu@zohocorp.com" target="_blank">gjprabu@zohocorp.com</a>></b> wrote ---- <br /></div><blockquote style="border-left: 1px solid #0000ff;padding-left: 6px; margin: 0 0 0 5px"><div><div style="font-size:10pt;">Hi Vijay,<br /><br /> We tired on physical machines but its doesn't improve speed.<br /><br /><blockquote style="border: 1px solid rgb(204, 204, 204); padding: 7px; background-color: rgb(245, 245, 245);"><div># gluster volume info<br /> <br />Volume Name: integvoltest<br />Type: Replicate<br />Volume ID: 6c66afb9-d466-428e-b944-e15d7a1be5f2<br />Status: Started<br />Number of Bricks: 1 x 2 = 2<br />Transport-type: tcp<br />Bricks:<br />Brick1: integ-gluster3:/srv/sdb1/brick7<br />Brick2: integ-gluster4:/srv/sdb1/brick7<br />Options
Reconfigured:<br />diagnostics.count-fop-hits: on<br />diagnostics.latency-measurement: on<br />cluster.ensure-durability: off<br />cluster.readdir-optimize: on<br />performance.readdir-ahead: on<br />server.event-threads: 30<br />client.event-threads: 30<br /></div></blockquote> <br /> <br /><blockquote style="border: 1px solid rgb(204, 204, 204); padding: 7px; background-color: rgb(245, 245, 245);"><div> <br />gluster volume profile integvoltest info<br />Brick: integ-gluster3:/srv/sdb1/brick7<br />--------------------------------------<br />Cumulative Stats:<br /> Block Size: 4b+ 8b+ 16b+ <br /> No. of
Reads: 0 0 1 <br />No. of Writes: 2 4 8 <br /> <br /> Block Size: 32b+
64b+ 128b+ <br /> No. of Reads: 2 2 2 <br />No. of Writes: 8 6 6 <br /> <br /> Block
Size: 256b+ 512b+ 1024b+ <br /> No. of Reads: 0 0 0 <br />No. of Writes: 4
2 6 <br /> <br /> Block Size: 2048b+ 4096b+ <br /> No. of Reads: 0 0 <br />No. of Writes: 2 6507 <br /> %-latency Avg-latency Min-Latency Max-Latency No. of
calls Fop<br /> --------- ----------- ----------- ----------- ------------ ----<br /> 0.00 0.00 us 0.00 us 0.00 us 37 FORGET<br /> 0.00 0.00 us 0.00 us 0.00 us 61 RELEASE<br /> 0.00 0.00 us 0.00 us 0.00
us 202442 RELEASEDIR<br /> 0.00 142.00 us 142.00 us 142.00 us 1 REMOVEXATTR<br /> 0.00 101.00 us 79.00 us 142.00 us 3 STAT<br /> 0.00 313.00 us 313.00 us 313.00 us 1 XATTROP<br /> 0.00 120.00 us 96.00 us 145.00
us 3 READ<br /> 0.00 101.75 us 69.00 us 158.00 us 4 STATFS<br /> 0.00 131.25 us 112.00 us 147.00 us 4 GETXATTR<br /> 0.00 256.00 us 216.00 us 309.00 us 3 UNLINK<br /> 0.00 820.00
us 820.00 us 820.00 us 1 SYMLINK<br /> 0.00 109.80 us 72.00 us 197.00 us 10 READDIR<br /> 0.00 125.58 us 100.00 us 161.00 us 12 SETATTR<br /> 0.00 138.36 us 102.00 us 196.00 us 11 OPEN<br />
0.00 55.38 us 24.00 us 240.00 us 29 FLUSH<br /> 0.00 445.00 us 125.00 us 937.00 us 4 READDIRP<br /> 0.01 306.43 us 165.00 us 394.00 us 7 RENAME<br /> 0.01 199.55 us 153.00 us 294.00 us 11 SETXATTR<br
/> 0.01 72.64 us 28.00 us 227.00 us 47 FINODELK<br /> 0.02 67.69 us 30.00 us 241.00 us 96 ENTRYLK<br /> 0.03 1038.18 us 943.00 us 1252.00 us 11 MKDIR<br /> 0.03 251.49 us 147.00 us 865.00 us 53
FXATTROP<br /> 0.06 1115.60 us 808.00 us 1860.00 us 20 CREATE<br /> 0.07 323.83 us 31.00 us 22132.00 us 88 INODELK<br /> 1.41 170.57 us 79.00 us 2022.00 us 3262 WRITE<br /> 26.35 103.15 us 4.00 us 260.00 us 100471 OPENDIR<br
/> 71.98 139.07 us 47.00 us 471.00 us 203591 LOOKUP<br /> <br /> Duration: 1349 seconds<br /> Data Read: 624 bytes<br />Data Written: 26675732 bytes<br /> <br />Interval 25 Stats:<br /> Block Size: 4096b+ <br /> No. of Reads: 0 <br />No. of Writes: 108 <br /> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br /> ---------
----------- ----------- ----------- ------------ ----<br /> 0.00 0.00 us 0.00 us 0.00 us 4887 RELEASEDIR<br /> 1.06 155.19 us 117.00 us 248.00 us 108 WRITE<br /> 25.69 83.24 us 35.00 us 260.00 us 4888 OPENDIR<br /> 73.25 117.83 us 51.00
us 406.00 us 9844 LOOKUP<br /> <br /> Duration: 17 seconds<br /> Data Read: 0 bytes<br />Data Written: 442368 bytes<br /> <br />Brick: integ-gluster4:/srv/sdb1/brick7<br />--------------------------------------<br />Cumulative Stats:<br /> Block Size: 4b+ 8b+ 16b+ <br /> No. of Reads:
0 0 1 <br />No. of Writes: 2 4 8 <br /> <br /> Block Size: 32b+ 64b+ 128b+ <br /> No.
of Reads: 2 2 2 <br />No. of Writes: 8 6 6 <br /> <br /> Block Size: 256b+
512b+ 1024b+ <br /> No. of Reads: 0 0 0 <br />No. of Writes: 4 2 6 <br /> <br /> Block Size:
2048b+ 4096b+ <br /> No. of Reads: 0 0 <br />No. of Writes: 2 6507 <br /> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br /> --------- ----------- ----------- ----------- ------------ ----<br />
0.00 0.00 us 0.00 us 0.00 us 37 FORGET<br /> 0.00 0.00 us 0.00 us 0.00 us 61 RELEASE<br /> 0.00 0.00 us 0.00 us 0.00 us 202444 RELEASEDIR<br /> 0.00 124.00 us 124.00 us 124.00 us 1
REMOVEXATTR<br /> 0.00 186.00 us 186.00 us 186.00 us 1 XATTROP<br /> 0.00 90.00 us 49.00 us 131.00 us 3 GETXATTR<br /> 0.00 276.00 us 276.00 us 276.00 us 1 SYMLINK<br /> 0.00 92.75 us 49.00 us 119.00
us 4 STATFS<br /> 0.00 125.00 us 58.00 us 161.00 us 3 UNLINK<br /> 0.00 135.75 us 90.00 us 173.00 us 4 READ<br /> 0.00 64.73 us 54.00 us 131.00 us 11 SETXATTR<br /> 0.00 93.80
us 40.00 us 129.00 us 10 READDIR<br /> 0.00 95.00 us 75.00 us 159.00 us 12 SETATTR<br /> 0.00 121.27 us 93.00 us 156.00 us 11 OPEN<br /> 0.00 257.14 us 106.00 us 404.00 us 7 RENAME<br
/> 0.00 62.76 us 26.00 us 245.00 us 29 FLUSH<br /> 0.00 313.73 us 256.00 us 397.00 us 11 MKDIR<br /> 0.00 74.08 us 23.00 us 243.00 us 49 FINODELK<br /> 0.01 62.12 us 20.00 us 145.00
us 88 INODELK<br /> 0.01 58.81 us 18.00 us 120.00 us 96 ENTRYLK<br /> 0.01 370.35 us 285.00 us 479.00 us 20 CREATE<br /> 0.01 149.85 us 70.00 us 230.00 us 53 FXATTROP<br /> 0.03 69.19 us 32.00
us 167.00 us 350 STAT<br /> 0.46 130.36 us 59.00 us 1042.00 us 3262 WRITE<br /> 10.07 91.73 us 5.00 us 759.00 us 100472 OPENDIR<br /> 24.90 111.87 us 40.00 us 954.00 us 203595 LOOKUP<br /> 64.49 293.59 us 28.00
us 2779.00 us 200938 READDIRP<br /> <br /> Duration: 1349 seconds<br /> Data Read: 624 bytes<br />Data Written: 26675732 bytes<br /> <br />Interval 25 Stats:<br /> Block Size: 4096b+ <br /> No. of Reads: 0 <br />No. of Writes: 108 <br /> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br /> --------- ----------- ----------- -----------
------------ ----<br /> 0.00 0.00 us 0.00 us 0.00 us 4888 RELEASEDIR<br /> 0.03 92.18 us 55.00 us 161.00 us 17 STAT<br /> 0.31 144.44 us 108.00 us 382.00 us 108 WRITE<br /> 11.20 113.73 us 53.00 us 203.00
us 4887 OPENDIR<br /> 26.70 134.65 us 79.00 us 265.00 us 9844 LOOKUP<br /> 61.76 313.65 us 56.00 us 652.00 us 9774 READDIRP<br /> <br /> Duration: 17 seconds<br /> Data Read: 0 bytes<br />Data Written: 442368 bytes</div></blockquote> <br /><blockquote style="border: 1px solid rgb(204, 204, 204); padding: 7px; background-color: rgb(245, 245, 245);"><div>strace -Tcf git clone<br />% time seconds usecs/call calls errors syscall<br
/>------ ----------- ----------- --------- --------- ----------------<br />100.00 0.000026 2 13 mmap<br /> 0.00 0.000000 0 3 read<br /> 0.00 0.000000 0 77 write<br /> 0.00 0.000000 0 4 open<br /> 0.00
0.000000 0 4 close<br /> 0.00 0.000000 0 4 fstat<br /> 0.00 0.000000 0 7 mprotect<br /> 0.00 0.000000 0 1 munmap<br /> 0.00 0.000000
0 3 brk<br /> 0.00 0.000000 0 2 rt_sigaction<br /> 0.00 0.000000 0 1 rt_sigprocmask<br /> 0.00 0.000000 0 1 1 access<br /> 0.00 0.000000 0
1 execve<br /> 0.00 0.000000 0 1 getrlimit<br /> 0.00 0.000000 0 1 arch_prctl<br /> 0.00 0.000000 0 2 1 futex<br /> 0.00 0.000000 0 1 set_tid_address<br />
0.00 0.000000 0 1 set_robust_list<br />------ ----------- ----------- --------- --------- ----------------<br />100.00 0.000026 127 2 total</div></blockquote> <br /><br /><div><div><span style="color: rgb(0, 0, 0);">Regards</span></div><span style="color: rgb(0, 0, 0);">Prabu<br /></span><div style="color: rgb(255, 0, 0);"><b><br clear="none" /></b></div><font size="3"><span style="color: rgb(192, 192, 192); font-family: arial,helvetica,sans-serif;"><font style="font-family: 'courier new',courier,monospace;" size="5"><span style="color: rgb(0, 0, 255);"></span></font><span></span></span></font></div><br /><div><div><br />---- On
Tue, 05 May 2015 15:51:58 +0530 <b>Vijay Bellur<<a subj="" mailid="vbellur%40redhat.com" href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></b> wrote ---- <br /></div><blockquote style="border-left: 1px solid #0000ff; padding-left: 6px; margin:0 0 0 5px"><div>On 05/05/2015 03:43 PM, Kamal wrote: <br />> Hi Vijay, <br />> <br />> We Tried the same. But its doesn't improve speed. <br />> <br />> For testing glusterfs, we are running storage in virtual machines. Is <br />> that will make any difference ? <br /> <br />Performance testing on physical machines is a better bet as it takes <br />some variables away from the equation. <br /> <br />Since you happen to use a replicated volume, can you try by enabling <br />this option? <br /> <br />gluster volume set <volname> cluster.ensure-durability off <br /> <br />Additionally you might want to try "strace -Tcf git clone .." and <br />"gluster volume profile ..." to figure out
where the latency is stemming <br />from. <br /> <br /> <br />> But, copying the same folder between two storage machines is really <br />> fast. fyi. <br />> <br /> <br />Does this copying involve gluster or not? <br /> <br />Regards, <br />Vijay <br /> <br />> Regards, <br />> Kamal <br />> <br />> <br />> <br />> ---- On Tue, 05 May 2015 15:08:27 +0530 *Vijay <br />> Bellur<<a subj="" mailid="vbellur%40redhat.com" href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>>* wrote ---- <br />> <br />> On 05/05/2015 12:59 PM, Kamal wrote: <br />> > Hi Amukher, <br />> > <br />> > Even after upgrade to 3.7 small files transfer rate is slow. <br />> > <br />> > Below is the volume info. <br />> > <br />> > Volume Name: integvol1 <br />> > Type: Replicate <br />> > Volume ID: 31793ba4-eeca-462a-a0cd-9adfb281225b <br />>
> Status: Started <br />> > Number of Bricks: 1 x 2 = 2 <br />> > Transport-type: tcp <br />> > Bricks: <br />> > Brick1: integ-gluster1:/srv/sdb2/brick4 <br />> > Brick2: integ-gluster2:/srv/sdb2/brick4 <br />> > Options Reconfigured: <br />> > server.event-threads: 30 <br />> > client.event-threads: 30 <br />> > ---- <br />> > <br />> > I understand that for replication it would take some more time, but <br />> > here its taking more time. <br />> > <br />> > <br />> > Time taken for git clone in non gluster directory = 25 sec <br />> > <br />> > Time taken for git clone in gluster directory = 14 minutes <br />> > <br />> > Its a huge difference. Plz let me know any other tuning <br />> parameters need <br />> > to be done. <br />> > <br />>
> <br />> <br />> I have seen this before and it primarily seems to be related to the <br />> readdir calls done by git clone. <br />> <br />> Turning on these options might help to some extent: <br />> <br />> gluster volume set <volname> performance.readdir-ahead on <br />> <br />> gluster volume set <volname> cluster.readdir-optimize on <br />> <br />> Please do let us know what you observe with these options enabled. <br />> <br />> Regards, <br />> Vijay <br />> <br />> <br />> <br /> <br />_______________________________________________ <br />Gluster-users mailing list <br /><a subj="" mailid="Gluster-users%40gluster.org" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <br /><a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a> <br /></div></blockquote><br
/></div><br /></div></div></blockquote><br /></div><br /></div></div></blockquote><br /></div><br /></div><p style="margin-top: 2.5em; margin-bottom: 1em; border-bottom: 1px solid #000"></p><pre class="k9mail"><hr /><br />Gluster-users mailing list<br />Gluster-users@gluster.org<br /><a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre></blockquote></div><br>
-- <br>
Sent from my Android device with K-9 Mail. Please excuse my brevity.</body></html>