<div dir="ltr">Hi Anuradha,<div><br></div><div>Please find the Gluster Volume Profile details</div><div><br></div><div><div>time cp -RPp drupal\ code/ /mnt/testmount/copytogluster</div><div><br></div><div><br></div><div><br></div><div><b><u><br></u></b></div><div><b><u>Profile info of the volume when you copy dirs/files into glusterfs.</u></b></div><div><br></div><div><br></div><div><b><u>Time taken to copy (70 MB files/Folder)</u></b></div><div><br></div><div>[root@GFSCLIENT01 temp]# time cp -RPp /mnt/testmount/ /mnt/testmount/copytogluster<br></div><div><br></div><div>real 29m40.985s</div><div>user 0m0.172s</div><div>sys 0m1.688s</div><div><br></div><div><br></div><div><br></div><div>[root@GFSNODE01 ~]# gluster volume profile gv1 info</div><div>Brick: GFSNODE01:/mnt/perfDisk/gv1</div><div><br></div><div>--------------------------------------</div><div>Cumulative Stats:</div><div> Block Size: 16b+ 32b+ 64b+</div><div> No. of Reads: 0 0 0</div><div>No. of Writes: 19 11 75</div><div><br></div><div> Block Size: 128b+ 256b+ 512b+</div><div> No. of Reads: 0 0 0</div><div>No. of Writes: 77 221 297</div><div><br></div><div> Block Size: 1024b+ 2048b+ 4096b+</div><div> No. of Reads: 0 0 0</div><div>No. of Writes: 344 305 336</div><div><br></div><div> Block Size: 8192b+ 16384b+ 32768b+</div><div> No. of Reads: 0 0 0</div><div>No. of Writes: 160 200 87</div><div><br></div><div> Block Size: 65536b+ 131072b+</div><div> No. of Reads: 0 0</div><div>No. of Writes: 59 38</div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 2198 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 18 RELEASEDIR</div><div> 0.00 39.75 us 22.00 us 59.00 us 4 READDIR</div><div> 0.01 63.12 us 3.00 us 143.00 us 8 OPENDIR</div><div> 0.01 108.83 us 27.00 us 194.00 us 6 GETXATTR</div><div> 0.11 58.07 us 28.00 us 124.00 us 170 STAT</div><div> 0.54 113.57 us 46.00 us 258.00 us 440 SETXATTR</div><div> 0.79 97.28 us 23.00 us 224.00 us 745 STATFS</div><div> 1.37 57.40 us 12.00 us 428.00 us 2198 FLUSH</div><div> 3.70 77.12 us 15.00 us 322.00 us 4420 FINODELK</div><div> 3.94 68.70 us 14.00 us 259.00 us 5278 ENTRYLK</div><div> 4.98 205.68 us 70.00 us 2874.00 us 2229 WRITE</div><div> 5.15 1077.38 us 202.00 us 112584.00 us 440 MKDIR</div><div> 5.27 110.26 us 33.00 us 5589.00 us 4397 REMOVEXATTR</div><div> 7.88 118.30 us 28.00 us 11471.00 us 6130 SETATTR</div><div> 9.23 190.97 us 33.00 us 107884.00 us 4450 FXATTROP</div><div> 16.06 672.52 us 112.00 us 177035.00 us 2199 CREATE</div><div> 20.24 80.67 us 11.00 us 454.00 us 23102 INODELK</div><div> 20.74 160.46 us 24.00 us 33476.00 us 11901 LOOKUP</div><div><br></div><div> Duration: 3007 seconds</div><div> Data Read: 0 bytes</div><div>Data Written: 24173066 bytes</div><div><br></div><div>-----------------------------------------------------------------------------------------------</div><div><u><br></u></div><div><b><u>Profile info of the volume when you copy dirs/files within glusterfs.</u></b></div><div><br></div><div>Time taken to copy (70 MB files/folders)</div><div><br></div><div>[root@GFSCLIENT01 testmount]# time cp -RPp copytogluster/data/ /mnt/testmount/copywithinglustervol/</div><div><br></div><div>real 37m50.407s</div><div>user 0m0.248s</div><div>sys 0m1.979s</div><div><br></div><div>[root@GFSNODE01 ~]# gluster volume profile gv1 info</div><div>Brick: GFSNODE01:/mnt/perfDisk/gv1</div><div><br></div><div><br></div><div>Interval 8 Stats:</div><div> Block Size: 64b+ 128b+ 256b+</div><div> No. of Reads: 0 0 0</div><div>No. of Writes: 11 5 11</div><div><br></div><div> Block Size: 512b+ 1024b+ 2048b+</div><div> No. of Reads: 0 0 0</div><div>No. of Writes: 14 18 13</div><div><br></div><div> Block Size: 4096b+ 8192b+ 16384b+</div><div> No. of Reads: 0 0 0</div><div>No. of Writes: 11 6 3</div><div><br></div><div> Block Size: 32768b+ 65536b+</div><div> No. of Reads: 0 1</div><div>No. of Writes: 1 1</div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 94 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 4 RELEASEDIR</div><div> 0.03 37.00 us 37.00 us 37.00 us 1 FSTAT</div><div> 0.04 22.50 us 12.00 us 33.00 us 2 READDIR</div><div> 0.05 20.67 us 17.00 us 28.00 us 3 STAT</div><div> 0.06 77.00 us 77.00 us 77.00 us 1 READ</div><div> 0.07 41.50 us 14.00 us 69.00 us 2 GETXATTR</div><div> 0.10 30.25 us 2.00 us 53.00 us 4 OPENDIR</div><div> 0.13 51.67 us 42.00 us 60.00 us 3 SETXATTR</div><div> 0.35 139.67 us 127.00 us 152.00 us 3 MKDIR</div><div> 0.59 25.46 us 16.00 us 40.00 us 28 STATFS</div><div> 1.48 18.90 us 11.00 us 38.00 us 94 FLUSH</div><div> 3.87 24.00 us 13.00 us 43.00 us 194 ENTRYLK</div><div> 4.00 25.57 us 16.00 us 55.00 us 188 FINODELK</div><div> 5.65 72.37 us 57.00 us 177.00 us 94 WRITE</div><div> 7.38 47.22 us 38.00 us 60.00 us 188 REMOVEXATTR</div><div> 8.28 50.55 us 27.00 us 108.00 us 197 SETATTR</div><div> 8.50 54.41 us 37.00 us 112.00 us 188 FXATTROP</div><div> 13.04 166.86 us 78.00 us 1050.00 us 94 CREATE</div><div> 17.43 26.27 us 11.00 us 85.00 us 798 INODELK</div><div> 28.95 68.04 us 14.00 us 233.00 us 512 LOOKUP</div><div><br></div><div> Duration: 29 seconds</div><div> Data Read: 78602 bytes</div><div>Data Written: 365315 bytes</div><div><br></div><div>Interval 28 Stats:</div><div> Block Size: 4b+ 32b+ 64b+</div><div> No. of Reads: 0 0 0</div><div>No. of Writes: 1 5 28</div><div><br></div><div> Block Size: 128b+ 256b+ 512b+</div><div> No. of Reads: 0 0 0</div><div>No. of Writes: 59 164 305</div><div><br></div><div> Block Size: 1024b+ 2048b+ 4096b+</div><div> No. of Reads: 0 0 1</div><div>No. of Writes: 232 171 165</div><div><br></div><div> Block Size: 8192b+ 16384b+ 32768b+</div><div> No. of Reads: 0 0 2</div><div>No. of Writes: 117 96 39</div><div><br></div><div> Block Size: 65536b+ 131072b+</div><div> No. of Reads: 11 8</div><div>No. of Writes: 24 22</div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 1406 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 265 RELEASEDIR</div><div> 0.00 22.50 us 13.00 us 32.00 us 2 READDIR</div><div> 0.00 30.50 us 19.00 us 42.00 us 2 GETXATTR</div><div> 0.02 35.12 us 28.00 us 45.00 us 16 FSTAT</div><div> 0.06 75.86 us 27.00 us 232.00 us 22 READ</div><div> 0.08 21.99 us 15.00 us 37.00 us 102 STAT</div><div> 0.31 32.43 us 2.00 us 55.00 us 265 OPENDIR</div><div> 0.43 44.88 us 33.00 us 69.00 us 264 SETXATTR</div><div> 0.54 24.82 us 15.00 us 91.00 us 598 STATFS</div><div> 0.84 16.54 us 10.00 us 74.00 us 1406 FLUSH</div><div> 1.89 199.11 us 99.00 us 6678.00 us 263 MKDIR</div><div> 2.39 23.27 us 14.00 us 91.00 us 2840 FINODELK</div><div> 2.58 21.39 us 11.00 us 62.00 us 3339 ENTRYLK</div><div> 2.83 288.06 us 25.00 us 2315.00 us 272 READDIRP</div><div> 3.93 76.22 us 54.00 us 355.00 us 1428 WRITE</div><div> 6.05 59.59 us 36.00 us 42015.00 us 2812 REMOVEXATTR</div><div> 6.34 61.60 us 24.00 us 28264.00 us 2850 FXATTROP</div><div> 6.40 45.78 us 25.00 us 146.00 us 3867 SETATTR</div><div> 10.96 215.75 us 79.00 us 14633.00 us 1406 CREATE</div><div> 12.49 23.78 us 10.00 us 100.00 us 14545 INODELK</div><div> 41.85 71.82 us 14.00 us 55712.00 us 16131 LOOKUP</div><div><br></div><div> Duration: 598 seconds</div><div> Data Read: 2150643 bytes</div><div>Data Written: 12210039 bytes</div><div><br></div><div>Do let me know if you need any other details</div></div><div><br></div><div>Thanks</div><div>Srikanth</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Dec 11, 2015 at 4:15 PM, Anuradha Talur <span dir="ltr"><<a href="mailto:atalur@redhat.com" target="_blank">atalur@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Response inline.<br>
<br>
----- Original Message -----<br>
> From: "Srikanth Mampilakal" <<a href="mailto:shrikanth1980@gmail.com">shrikanth1980@gmail.com</a>><br>
> To: <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
> Sent: Thursday, December 10, 2015 7:59:04 PM<br>
> Subject: Re: [Gluster-users] Gluster - Performance issue while copying bulk files/folders<br>
><br>
><br>
><br>
> Hi members,<br>
><br>
> Really appreciate if you can share your thoughts or any feedback for<br>
> resolving the slow copy issue<br>
><br>
> Regards<br>
> Srikanth<br>
> On 10-Dec-2015 2:12 AM, "Srikanth Mampilakal" < <a href="mailto:srikanth.mampilakal@gmail.com">srikanth.mampilakal@gmail.com</a><br>
> > wrote:<br>
><br>
><br>
><br>
> Hi,<br>
><br>
><br>
> I have production gluster file service used as a shared storage where the<br>
> content management system uses it as document root. I have run in to a<br>
> performance issue with the gluster/fuse client.<br>
><br>
> Looking for your thoughts and experience in resolving Gluster performance<br>
> issues:<br>
><br>
> Gluster Infrastructure<br>
><br>
> Gluster version :GlusterFS 3.7.6<br>
><br>
> 2 gluster nodes of the same config below<br>
><br>
> Redhat EL7.0-64<br>
> Memory : 4GB<br>
> Processor : 2 x 2.0 Ghz<br>
> Network : 100 Mbps<br>
> File Storage Volume : NETAPP Storage LUN with 2.0 IOPS/GB<br>
><br>
> Gluster Volume information:<br>
><br>
> [root@GlusterFileServe1 ~]# gluster volume info<br>
><br>
> Volume Name: prodcmsroot<br>
> Type: Replicate<br>
> Volume ID: f1284bf0-1939-46f9-a672-a7716e362947<br>
> Status: Started<br>
> Number of Bricks: 1 x 2 = 2<br>
> Transport-type: tcp<br>
> Bricks:<br>
> Brick1: Server1:/glusterfs/brick1/prodcmsroot<br>
> Brick2: Server2:/glusterfs/brick1/prodcmsroot<br>
> Options Reconfigured:<br>
> performance.io-thread-count: 64<br>
> performance.cache-size: 1073741824<br>
> performance.readdir-ahead: on<br>
> performance.write-behind-window-size: 524288<br>
><br>
> [root@GlusterFileServe1 ~]#<br>
><br>
> The replication between Gluster node are quick and consistent.<br>
><br>
> The apache webservers are accessing the Gluster volume using native gluster<br>
> fuse client and located in the same VLAN as the Gluster Server.<br>
><br>
> GlusterFileServe1:/prodcmsroot /mnt/glusterfs glusterfs<br>
> direct-io-mode=disable,defaults,_netdev 0 0<br>
><br>
> The server utilization (memory,cpu,network and disk 1/0) is relatively low<br>
><br>
> I am experiencing very slow performance while copying multiple file/folders<br>
> (approx 75 MB) and it takes atleast approx 35 min. Even copy a folder (with<br>
> multiple files/subfolders) within the Gluster volume take the same time.<br>
><br>
> However, if I do dd to check the copy speed, I get the below result.<br>
><br>
> [root@ClientServer ~]# time sh -c "dd if=/dev/zero of=/mnt/testmount/test.tmp<br>
> bs=4k count=20000 && sync"<br>
> 20000+0 records in<br>
> 20000+0 records out<br>
> 81920000 bytes (82 MB) copied, 17.1357 s, 4.8 MB/s<br>
><br>
> real 0m17.337s<br>
> user 0m0.031s<br>
> sys 0m0.317s<br>
><br>
><br>
> Anyone experience the same kind of performance issue, please let me know your<br>
> thoughts.<br>
><br>
Hi Srikanth,<br>
<br>
Could you please provide the following information so that the reason behind<br>
slow copy can be deduced?<br>
<br>
1) Profile info of the volume when you copy dirs/files into glusterfs.<br>
2) Profile info of the volume when you copy dirs/files within glusterfs.<br>
<br>
The following steps should help you with profile info:<br>
1) gluster volume profile <VOLNAME> start<br>
2) Perform copy operations<br>
3) gluster volume profile <VOLNAME> info (you will get stats of the FOPs at this point)<br>
4) gluster volume profile <VOLNAME> stop<br>
<br>
Please follow steps 1 through 4 twice. Once for copy into glusterfs and once for copy<br>
within.<br>
<br>
> Cheers<br>
> Srikanth<br>
><br>
> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
><br>
> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Thanks,<br>
Anuradha.<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Cheers<br>Shrikanth</div>
</div>