<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head><meta content="text/html;charset=UTF-8" http-equiv="Content-Type"></head><body ><div style='font-size:10pt;'><div><div style="font-size:10pt;">Hi Amukher, <br><br> Even after upgrade to 3.7 small files transfer rate is slow.<br><br>Below is the volume info. <br><br>Volume Name: integvol1<br>Type: Replicate<br>Volume ID: 31793ba4-eeca-462a-a0cd-9adfb281225b<br>Status: Started<br>Number of Bricks: 1 x 2 = 2<br>Transport-type: tcp<br>Bricks:<br>Brick1: integ-gluster1:/srv/sdb2/brick4<br>Brick2: integ-gluster2:/srv/sdb2/brick4<br>Options Reconfigured:<br>server.event-threads: 30<br>client.event-threads: 30<br>----<br><br> I understand that for replication it would take some more time, but here its taking more time. <br><br><br>Time taken for git clone in non gluster directory = 25 sec<br><br>Time taken for git clone in gluster directory = 14 minutes<br><br>Its a huge difference. Plz let me know any other tuning parameters need to be done. <br><br><br>Regards,<br>Kamal<br><br><br> <br><br><br><br><br><br><br><br><div style=""><blockquote style="border-left: 1px solid #cccccc; padding-left: 6px; margin:0 0 0 5px"><div><div style="font-size:10pt;"><div><blockquote style="border-left: 1px solid #0000ff;padding-left: 6px; margin: 0 0 0 5px"><div><div style="font-size:10pt;"><div><blockquote style="border-left: 1px solid #cccccc; padding-left: 6px; margin:0 0 0 5px"><div><div style="font-size:10pt;"><div><blockquote><div><br></div><div>============ Forwarded Message ============<br></div><div>From : <a subj="" mailid="bturner%40redhat.com" href="mailto:bturner@redhat.com" target="_blank">bturner@redhat.com</a> <br></div><div>To : <a subj="" mailid="gjprabu%40zohocorp.com" href="mailto:gjprabu@zohocorp.com" target="_blank">gjprabu@zohocorp.com</a> <br></div><div>Cc : <a subj="" mailid="gluster-users%40gluster.org" href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>,<a subj="" mailid="amukherj%40redhat.com" href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a> <br></div><div>Date : Thu, 30 Apr 2015 17:14:00 +0530 <br></div><div>Subject : Re: [Gluster-users] client is terrible with large amount of small files<br></div><div>============ Forward Message ============<br></div><div><br></div><div>----- Original Message ----- <br></div><div>> From: "Atin Mukherjee" <<a subj="" mailid="amukherj%40redhat.com" href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>> <br></div><div>> To: "gjprabu" <<a subj="" mailid="gjprabu%40zohocorp.com" href="mailto:gjprabu@zohocorp.com" target="_blank">gjprabu@zohocorp.com</a>> <br></div><div>> Cc: "Ben Turner" <<a subj="" mailid="bturner%40redhat.com" href="mailto:bturner@redhat.com" target="_blank">bturner@redhat.com</a>>, <a subj="" mailid="gluster-users%40gluster.org" href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a> <br></div><div>> Sent: Thursday, April 30, 2015 7:37:19 AM <br></div><div>> Subject: Re: [Gluster-users] client is terrible with large amount of small files <br></div><div>> <br></div><div>> <br></div><div>> On 04/30/2015 03:09 PM, gjprabu wrote: <br></div><div>> > Hi Amukher, <br></div><div>> > <br></div><div>> > How to resolve this issue, till we need to wait for 3.7 release <br></div><div>> > or any work around is there. <br></div><div>> You will have to as this feature is in for 3.7. <br></div><div><br></div><div>My apologies, I didn't realize that MT epoll didn't land in 3.6. If you want to test it out there is an alpha build available: <br></div><div><br></div><div><a href="http://download.gluster.org/pub/gluster/glusterfs/nightly/glusterfs-3.7/epel-6-x86_64" target="_blank">http://download.gluster.org/pub/gluster/glusterfs/nightly/glusterfs-3.7/epel-6-x86_64</a> <br></div><div><br></div><div>I wouldn't run this in production until 3.7 is released though. Again sorry for the confusion. <br></div><div><br></div><div>-b <br></div><div><br></div><div>> > <br></div><div>> > RegardsPrabu <br></div><div>> > <br></div><div>> > <br></div><div>> > <br></div><div>> > <br></div><div>> > <br></div><div>> > ---- On Thu, 30 Apr 2015 14:49:46 +0530 Atin <br></div><div>> > Mukherjee&lt;<a subj="" mailid="amukherj%40redhat.com" href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>&gt; wrote ---- <br></div><div>> > <br></div><div>> > <br></div><div>> > <br></div><div>> > On 04/30/2015 02:32 PM, gjprabu wrote: <br></div><div>> > &gt; Hi bturner, <br></div><div>> > &gt; <br></div><div>> > &gt; <br></div><div>> > &gt; I am getting below error while adding server.event <br></div><div>> > &gt; <br></div><div>> > &gt; gluster v set integvol server.event-threads 3 <br></div><div>> > &gt; volume set: failed: option : server.event-threads does not exist <br></div><div>> > &gt; Did you mean server.gid-timeout or ...manage-gids? <br></div><div>> > This option is not available in 3.6, its going to come in 3.7 <br></div><div>> > <br></div><div>> > &gt; <br></div><div>> > &gt; <br></div><div>> > &gt; Glusterfs version has been upgraded to 3.6.3 <br></div><div>> > &gt; Also os kernel upgraded to 6.6 kernel <br></div><div>> > &gt; Yes two brick are running in KVM and one is physical machine and we <br></div><div>> > are not using thinp. <br></div><div>> > &gt; <br></div><div>> > &gt; Regards <br></div><div>> > &gt; G.J <br></div><div>> > &gt; <br></div><div>> > &gt; <br></div><div>> > &gt; <br></div><div>> > &gt; <br></div><div>> > &gt; <br></div><div>> > &gt; ---- On Thu, 30 Apr 2015 00:37:44 +0530 Ben <br></div><div>> > Turner&amp;lt;<a subj="" mailid="bturner%40redhat.com" href="mailto:bturner@redhat.com" target="_blank">bturner@redhat.com</a>&amp;gt; wrote ---- <br></div><div>> > &gt; <br></div><div>> > &gt; ----- Original Message ----- <br></div><div>> > &gt; &amp;gt; From: "gjprabu" &amp;lt;<a subj="" mailid="gjprabu%40zohocorp.com" href="mailto:gjprabu@zohocorp.com" target="_blank">gjprabu@zohocorp.com</a>&amp;gt; <br></div><div>> > &gt; &amp;gt; To: "A Ghoshal" &amp;lt;<a subj="" mailid="a.ghoshal%40tcs.com" href="mailto:a.ghoshal@tcs.com" target="_blank">a.ghoshal@tcs.com</a>&amp;gt; <br></div><div>> > &gt; &amp;gt; Cc: <a subj="" mailid="gluster-users%40gluster.org" href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>, <br></div><div>> > <a subj="" mailid="gluster-users-bounces%40gluster.org" href="mailto:gluster-users-bounces@gluster.org" target="_blank">gluster-users-bounces@gluster.org</a> <br></div><div>> > &gt; &amp;gt; Sent: Wednesday, April 29, 2015 9:07:07 AM <br></div><div>> > &gt; &amp;gt; Subject: Re: [Gluster-users] client is terrible with large <br></div><div>> > amount of small files <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; Hi Ghoshal, <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; Please find the details below. <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; A) Glusterfs version <br></div><div>> > &gt; &amp;gt; glusterfs 3.6.2 <br></div><div>> > &gt; <br></div><div>> > &gt; Upgrade to 3.6.3 and set client.event-threads and server.event-threads <br></div><div>> > to at least 4. Here is a guide on tuning MT epoll: <br></div><div>> > &gt; <br></div><div>> > &gt; <br></div><div>> > <a href="https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/Small_File_Performance_Enhancements.html" target="_blank">https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/Small_File_Performance_Enhancements.html</a> <br></div><div>> > &gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; B) volume configuration (gluster v &amp;lt;volname&amp;gt; <br></div><div>> > info) <br></div><div>> > &gt; &amp;gt; gluster volume info <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; Volume Name: integvol <br></div><div>> > &gt; &amp;gt; Type: Replicate <br></div><div>> > &gt; &amp;gt; Volume ID: b8f3a19e-59bc-41dc-a55a-6423ec834492 <br></div><div>> > &gt; &amp;gt; Status: Started <br></div><div>> > &gt; &amp;gt; Number of Bricks: 1 x 3 = 3 <br></div><div>> > &gt; &amp;gt; Transport-type: tcp <br></div><div>> > &gt; &amp;gt; Bricks: <br></div><div>> > &gt; &amp;gt; Brick1: integ-gluster2:/srv/sdb1/brick <br></div><div>> > &gt; &amp;gt; Brick2: integ-gluster1:/srv/sdb1/brick <br></div><div>> > &gt; &amp;gt; Brick3: integ-gluster3:/srv/sdb1/brick <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; C) host linux version <br></div><div>> > &gt; &amp;gt; CentOS release 6.5 (Final) <br></div><div>> > &gt; <br></div><div>> > &gt; Are your bricks on LVM? Are you using thinp? If so update to the <br></div><div>> > latest kernel as thinp perf was really bad in 6.5 and early 6.6 kernels. <br></div><div>> > &gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; D) details about the kind of network you use to connect your <br></div><div>> > servers making <br></div><div>> > &gt; &amp;gt; up your storage pool. <br></div><div>> > &gt; &amp;gt; We are connecting LAN to LAN there is no special network <br></div><div>> > configuration done <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; Frome client we use to mount like below <br></div><div>> > &gt; &amp;gt; mount -t glusterfs gluster1:/integvol /mnt/gluster/ <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; Regards <br></div><div>> > &gt; &amp;gt; Prabu <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; ---- On Wed, 29 Apr 2015 17:58:16 +0530 A <br></div><div>> > Ghoshal&amp;lt;<a subj="" mailid="a.ghoshal%40tcs.com" href="mailto:a.ghoshal@tcs.com" target="_blank">a.ghoshal@tcs.com</a>&amp;gt; wrote <br></div><div>> > &gt; &amp;gt; ---- <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; Performance would largely depend upon setup. While I cannot <br></div><div>> > think of any <br></div><div>> > &gt; &amp;gt; setup that would cause write to be this slow, if would help <br></div><div>> > if you share the <br></div><div>> > &gt; &amp;gt; following details: <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; A) Glusterfs version <br></div><div>> > &gt; &amp;gt; B) volume configuration (gluster v &amp;lt;volname&amp;gt; <br></div><div>> > info) <br></div><div>> > &gt; &amp;gt; C) host linux version <br></div><div>> > &gt; &amp;gt; D) details about the kind of network you use to connect your <br></div><div>> > servers making <br></div><div>> > &gt; &amp;gt; up your storage pool. <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; Thanks, <br></div><div>> > &gt; &amp;gt; Anirban <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; From: gjprabu &amp;lt; <a subj="" mailid="gjprabu%40zohocorp.com" href="mailto:gjprabu@zohocorp.com" target="_blank">gjprabu@zohocorp.com</a> &amp;gt; <br></div><div>> > &gt; &amp;gt; To: &amp;lt; <a subj="" mailid="gluster-users%40gluster.org" href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a> &amp;gt; <br></div><div>> > &gt; &amp;gt; Date: 04/29/2015 05:52 PM <br></div><div>> > &gt; &amp;gt; Subject: Re: [Gluster-users] client is terrible with large <br></div><div>> > amount of small <br></div><div>> > &gt; &amp;gt; files <br></div><div>> > &gt; &amp;gt; Sent by: <a subj="" mailid="gluster-users-bounces%40gluster.org" href="mailto:gluster-users-bounces@gluster.org" target="_blank">gluster-users-bounces@gluster.org</a> <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; Hi Team, <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; If anybody know the solution please share us. <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; Regards <br></div><div>> > &gt; &amp;gt; Prabu <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; ---- On Tue, 28 Apr 2015 19:32:40 +0530 gjprabu &amp;lt; <br></div><div>> > <a subj="" mailid="gjprabu%40zohocorp.com" href="mailto:gjprabu@zohocorp.com" target="_blank">gjprabu@zohocorp.com</a> &amp;gt; <br></div><div>> > &gt; &amp;gt; wrote ---- <br></div><div>> > &gt; &amp;gt; Hi Team, <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; We are using glusterfs newly and testing data transfer part <br></div><div>> > in client using <br></div><div>> > &gt; &amp;gt; fuse.glusterfs file system but it is terrible with large <br></div><div>> > amount of small <br></div><div>> > &gt; &amp;gt; files (Large amount of small file 150MB of size it's writing <br></div><div>> > around 18min). <br></div><div>> > &gt; &amp;gt; I can able copy small files and syncing between the server <br></div><div>> > brick are working <br></div><div>> > &gt; &amp;gt; fine but it is terrible with large amount of small files. <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; if anybody please share the solution for the above issue. <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; Regards <br></div><div>> > &gt; &amp;gt; Prabu <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; _______________________________________________ <br></div><div>> > &gt; &amp;gt; Gluster-users mailing list <br></div><div>> > &gt; &amp;gt; <a subj="" mailid="Gluster-users%40gluster.org" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <br></div><div>> > &gt; &amp;gt; <a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a> <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; _______________________________________________ <br></div><div>> > &gt; &amp;gt; Gluster-users mailing list <br></div><div>> > &gt; &amp;gt; <a subj="" mailid="Gluster-users%40gluster.org" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <br></div><div>> > &gt; &amp;gt; <a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a> <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; =====-----=====-----===== <br></div><div>> > &gt; &amp;gt; Notice: The information contained in this e-mail <br></div><div>> > &gt; &amp;gt; message and/or attachments to it may contain <br></div><div>> > &gt; &amp;gt; confidential or privileged information. If you are <br></div><div>> > &gt; &amp;gt; not the intended recipient, any dissemination, use, <br></div><div>> > &gt; &amp;gt; review, distribution, printing or copying of the <br></div><div>> > &gt; &amp;gt; information contained in this e-mail message <br></div><div>> > &gt; &amp;gt; and/or attachments to it are strictly prohibited. If <br></div><div>> > &gt; &amp;gt; you have received this communication in error, <br></div><div>> > &gt; &amp;gt; please notify us by reply e-mail or telephone and <br></div><div>> > &gt; &amp;gt; immediately and permanently delete the message <br></div><div>> > &gt; &amp;gt; and any attachments. Thank you <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; <br></div><div>> > &gt; &amp;gt; _______________________________________________ <br></div><div>> > &gt; &amp;gt; Gluster-users mailing list <br></div><div>> > &gt; &amp;gt; <a subj="" mailid="Gluster-users%40gluster.org" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <br></div><div>> > &gt; &amp;gt; <a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a> <br></div><div>> > &gt; <br></div><div>> > &gt; <br></div><div>> > &gt; <br></div><div>> > &gt; <br></div><div>> > &gt; <br></div><div>> > &gt; <br></div><div>> > &gt; <br></div><div>> > &gt; <br></div><div>> > &gt; _______________________________________________ <br></div><div>> > &gt; Gluster-users mailing list <br></div><div>> > &gt; <a subj="" mailid="Gluster-users%40gluster.org" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <br></div><div>> > &gt; <a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a> <br></div><div>> > &gt; <br></div><div>> > <br></div><div>> > <br></div><div>> <br></div><div>> -- <br></div><div>> ~Atin <br></div><div>> <br></div></blockquote></div></div></div></blockquote><div><br></div></div></div></div></blockquote><div><br></div></div><div><br></div></div></div></blockquote><div><br></div></div><div><br></div></div></div></div></body></html>