<p dir="ltr">Hi Ben,</p>
<p dir="ltr">Regarding <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1250241which">https://bugzilla.redhat.com/show_bug.cgi?id=1250241which</a> does look like a serious regression for small file performance, do you know which versions are affected, or is there a way to find out?</p>
<p dir="ltr">Also the patch didn't make it: do you have visibility on whether another patch is likely to land soon?</p>
<p dir="ltr">If not I may try the version before the regression was introduced...</p>
<p dir="ltr">Thanks,<br>
Thibault.</p>
<div class="gmail_quote">On 14 Sep 2015 4:22 pm, "Ben Turner" <<a href="mailto:bturner@redhat.com">bturner@redhat.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">----- Original Message -----<br>
> From: "Diego Remolina" <<a href="mailto:dijuremo@gmail.com">dijuremo@gmail.com</a>><br>
> To: "Alex Crow" <<a href="mailto:acrow@integrafin.co.uk">acrow@integrafin.co.uk</a>><br>
> Cc: <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
> Sent: Monday, September 14, 2015 9:26:17 AM<br>
> Subject: Re: [Gluster-users] Very slow roaming profiles on top of glusterfs<br>
><br>
> Hi Alex,<br>
><br>
> Thanks for the reply, I was aware of the performance issues with small<br>
> files, but never expected an order of magnitude slower. I understand<br>
> some improvements were made to 3.7.x to help with low small file<br>
> performance, however I did not see any big changes after upgrading<br>
> from 3.6.x to 3.7.x.<br>
><br>
> <a href="http://www.gluster.org/community/documentation/index.php/Features/Feature_Smallfile_Perf" rel="noreferrer" target="_blank">http://www.gluster.org/community/documentation/index.php/Features/Feature_Smallfile_Perf</a><br>
><br>
> And the ssd metadata support feature seems to have not had any changes<br>
> since September 2014:<br>
><br>
> <a href="https://forge.gluster.org/gluster-meta-data-on-ssd" rel="noreferrer" target="_blank">https://forge.gluster.org/gluster-meta-data-on-ssd</a><br>
><br>
> Am I just totally out of luck with gluster for now?<br>
<br>
Are you using glusterFS mounts or SMB mounts? As for SMB mounts we are working VERY hard to improve metadata / smallfile performance but as it sits right now we are limited by the number of lookup / stat calls that are issued. When we can reduce the number of lookups and prefetch the xattrs that SMB / windows needs(I am working on the stat prefetch but don't have a testable solution yet) I expect to see a vast perf improvement but I don't have an ETA for you.<br>
<br>
On the glusterFS side I see ~300% improvement in smallfile create performance between 3.6 and 3.7. Try setting:<br>
<br>
gluster volume set testvol server.event-threads 4<br>
gluster volume set testvol client.event-threads 4<br>
gluster volume set testvol cluster.lookup-optimize on<br>
<br>
Unfortunately WRT to metadata operations a fix went in that has negatively affected performance:<br>
<br>
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1250241" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1250241</a><br>
<br>
I used to see about 25k metatdata operations per second, now I am only seeking 6k. It looks like there is a patch but I don't know if the fix will get us back to the 25k OPs per second, maybe Pranith can comment on expectations for:<br>
<br>
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1250803" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1250803</a><br>
<br>
To summarize:<br>
<br>
SMB - no ETA for improvement<br>
GlusterFS smallfile create - 300% increase in my env between 3.6 and 3.7<br>
GlusterFS metadata - BZ is in POST(patch is submitted) but I am not sure on the ETA of the fix and if the fix will get back to what I was seeing in 3.6<br>
<br>
Hope this helps.<br>
<br>
-b<br>
<br>
><br>
> Diego<br>
><br>
> On Mon, Sep 14, 2015 at 8:37 AM, Alex Crow <<a href="mailto:acrow@integrafin.co.uk">acrow@integrafin.co.uk</a>> wrote:<br>
> > Hi Diego,<br>
> ><br>
> > I think it's the overhead of fstat() calls. Gluster keeps its metadata on<br>
> > the bricks themselves, and this has to be looked up for every file access.<br>
> > For big files this is not an issue as it only happens once, but when<br>
> > accessing lots of small files this overhead rapidly builds up, the smaller<br>
> > the file the worse the issue. Profiles do have hundreds of very small<br>
> > files!<br>
> ><br>
> > I was looking to use GlusterFS for generic file sharing as well, but I<br>
> > noticed the same issue while testing backups from a GlusterFS volume. On<br>
> > one<br>
> > vol (scanned 4-bit greyscale images and small PDFs) backups were taking<br>
> > over<br>
> > 16 hours whereas with a traditional FS they were completing in just over 1<br>
> > hour.<br>
> ><br>
> > It may be worth trying out one of the distributed filesystems that use a<br>
> > separate in-memory metadata server. I've tried LizardFS and MooseFS and<br>
> > they<br>
> > are both much faster than GlusterFS for small files, although large-file<br>
> > sequential performance is not as good (but still plenty for a Samba<br>
> > server).<br>
> ><br>
> > Alex<br>
> ><br>
> ><br>
> > On 14/09/15 13:21, Diego Remolina wrote:<br>
> >><br>
> >> Bump...<br>
> >><br>
> >> Anybody has any clues as to how I can try and identify the cause of<br>
> >> the slowness?<br>
> >><br>
> >> Diego<br>
> >><br>
> >> On Wed, Sep 9, 2015 at 7:42 PM, Diego Remolina <<a href="mailto:dijuremo@gmail.com">dijuremo@gmail.com</a>> wrote:<br>
> >>><br>
> >>> Hi,<br>
> >>><br>
> >>> I am running two glusterfs servers as replicas. I have a 3rd server<br>
> >>> which provides quorum. Since gluster was introduced, we have had an<br>
> >>> issue where windows roaming profiles are extremely slow. The initial<br>
> >>> setup was done on 3.6.x and since 3.7.x has small file performance<br>
> >>> improvements, I upgraded to 3.7.3, but that has not helped.<br>
> >>><br>
> >>> It seems that for some reason gluster is very slow when dealing with<br>
> >>> lots of small files. I am not sure how to really troubleshoot this via<br>
> >>> samba, but I have come up with other tests that produce rather<br>
> >>> disconcerting results as shown below.<br>
> >>><br>
> >>> If I run directly on the brick:<br>
> >>> [root@ysmha01 /]# time ( find<br>
> >>> /bricks/hdds/brick/home/jgibbs/.winprofile.V2 -type f > /dev/null )<br>
> >>> real 0m3.683s<br>
> >>> user 0m0.042s<br>
> >>> sys 0m0.154s<br>
> >>><br>
> >>> Now running on the gluster volume mounted via fuse:<br>
> >>> [root@ysmha01 /]# mount | grep export<br>
> >>> 10.0.1.6:/export on /export type fuse.glusterfs<br>
> >>> (rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072)<br>
> >>><br>
> >>> [root@ysmha01 /]# time ( find /export/home/jgibbs/.winprofile.V2 -type<br>
> >>> f > /dev/null )<br>
> >>> real 0m57.812s<br>
> >>> user 0m0.118s<br>
> >>> sys 0m0.374s<br>
> >>><br>
> >>> In general, the time to run the command on this particular user can be<br>
> >>> up to 2 minutes. If I run the command on the brick first, then it<br>
> >>> seems the time to run on the mounted gluster volume is lower like in<br>
> >>> the example above. I assume some caching in preserved.<br>
> >>><br>
> >>> This particular user has 13,216 files in his roaming profile, which<br>
> >>> adds up to about 452MB of data.<br>
> >>><br>
> >>> The server performance over samba for copying big files (both read and<br>
> >>> write) is great, I can almost max out the gigabit connections on the<br>
> >>> desktops.<br>
> >>><br>
> >>> Reading from samba share on the server and writing to local drive:<br>
> >>> 111MB/s (Copying a 650MB iso file)<br>
> >>> Reading from local drive and writing to server samba share: 94MB/s<br>
> >>> (Copying a 3.2GB ISO file)<br>
> >>><br>
> >>> The servers are connected to the network with 10Gbit adapters and also<br>
> >>> use separate adapters; one 10 Gbit adapter is used for services, and<br>
> >>> other for the backend storage communication.<br>
> >>><br>
> >>> The servers have hardware raid controllers and the samba shares are on<br>
> >>> top of an Areca ARC-1882 controller, with a volume made out of 12 2TB<br>
> >>> drives in raid 6.<br>
> >>><br>
> >>> If you can provide any steps to better troubleshoot this problem and<br>
> >>> fix the issue, I will really appreciate it.<br>
> >>><br>
> >>> Diego<br>
> >>><br>
> >>> Further details about the machines below:<br>
> >>><br>
> >>> [root@ysmha01 /]# cat /etc/redhat-release<br>
> >>> CentOS Linux release 7.1.1503 (Core)<br>
> >>><br>
> >>> [root@ysmha01 /]# gluster volume info export<br>
> >>> Volume Name: export<br>
> >>> Type: Replicate<br>
> >>> Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff<br>
> >>> Status: Started<br>
> >>> Number of Bricks: 1 x 2 = 2<br>
> >>> Transport-type: tcp<br>
> >>> Bricks:<br>
> >>> Brick1: 10.0.1.7:/bricks/hdds/brick<br>
> >>> Brick2: 10.0.1.6:/bricks/hdds/brick<br>
> >>> Options Reconfigured:<br>
> >>> performance.io-cache: on<br>
> >>> performance.io-thread-count: 64<br>
> >>> nfs.disable: on<br>
> >>> cluster.server-quorum-type: server<br>
> >>> performance.cache-size: 1024MB<br>
> >>> server.allow-insecure: on<br>
> >>> cluster.server-quorum-ratio: 51%<br>
> >>><br>
> >>> Each server has dual Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz with<br>
> >>> 32GB of memory.<br>
> >><br>
> >> _______________________________________________<br>
> >> Gluster-users mailing list<br>
> >> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> >> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> ><br>
> ><br>
> > --<br>
> > This message is intended only for the addressee and may contain<br>
> > confidential information. Unless you are that person, you may not<br>
> > disclose its contents or use it in any way and are requested to delete<br>
> > the message along with any attachments and notify us immediately.<br>
> > "Transact" is operated by Integrated Financial Arrangements plc. 29<br>
> > Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608<br>
> > 5300. (Registered office: as above; Registered in England and Wales<br>
> > under number: 3727592). Authorised and regulated by the Financial<br>
> > Conduct Authority (entered on the Financial Services Register; no. 190856).<br>
> ><br>
> > .<br>
> ><br>
> > _______________________________________________<br>
> > Gluster-users mailing list<br>
> > <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
><br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div>