<div dir="ltr">Thank you Ben,<div><br></div><div>I'm already running on 7, I've set the options you recommended and indeed file creation is faster now.</div><div><br></div><div>I double-checked my config and found out that the filesystem of the brick on the arbiter node doesn't support ACLs: underlying fs is ext4 without acl mount option, while the other bricks are XFS ( where it's always enabled). Do all the bricks need to support ACLs?</div><div><br></div><div>To keep things simple I suppose it makes sense to remove ACLs as it's not strictly needed for my setup, so I ran some tests and I can now confirm that I don't see the self-heals when the volume isn't mounted with --acl. I don't have exact numbers but I have the impression that syncing is faster as well. </div></div><div class="gmail_extra"><br><div class="gmail_quote">On 16 November 2015 at 19:02, Ben Turner <span dir="ltr"><<a href="mailto:bturner@redhat.com" target="_blank">bturner@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">----- Original Message -----<br>
> From: "Tiemen Ruiten" <<a href="mailto:t.ruiten@rdmedia.com">t.ruiten@rdmedia.com</a>><br>
</span><span class="">> To: "Ben Turner" <<a href="mailto:bturner@redhat.com">bturner@redhat.com</a>><br>
> Cc: "gluster-users" <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
> Sent: Monday, November 16, 2015 5:00:20 AM<br>
> Subject: Re: [Gluster-users] rsync to gluster mount: self-heal and bad performance<br>
><br>
</span><span class="">> Hello Ben,<br>
><br>
> Thank you for your answer. I don't see the same errors when just creating a<br>
> number of files, eg. touch test{0000..9999}. Performance is not great, but<br>
> after a few minutes it finishes successfully.<br>
<br>
</span>We have made alot of small file perf enhancements, try the following:<br>
<br>
-Run on RHEL 7, I have seen a good improvement on el7 over el6.<br>
-Set lookup optimize on<br>
-Set client and server event threads to 4<br>
-There is a metadata perf regression that could be affecting your rsync as well, keep an eye on - <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1250803" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1250803</a> for the fix.<br>
<span class=""><br>
<br>
><br>
> I'm running rsync through lsyncd, the options are:<br>
><br>
> /usr/bin/rsync --delete --ignore-errors -zslt -r $source $destination<br>
><br>
> I'm running it over a LAN network, between two VMs. The volume is indeed<br>
> mounted with --acl, but on the directory I'm syncing to I haven't set them<br>
> explicitly:<br>
<br>
</span>Do you need ACLs? If not can you try without that option? I am wondering if there is a bug with ACLs that could be causing the self heals to happen. If we don't see it without ACLs that can give us somewhere to look at.<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
> [tiemen@iron2 test]$ getfacl stg/<br>
> # file: stg/<br>
> # owner: root<br>
> # group: rdcompany<br>
> # flags: -s-<br>
> user::rwx<br>
> group::rwx<br>
> other::r-x<br>
><br>
> Volume options:<br>
><br>
> [tiemen@iron2 test]$ sudo gluster volume info lpxassets<br>
><br>
> Volume Name: lpxassets<br>
> Type: Replicate<br>
> Volume ID: fea00430-63b1-4a4e-bc38-b74d3732acf4<br>
> Status: Started<br>
> Number of Bricks: 1 x 3 = 3<br>
> Transport-type: tcp<br>
> Bricks:<br>
> Brick1: iron2:/data/brick/lpxassets<br>
> Brick2: cobalt2:/data/brick/lpxassets<br>
> Brick3: arbiter:/data/arbiter/lpxassets<br>
> Options Reconfigured:<br>
> nfs.disable: on<br>
> performance.readdir-ahead: on<br>
> cluster.quorum-type: auto<br>
> cluster.enable-shared-storage: enable<br>
> nfs-ganesha: disable<br>
><br>
> Any other info I could provide?<br>
><br>
><br>
> On 15 November 2015 at 18:11, Ben Turner <<a href="mailto:bturner@redhat.com">bturner@redhat.com</a>> wrote:<br>
><br>
> > ----- Original Message -----<br>
> > > From: "Tiemen Ruiten" <<a href="mailto:t.ruiten@rdmedia.com">t.ruiten@rdmedia.com</a>><br>
> > > To: "gluster-users" <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
> > > Sent: Sunday, November 15, 2015 5:22:08 AM<br>
> > > Subject: Re: [Gluster-users] rsync to gluster mount: self-heal and bad<br>
> > performance<br>
> > ><br>
> > > Any other suggestions?<br>
> ><br>
> > You are correct, rsync should not cause self heal on every file. It makes<br>
> > me think that Ernie is correct and that something isn't correct. If you<br>
> > just create a bunch of files out side of rsync do you see the same<br>
> > behavior? What rsync command are you running, where are you syncing the<br>
> > data from? I see you have the acl mount option, are you using ACLs?<br>
> ><br>
> > -b<br>
> ><br>
> ><br>
> > ><br>
> > > On 13 November 2015 at 09:56, Tiemen Ruiten < <a href="mailto:t.ruiten@rdmedia.com">t.ruiten@rdmedia.com</a> ><br>
> > wrote:<br>
> > ><br>
> > ><br>
> > ><br>
> > > Hello Ernie, list,<br>
> > ><br>
> > > No, that's not the case. The volume is mounted through glusterfs-fuse -<br>
> > on<br>
> > > the same server running one of the bricks. The fstab:<br>
> > ><br>
> > > # /etc/fstab<br>
> > > # Created by anaconda on Tue Aug 18 18:10:49 2015<br>
> > > #<br>
> > > # Accessible filesystems, by reference, are maintained under '/dev/disk'<br>
> > > # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more<br>
> > info<br>
> > > #<br>
> > > UUID=56778fed-bf3f-435e-8c32-edaa8c707f29 / xfs defaults 0 0<br>
> > > UUID=a44e32ed-cfbe-4ba0-896f-1efff9397ba1 /boot xfs defaults 0 0<br>
> > > UUID=a344d2bc-266d-4905-85b1-fbb7fe927659 swap swap defaults 0 0<br>
> > > /dev/vdb1 /data/brick xfs defaults 1 2<br>
> > > iron2:/lpxassets /mnt/lpxassets glusterfs _netdev,acl 0 0<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > On 12 November 2015 at 22:50, Ernie Dunbar < <a href="mailto:maillist@lightspeed.ca">maillist@lightspeed.ca</a> ><br>
> > wrote:<br>
> > ><br>
> > ><br>
> > > Hi Tiemen<br>
> > ><br>
> > > It sounds like you're trying to rsync files onto your Gluster server,<br>
> > rather<br>
> > > than to the Gluster filesystem. You want to copy these files into the<br>
> > > mounted filesystem (typically on some other system than the Gluster<br>
> > > servers), because Gluster is designed to handle it that way.<br>
> > ><br>
> > > I can't remember the nitty gritty details about why this is, but I've<br>
> > made<br>
> > > this mistake before as well. Hope that helps. :)<br>
> > ><br>
> > ><br>
> > > On 2015-11-12 11:31, Tiemen Ruiten wrote:<br>
> > ><br>
> > ><br>
> > ><br>
> > > Hello,<br>
> > ><br>
> > > While rsyncing to a directory mounted through glusterfs fuse,<br>
> > > performance is very bad and it appears every synced file generates a<br>
> > > (metadata) self-heal.<br>
> > ><br>
> > > The volume is mounted with option acl and acl's are set on a<br>
> > > subdirectory.<br>
> > ><br>
> > > Setup is as follows:<br>
> > ><br>
> > > Two Centos 7 VM's (KVM), with Gluster 3.7.6 and one physical CentOS 6<br>
> > > node, also Gluster 3.7.6. Physical node functions as arbiter. So it's<br>
> > > a replica 3 arbiter 1 volume. The bricks are LVM's with XFS<br>
> > > filesystem.<br>
> > ><br>
> > > While I don't think I should expect top performance for rsync on<br>
> > > Gluster, I wouldn't expect every file synced to trigger a self-heal.<br>
> > > Anything I can do to improve this? Should I file a bug?<br>
> > ><br>
> > > Another thing that looks related, I see a lot of these messages,<br>
> > > especially when doing IO:<br>
> > ><br>
> > > [2015-11-12 19:25:42.185904] I [dict.c:473:dict_get]<br>
> > ><br>
> > (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x121)<br>
> > > [0x7fdcc2d31161]<br>
> > ><br>
> > -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x242)<br>
> > > [0x7fdcc2b1b212] -->/lib64/libglusterfs.so.0(dict_get+0xac)<br>
> > > [0x7fdcd5e770cc] ) 0-dict: !this || key=system.posix_acl_default<br>
> > > [Invalid argument]<br>
> > ><br>
> > > --<br>
> > ><br>
> > > Tiemen Ruiten<br>
> > > Systems Engineer<br>
> > > R&D Media<br>
> > ><br>
> > > _______________________________________________<br>
> > > Gluster-users mailing list<br>
> > > <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> > > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> > ><br>
> > > _______________________________________________<br>
> > > Gluster-users mailing list<br>
> > > <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> > > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > Tiemen Ruiten<br>
> > > Systems Engineer<br>
> > > R&D Media<br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > Tiemen Ruiten<br>
> > > Systems Engineer<br>
> > > R&D Media<br>
> > ><br>
> > > _______________________________________________<br>
> > > Gluster-users mailing list<br>
> > > <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> > > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> ><br>
><br>
><br>
><br>
> --<br>
> Tiemen Ruiten<br>
> Systems Engineer<br>
> R&D Media<br>
><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">Tiemen Ruiten<br>Systems Engineer<br>R&D Media<br></div></div>
</div>