<div dir="ltr">That's more a limitation in the gluster protocol as it currently sits, not so much with whether you're using FUSE or libgfapi. There are probably other things you can do, though.<div><br></div><div>One might be to use dispersion instead of replication. Another might be to disable atime on the server. Per this thread ( <a href="http://lists.gnu.org/archive/html/gluster-devel/2008-03/msg00149.html">http://lists.gnu.org/archive/html/gluster-devel/2008-03/msg00149.html</a> ), if the server fs does not have atime enabled, gluster won't update it. That may well save you a few network round trips.</div><div><br></div><div>Another thing you might try is taking steps to ensure your ARP caches on your clients and bricks remain hot. (Random guess, here).</div><div><br></div><div>I've seen feature request tickets about bundling IOPs on the wire to opportunistically eliminate round trips, but I don't know what the status is of those.<br><div><br><div class="gmail_quote"><div dir="ltr">On Thu, Jul 30, 2015 at 12:07 PM David Robinson <<a href="mailto:david.robinson@corvidtec.com">david.robinson@corvidtec.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Copy that. I am very aware of the small file performance issues and I<br>
have been tracking 3.7. I am waiting for 3.7 to stabilize before I roll<br>
it out to my production system.<br>
<br>
My thought was that it looks like Bareos now has built in use of<br>
libgfapi, which from my understanding should improve performance for<br>
gluster. My issues with rsync aren't with the actual transfer speed of<br>
the files. My issue is with the time it takes to search through the<br>
filesystem to figure out which files to transfer. This takes an<br>
extremely long time on 400TB of data, especially when it is going<br>
through the directories with large numbers of small files.<br>
<br>
I was curious if Bareos with built-in support for libgfapi would be<br>
faster than rsync backups between gluster machines.<br>
Anyone on the email list care to comment? Thanks in advance for any<br>
info that can be provided.<br>
<br>
David<br>
<br>
<br>
------ Original Message ------<br>
From: "André Bauer" <<a href="mailto:abauer@magix.net" target="_blank">abauer@magix.net</a>><br>
To: "David F. Robinson" <<a href="mailto:david.robinson@corvidtec.com" target="_blank">david.robinson@corvidtec.com</a>><br>
Sent: 7/30/2015 11:35:21 AM<br>
Subject: Re: [Gluster-users] Bareos backup from Gluster mount<br>
<br>
>Hi David,<br>
><br>
>i never used Bareos until now. We like to switch from Bacula in the<br>
>future but i think this will not happen before next Ubuntu LTS release<br>
>(16.04).<br>
><br>
>I also never directly compared with rsync but i think rsync is faster<br>
>in<br>
>transfering because it does not have to do any compression and so on...<br>
><br>
>What i can say about Bacula on Glusterfs volumes is, that copying big<br>
>files works at reasonable speed while small files (especialy if there<br>
>are a lot) are a bit slow, whats in Glusterfs nature until versions<br>
>prior 3.6(?).<br>
><br>
>With Glusterfs 3.6 / 3.7 this should be a bit faster in the meantime<br>
>but<br>
>i have no experience with the performance gains because i'm still on<br>
>Glusterfs 3.5.5.<br>
><br>
>In conclusion i still prefer Bacula over Rsync even if its slower.<br>
><br>
>Some more info about Glusterfs small file performance can be found<br>
>here:<br>
><br>
><a href="https://gluster.readthedocs.org/en/latest/Feature%20Planning/GlusterFS%203.7/Small%20File%20Performance/" rel="noreferrer" target="_blank">https://gluster.readthedocs.org/en/latest/Feature%20Planning/GlusterFS%203.7/Small%20File%20Performance/</a><br>
><br>
>Regards<br>
>André<br>
><br>
><br>
><br>
>Am 30.07.2015 um 15:23 schrieb David F. Robinson:<br>
>> Andre,<br>
>><br>
>> I am looking at a backup alternative to rsnc for gluster. My storage<br>
>>system is growing and rsync takes too long on my system (300TB). Do<br>
>>you have any idea of the relative performance of bareos as compared to<br>
>>that of rsync? Can it be run in a multi-threaded mode? Rsync takes an<br>
>>extremely long time just searching the directory tree to figure out<br>
>>what to copy. Before digging into bareos, I was wondering if you had<br>
>>any thoughts on performance for gluster.<br>
>><br>
>> David (Sent from mobile)<br>
>><br>
>> ===============================<br>
>> David F. Robinson, Ph.D.<br>
>> President - Corvid Technologies<br>
>> 704.799.6944 x101 [office]<br>
>> 704.252.1310 [cell]<br>
>> 704.799.7974 [fax]<br>
>> <a href="mailto:David.Robinson@corvidtec.com" target="_blank">David.Robinson@corvidtec.com</a><br>
>> <a href="http://www.corvidtechnologies.com" rel="noreferrer" target="_blank">http://www.corvidtechnologies.com</a><br>
>><br>
>>> On Jul 29, 2015, at 1:36 PM, André Bauer <<a href="mailto:abauer@magix.net" target="_blank">abauer@magix.net</a>> wrote:<br>
>>><br>
>>> We're using Bacula (Bareos is a fork of it) for backups.<br>
>>> Never had any problems doing backups of Gluster volumes.<br>
>>><br>
>>>> Am 27.07.2015 um 23:02 schrieb Ryan Clough:<br>
>>>> Hello,<br>
>>>><br>
>>>> I have cross-posted this question in the bareos-users mailing list.<br>
>>>><br>
>>>> Wondering if anyone has tried this because I am unable to backup<br>
>>>>data<br>
>>>> that is mounted via Gluster Fuse or Gluster NFS. Basically, I have<br>
>>>>the<br>
>>>> Gluster volume mounted on the Bareos Director which also has the<br>
>>>>tape<br>
>>>> changer attached.<br>
>>>><br>
>>>> Here is some information about versions:<br>
>>>> Bareos version 14.2.2<br>
>>>> Gluster version 3.7.2<br>
>>>> Scientific Linux version 6.6<br>
>>>><br>
>>>> Our Gluster volume consists of two nodes in distribute only. Here<br>
>>>>is the<br>
>>>> configuration of our volume:<br>
>>>> [root@hgluster02 ~]# gluster volume info<br>
>>>><br>
>>>> Volume Name: export_volume<br>
>>>> Type: Distribute<br>
>>>> Volume ID: c74cc970-31e2-4924-a244-4c70d958dadb<br>
>>>> Status: Started<br>
>>>> Number of Bricks: 2<br>
>>>> Transport-type: tcp<br>
>>>> Bricks:<br>
>>>> Brick1: hgluster01:/gluster_data<br>
>>>> Brick2: hgluster02:/gluster_data<br>
>>>> Options Reconfigured:<br>
>>>> performance.io-thread-count: 24<br>
>>>> server.event-threads: 20<br>
>>>> client.event-threads: 4<br>
>>>> performance.readdir-ahead: on<br>
>>>> features.inode-quota: on<br>
>>>> features.quota: on<br>
>>>> nfs.disable: off<br>
>>>> auth.allow: 192.168.10.*,10.0.10.*,10.8.0.*,10.2.0.*,10.0.60.*<br>
>>>> server.allow-insecure: on<br>
>>>> server.root-squash: on<br>
>>>> performance.read-ahead: on<br>
>>>> features.quota-deem-statfs: on<br>
>>>> diagnostics.brick-log-level: WARNING<br>
>>>><br>
>>>> When I try to backup a directory from Gluster Fuse or Gluster NFS<br>
>>>>mount<br>
>>>> and I monitor the network communication I only see data being<br>
>>>>pulled<br>
>>>> from the hgluster01 brick. When the job finishes Bareos thinks that<br>
>>>>it<br>
>>>> completed without error but included in the messages for the job<br>
>>>>are<br>
>>>> lots and lots of permission denied errors like this:<br>
>>>> 15-Jul 02:03 ripper.red.dsic.com-fd JobId 613: Cannot open<br>
>>>> "/export/rclough/psdv-2014-archives-2/scan_111.tar.bak":<br>
>>>>ERR=Permission<br>
>>>> denied.<br>
>>>> 15-Jul 02:03 ripper.red.dsic.com-fd JobId 613: Cannot open<br>
>>>> "/export/rclough/psdv-2014-archives-2/run_219.tar.bak":<br>
>>>>ERR=Permission<br>
>>>> denied.<br>
>>>> 15-Jul 02:03 ripper.red.dsic.com-fd JobId 613: Cannot open<br>
>>>> "/export/rclough/psdv-2014-archives-2/scan_112.tar.bak":<br>
>>>>ERR=Permission<br>
>>>> denied.<br>
>>>> 15-Jul 02:03 ripper.red.dsic.com-fd JobId 613: Cannot open<br>
>>>> "/export/rclough/psdv-2014-archives-2/run_220.tar.bak":<br>
>>>>ERR=Permission<br>
>>>> denied.<br>
>>>> 15-Jul 02:03 ripper.red.dsic.com-fd JobId 613: Cannot open<br>
>>>> "/export/rclough/psdv-2014-archives-2/scan_114.tar.bak":<br>
>>>>ERR=Permission<br>
>>>> denied.<br>
>>>><br>
>>>> At first I thought this might be a root-squash problem but, if I<br>
>>>>try to<br>
>>>> read/copy a file using the root user from the Bareos server that is<br>
>>>> trying to do the backup, I can read files just fine.<br>
>>>><br>
>>>> When the job finishes is reports that it finished "OK -- with<br>
>>>>warnings"<br>
>>>> but, again the log for the job is filled with "ERR=Permission<br>
>>>>denied"<br>
>>>> messages. In my opinion, this job did not finish OK and should be<br>
>>>> Failed. Some of the files from the HGluster02 brick are backed up<br>
>>>>but<br>
>>>> all of the ones with permission errors do not. When I restore the<br>
>>>>job,<br>
>>>> all of the files with permission errors are empty.<br>
>>>><br>
>>>> Has anyone successfully used Bareos to backup data from Gluster<br>
>>>>mounts?<br>
>>>> This is an important use case for us because this is the largest<br>
>>>>single<br>
>>>> volume that we have to prepare large amounts of data to be<br>
>>>>archived.<br>
>>>><br>
>>>> Thank you for your time,<br>
>>>> ___________________________________________<br>
>>>> ¯\_(ツ)_/¯<br>
>>>> Ryan Clough<br>
>>>> Information Systems<br>
>>>> Decision Sciences International Corporation<br>
>>>><br>
>>>><<a href="http://www.decisionsciencescorp.com/" rel="noreferrer" target="_blank">http://www.decisionsciencescorp.com/</a>><<a href="http://www.decisionsciencescorp.com/" rel="noreferrer" target="_blank">http://www.decisionsciencescorp.com/</a>><br>
>>>><br>
>>>> This email and its contents are confidential. If you are not the<br>
>>>> intended recipient, please do not disclose or use the information<br>
>>>>within<br>
>>>> this email or its attachments. If you have received this email in<br>
>>>>error,<br>
>>>> please report the error to the sender by return email and delete<br>
>>>>this<br>
>>>> communication from your records.<br>
>>>><br>
>>>><br>
>>>> _______________________________________________<br>
>>>> Gluster-users mailing list<br>
>>>> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
>>>> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
>>>><br>
>>><br>
>>><br>
>>> --<br>
>>> Mit freundlichen Grüßen<br>
>>> André Bauer<br>
>>><br>
>>> MAGIX Software GmbH<br>
>>> André Bauer<br>
>>> Administrator<br>
>>> August-Bebel-Straße 48<br>
>>> 01219 Dresden<br>
>>> GERMANY<br>
>>><br>
>>> tel.: 0351 41884875<br>
>>> e-mail: <a href="mailto:abauer@magix.net" target="_blank">abauer@magix.net</a><br>
>>> <a href="mailto:abauer@magix.net" target="_blank">abauer@magix.net</a> <mailto:<a href="mailto:Email" target="_blank">Email</a>><br>
>>> <a href="http://www.magix.com" rel="noreferrer" target="_blank">www.magix.com</a> <<a href="http://www.magix.com/" rel="noreferrer" target="_blank">http://www.magix.com/</a>><br>
>>><br>
>>><br>
>>> Geschäftsführer | Managing Directors: Dr. Arnd Schröder, Michael<br>
>>>Keith<br>
>>> Amtsgericht | Commercial Register: Berlin Charlottenburg, HRB 127205<br>
>>><br>
>>> Find us on:<br>
>>><br>
>>> <<a href="http://www.facebook.com/MAGIX" rel="noreferrer" target="_blank">http://www.facebook.com/MAGIX</a>> <<a href="http://www.twitter.com/magix_de" rel="noreferrer" target="_blank">http://www.twitter.com/magix_de</a>><br>
>>> <<a href="http://www.youtube.com/wwwmagixcom" rel="noreferrer" target="_blank">http://www.youtube.com/wwwmagixcom</a>> <<a href="http://www.magixmagazin.de" rel="noreferrer" target="_blank">http://www.magixmagazin.de</a>><br>
>>><br>
>>>----------------------------------------------------------------------<br>
>>> The information in this email is intended only for the addressee<br>
>>>named<br>
>>> above. Access to this email by anyone else is unauthorized. If you<br>
>>>are<br>
>>> not the intended recipient of this message any disclosure, copying,<br>
>>> distribution or any action taken in reliance on it is prohibited and<br>
>>> may be unlawful. MAGIX does not warrant that any attachments are<br>
>>>free<br>
>>> from viruses or other defects and accepts no liability for any<br>
>>>losses<br>
>>> resulting from infected email transmissions. Please note that any<br>
>>> views expressed in this email may be those of the originator and do<br>
>>> not necessarily represent the agenda of the company.<br>
>>><br>
>>>----------------------------------------------------------------------<br>
>>> _______________________________________________<br>
>>> Gluster-users mailing list<br>
>>> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
>>> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
>><br>
><br>
><br>
>--<br>
>Mit freundlichen Grüßen<br>
>André Bauer<br>
><br>
>MAGIX Software GmbH<br>
>André Bauer<br>
>Administrator<br>
>August-Bebel-Straße 48<br>
>01219 Dresden<br>
>GERMANY<br>
><br>
>tel.: 0351 41884875<br>
>e-mail: <a href="mailto:abauer@magix.net" target="_blank">abauer@magix.net</a><br>
><a href="mailto:abauer@magix.net" target="_blank">abauer@magix.net</a> <mailto:<a href="mailto:Email" target="_blank">Email</a>><br>
><a href="http://www.magix.com" rel="noreferrer" target="_blank">www.magix.com</a> <<a href="http://www.magix.com/" rel="noreferrer" target="_blank">http://www.magix.com/</a>><br>
><br>
><br>
>Geschäftsführer | Managing Directors: Dr. Arnd Schröder, Michael Keith<br>
>Amtsgericht | Commercial Register: Berlin Charlottenburg, HRB 127205<br>
><br>
>Find us on:<br>
><br>
><<a href="http://www.facebook.com/MAGIX" rel="noreferrer" target="_blank">http://www.facebook.com/MAGIX</a>> <<a href="http://www.twitter.com/magix_de" rel="noreferrer" target="_blank">http://www.twitter.com/magix_de</a>><br>
><<a href="http://www.youtube.com/wwwmagixcom" rel="noreferrer" target="_blank">http://www.youtube.com/wwwmagixcom</a>> <<a href="http://www.magixmagazin.de" rel="noreferrer" target="_blank">http://www.magixmagazin.de</a>><br>
>----------------------------------------------------------------------<br>
>The information in this email is intended only for the addressee named<br>
>above. Access to this email by anyone else is unauthorized. If you are<br>
>not the intended recipient of this message any disclosure, copying,<br>
>distribution or any action taken in reliance on it is prohibited and<br>
>may be unlawful. MAGIX does not warrant that any attachments are free<br>
>from viruses or other defects and accepts no liability for any losses<br>
>resulting from infected email transmissions. Please note that any<br>
>views expressed in this email may be those of the originator and do<br>
>not necessarily represent the agenda of the company.<br>
>----------------------------------------------------------------------<br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></div></div></div>