<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I suppose some of it may depend on what you consider a small file and how many of them there are and the operation read/write/sequential/etc as well as the performance expectations are. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I had looked at zfs replication also but redhat support was a problem. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Does anyone know if a beta is available for: <a href="http://www.gluster.org/community/documentation/index.php/Features/Feature_Smallfile_Perf">http://www.gluster.org/community/documentation/index.php/Features/Feature_Smallfile_Perf</a> <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Are there any benchmarks published comparing nfs client mount vs. gluster fuse vs. boost (for apache)? I know this is very current but <a href="https://s3.amazonaws.com/aws001/guided_trek/Performance_in_a_Gluster_Systemv6F.pdf%20pages%2019-20">https://s3.amazonaws.com/aws001/guided_trek/Performance_in_a_Gluster_Systemv6F.pdf pages 19-20</a> has some data for small files – from that doc “</span>As can be seen in Figure 8 below, Gluster delivers good single storage node performance for a variety of small file operations. Generally speaking, Gluster Native FUSE will deliver better small file performance than Gluster NFS, although Gluster NFS is often better for very small block sizes. Perhaps most important, IOPS performance in Gluster scales out just as throughput performance scales out.<span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>” This is from 2013 and has a few suggestions to try: <a href="https://rhsummit.files.wordpress.com/2013/07/england_th_0450_rhs_perf_practices-4_neependra.pdf">https://rhsummit.files.wordpress.com/2013/07/england_th_0450_rhs_perf_practices-4_neependra.pdf</a>. If anyone has newer information that would be appreciated. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>-john <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> gluster-users-bounces@gluster.org [mailto:gluster-users-bounces@gluster.org] <b>On Behalf Of </b>Matan Safriel<br><b>Sent:</b> Thursday, January 29, 2015 2:31 PM<br><b>To:</b> Liam Slusser<br><b>Cc:</b> gluster-users@gluster.org<br><b>Subject:</b> Re: [Gluster-users] Small files<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>Hi Liam,<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Thanks for the comprehensive reply (!)<o:p></o:p></p></div><div><p class=MsoNormal>How many nodes do you safely replicate to with ZFS? <o:p></o:p></p></div><div><p class=MsoNormal>I don't think seek time is much of a concern with SSD by the way, so it does seem that glusterfs is much better for the small files scenario than HDFS, which as you say is very different in key aspects, and couldn't quite follow why rebalancing is slow or slower than in the case of HDFS actually, unless you just meant that HDFS works at a large block level and no more.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Perhaps you'd care to comment ;)<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Matan<o:p></o:p></p></div></div><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>On Thu, Jan 29, 2015 at 9:15 PM, Liam Slusser <<a href="mailto:lslusser@gmail.com" target="_blank">lslusser@gmail.com</a>> wrote:<o:p></o:p></p><div><p class=MsoNormal>Matan - I'll do my best to take a shot at answering this...<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>They're completely different technologies. HDFS is not posix compliant and is not a "mountable" filesystem while Gluster is.<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>In HDFS land, every file, directory and block in HDFS is represented as an object in the namenode’s memory, each of which occupies 150 bytes. So 10 million files would each up about 3 gigs of memory. Furthermore was designed for streaming large files - the default blocksize in HDFS is 64MB.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Gluster doesn't have a central namenode, so having millions of files doesn't put a tax on it in the same way. But, again, small files causes lots of small seeks to handle the replication tasks/checks and generally isn't very efficient. So don't expect blazing performance... Doing rebalancing and rebuilding of Gluster bricks can be extremely painful since Gluster isn't a block level filesystem - so it will have to read each file one at a time.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal>If you want to use HDFS and don't need a mountable filesystem have a look at HBASE.<o:p></o:p></p></div></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>We tacked the small files problem by using a different technology. I have an image store of about 120 million+ small-file images, I needed a "mountable" filesystem which was posix compliant and ended up doing a ZFS setup - using the built in replication to create a few identical copies on different servers for both load balancing and reliability. So we update one server and than have a few read-only copies serving the data. Changes get replicated, at a block level, every few minutes.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>thanks,<o:p></o:p></p></div><div><p class=MsoNormal>liam<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div></div></div><div><p class=MsoNormal><o:p> </o:p></p><div><div><div><p class=MsoNormal>On Thu, Jan 29, 2015 at 4:29 AM, Matan Safriel <<a href="mailto:dev.matan@gmail.com" target="_blank">dev.matan@gmail.com</a>> wrote:<o:p></o:p></p></div></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><div><div><div><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><p class=MsoNormal>Hi,<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Is glusterfs much better than hdfs for the many small files scenario?<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Thanks,<o:p></o:p></p></div><div><p class=MsoNormal>Matan<o:p></o:p></p></div></div></blockquote></div><p class=MsoNormal><o:p> </o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p></div></div><p class=MsoNormal>_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></p></blockquote></div><p class=MsoNormal><o:p> </o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p></div></div></body></html>