<div dir="ltr">Thanks Liam,<div><br></div><div>Indeed I started this journey from that blog post. I learned a lot from your last reply, but basically it confirms my thoughts that gluster has the potential to be better at small files. I'm however quite reluctant to delve into testing it, as the status of some blocking (?) development items, mentioned by John on this thread, remains unknown to me. </div><div><br></div><div>I will use replication at the application level, until I find a file system that behaves with reliable semantics around mass replication of small files. It will be slower, but predictable and won't require much communication on user groups for every step....</div><div><br></div><div>Also I am looking at cloud setup, and have less freedom in choosing communication speeds and bandwidth between nodes, than you'd get in your own datacenter (I think).</div><div><br></div><div>Matan</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 30, 2015 at 4:28 AM, Liam Slusser <span dir="ltr"><<a href="mailto:lslusser@gmail.com" target="_blank">lslusser@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span class=""><div>Matan -</div><div><br></div><div>We replicate to two nodes. But since a zfs send | zfs recv communicates one-way, I'd think you could do as many as you want. It just might take a little bit longer - although you should be able to run multiple at a time as long as you had enough bandwidth over the network. Ours are connected via a dedicated 10gigabit network and see around 4-5gbit/sec on a large commit. How long the replication job takes depends on how much is changed between the two snapshots.</div><div><br></div><div>Even though the seek time with a SSD is quick, you'll still get far greater throughput in sequential read/writing vs small random accesses.</div><div><br></div><div>You can test it yourself. Create a directory with 100 64MB files and another directory with 64,000 100K files. Now copy it from one place to another and see for yourself which is faster. Sequential reading always wins. And this is true with both Gluster and HDFS.</div><div><br></div><div>In HDFS small files exacerbates the problem because you need to contact the NameNode to get the block information and then contact the DataNode to get the block. Think of it like this. Reading 1000 64KB files in HDFS means 1000 requests to the NameNode and 1000 requests to the datanodes while reading 1 64MB file is one trip to the NameNode and one trip the the Datanode to get the same amount of data.</div><div><br></div><div>You can read more about this issue here: <a href="http://blog.cloudera.com/blog/2009/02/the-small-files-problem/" target="_blank">http://blog.cloudera.com/blog/2009/02/the-small-files-problem/</a></div><div><br></div></span><div>thanks,</div><div>liam</div></div><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Thu, Jan 29, 2015 at 12:30 PM, Matan Safriel <span dir="ltr"><<a href="mailto:dev.matan@gmail.com" target="_blank">dev.matan@gmail.com</a>></span> wrote:<br></span><div><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Liam,<div><br></div><div>Thanks for the comprehensive reply (!)</div><div>How many nodes do you safely replicate to with ZFS? </div><div>I don't think seek time is much of a concern with SSD by the way, so it does seem that glusterfs is much better for the small files scenario than HDFS, which as you say is very different in key aspects, and couldn't quite follow why rebalancing is slow or slower than in the case of HDFS actually, unless you just meant that HDFS works at a large block level and no more.</div><div><br></div><div>Perhaps you'd care to comment ;)</div><span><font color="#888888"><div><br></div><div>Matan</div></font></span></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 29, 2015 at 9:15 PM, Liam Slusser <span dir="ltr"><<a href="mailto:lslusser@gmail.com" target="_blank">lslusser@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Matan - I'll do my best to take a shot at answering this...<div><br></div><div>They're completely different technologies. HDFS is not posix compliant and is not a "mountable" filesystem while Gluster is.<div><br></div><div>In HDFS land, every file, directory and block in HDFS is represented as an object in the namenode’s memory, each of which occupies 150 bytes. So 10 million files would each up about 3 gigs of memory. Furthermore was designed for streaming large files - the default blocksize in HDFS is 64MB.</div><div><br></div><div>Gluster doesn't have a central namenode, so having millions of files doesn't put a tax on it in the same way. But, again, small files causes lots of small seeks to handle the replication tasks/checks and generally isn't very efficient. So don't expect blazing performance... Doing rebalancing and rebuilding of Gluster bricks can be extremely painful since Gluster isn't a block level filesystem - so it will have to read each file one at a time.</div><div><br></div><div><div>If you want to use HDFS and don't need a mountable filesystem have a look at HBASE.<br></div></div><div><br></div><div>We tacked the small files problem by using a different technology. I have an image store of about 120 million+ small-file images, I needed a "mountable" filesystem which was posix compliant and ended up doing a ZFS setup - using the built in replication to create a few identical copies on different servers for both load balancing and reliability. So we update one server and than have a few read-only copies serving the data. Changes get replicated, at a block level, every few minutes.</div><div><br></div><div>thanks,</div><div>liam</div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Thu, Jan 29, 2015 at 4:29 AM, Matan Safriel <span dir="ltr"><<a href="mailto:dev.matan@gmail.com" target="_blank">dev.matan@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>Is glusterfs much better than hdfs for the many small files scenario?</div><div><br></div><div>Thanks,</div><div>Matan</div></div>
</blockquote></div><br></div></div>
<br></div></div>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div></div></div><br></div>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>