Gluster blog stories provide high-level spotlights on our users all over the world
Chitika Inc., an online advertising network based in Westborough, MA, sought to provide its data scientists with faster and simpler access to its massive store of ad impression data. The company managed to boost availability and broaden access to its data by swapping out HDFS for GlusterFS as the filesystem backend for its Hadoop deployment.
“There are a number of benefits to the utilization of Gluster, arguably the biggest of which is the system’s POSIX compliance. This allows us to very easily mount Gluster anywhere as if it was a disk local to the system, meaning that we can expose 60TB of data to anything in our data center, across any amount of servers, users, and applications.”
I talked to Chitika Senior Systems Administrator Nick Wood about his company’s GlusterFS deployment, and we discussed the challenges, opportunities, and next steps for GlusterFS in their environment.
Chitika’s GlusterFS storage deployment consists of four GlusterFS 3.5 host running Debian Wheezy. Each host is packed with disks, sporting 43TB of storage across a set of 6 RAID arrays. Each of these twenty-four arrays hosts a single GlusterFS brick, which together form a triple-replicated GlusterFS volume providing roughly 59TB of total storage, 40TB of which is currently consumed.
Chitika’s client machines, which also run Debian Wheezy, primarily access this volume via the GlusterFS FUSE client, although one of their clients makes use of GlusterFS’s NFS support.
Bridging Chikika’s GlusterFS cluster and the company’s cluster of 36 Hadoop nodes is a customized version of the glusterfs-hadoop plugin. On the hardware side, the company taps Infiniband gear to link up its GlusterFS and Hadoop clusters, using an IP over Infiniband connection.
Wood explained that he’s keen to see the RDMA (Remote Direct Memory Access) support in GlusterFS stablize enough for Chitika to shift from TCP to RDMA as their GlusterFS transport type, thereby allowing the company to take full advantage of its Infiniband hardware.
Since their June 2013 deployment, Chitika’s GlusterFS storage solution has undergone a two software upgrades while in production, both of which ran smoothly.
However, the team’s experience with its deployment hasn’t been trouble-free. At one point, the deployment suffered an issue in which problems with the operating system hard drive in one of the GlusterFS hosts that led to inconsistency between some of the replicated data in their volume — a state also known as split-brain.
GlusterFS includes a self-heal daemon for repairing inconsistencies between replicated files, but there are scenarios that require manual intervention to determine which copies to retain and which to discard, which Wood and the Chitika team experienced first-hand.
“The self-heal didn’t really work as we expected. It correctly identified some corrupt files, correctly healed others, and completely ignored most,” Wood explained. “Luckily, we store compressed files and could easily tell what was corrupted. However, it took a 50TB filesystem crawl and semi-automated identification/restoration of good copies from the raw bricks to recover.”
Also among the challenges the Chitika team have encountered has been slow performance with common system utilities that carry out file stat operations, leading the team to develop alternative utilities that avoid the stat system call or that operate in parallel.
Despite these bumps in the road, the team at Chitika is enthusiastic about its GlusterFS deployment, and are mulling plans to double their GlusterFS host count to eight, to accomodate the addition of more compute nodes.
2020 has not been a year we would have been able to predict. With a worldwide pandemic and lives thrown out of gear, as we head into 2021, we are thankful that our community and project continued to receive new developers, users and make small gains. For that and a...
It has been a while since we provided an update to the Gluster community. Across the world various nations, states and localities have put together sets of guidelines around shelter-in-place and quarantine. We request our community members to stay safe, to care for their loved ones, to continue to be...
The initial rounds of conversation around the planning of content for release 8 has helped the project identify one key thing – the need to stagger out features and enhancements over multiple releases. Thus, while release 8 is unlikely to be feature heavy as previous releases, it will be the...