Gluster Volume Setup
Now that all the nodes are setup and added to the cluster we need to tweak a few settings create our 1st volume and add it to the data center. Before completing these steps please make sure you have gone though and confirm you hav…

Functional languages replace the need for mutability by providing us with convenient, idiomatic mechanisms for defining transformations without explicitly modifying or assigning variables.  This level of decoupling transparently provides us with n…

Distributed Storage Across Four Storage Nodes With GlusterFS 3.2.x On Ubuntu 12.04

This tutorial shows how to combine four single storage servers
(running Ubuntu 12.04) to one large storage server (distributed storage)
with GlusterFS.
The client sy…

As people who attended my recent Red Hat Summit talk are aware, one of the big issues with GlusterFS replication is “split brain” which occurs when conflicting updates are made to replicas of a file. Self-healing in the wrong direction risks data loss, so we won’t do self-heal if we detect split brain and we’ll […]

Starting with GlusterFS 3.3, one change has been the check to see if a directory (or any of it’s ancestors) is already part of a volume. This is causing many support questions in #gluster.
This was implemented because if you remove a brick from a volum…

The big, bad Eclipse is great for whipping a massive java source tree into shape, fast.  But for Clojure – its a different, more thoughtful development idiom.  You don’t need to browse complex type hierarchies and fold 1000s of lines of code …

Within the past couple weeks, Fedora and Gluster rolled out new versions, packed with too many features to discuss in a single blog post. However, a couple of the stand-out updates in each release overlap neatly enough to tackle them together–namely, the inclusion of OpenStack Essex in Fedora 17 and support for using Gluster 3.3 as a […]

Probably everybody who cares already knows that GlusterFS 3.3 has been released. I find it amusing that quorum enforcement is listed as one of the marquee features, since it’s really such a trivial bit of code compared to the other features on the list, or to internal but significant changes such as the new GFID-based […]

Our fearless engineering leader, Vijay, aka hagarth, has kindly posted a HOWTO on upgrading to GlusterFS 3.3. Please read before doing anything dumb!  

Now that GlusterFS 3.3.0 is out, here is a quick primer on upgrading from earlier installed versions of GlusterFS. This howto covers upgrades from 3.1.x and 3.2.x versions of GlusterFS. 1) GlusterFS 3.3.0 is not compatible with any earlier released versions. Please make sure that you schedule a downtime before you upgrade. 2) Stop all […]

Today, we’re announcing the next generation of GlusterFS, version 3.3. The release has been a year in the making and marks several firsts: the first post-acquisition release under Red Hat, our first major act as an openly-governed project and our first foray beyond NAS. We’ve also taken our first steps towards merging big data and …Read more

The release of GlusterFS 3.3.0 by the Gluster Community marks a major milestone in Clustered File Storage. GlusterFS is the leading open source solution for the dramatically increasing volume of unstructured data. It is a software-only, highly available, scale-out, centrally managed storage pool that can be backed by any POSIX filesystem that supports extended attributes, such as Ext3/4, XFS, BTRFS and many more.

As an example of Red Hat’s goal of building strong, independent, open source communities, GlusterFS 3.3.0 marks the first release as an “upstream” project with its own release schedule. This release addresses many of the most commonly requested features including proactive self-healing, quorum enforcement, and granular locking for self-healing, as well as many more bug fixes and enhancements.

Some of the more noteworthy features include:

• Unified File and Object storage – Blending OpenStack’s Object Storage API  with GlusterFS provides simultaneous read and write access to data as files or as objects.
• HDFS compatibility – Gives Hadoop admins the ability to run map/reduce jobs on unstructured data on GlusterFS and access the data with well-known tools and shell scripts.
• Proactive self-healing – GlusterFS volumes will now automatically restore file integrity after a replica recovers from failure.
• Granular locking – Allows large files to be accessed even during self-healing, a feature that is particularly important for VM images.
• Replication improvements – with quorum enforcement you are assured that  your data has been written in at least the configured number of places before the file operation returns, allowing a user-configurable adjustment to fault tolerance vs performance.

Visit Gluster.org to download. Packages are available for most distributions, including Ubuntu, Debian, Fedora, RHEL, and CentOS.

Get involved! Join us on #gluster on freenode, join our mailing list, ‘like’ our Facebook page, follow us on twitter, or check out our LinkedIn group.

GlusterFS is an open source project sponsored by Red Hat®, who uses it in its line of Red Hat Storage products.

A lot of changes were committed recently to the gluster-wireshark repository. A lot of effort was put into the details (click on the image to enlarge):UUIDs and GFIDs are now displayed as 4-2-2-2-6 bytesflags for OPEN, CREATE etc are now shown in detai…

A lot of changes were committed recently to the gluster-wireshark repository. A lot of effort was put into the details (click on the image to enlarge):UUIDs and GFIDs are now displayed as 4-2-2-2-6 bytesflags for OPEN, CREATE etc are now shown in detai…

[From an email to the gluster-devel mailinglist]  today I have merged support for GlusterFS 3.2 and 3.3 into one Wireshark ‘dissector’. The packages with date 20120516 in the version support both the current stable 3.2.x version, and the latest 3….

[From an email to the gluster-devel mailinglist]  today I have merged support for GlusterFS 3.2 and 3.3 into one Wireshark ‘dissector’. The packages with date 20120516 in the version support both the current stable 3.2.x version, and the latest 3….

The Problem

Over the last couple days, in #gluster, users have come in complaining that their application can’t open a file, but that if they try accessing the file from the shell as the same user, it works fine. This was reported with apache’s tomcat and mod_fcgid and courier imap.

My first thought on this, and it still would be, is selinux. Selinux’s role is to prevent the wrong thing from doing what it’s not expected to do. It will make some applications be unable to even access a file that every other test proves should work. Always check this first if you’re experiencing unexpected access issues.

But in this case, it turned out to be the application itself. The users were running 32 bit apps on 64 bit platforms. As it turns out, the applications were tracking the inode numbers of files. They would call stat() which would return a 64bit inode. The apache programs would copy the results of that stat call into it’s own structure. If the apps were built on 32bit platforms, apache’s struct would have a 32 bit field. The 64 bit result wouldn’t fit. Apache tested for that and would error out with the ambiguous error message:

Syntax error on line ## of {filename}
Wrapper  {filename} cannot be accessed: (70008)Partial results are valid but processing is incomplete

What it really meant was that the 64 bit inode overflowed the 32 bit field it allocated for storing it.

To identify this is the problem, run

file $FILENAME

where $FILENAME is the binary that’s producing the error. If it contains “32-bit” then that’s a pretty good indication that this might be a problem.

The Solution

The best solution, of course, is to use 64 bit applications on your 64 bit clients. This wasn’t possible for this user.

To solve this problem, we configured the volume to enable the 32bit inode workaround in it

gluster volume set $VOLUME nfs.enable-ino32 on

Then we mounted via nfs instead of using the fuse client.

This passed 32 bit inode translations to the application, eliminating the overflow. This worked for the apache programs, but not the 32 bit courier imap. GlusterFS 3.2 doesn’t support nfs locks. Since courier requires those, this wouldn’t work.

What about redundancy?

Redundancy was maintained by installing the server package for GlusterFS on the client, starting glusterd, and adding the client to the peer group for the volume. This starts the nfs daemon on the client, allowing the client to do an nfs mount from localhost. The nfs daemon then handles connecting to the brick servers, maintaining redundancy.

My standing search for “glusterfs” on Twitter got me into an interesting discussion with Dr. Shawn Tan about an interesting GlusterFS configuration for several workstations. At first my reaction was panic, because I could see potential for data loss in that configuration, but “you’re using it wrong” is rarely a productive response from a developer. […]

As part of a continuing effort to broaden the potential uses of GlusterFS, the license for (most of) GlusterFS has been changed as follows (some details might vary between files). /* Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com> This file is part of GlusterFS.   This file is licensed to you under your choice of […]

One of the key features of GlusterFS, or any horizontally scalable system like it, is the ability to rebalance data as servers are added, removed, etc. How is that done? Come to think of it, what does “balance” even mean in such a system, and why is it so important to have it? Intuitively, balance […]