Broken 32bit apps on GlusterFS

The Problem

Over the last couple days, in #gluster, users have come in complaining that their application can’t open a file, but that if they try accessing the file from the shell as the same user, it works fine. This was reported with apache’s tomcat and mod_fcgid and courier imap.

My first thought on this, and it still would be, is selinux. Selinux’s role is to prevent the wrong thing from doing what it’s not expected to do. It will make some applications be unable to even access a file that every other test proves should work. Always check this first if you’re experiencing unexpected access issues.

But in this case, it turned out to be the application itself. The users were running 32 bit apps on 64 bit platforms. As it turns out, the applications were tracking the inode numbers of files. They would call stat() which would return a 64bit inode. The apache programs would copy the results of that stat call into it’s own structure. If the apps were built on 32bit platforms, apache’s struct would have a 32 bit field. The 64 bit result wouldn’t fit. Apache tested for that and would error out with the ambiguous error message:

Syntax error on line ## of {filename}
Wrapper  {filename} cannot be accessed: (70008)Partial results are valid but processing is incomplete

What it really meant was that the 64 bit inode overflowed the 32 bit field it allocated for storing it.

To identify this is the problem, run

file $FILENAME

where $FILENAME is the binary that’s producing the error. If it contains “32-bit” then that’s a pretty good indication that this might be a problem.

The Solution

The best solution, of course, is to use 64 bit applications on your 64 bit clients. This wasn’t possible for this user.

To solve this problem, we configured the volume to enable the 32bit inode workaround in it

gluster volume set $VOLUME nfs.enable-ino32 on

Then we mounted via nfs instead of using the fuse client.

This passed 32 bit inode translations to the application, eliminating the overflow. This worked for the apache programs, but not the 32 bit courier imap. GlusterFS 3.2 doesn’t support nfs locks. Since courier requires those, this wouldn’t work.

What about redundancy?

Redundancy was maintained by installing the server package for GlusterFS on the client, starting glusterd, and adding the client to the peer group for the volume. This starts the nfs daemon on the client, allowing the client to do an nfs mount from localhost. The nfs daemon then handles connecting to the brick servers, maintaining redundancy.