The Gluster Blog

Gluster blog stories provide high-level spotlights on our users all over the world

Broken 32bit apps on GlusterFS

Gluster
2012-05-15

The Problem

Over the last couple days, in #gluster, users have come in complaining that their application can’t open a file, but that if they try accessing the file from the shell as the same user, it works fine. This was reported with apache’s tomcat and mod_fcgid and courier imap.

My first thought on this, and it still would be, is selinux. Selinux’s role is to prevent the wrong thing from doing what it’s not expected to do. It will make some applications be unable to even access a file that every other test proves should work. Always check this first if you’re experiencing unexpected access issues.

But in this case, it turned out to be the application itself. The users were running 32 bit apps on 64 bit platforms. As it turns out, the applications were tracking the inode numbers of files. They would call stat() which would return a 64bit inode. The apache programs would copy the results of that stat call into it’s own structure. If the apps were built on 32bit platforms, apache’s struct would have a 32 bit field. The 64 bit result wouldn’t fit. Apache tested for that and would error out with the ambiguous error message:

Syntax error on line ## of {filename}
Wrapper  {filename} cannot be accessed: (70008)Partial results are valid but processing is incomplete

What it really meant was that the 64 bit inode overflowed the 32 bit field it allocated for storing it.

To identify this is the problem, run

file $FILENAME

where $FILENAME is the binary that’s producing the error. If it contains “32-bit” then that’s a pretty good indication that this might be a problem.

The Solution

The best solution, of course, is to use 64 bit applications on your 64 bit clients. This wasn’t possible for this user.

To solve this problem, we configured the volume to enable the 32bit inode workaround in it

gluster volume set $VOLUME nfs.enable-ino32 on

Then we mounted via nfs instead of using the fuse client.

This passed 32 bit inode translations to the application, eliminating the overflow. This worked for the apache programs, but not the 32 bit courier imap. GlusterFS 3.2 doesn’t support nfs locks. Since courier requires those, this wouldn’t work.

What about redundancy?

Redundancy was maintained by installing the server package for GlusterFS on the client, starting glusterd, and adding the client to the peer group for the volume. This starts the nfs daemon on the client, allowing the client to do an nfs mount from localhost. The nfs daemon then handles connecting to the brick servers, maintaining redundancy.

BLOG

  • 06 Dec 2020
    Looking back at 2020 – with g...

    2020 has not been a year we would have been able to predict. With a worldwide pandemic and lives thrown out of gear, as we head into 2021, we are thankful that our community and project continued to receive new developers, users and make small gains. For that and a...

    Read more
  • 27 Apr 2020
    Update from the team

    It has been a while since we provided an update to the Gluster community. Across the world various nations, states and localities have put together sets of guidelines around shelter-in-place and quarantine. We request our community members to stay safe, to care for their loved ones, to continue to be...

    Read more
  • 03 Feb 2020
    Building a longer term focus for Gl...

    The initial rounds of conversation around the planning of content for release 8 has helped the project identify one key thing – the need to stagger out features and enhancements over multiple releases. Thus, while release 8 is unlikely to be feature heavy as previous releases, it will be the...

    Read more