The Gluster Blog

Gluster blog stories provide high-level spotlights on our users all over the world

Fixing split-brain with GlusterFS 3.3

Gluster
2012-07-19

With the addition of automated self-heal in GlusterFS 3.3, a new hidden directory structure was added to each brick: “.glusterfs”. This complicates split-brain resolution as you now not only have to remove the “bad” file from the brick, but it’s counterpart in .glusterfs.

Identify that you have a split-brain file:

VOLUME=testvol
gluster volume heal $VOLUME info split-brain
Heal operation on volume testvol has been successful

Brick server1:/data/testvol/brick1
Number of entries: 1
at                   path on brick
----------------------------------
2012-06-13 04:02:05  /foo/bar

Brick server2:/data/testvol/brick1
Number of entries: 1
at                   path on brick
----------------------------------
2012-06-13 04:02:05  /foo/bar

Ok, this says that I have one file that’s marked split-brain, “bar” in the “/foo” directory in volume “testvol”. After looking at that file in /data/testvol/brick1/foo/bar on both servers, I decided that the one on server1 is the good one, so I log into server2.

For my examples I like to set shell variables to represent things that are unique to my example. If you’re lazy like me, simply set your own shell variables and you should be able to just copy/paste the rest.

BRICK=/data/testvol/brick1
SBFILE=/foo/bar
GFID=$(getfattr -n trusted.gfid --absolute-names -e hex ${BRICK}${SBFILE} | grep 0x | cut -d'x' -f2)
rm -f ${BRICK}${SBFILE}
rm -f ${BRICK}/.glusterfs/${GFID:0:2}/${GFID:2:2}/${GFID:0:8}-${GFID:8:4}-${GFID:12:4}-${GFID:16:4}-${GFID:20:12}

At this point, I’ve always gone back to the old method of calling stat on the file through the client mount. I don’t know if it would heal automatically though.

If you have any questions, come see us in #gluster.

BLOG

  • 06 Dec 2020
    Looking back at 2020 – with g...

    2020 has not been a year we would have been able to predict. With a worldwide pandemic and lives thrown out of gear, as we head into 2021, we are thankful that our community and project continued to receive new developers, users and make small gains. For that and a...

    Read more
  • 27 Apr 2020
    Update from the team

    It has been a while since we provided an update to the Gluster community. Across the world various nations, states and localities have put together sets of guidelines around shelter-in-place and quarantine. We request our community members to stay safe, to care for their loved ones, to continue to be...

    Read more
  • 03 Feb 2020
    Building a longer term focus for Gl...

    The initial rounds of conversation around the planning of content for release 8 has helped the project identify one key thing – the need to stagger out features and enhancements over multiple releases. Thus, while release 8 is unlikely to be feature heavy as previous releases, it will be the...

    Read more