The Gluster Blog

Gluster blog stories provide high-level spotlights on our users all over the world

Running GlusterFS inside docker container

Gluster
2014-02-16

As a part of GlusterFS 3.5 testing and hackathon, I decided to put GlusterFS inside a docker container.So I installed docker on my Fedora20 desktop

$ yum install docker-io -y
$ systemctl enable docker.service
ln -s '/usr/lib/systemd/system/docker.service' '/etc/systemd/system/multi-user.target.wants/docker.service'
$ systemctl start docker.service
$ docker version
Client version: 0.7.6
..
Server version: 0.7.6

and then started a Fedora container

$ docker run -i -t mattdm/fedora /bin/bash

Once I am inside the container I installed GlusterFS packages

bash-4.2# yum install glusterfs glusterfs-server -y

And then tried to create volume

bash-4.2# /usr/sbin/glusterd
bash-4.2# gluster volume create vol 172.17.0.3:/mnt/brick/ force


but I got following error:-

volume create: vol: failed: Glusterfs is not supported on brick: 172.17.0.3:/mnt/brick.
Setting extended attributes failed, reason: Operation not permitted.

From above error it looked as setting up extended attributes are not supported, which is a basic need to use GlusteFS. So I tried to test them manually. I was able to set extended attributes in user namespace but not in trusted namespace.

bash-4.2# yum install attr -y
bash-4.2# setfattr -n user.foo1 -v "bar" a
bash-4.2# touch a; setfattr -n trusted.foo1 -v "bar" a
setfattr: a: Operation not permitted

With some internet search I figured out that CAP_SYS_ADMIN is needed for setting up extended attributes in trusted namespace and to get that inside docker we need to run an image with –privileged=true option like

$ docker run --privileged=true -i -t mattdm/fedora /bin/bash

With that I was able to create the volume and start it

bash-4.2# gluster volume create vol 172.17.0.3:/mnt/brick/ force
bash-4.2# gluster volume start vol

But when I tried to mount the volume I got following error:-

E [mount.c:267:gf_fuse_mount] 0-glusterfs-fuse: cannot open /dev/fuse (No such file or directory)

this turned out to be image specific problem, which I am using (mattdm/fedora). I had to mknod for /dev/fuse

bash-4.2# mknod /dev/fuse c 10 229

and after that I was able to mount volume.

On Fedroa 20 for docker version 0.7.6 the default storage driver for docker is device-mapper on which extended attributes are supported. AUFS storage driver does not support extended attributes as of now.I have tried with btrfs storage driver with docker 0.8 as well and was able to use GlusterFS. To use btrfs storage driver, we need to start docker daemon with following command :-

$ docker -d -s btrfs

Above will only work if Docker is running on a btrfs partition already prepared by the host system.

BLOG

  • 28 Nov 2019
    Planning ahead for Gluster releases

    In order to plan the content for upcoming releases, it is good to take a moment of pause, step back and attempt to look at the consumption of GlusterFS within large enterprises. With the enterprise architecture taking large strides towards cloud and more specifically, the hybrid cloud, continued efforts towards...

    Read more
  • 13 Nov 2019
    Announcing Gluster 7.0

    The Gluster community is pleased to announce the release of 7.0, our latest release. This is a major release that includes a range of code improvements and stability fixes along with a few features as noted below. A selection of the key features and bugs addressed are documented in this...

    Read more
  • 15 Oct 2019
    Gluster and CentOS Stream

    Progress cannot be made without change. As technologists, we recognize this every day. Most of the time, these changes are iterative: progresssive additions of features to projects like Gluster. Sometimes those changes are small, and sometimes not. And that’s, of course, just talking about our project. But one of the...

    Read more