Running GlusterFS inside docker container

As a part of GlusterFS 3.5 testing and hackathon, I decided to put GlusterFS inside a docker container.So I installed docker on my Fedora20 desktop

$ yum install docker-io -y
$ systemctl enable docker.service
ln -s '/usr/lib/systemd/system/docker.service' '/etc/systemd/system/multi-user.target.wants/docker.service'
$ systemctl start docker.service
$ docker version
Client version: 0.7.6
..
Server version: 0.7.6

and then started a Fedora container

$ docker run -i -t mattdm/fedora /bin/bash

Once I am inside the container I installed GlusterFS packages

bash-4.2# yum install glusterfs glusterfs-server -y

And then tried to create volume

bash-4.2# /usr/sbin/glusterd
bash-4.2# gluster volume create vol 172.17.0.3:/mnt/brick/ force

but I got following error:-

volume create: vol: failed: Glusterfs is not supported on brick: 172.17.0.3:/mnt/brick.
Setting extended attributes failed, reason: Operation not permitted.

From above error it looked as setting up extended attributes are not supported, which is a basic need to use GlusteFS. So I tried to test them manually. I was able to set extended attributes in user namespace but not in trusted namespace.

bash-4.2# yum install attr -y
bash-4.2# setfattr -n user.foo1 -v "bar" a
bash-4.2# touch a; setfattr -n trusted.foo1 -v "bar" a
setfattr: a: Operation not permitted

With some internet search I figured out that CAP_SYS_ADMIN is needed for setting up extended attributes in trusted namespace and to get that inside docker we need to run an image with –privileged=true option like

$ docker run --privileged=true -i -t mattdm/fedora /bin/bash

With that I was able to create the volume and start it

bash-4.2# gluster volume create vol 172.17.0.3:/mnt/brick/ force
bash-4.2# gluster volume start vol

But when I tried to mount the volume I got following error:-

E [mount.c:267:gf_fuse_mount] 0-glusterfs-fuse: cannot open /dev/fuse (No such file or directory)

this turned out to be image specific problem, which I am using (mattdm/fedora). I had to mknod for /dev/fuse

bash-4.2# mknod /dev/fuse c 10 229

and after that I was able to mount volume.

On Fedroa 20 for docker version 0.7.6 the default storage driver for docker is device-mapper on which extended attributes are supported. AUFS storage driver does not support extended attributes as of now.I have tried with btrfs storage driver with docker 0.8 as well and was able to use GlusterFS. To use btrfs storage driver, we need to start docker daemon with following command :-

$ docker -d -s btrfs

Above will only work if Docker is running on a btrfs partition already prepared by the host system.