GlusterFS 1.2 Configuration Example for Four Bricks

From GlusterDocumentation

Jump to: navigation, search

This article was last updated for the version 1.2.3 of GlusterFS

Contents

[edit]

Introduction

In this article, I will show you how to configure GlusterFS to simulate a cluster with:

  • one single mount point
  • 4 storage nodes serving the files

I said simulate since we will deploy such a cluster on only one computer, but you can as well set it up over physical different computers by changing the IP addresses.

[edit]

Configure the 4 storage server nodes

We create a directory for each storage server to be created: 

toad@vlk:~/gfs$ sudo mkdir /home/cfs{0,1,2,3}
toad@vlk:~/gfs$ sudo chmod 1777 /home/cfs{0,1,2,3}
toad@vlk:~/gfs$ ls -ld /home/cfs[0-3]
drwxrwxrwt  2 root root 4096 2007-01-14 13:42 /home/cfs0
drwxrwxrwt  2 root root 4096 2007-01-14 13:42 /home/cfs1
drwxrwxrwt  2 root root 4096 2007-01-14 13:42 /home/cfs2
drwxrwxrwt  2 root root 4096 2007-01-14 13:42 /home/cfs3
toad@vlk:~/gfs$

Then we write 4 configuration files for server volume specification where we will merely change the storage destination and the listening port

[edit]

Server volume for the 1st node: server0.vol 

volume brick
        type storage/posix
        option directory /home/cfs0
end-volume

volume server
        type protocol/server
        option transport-type tcp/server
        option listen-port 6996
        option bind-address 127.0.0.1
        subvolumes brick
        option auth.ip.brick.allow 127.0.0.1
end-volume
[edit]

Server volume for the 2nd node: server1.vol 

volume brick
        type storage/posix
        option directory /home/cfs1
end-volume

volume server
        type protocol/server
        option transport-type tcp/server
        option listen-port 6997
        option bind-address 127.0.0.1
        subvolumes brick
        option auth.ip.brick.allow 127.0.0.1
end-volume
[edit]

Server volume for the 3rd node: server2.vol 

volume brick
        type storage/posix
        option directory /home/cfs2
end-volume

volume server
        type protocol/server
        option transport-type tcp/server
        option listen-port 6998
        option bind-address 127.0.0.1
        subvolumes brick
        option auth.ip.brick.allow 127.0.0.1
end-volume
[edit]

Server volume for the 4th node: server3.vol 

volume brick
        type storage/posix
        option directory /home/cfs3
end-volume

volume server
        type protocol/server
        option transport-type tcp/server
        option listen-port 6999
        option bind-address 127.0.0.1
        subvolumes brick
        option auth.ip.brick.allow 127.0.0.1
end-volume
[edit]

Configure the client node: client.vol 

volume client0
 type protocol/client
 option transport-type tcp/client
 option remote-host 127.0.0.1
 option remote-port 6996
 option remote-subvolume brick
end-volume

volume client1
 type protocol/client
 option transport-type tcp/client
 option remote-host 127.0.0.1
 option remote-port 6997
 option remote-subvolume brick
end-volume

volume client2
 type protocol/client
 option transport-type tcp/client
 option remote-host 127.0.0.1
 option remote-port 6998
 option remote-subvolume brick
end-volume

volume client3
 type protocol/client
 option transport-type tcp/client
 option remote-host 127.0.0.1
 option remote-port 6999
 option remote-subvolume brick
end-volume

volume bricks
  type cluster/unify
  subvolumes client0 client1 client2 client3
  option rr.limits.min-free-disk 10GB # In later versions, its changed to % based value.
  option scheduler rr
end-volume

### Add writebehind feature
volume writebehind
  type performance/write-behind
  option aggregate-size 131072 # unit in bytes
  subvolumes bricks
end-volume

### Add readahead feature
volume readahead
  type performance/read-ahead
  option page-size 65536     # unit in bytes
  option page-count 16       # cache per file  = (page-count x page-size)
  subvolumes writebehind
end-volume
[edit]

The big picture

Here is how the previous configuration is looking like now:

Image:Glusterfs-cluster.png

[edit]

Start the storage cluster

We need to start the four instances of the glusterfs daemon to serve each directory: 

toad@vlk:~/gfs$ sudo glusterfsd -f ./server0.vol
toad@vlk:~/gfs$ sudo glusterfsd -f ./server1.vol
toad@vlk:~/gfs$ sudo glusterfsd -f ./server2.vol
toad@vlk:~/gfs$ sudo glusterfsd -f ./server3.vol
toad@vlk:~/gfs$ ps w -C glusterfsd
  PID TTY      STAT   TIME COMMAND
13524 ?        Ss     0:00 [glusterfsd]
13526 ?        Ss     0:00 [glusterfsd]
13528 ?        Ss     0:00 [glusterfsd]
13532 ?        Ss     0:00 [glusterfsd]
toad@vlk:~/gfs$

Then we mount the cluster on one mount point: 

toad@vlk:~/gfs$ mkdir ./glusterfs
toad@vlk:~/gfs$ sudo glusterfs -f ./client.vol ./glusterfs
toad@vlk:~/gfs$

We check if the partition is mounted: 

toad@vlk:~/gfs$ mount | tail -1
glusterfs:13548 on /home/toad/gfs/glusterfs type fuse (rw,allow_other,default_permissions)

Hooray ! You have now a glusterFS cluster running on your local computer :)

[edit]

Use your cluster

Now you can create files... 

toad@vlk:~/gfs$ touch ./glusterfs/file{0,1,2,3,4,5,6,7,8}
toad@vlk:~/gfs$

.. and check how the scheduler rr (round-robin) is behaving: 

toad@vlk:~/gfs$ find /home/cfs[0-3] -type f
/home/cfs0/file3
/home/cfs0/file7
/home/cfs1/file2
/home/cfs1/file6
/home/cfs2/file1
/home/cfs2/file5
/home/cfs3/file0
/home/cfs3/file4
/home/cfs3/file8
toad@vlk:~/gfs$
[edit]

Links

  • GlusterFS_Utility_Storage_Cluster Storage cluster is an utility I wrote to quickly set up clusters of X servers and 1 local client node. Its in Perl, and its ugly, it was not supposed to be a release of any kind.
[edit]

What's next ?

Now that you now how to set up a cluster of GlusterFS nodes, you can learn how to encrypt your communication channel using Stunnel on the next tutorial right here.

[edit]

About

Retrieved from "http://www.gluster.org/docs/index.php/GlusterFS_1.2_Configuration_Example_for_Four_Bricks"
Personal tools