Setting up AFR on two servers with server side replication
From GlusterDocumentation
Contents |
Introduction
NOTE: Before beginning, BE SURE TO HAVE A BACKUP OF ALL YOUR DATA BEFOREHAND! Remember to always keep a backup handy if you are working on live/important data. Things that CAN go wrong have a way of going wrong!
In this tutorial we will walk through setting up an AFR cluster using two servers and a single client, using server side replication.
This tutorial is based on the assumption that you have already gone through installing GlusterFS and it is in a working state. If not please refer to Install and run GlusterFS v1.3 in 10mins.
NOTE: Also see Setting up AFR on two servers with client side replication
The Scenario
Here we assume we are working with three machines. Two are servers and one is the client. Server1 and server2 both have raids mounted under /mnt/raid. Each of these will have a glusterfs exported volume directory of web which the client will mount as /mnt/web. We will be using server side AFR, meaning the servers will handle the file replication as opposed to doing this client side. Be sure to read the "Things to know / gotchas".
Below are the machines defined.
Server1
- IP address : 192.168.0.1
- Export volume directory : /mnt/raid/web
Server2
- IP address : 192.168.0.2
- Export volume directory : /mnt/raid/web
Client
- IP address : 192.168.0.3
- Server volume mount point : /mnt/web
Preparing the directories
Ok first lets create our server volume directories. On both server1 and server2, create a directory called web under /mnt/raid/.
$ mkdir /mnt/raid/web
Now lets make sure we do not have any extended attributes, we remove them here if they happen to exist (for instance if you are on your second or more attempt at this). Type these commands on both server1 and server2:
$ setfattr -x trusted.glusterfs.version /mnt/raid/web $ setfattr -x trusted.glusterfs.createtime /mnt/raid/web
Configuring the volume spec files
Now that we have our volume directories on both servers, we will create the server and client volume spec files. The volume spec files provided here are very basic and only provided to illustrate getting AFR to work. Once you have a working setup you can start adding performance translators, but that is beyond the scope of this particular tutorial.
Server volume spec file
Both servers will use a similiar server volume spec file, exporting the /mnt/raid/web directory. This example does not use/include any performance translators. Please refer to GlusterFS Volume Specification Examples for more ideas on how to configure spec files.
The spec file for the servers will be different only in the IP address for brick2.
On server1, create or edit the server volume specification file (/etc/glusterfs/glusterfs-server.vol) to match the example below.
volume brick1 type storage/posix option directory /mnt/raid/web end-volume volume brick2 type protocol/client option transport-type tcp/client option remote-host 192.168.0.2 # IP address of server2 option remote-subvolume brick1 # use brick1 on server2 end-volume volume afr type cluster/afr subvolumes brick1 brick2 end-volume volume server type protocol/server option transport-type tcp/server subvolumes brick1 afr option auth.ip.brick1.allow 192.168.0*,127.0.0.1 option auth.ip.afr.allow 192.168.0*,127.0.0.1 end-volume
On server2, create or edit the server volume specification file (/etc/glusterfs/glusterfs-server.vol) to match the example below.
volume brick1 type storage/posix option directory /mnt/raid/web end-volume volume brick2 type protocol/client option transport-type tcp/client option remote-host 192.168.0.1 # IP address of server1 option remote-subvolume brick1 # use brick1 on server1 end-volume volume afr type cluster/afr subvolumes brick2 brick1 end-volume volume server type protocol/server option transport-type tcp/server subvolumes brick1 afr option auth.ip.brick1.allow 192.168.0*,127.0.0.1 option auth.ip.afr.allow 192.168.0*,127.0.0.1 end-volume
Client volume spec file
In this tutorial we are assuming there is only a single client, but you can add more simply by copying the below client volume spec file to other machines.
Create or edit the client volume specification file (/etc/glusterfs/glusterfs-client.vol) to match the example below.
volume brick type protocol/client option transport-type tcp/client # for TCP/IP transport option remote-host 192.168.0.1 # IP address of the server option remote-subvolume afr # name of the remote volume end-volume
Start the GlusterFS servers
Start the GlusterFS server (glusterfsd) on both servers. Then tail the log to make sure we had no errors.
NOTE: The location of your log files may be different if you specified a --prefix during installation.
$ glusterfsd -f /etc/glusterfs/glusterfs-server.vol $ tail /var/log/glusterfsd.log
Mount the server volume
Now on the client machine mount the server volume.
$ glusterfs -f /etc/glusterfs/glusterfs-client.vol /mnt/web $ cd /mnt/web $ ls -l
You should see no files or directories listed. Now lets test out our cluster.
Verify AFR replication
Create a new file
You should be on the client machine. To test out our setup, we will touch a single file and then look on both servers to make sure it was created successfully.
On the client machine, first make sure you are in the /mnt/gfs directory.
$ cd /mnt/raid/web $ touch file.txt $ ls file.txt
Check for new file on both servers
Now that we created a file, lets make sure that it was replicated on both servers. To do this, get on each server and go to the /mnt/raid/web/ directory and do an 'ls'.
$ cd /mnt/raid/web $ ls file.txt
Great! The file was created successfully.
Conclusion
What you have here is a client machine connecting to server1 only. When a client performs a write, it sends it to server1. Since server1 has an afr brick defined using a local directory and a remote volume on server2, it writes the file to itself and sends a write to server.
Now what if you want to add a second client? How do you make it connect to the other server? Well on a second client machine you could edit its client volume spec file to look like the example below.
On a new client:
$ cat /etc/glusterfs/glusterfs-client.vol volume brick type protocol/client option transport-type tcp/client # for TCP/IP transport option remote-host 192.168.0.2 # IP address of the server option remote-subvolume afr # name of the remote volume end-volume
The option remote-host 192.168.0.2 would tell this client machine to mount the afr volume on server2. At the same time, the first client is connected to server1. See the section below for 'Things to know / gotchas'.
Breaking things
Ok for this section I am departing from the two server one client model, instead at this point we should have two servers and two clients. Client1 is connected to server1 and client2 is connected to server2.
Now for some information on what happens if a server goes down.
Lets kill glusterfsd on server2 to simulate it crashing.
On server2:
$ killall glusterfsd
Are we still working? Lets check on client1:
$ cd /mnt/web $ ls file.txt
Ok its working, now for client2:
$ cd /mnt/web $ ls ls: cannot open directory .: Transport endpoint is not connected
Uh oh, client2 was connected to server2 which is now down, it has no mount point! Ok for now lets go back to client1 and see if we can perform a write operation:
$ echo "hello" > file.txt -bash: file.txt: Input/output error
Yea things are looking pretty bad from here. Lets try starting glusterfsd back up on server2 and see if it picks back up.
On server2:
$ glusterfsd -f /etc/glusterfs/glusterfs-server.vol
Now lets see if client2 is back and working:
$ ls file.txt
Great! Now lets try a write:
$ echo "hello" > file.txt $ cat file.txt hello
As you can see the cluster came back. During the time that server2 is down, the file system does not seem to be available. This does not bode well when we need or want to be able to down one of the data servers for whatever reason. Hence client side AFR is recommended over server side.
Things to keep in mind / gotchas
Refer to AFR (Automatic File Replication) - Things to keep in mind and gotchas.
Credits
- Author : Brandon Lamb
- Email : brandonlamb@gmail.com
- GlusterFS rules!
