GlusterFS Encrypted network
From GlusterDocumentation
This article was last updated for the version 1.2.3 of GlusterFS
Contents |
Introduction
One possible concern about setting up GlusterFS in a WAN is the security of the data on the network. As of today (version 1.2.3 is the latest release), there is no mean to protect communication channels between a GlusterFS client and a GlusterFS server. In this article, I will quickly show you how it is possible to very simply protect those channels.
Prerequisites
Architecture
The network will be composed by 4 storage nodes (GlusterFS server daemons) and 1 client node (GlusterFS client daemon). Please refer to GlusterFS_Configuration_Example_for_Four_Bricks to discover how to set it up.
So far, each client is connecting directly to the server, without a protection against a malicious eye on the network. The current connection could look like this:
Each couple of client/server has a single connection. This network design of GlusterFS makes the channel very easy to protect by setting up a single tunnel for the whole connection. All we need to do now is to run 2 instances of stunnel for each connection (one on the client, one on the server). The instance of stunnel running on the client will simply redirect a local port to a remote one, using SSL and our own x509 certificate to encrypt the telecommunication. On the server side, stunnel will listen for incoming connections on a port (let's say 7996), decrypt the telecommunication and forward the stream to the GlusterFS daemon.
This is what we want to achieve:
The problem
There is a problem with all this nice design though. Indeed, GlusterFS team decided not to allow connections on the server that would come from unpriviligied tcp ports (< 1024), and that for security concerns. To bypass this limitation, there are 2 possibilities:
- patch glusterfs to allow unpriviligied source ports;
- patch stunnel to be able to set a specific source port for the connection.
Patch glusterfs
Write the patch:
--- ./xlators/protocol/server/src/proto-srv.c 2007-03-29 16:09:52.000000000 +0200
+++ ./xlators/protocol/server/src/proto-srv.c.new 2007-03-29 16:09:43.000000000 +0200
@@ -2417,7 +2417,7 @@
"mop_setvolume: received port = %d",
ntohs (_sock->sin_port));
- if (ntohs (_sock->sin_port) < 1024) {
+ // if (ntohs (_sock->sin_port) < 1024) {
char *ip_addr_str = NULL;
char *tmp;
char *ip_addr_cpy = strdup (allow_ip->data);
@@ -2456,11 +2456,11 @@
}
free (ip_addr_cpy);
goto fail;
- } else {
- dict_set (dict, "ERROR",
- str_to_data ("Authentication Range not specified in volume spec"));
- goto fail;
- }
+ // } else {
+ // dict_set (dict, "ERROR",
+ // str_to_data ("Authentication Range not specified in volume spec"));
+ // goto fail;
+ // }
} else {
char msg[256] = {0,};
sprintf (msg,
And apply it inside glusterfs root:
$ patch -p0 < ./no-priviligied-ports.patch patching file ./xlators/protocol/server/src/proto-srv.c
Patch glusterfs 1.3.8
Patching glusterfs 1.3.8 os more easy. You need only to modify a line within the file auth/ip/src/ip.c. Replace:
#define PRIVILEGED_PORT_CIELING 1024
with:
#define PRIVILEGED_PORT_CIELING 65536
Patch stunnel
FIXME: patch missing, if someone is coding it one day, she would be nice to include it here
Creation of the x509 certificates
To quickly generate a certificate for stunnel, use the following command:
openssl req -new -x509 -days 365 -nodes -out certificate.pem -keyout certificate.pem -batch
To generate your 4 certificates for your GlusterFS servers, you can proceed as following:
toad@vlk:~/gfs$ for i in {0..3}; do \
echo generating certificate for storage node $i...; \
openssl req -new -x509 -days 365 -nodes -out ./server$i.pem -keyout ./server$i.pem -batch 1> /dev/null \
; done
generating certificate for storage node 0...
Generating a 1024 bit RSA private key
........++++++
..........++++++
unable to write 'random state'
writing new private key to './server0.pem'
-----
generating certificate for storage node 1...
Generating a 1024 bit RSA private key
......++++++
.............++++++
unable to write 'random state'
writing new private key to './server1.pem'
-----
generating certificate for storage node 2...
Generating a 1024 bit RSA private key
.....++++++
...................................................................++++++
unable to write 'random state'
writing new private key to './server2.pem'
-----
generating certificate for storage node 3...
Generating a 1024 bit RSA private key
.......++++++
...............................................++++++
unable to write 'random state'
writing new private key to './server3.pem'
-----
Raising the protected tunnels
On the servers' side...
Now we will configure 4 tunnels on the server side of the storage. For this we need to write down 4 configuration files describing how each instance of stunnel should behave. As we are lazy and smart, we will write a script to do it for us.
Open your favorite editor (vi or emacs ;)
vi ./gen_stunnel_conf_for_server
and copy / paste the following shell script (working for Bash):
#! /bin/sh server_id=$1 default_port=6996 server_port=`expr $default_port + $server_id` listen_port=`expr $server_port + 1000` pid=$PWD/stunnel_server_$server_id.pid cat <<EOF cert = ./server$server_id.pem pid = $pid [glusterfsd] accept = 127.0.0.1:$listen_port connect = 127.0.0.1:$server_port EOF
Give the ability to execute the shell script directly:
toad@vlk:~/gfs$ chmod a+x ./gen_stunnel_conf_for_server
And finally test it, it should output the content of the stunnel configuration file we need:
toad@vlk:~/gfs$ ./gen_stunnel_conf_for_server 0 cert = ./server0.pem pid = /home/toad/gfs/stunnel_server_0.pid [glusterfsd] accept = 127.0.0.1:7996 connect = 127.0.0.1:6996
Let's now massively write down the configuration and launch our stunnel daemons:
toad@vlk:~/gfs$ for id in {0..3}; do \
./gen_stunnel_conf_for_server $id > ./server$id.stunnel \
; done
toad@vlk:~/gfs$ ls -l ./server*.stunnel
-rw-r--r-- 1 toad toad 102 2007-01-16 20:55 ./server0.stunnel
-rw-r--r-- 1 toad toad 102 2007-01-16 20:55 ./server1.stunnel
-rw-r--r-- 1 toad toad 102 2007-01-16 20:55 ./server2.stunnel
-rw-r--r-- 1 toad toad 102 2007-01-16 20:55 ./server3.stunnel
toad@vlk:~/gfs$ for id in {0..3}; do \
stunnel4 ./server$id.stunnel \
; done
toad@vlk:~/gfs$ ps w -C stunnel4
PID TTY STAT TIME COMMAND
25674 ? Ss 0:00 stunnel4 ./server0.stunnel
25676 ? Ss 0:00 stunnel4 ./server1.stunnel
25678 ? Ss 0:00 stunnel4 ./server2.stunnel
25680 ? Ss 0:00 stunnel4 ./server3.stunnel
... and on the client's side
Same thing on the client side, we need to write the configuration files for 4 stunnel daemons, plus to modify the client's volume specification file of GlusterFS to take into account those changes.
Let's proceed:
toad@vlk:~/gfs$ emacs ./gen_stunnel_conf_for_client
Write down the following script:
toad@vlk:~/gfs$ cat ./gen_stunnel_conf_for_client #! /bin/sh server_id=$1 default_port=`expr 6996 + $server_id` listen_port=`expr $default_port - 1000` forward_port=`expr $default_port + 1000` cat <<EOF client = yes pid = [glusterfs] accept = 127.0.0.1:$listen_port connect = 127.0.0.1:$forward_port EOF
Let's test if the output is correct:
toad@vlk:~/gfs$ ./gen_stunnel_conf_for_client 2 client = yes pid = [glusterfs] accept = 127.0.0.1:5998 connect = 127.0.0.1:7998
Let's set the execute bit on the file:
toad@vlk:~/gfs$ chmod a+x ./gen_stunnel_conf_for_client
We can finally set up all tunnels for the clients:
toad@vlk:~/gfs$ for id in {0..3}; do ./gen_stunnel_conf_for_client $id > ./client$id.stunnel; done
toad@vlk:~/gfs$ for id in {0..3}; do stunnel4 ./client$id.stunnel; done
toad@vlk:~/gfs$ ps w -C stunnel4 | grep client
26284 ? Ss 0:00 stunnel4 ./client0.stunnel
26286 ? Ss 0:00 stunnel4 ./client1.stunnel
26288 ? Ss 0:00 stunnel4 ./client2.stunnel
26290 ? Ss 0:00 stunnel4 ./client3.stunnel
And check if they are correctly waiting for incoming connections:
toad@vlk:~/gfs$ sudo netstat --inet -lnp | grep stunnel4 tcp 0 0 127.0.0.1:5996 0.0.0.0:* LISTEN 26284/stunnel4 tcp 0 0 127.0.0.1:5997 0.0.0.0:* LISTEN 26286/stunnel4 tcp 0 0 127.0.0.1:5998 0.0.0.0:* LISTEN 26288/stunnel4 tcp 0 0 127.0.0.1:5999 0.0.0.0:* LISTEN 26290/stunnel4 tcp 0 0 127.0.0.1:7996 0.0.0.0:* LISTEN 25674/stunnel4 tcp 0 0 127.0.0.1:7997 0.0.0.0:* LISTEN 25676/stunnel4 tcp 0 0 127.0.0.1:7998 0.0.0.0:* LISTEN 25678/stunnel4 tcp 0 0 127.0.0.1:7999 0.0.0.0:* LISTEN 25680/stunnel4
Clients' volumes specification
Now that all tunnels are ready to operate, we can finally modify the clients' volumes specification file in order to force GlusterFS to connect to stunnel instead of the GlusterFS servers:
volume client0 type protocol/client option transport-type tcp/client option remote-host 127.0.0.1 option remote-port 5996 # instead of 6996 option remote-subvolume brick end-volume volume client1 type protocol/client option transport-type tcp/client option remote-host 127.0.0.1 option remote-port 5997 # instead of 6997 option remote-subvolume brick end-volume volume client2 type protocol/client option transport-type tcp/client option remote-host 127.0.0.1 option remote-port 5998 # instead of 6998 option remote-subvolume brick end-volume volume client3 type protocol/client option transport-type tcp/client option remote-host 127.0.0.1 option remote-port 5999 # instead of 6999 option remote-subvolume brick end-volume
Launching GlusterFS
At last, we start the GlusterFS cluster to see if everything goes ok:
toad@vlk:~/gfs$ sudo glusterfsd -f $PWD/server0.vol toad@vlk:~/gfs$ sudo glusterfsd -f $PWD/server1.vol toad@vlk:~/gfs$ sudo glusterfsd -f $PWD/server2.vol toad@vlk:~/gfs$ sudo glusterfsd -f $PWD/server3.vol toad@vlk:~/gfs$ sudo glusterfs -f $PWD/client.vol ./glusterfs toad@vlk:~/gfs$ df -H ./glusterfs Filesystem Size Used Avail Use% Mounted on glusterfs:1002 115G 108G 408M 100% /home/toad/gfs/glusterfs
You have now your cluster's connections encrypted, enjoy ! :)
About
- Author: Julien Perez <mailto:julien.perez@epsylonia.net>
- Copyright © 2007 J&J Associates, s.r.o.
