A Gluster Block Interface – Performance and Configuration
This post shares some experiences I’ve had in simulating a block device in gluster.
The block device is a file based image, which acts as the backend for the Linux SCSI target. The file resides in gluster, so enjoys gluster’s feature set. But the client only sees a block device. The Linux SCSI target presents it over iSCSI.
Some more information on how to set this up is at the bottom of this post.
But where to mount gluster?
There are three options to consider. Call them client, server, and gateway.
The configurations are depicted in the diagram below:
-
Configuring the block device at the server means the client only requires an iSCSI initiator.
-
Configuring the block device at the client allows I/O to fan out to different nodes.
-
Configuring the block device at a gateway allows I/O to fan out to different nodes without changing the client.
These options have their pros and cons. For example, a gateway minimizes client overhead while providing fan-out. OTOH, the customer must provide an additional node.
In my case, the objective was to minimize customer burden. Server-side configuration is probably the best choice. After chatting with some colleagues here at Red Hat, thats what we settled on.
I ran some simple performance tests using the “fio” tool to generate I/O.
-
Up to 10 fio processes were started.
-
The queue depth for each was 32.
-
Each process sends I/O to its own slice of the volume.
-
client and server block cache is flushed between tests
-
echo 3 > /proc/sys/vm/drop_caches
-
-
64k records
-
50G volume replicated over two nodes
-
Gluster version: 3.4.0.33
Warning!
-
Only two nodes were used in performance testing. Ideally more nodes would be used, but that equipment is not readily available.
-
There are numerous options to tune, including the queue depth, number of streams, number of paths, gluster volume configuration, block size, number of targets etc. Parameters were chosen based on trial and error rather than formal methodology.
1 KVM client – 6VCPU x 20GB client (pinned) <===> 2 file servers baremetal 12x20GB
4 x 64GB blk virtio file systems (cache=none)
(1 x 12disk RAID6 gluster brick / server)
—————–
For the interested, here is a cookbook to set it up. For more information, see [1] and [2].
-
Mount gluster locally on your server.
$ mount -t glusterfs 127.0.0.1:gserver /mnt
-
Create a large file representing your block device within the gluster fs.
$ dd if=/dev/zero of=/mnt/disk bs=2G count=25
-
Create a target using the file as the backend storage.
If necessary, download the Linux SCSI target. Then start the service.
$ yum install scsi-target-utils
$ service tgtd start
You must give an iSCSI Qualified name (IQN), in the format :
iqn.yyyy-mm.reversed.domain.name:OptionalIdentifierText
where:
yyyy-mm represents the 4-digit year and 2-digit month the device was started (for example: 2011-07);
$ tgtadm –lld iscsi –op new –mode target –tid 1 -T iqn.20013-10.com.redhat
You can look at the target:
$ tgtadm –lld iscsi –op show –mode conn –tid 1
Session: 11
Connection: 0
Initiator: iqn.1994-05.com.redhat:cf75c8d4274d
IP Address: 172.17.11.25
Next, add a logical unit to the target
$ tgtadm –lld iscsi –op new –mode logicalunit –tid 1 –lun 1 -b /mnt/disk
Allow any initiator to access the target.
$ tgtadm –lld iscsi –op bind –mode target –tid 1 -I ALL
-
Now it’s time to set up your client.
Discover your targets.
$ iscsiadm –mode discovery –type sendtargets –portal 192.168.1.2
Login to your target session.
$ iscsiadm –mode node –targetname iqn.2001-04.com.example:storage.disk1.amiens.sys1.xyz –portal 192.168.1.2:3260 –login
You should have a new SCSI disk. You will see it created in /var/log/messages. You will see it in lsblk.
You can send I/O to it:
$ dd if=/dev/zero of=/dev/sda bs=4K count=100
To tear down the session when you are finished:
$ iscsiadm -m node -T iqn.2001-04.com.redhat -p 192.168.1.2 -u
[1]
Linux Journal article on making software backed iSCSI targets
[2]
How to set up Linux iSCSI targets with tgt