Recently, I co-authored a patch for Nova that allows for GlusterFS volumes to be accessed directly from Qemu using libgfapi. Previous to this patch, it was only possible to access glusterfs by mounting them using GlusterFS FUSE client.
Recently, in version 1.3, QEMU introduced a GlusterFS block driver based on libgfapi.
Libgfapi is a POSIX-like C library shipped along with GlusterFS, which allows to access Gluster’s volumes without passing through its FUSE client. This integration brings in some benefits but, the most relevant ones are:
There’s no special configuration needed to use this, as long as you have QEMU >=1.3 and GlusterFS>=3.4 you should be fine. This is an example of what you can do:
qemu-img create gluster://$GLUSTER_HOST/$GLUSTER_VOLUME/images 5G
If you’d like to know more about this specific implementation, I suggest you to read this well-explained blog post where all the details about GlusterFS’s driver are explained.
Small changes were required in order to make this work.
A new configuration parameter – qemu_allowed_storage_drivers
– was added to enable or disable this feature. Direct access is disabled by default – this means LibvirtGlusterfsVolumeDriver
will mount gluster’s volume – and can be enabled by adding ‘gluster’ to the new configuration parameter.
The patch is quite small – pasted right bellow – and just required to modify libvirt’s configuration object to let QEMU know it should use use a network device and access it using gluster’s protocol.
- options = connection_info['data'].get('options')
- path = self._ensure_mounted(connection_info['data']['export'], options)
- path = os.path.join(path, connection_info['data']['name'])
- conf.source_type = 'file'
- conf.source_path = path
+
+ data = connection_info['data']
+
+ if 'gluster' in CONF.qemu_allowed_storage_drivers:
+ vol_name = data['export'].split('/')[1]
+ source_host = data['export'].split('/')[0][:-1]
+
+ conf.source_ports = [None]
+ conf.source_type = 'network'
+ conf.source_protocol = 'gluster'
+ conf.source_hosts = [source_host]
+ conf.source_name = '%s/%s' % (vol_name, data['name'])
+ else:
+ path = self._ensure_mounted(data['export'], data.get('options'))
+ path = os.path.join(path, data['name'])
+ conf.source_type = 'file'
+ conf.source_path = path
NOTE: From now on, I’ll assume you’ve installed nova, cinder, GlusterFS, libvirt and all the required pieces (I suggest you to use devstack if you’re just testing this feature).
$ # Edit nova.conf and add gluster to the qemu_allowed_storage_drivers list.
$ dd if=/dev/zero of=gluster-volume bs=1M count=4096
$ mkfs.xfs -f gluster-volume
So far we created an XFS loop device. We now have to mount it and add it to glusterfs. Execute as root:
$ mkdir /srv/brick2
$ mount -o loop -t xfs gluster-volume /srv/brick2
$ gluster volume create <ip>:/srv/brick2 testvol1
$ gluster volume start testvol1
$ # Edit /etc/cinder/glusterfs_shares and add "$GLUSTER_HOST:/testvol1"
$ # Restart cinder-volume
We just created a directory for our brick and mounted our loop device there. Then we created a volume in glusterfs, started it and then we added it to the glusterfs_shares
file, which is were all glusterfs “shares” are specified. It is important to restart Cinder’s volume service so it can reload the glusterfs_shares
file.
We can now create our volume and attach it to our running instance.
$ cinder create 1 # Creates a 1GB volume
$ nova volume-attach <vm> <vol> auto
If nothing bad happened – you know, Murphy – you should see a new device attached to your instance. You can verify this by either ssh’ing into the running instance and listing all available devices – $ ls /dev/vd* – or dumping libvirt instance’s XML.
Enjoy!
2020 has not been a year we would have been able to predict. With a worldwide pandemic and lives thrown out of gear, as we head into 2021, we are thankful that our community and project continued to receive new developers, users and make small gains. For that and a...
It has been a while since we provided an update to the Gluster community. Across the world various nations, states and localities have put together sets of guidelines around shelter-in-place and quarantine. We request our community members to stay safe, to care for their loved ones, to continue to be...
The initial rounds of conversation around the planning of content for release 8 has helped the project identify one key thing – the need to stagger out features and enhancements over multiple releases. Thus, while release 8 is unlikely to be feature heavy as previous releases, it will be the...