The Gluster Blog

Gluster blog stories provide high-level spotlights on our users all over the world

Gluster 4.0 & Kubernetes

vbellur
February 27, 2018

In the recent past, the Gluster community has been focusing on persistent storage for containers as a key use case for the project and Gluster has been making rapid strides in its integration with Kubernetes. The release of 4.0 will deepen that integration and provide a foundation for building more functionality using these two popular open source ecosystems.

If you have not been following our progress in this space, Gluster has been integrated with Kubernetes by using an intelligent middleware known as Heketi. Heketi manages storage on Gluster servers and exposes high level RESTful APIs for consumers to dynamically provision storage from multiple Gluster trusted storage pools. This allows users to create a new volume by simply specifying the desired size and the replication factor. Upon receiving such a request, Heketi figures out the nodes on which bricks have to be placed and creates bricks in a way that Gluster expects. Not only does Heketi provide interfaces for managing the lifecycle of file-based GlusterFS volumes, it also provides lifecycle management for block devices created using gluster-block. Additionally, Heketi provides interfaces for day-2 operations like volume expansion, disk replacement, and node replacement.

Gluster 4.0 introduces glusterd2, the next generation management engine for Gluster. Glusterd2 provides better scale for membership of servers in a trusted storage pool. It provides RESTful interfaces for volume & membership operations and provides a high degree of consistency for state management within a trusted storage pool by integrating with an embedded etcd store. The RESTful interface from Glusterd2 is useful for projects like Heketi which currently have to invoke Gluster’s CLI through ssh or kubectl exec. All such invocations can be replaced with RESTful calls to Glusterd2 to provide a complete service oriented architecture.

For the future, Glusterd2 aims to provide capabilities to manage storage devices, expose higher level APIs for volume management and make it easier for operators to monitor & manage storage. Heketi was originally implemented to address these gaps in Glusterd and already has these key capabilities. Hence we will be incorporating most of Heketi’s key logic in Glusterd2 in the near future. This will result in Glusterd2 exposing both the high level and the classical volume interfaces behind the same consistent API and CLI.  As a consequence, the cluster state that Heketi maintains in its database will be greatly simplified and will only contain minimal information about multiple Gluster trusted storage pools.

Heketi has evolved significantly with the recent release of v6.0.0. As a consequence, the integration with Kubernetes has been improved vastly with the introduction of the following features:

  • Support for provisioning gluster-block backed persistent volumes with the new external gluster-block-provisioner ()
  • Support for expanding persistent volumes (via Heketi)
  • Custom volume names for persistent volumes (via Heketi)
  • Prometheus metrics collection for gluster volumes (directly in Kubernetes)
  • Improved device management with resync API
  • Enhanced robustness for Heketi’s database

Future releases of Heketi are expected to add support for arbiter-volumes, complement features being developed in upstream Kubernetes, and expose interfaces for upcoming features in Kubernetes like Snapshots & Cloning. There are ongoing efforts to provide an enhanced disaster recovery mechanism for persistent volumes using the geo-replication feature in Gluster.

Gluster-block also has observed a slew of improvements in the recent past. The upcoming release of gluster-block will contain:

  • Ability to migrate configured block-devices across nodes
  • Creation of  new block devices with existing backing files
  • Configuration option for deletion of backing files

Further releases of gluster-block are expected to add support for loopback devices and provide mechanisms to snapshot and clone block devices.

With work ongoing in projects like gluster-kubernetes, gluster-subvol, gluster-csi-driver & gluster-s3 it has been an exciting phase for evolving Gluster as a robust & flexible storage backend for containers. Stay tuned as we build out more features on the foundation that Gluster 4.0 provides for container storage and help the broader cause of making application deployment simpler with microservices!

BLOG

  • 26 Apr 2019
    Gluster Monthly Newsletter, April 2...

    Upcoming Community Happy Hour at Red Hat Summit! Tue, May 7, 2019, 6:30 PM – 7:30 PM EDT https://cephandglusterhappyhour_rhsummit.eventbrite.com has all the details. Gluster 7 Roadmap Discussion kicked off for our 7 roadmap on the mailing lists, see [Gluster-users] GlusterFS v7.0 (and v8.0) roadmap discussion https://lists.gluster.org/pipermail/gluster-users/2019-March/036139.html for more details. Community...

    Read more
  • 24 Apr 2019
    Community Survey Feedback, 2019

    In this year’s survey, we asked quite a few questions about how people are using Gluster, how much storage they’re managing, their primary use for Gluster, and what they’d like to see added. Here’s some of the highlights from this year!

    Read more
  • 24 Apr 2019
    How to Deploy the OpenVPN Encryptio...

    This is part of a new series on using Gluster! OpenVPN is open source software that serves as the basis for a Virtual Private Network capable of supporting a point-to-point or site-to-site connection. Along with the fact that it’s free to use, it also has the benefit of being one...

    Read more