<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 17, 2015 at 8:35 PM, Kaushal M <span dir="ltr"><<a href="mailto:kshlmster@gmail.com" target="_blank">kshlmster@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">At the Gluster Summit, everyone agreed that GlusterD should be the<br>
first component to be targeted for GlusterFS-4.0. We had good<br>
discussions on what GlusterD lacks currently and what is required for<br>
GlusterD-2.0. KP and I had promised to send an update to mailing list<br>
summarizing the discussions, and this is it.<br>
<br>
Along with the summary, we'll also be discussing our plans for Manila<br>
integration and how we are planning to do it with GlusterD-2.0.<br>
<br>
## Gluster Summit Summary<br>
In the summit, KP and I presented a talk titled *GLUSTERD NEXT GEN<br>
OR: HOW WE CAN STOP BEING A ROADBLOCK AND INSTEAD BECOME THE ROAD TO<br>
SCALABILITY*. The slides can be viewed at [1][1]. There is no video<br>
recording of it unfortunately.<br>
<br>
The summary of the presentation is below.<br>
<br>
### Problems<br>
GlusterD, as it is currently, is not scalable which will prevent<br>
GlusterFS as whole from scaling. The scaling issues can be classified<br>
into,<br>
- Node scalability<br>
- Maintenance scalability<br>
- Integration scalability<br>
- Contribution scalability<br>
<br>
#### Node scalability<br>
This is the general scalability we all think about, scale in terms of<br>
node/machine/clients. GlusterD scalability is help back in this<br>
because of the store, transaction and synchronization mechanisms used<br>
in GlusterD.<br>
<br>
#### Maintenance scalability<br>
Maintenance scalability is to with the problems we as GlusterD<br>
maintainers faced. This is mainly related to the huge, monolithic code<br>
base of the current GlusterD, which makes splitting of maintenance and<br>
ownership tasks hard.<br>
<br>
#### Integration scalability<br>
Integration scalability can split into internal and external integration.<br>
Internal integration is the integration dealing with new features<br>
being added to GlusterFS. Every new feature being added needs to touch<br>
GlusterD or CLI in some way. This has generally been done with<br>
copy/paste coding which has added to the maintenence overhead.<br>
External integration is the integration of Gluster with other projects<br>
or other projects with Gluster. This is hurt due to the<br>
non-availability of a proper API for GlusterD operations. All<br>
interaction with GlusterD currently happens only via the CLI. And the<br>
output we provide is generally inconsistent to be programatically<br>
parsed.<br>
<br>
#### Contribution scalability<br>
Getting new contributors for GlusterD is hard. New contributors are<br>
put off because GlusterD is hard to understand, and because there<br>
isn't enough documentation.<br>
<br>
<br>
So GlusterD-2.0 will need to<br>
- be scalable to 1000s of nodes<br>
- have lower maintenance costs<br>
- enable better external and internal integrations<br>
- make it easier for newer contributors<br>
<br>
### Design characteristics for GlusterD-2.0<br>
For GlusterD-2.0 to satisfy the above listed requirements and solve<br>
the problems listed before, it should have the following<br>
characteristics.<br></blockquote><div><br></div><div>One other design characteristic I can think of is to have GlusterD-2.0<br></div><div>be able to make best use of Nodes based on the Node topology, not sure<br></div><div>if this is covered by any other feature below ?<br><br></div><div>IOW, admin should be able to provide a hierarchy of how nodes are provisioned<br></div><div>(rack, site, zone, region, geo etc) and glusterd should use this info while<br></div><div>provisioning bricks for volume creation<br><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
#### Centralized store<br>
This should help with our numbers scalability issues. GlusterD-2.0<br>
will be built around a centralized store. This means, instead of the<br>
GlusterD volume and peer information being persisted on all nodes, it<br>
will be stored only on a subset of the nodes in a trusted storage<br>
pool.<br>
<br>
We are looking at solutions like etcd and consul, both of which<br>
provide a distributed key/value store (and some more useful features<br>
on top), to provide the centralized store. The transaction mechanism<br>
for Gluster operations will be built around this centralized store.<br>
<br>
Moving to an external store provider and a transaction framework built<br>
around it will reduce a lot of the complexity in GlusterD.<br>
<br>
#### Plugins<br>
This mainly for the maintainability and internal integration aspects.<br>
GlusterD-2.0 will have a plug-gable, modular design. We expect all the<br>
commands of GlusterD to be implemented as plugins. Certain other parts<br>
of GlusterD, including things like volgen, volume-set, rest api, etc.<br>
This will allow new features to be integrated in to GlusterD easily.<br>
The code for these plugins is expected to live with the feature, and<br>
not in GlusterD.<br>
<br>
Doing a plugin design requires the defining of well defined plugin<br>
interfaces to not just plug into GlusterD-2.0, but also interact with<br>
GlusterD well. I'll be posting a document on this to the mailing list<br>
soon.<br>
<br>
The GlusterD maintainers/team will be doing the implementation of a<br>
few core commands required including volume create, start, stop,<br>
delete, expand, shrink and set/reset options.<br>
<br>
#### Rest API<br>
To provide better a better method for external projects to hook on to.<br>
GlusterD-2.0 will provide a HTTP REST API, with JSON output and proper<br>
API versioning. We will be looking for inspiration from other storage<br>
solutions to define the APIs presented to the users and admins.<br>
<br>
#### Logging<br>
To help with debuggability, GlusterD-2.0 will provide better logs that<br>
should allow easier tracking of operations across machines. We are<br>
currently looking at the oVirt logging mechanism, which tags all logs<br>
of an operation with a unique id, for inspiration.<br>
<br>
#### Documentation<br>
We hope to do a much better job of documenting the technical details<br>
of GlusterD-2.0, starting with design documents. We still don't have<br>
<br>
<br>
## Manila and GlusterD-2.0<br>
<br>
Manila is the File System as a Service component of OpenStack. Manila<br>
has two GlusterFS providers neither of which currently support the<br>
full Manila API. One uses a sub-directory over NFS approach and the<br>
other uses a volume based approach. Both of them require specific<br>
features from GlusterFS to be completely functional by the next<br>
OpenStack release, which is expected for October 2015. Csaba listed<br>
these requirements at [2][2]. This lists two possible approaches,<br>
1. Implement directory level operations (required by the sub-directory approach)<br>
2. Implement intelligent volume creation (IVC) (required for the<br>
volume based approach)<br>
(Please read the linked mail-thread for more context)<br>
<br></blockquote><div><br></div><div>I have 2 things to say here :<br><br>1) Manila uses GlusterFS SSL auth mode, so is there any discussion happening<br></div><div>around adding support in GlusterD-2.0 for managing SSL certificates ?<br><br></div><div>For eg: Other (eg Netapp) storage arrays do provide basic digital cert mgmt<br></div><div>support for server and client auth.<br><br></div><div>I feel it would be good to have glusterd provide support to generate, install, manage<br></div><div>self-signed and/or CA signed digital certs.<br><br></div><div>This will not just help Manila usecase, but also help GlusterFS provide a complete<br></div><div>solution for digital cert management which will aid the SSL auth support feature of GlusterFS<br><br></div><div>In fact, this can be done in a modular way where in the default implm will be that of GlusterD<br></div><div>cert module, but optionally one can use openstack Barbican service too as a cert mgmt service<br><br></div><div>2) In the manila community there was some intense discussion on the definition of multi-tenancy <br>when applied to storage and network level multi-tenancy was voted as being very important in<br></div><div>Manila cloud usecase. Do we have any thoughts on how GlusterD-2.0 / GlusterFS-4.0 can <br></div><div>look at providing tenant separation at network layer ?<br><br></div><div>thanx,<br></div><div>deepak<br></div></div><br></div></div>