<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 19, 2016 at 9:22 PM, Niels de Vos <span dir="ltr"><<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Mon, Sep 19, 2016 at 10:31:11AM -0400, Luis Pabón wrote:<br>
> Using qemu is interesting, but the I/O should be using the IO path of QEMU block API. If not,<br>
> TCMU would not know how to work with QEMU dynamic QCOW2 files.<br>
><br>
> Now, if TCMU already has this, then that would be great!<br>
<br>
</span>It has a qcow2 header, maybe you guys are lucky!<br>
<a href="https://github.com/open-iscsi/tcmu-runner/blob/master/qcow2.h" rel="noreferrer" target="_blank">https://github.com/open-iscsi/<wbr>tcmu-runner/blob/master/qcow2.<wbr>h</a></blockquote><div><br></div><div>Sent the earlier mail before seeing this mail :-). So yes, what we discussed is to see if this qemu in tcmu can internally use gfapi for doing the operations or not is something we are trying to find out.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
<span class="HOEnZb"><font color="#888888"><br>
Niels<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
><br>
> - Luis<br>
><br>
> ----- Original Message -----<br>
> From: "Prasanna Kalever" <<a href="mailto:pkalever@redhat.com">pkalever@redhat.com</a>><br>
> To: "Niels de Vos" <<a href="mailto:ndevos@redhat.com">ndevos@redhat.com</a>><br>
> Cc: "Luis Pabón" <<a href="mailto:lpabon@redhat.com">lpabon@redhat.com</a>>, "Stephen Watt" <<a href="mailto:swatt@redhat.com">swatt@redhat.com</a>>, "gluster-devel" <<a href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a>>, "Ramakrishna Yekulla" <<a href="mailto:rreddy@redhat.com">rreddy@redhat.com</a>>, "Humble Chirammal" <<a href="mailto:hchiramm@redhat.com">hchiramm@redhat.com</a>><br>
> Sent: Monday, September 19, 2016 7:13:36 AM<br>
> Subject: Re: [Gluster-devel] [Heketi] Block store related API design discussion<br>
><br>
> On Mon, Sep 19, 2016 at 4:09 PM, Niels de Vos <<a href="mailto:ndevos@redhat.com">ndevos@redhat.com</a>> wrote:<br>
> ><br>
> > On Mon, Sep 19, 2016 at 03:34:29PM +0530, Prasanna Kalever wrote:<br>
> > > On Mon, Sep 19, 2016 at 10:13 AM, Niels de Vos <<a href="mailto:ndevos@redhat.com">ndevos@redhat.com</a>> wrote:<br>
> > > > On Tue, Sep 13, 2016 at 12:06:00PM -0400, Luis Pabón wrote:<br>
> > > >> Very good points. Thanks Prasanna for putting this together. I agree with<br>
> > > >> your comments in that Heketi is the high level abstraction API and it should have<br>
> > > >> an API similar of what is described by Prasanna.<br>
> > > >><br>
> > > >> I definitely do not think any File Api should be available in Heketi,<br>
> > > >> because that is an implementation of the Block API. The Heketi API should<br>
> > > >> be similar to something like OpenStack Cinder.<br>
> > > >><br>
> > > >> I think that the actual management of the Volumes used for Block storage<br>
> > > >> and the files in them should be all managed by Heketi. How they are<br>
> > > >> actually created is still to be determined, but we could have Heketi<br>
> > > >> create them, or have helper programs do that.<br>
> > > ><br>
> > > > Maybe a tool like qemu-img? If whatever iscsi service understand the<br>
> > > > format (at the very least 'raw'), you could get functionality like<br>
> > > > snapshots pretty simple.<br>
> > ><br>
> > > Niels,<br>
> > ><br>
> > > This is brilliant and subset of the Idea falls in one among my<br>
> > > thoughts, only concern is about building dependencies of qemu with<br>
> > > Heketi.<br>
> > > But at an advantage of easy and cool snapshots solution.<br>
> ><br>
> > And well tested as I understand that oVirt is moving to use qemu-img as<br>
> > well. Other tools are able to use the qcow2 format, maybe the iscsi<br>
> > servce that gets used does so too.<br>
> ><br>
> > Has there already been a decision on what Heketi will configure as iscsi<br>
> > service? I am aware of the tgt [1] and LIO/TCMU [2] projects.<br>
><br>
> Niels,<br>
><br>
> yes we will be using TCMU (Kernel Module) and TCMU-runner (user space<br>
> service) to expose file in Gluster volume as an iSCSI target.<br>
> more at [1], [2] & [3]<br>
><br>
> [1] <a href="https://pkalever.wordpress.com/2016/06/23/gluster-solution-for-non-shared-persistent-storage-in-docker-container/" rel="noreferrer" target="_blank">https://pkalever.wordpress.<wbr>com/2016/06/23/gluster-<wbr>solution-for-non-shared-<wbr>persistent-storage-in-docker-<wbr>container/</a><br>
> [2] <a href="https://pkalever.wordpress.com/2016/06/29/non-shared-persistent-gluster-storage-with-kubernetes/" rel="noreferrer" target="_blank">https://pkalever.wordpress.<wbr>com/2016/06/29/non-shared-<wbr>persistent-gluster-storage-<wbr>with-kubernetes/</a><br>
> [3] <a href="https://pkalever.wordpress.com/2016/08/16/read-write-once-persistent-storage-for-openshift-origin-using-gluster/" rel="noreferrer" target="_blank">https://pkalever.wordpress.<wbr>com/2016/08/16/read-write-<wbr>once-persistent-storage-for-<wbr>openshift-origin-using-<wbr>gluster/</a><br>
><br>
> --<br>
> Prasanna<br>
><br>
> ><br>
> > Niels<br>
> ><br>
> > 1. <a href="http://stgt.sourceforge.net/" rel="noreferrer" target="_blank">http://stgt.sourceforge.net/</a><br>
> > 2. <a href="https://github.com/open-iscsi/tcmu-runner" rel="noreferrer" target="_blank">https://github.com/open-iscsi/<wbr>tcmu-runner</a><br>
> > <a href="http://blog.gluster.org/2016/04/using-lio-with-gluster/" rel="noreferrer" target="_blank">http://blog.gluster.org/2016/<wbr>04/using-lio-with-gluster/</a><br>
> ><br>
> > ><br>
> > > --<br>
> > > Prasanna<br>
> > ><br>
> > > ><br>
> > > > Niels<br>
> > > ><br>
> > > ><br>
> > > >> We also need to document the exact workflow to enable a file in<br>
> > > >> a Gluster volume to be exposed as a block device. This will help<br>
> > > >> determine where the creation of the file could take place.<br>
> > > >><br>
> > > >> We can capture our decisions from these discussions in the<br>
> > > >> following page:<br>
> > > >><br>
> > > >> <a href="https://github.com/heketi/heketi/wiki/Proposed-Changes" rel="noreferrer" target="_blank">https://github.com/heketi/<wbr>heketi/wiki/Proposed-Changes</a><br>
> > > >><br>
> > > >> - Luis<br>
> > > >><br>
> > > >><br>
> > > >> ----- Original Message -----<br>
> > > >> From: "Humble Chirammal" <<a href="mailto:hchiramm@redhat.com">hchiramm@redhat.com</a>><br>
> > > >> To: "Raghavendra Talur" <<a href="mailto:rtalur@redhat.com">rtalur@redhat.com</a>><br>
> > > >> Cc: "Prasanna Kalever" <<a href="mailto:pkalever@redhat.com">pkalever@redhat.com</a>>, "gluster-devel" <<a href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a>>, "Stephen Watt" <<a href="mailto:swatt@redhat.com">swatt@redhat.com</a>>, "Luis Pabon" <<a href="mailto:lpabon@redhat.com">lpabon@redhat.com</a>>, "Michael Adam" <<a href="mailto:madam@redhat.com">madam@redhat.com</a>>, "Ramakrishna Yekulla" <<a href="mailto:rreddy@redhat.com">rreddy@redhat.com</a>>, "Mohamed Ashiq Liyazudeen" <<a href="mailto:mliyazud@redhat.com">mliyazud@redhat.com</a>><br>
> > > >> Sent: Tuesday, September 13, 2016 2:23:39 AM<br>
> > > >> Subject: Re: [Gluster-devel] [Heketi] Block store related API design discussion<br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >> ----- Original Message -----<br>
> > > >> | From: "Raghavendra Talur" <<a href="mailto:rtalur@redhat.com">rtalur@redhat.com</a>><br>
> > > >> | To: "Prasanna Kalever" <<a href="mailto:pkalever@redhat.com">pkalever@redhat.com</a>><br>
> > > >> | Cc: "gluster-devel" <<a href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a>>, "Stephen Watt" <<a href="mailto:swatt@redhat.com">swatt@redhat.com</a>>, "Luis Pabon" <<a href="mailto:lpabon@redhat.com">lpabon@redhat.com</a>>,<br>
> > > >> | "Michael Adam" <<a href="mailto:madam@redhat.com">madam@redhat.com</a>>, "Humble Chirammal" <<a href="mailto:hchiramm@redhat.com">hchiramm@redhat.com</a>>, "Ramakrishna Yekulla"<br>
> > > >> | <<a href="mailto:rreddy@redhat.com">rreddy@redhat.com</a>>, "Mohamed Ashiq Liyazudeen" <<a href="mailto:mliyazud@redhat.com">mliyazud@redhat.com</a>><br>
> > > >> | Sent: Tuesday, September 13, 2016 11:08:44 AM<br>
> > > >> | Subject: Re: [Gluster-devel] [Heketi] Block store related API design discussion<br>
> > > >> |<br>
> > > >> | On Mon, Sep 12, 2016 at 11:30 PM, Prasanna Kalever <<a href="mailto:pkalever@redhat.com">pkalever@redhat.com</a>><br>
> > > >> | wrote:<br>
> > > >> |<br>
> > > >> | > Hi all,<br>
> > > >> | ><br>
> > > >> | > This mail is open for discussion on gluster block store integration with<br>
> > > >> | > heketi and its REST API interface design constraints.<br>
> > > >> | ><br>
> > > >> | ><br>
> > > >> | > ___ Volume Request ...<br>
> > > >> | > |<br>
> > > >> | > |<br>
> > > >> | > PVC claim -> Heketi --->|<br>
> > > >> | > |<br>
> > > >> | > |<br>
> > > >> | > |<br>
> > > >> | > |<br>
> > > >> | > | __ BlockCreate<br>
> > > >> | > | |<br>
> > > >> | > | |__ BlockInfo<br>
> > > >> | > | |<br>
> > > >> | > |___ Block Request (APIS)-> |__ BlockResize<br>
> > > >> | > |<br>
> > > >> | > |__ BlockList<br>
> > > >> | > |<br>
> > > >> | > |__ BlockDelete<br>
> > > >> | ><br>
> > > >> | > Heketi will have block API and volume API, when user submit a Persistent<br>
> > > >> | > volume claim, Kubernetes provisioner based on the storage class(from PVC)<br>
> > > >> | > talks to heketi for storage, heketi intern calls block or volume API's<br>
> > > >> | > based on request.<br>
> > > >> | ><br>
> > > >> |<br>
> > > >> | This is probably wrong. It won't be Heketi calling block or volume APIs. It<br>
> > > >> | would be Kubernetes calling block or volume API *of* Heketi.<br>
> > > >> |<br>
> > > >> |<br>
> > > >> | > With my limited understanding, heketi currently creates clusters from<br>
> > > >> | > provided nodes, creates volumes and handover them to the user.<br>
> > > >> | > For block related API's, it has to deal with files right ?<br>
> > > >> | ><br>
> > > >> | > Here is how block API's look like in short-<br>
> > > >> | > Create: heketi has to create file in the volume and export it as a iscsi<br>
> > > >> | > target device and hand it over to user.<br>
> > > >> | > Info: show block stores information across all the clusters, connection<br>
> > > >> | > info, size etc.<br>
> > > >> | > resize: resize the file in the volume, refresh connections from initiator<br>
> > > >> | > side<br>
> > > >> | > List: List the connections<br>
> > > >> | > Delete: logout the connections and delete the file in the gluster volume<br>
> > > >> | ><br>
> > > >> | > Couple of questions:<br>
> > > >> | > 1. Should Block API have sub API's such as FileCreate, FileList,<br>
> > > >> | > FileResize, File delete and etc then get it used in Block API as they<br>
> > > >> | > mostly deal with files.<br>
> > > >> | ><br>
> > > >> |<br>
> > > >> | IMO, Heketi should not expose any File related API. It should only have<br>
> > > >> | APIs to service request for block devices; how the block devices are<br>
> > > >> | created and modified is an implementation detail.<br>
> > > >> |<br>
> > > >> |<br>
> > > >> | > 2. How do we create the actual file in the volume, meaning using FUSE<br>
> > > >> | > mount (which may involve an extra process running) or gfapi, again if gfapi<br>
> > > >> | > should we go with c API's, python bindings or go bindings ?<br>
> > > >> | ><br>
> > > >> | 3. Should we get targetcli related (LUN exporting) setup done from heketi<br>
> > > >> | > or do we seek help from gdeploy for this ?<br>
> > > >> | ><br>
> > > >> |<br>
> > > >> | I would prefer to either have it in Heketi or in Kubernetes. If the API in<br>
> > > >> | Heketi promises just the creation of block device, then the rest of the<br>
> > > >> | implementation should be in Kubernetes(the export part). If the API in<br>
> > > >> | Heketi promises create and export both, I would say Heketi should have the<br>
> > > >> | implementation within itself.<br>
> > > >> |<br>
> > > >> |<br>
> > > >><br>
> > > >> IMO, we should not think about how the clients ( ex: k8s) use it, because there may be different clients.<br>
> > > >> We should concentrate mainly on 'in/out' of block API in Heketi. Regardless of which client, the API should act the same way.<br>
> > > >><br>
> > > >> --Humble<br>
> > > >> ______________________________<wbr>_________________<br>
> > > >> Gluster-devel mailing list<br>
> > > >> <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> > > >> <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br>
> > > ><br>
> > > > ______________________________<wbr>_________________<br>
> > > > Gluster-devel mailing list<br>
> > > > <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> > > > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br>
</div></div><br>______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>