HTTP GET-outta here!

In the up coming Havana release of OpenStack Virtual Machine images can be downloaded a lot more efficiently.  This post will explain how to configure Glance and Nova so that VM images can be directly copied via the file system instead of routing all of the data over HTTP.

Historically How An Image Becomes a Law

Often times the architecture of an OpenStack deployment looks like the following:

In the above Glance and Nova-compute are both backed by the same file system.  Glance stores VM images available for boot on the same file system onto which Nova-compute downloads these images and uses them as an instance store.  Even tho the file system is the same, in previous releases of OpenStack a lot of unnecessary work must be done to retrieve the image.  The following steps were needed:

1. Glance opened the file on disk and read it into user space memory.
2. Glance marshaled the data in the image into the HTTP protocol
3. The data was sent through the TCP stack
4. The data was received through the TCP stack
5. Nova compute marshaled the HTTP data into memory buffers.
6. Nova compute sent the data buffers back to the file system.

That is a lot of unneeded memory copies and processing.  If HTTPS is used the processes is even more laborious.

Direct Copy

In the Havana release a couple of patches have made it so the file can be directly accessed, and thus all of the HTTP protocol processing is skipped.  The specifics involved with these patches can be found in the following links, but will otherwise not be discussed in this post.

• Multiple Locations Blueprint
• Direct URL Meta Data Blueprint
• Download plug-ins

Setting Up Glance

In order to make this process work the first thing that must be done is to describe the Glance file system store in a JSON document.  There are two mandatory pieces of information that must be determined.

1. The mount point.  This is simply at what point the associated file system is mounted.  the command line tool df can help you determine this.
2. A unique ID.  This ID is what Glance and Nova use to determine that they are talking about the same file system.This can be anything you like.  The only requirement is that it must match what is give to nova (described later).   You can use uuidgen to determine this value.

Once you have this information create a file that looks like the following:
{
"id": "b9d69795-5951-4cb0-bb5c-29491e1e2daf",
"mountpoint": "/"
}

Now edit your glance-api.conf file and make the following changes:
show_multiple_locations = True
filesystem_store_metadata_file =<path to the JSON file created above>


This tells Glance to expose direct URLs to clients and to associate the meta data described in your JSON file with all URLs that come from the file system store.

Note that this metadata will only apply to new images.  Anything that was in the store prior to this configuration change will not have the associated metadata.

Setting Up Nova Compute

Now we must configure Nova Compute to make use of the new information that Glance is exposing.  Edit nova.conf and add the folowing values:


allowed_direct_url_schemes=file

[image_file_url]
filesystems=myfs

[image_file_url:myfs]
id=b9d69795-5951-4cb0-bb5c-29491e1e2daf
mountpoint=/

This tells Nova to use direct URLs if they have the file:// scheme and they are advertised with the id b9d69795-5951-4cb0-bb5c-29491e1e2daf.  It also sets where Nova has this file system mounted.  This may be different from what Glance has it mounted, if so the correct path will be calculated. For example, if glance has the file system mounted at /mnt/gluster, and nova has it mounted at /opt/gluster the paths with which each accesses the file will be different. However, because Glance tells nova where it has it mounted, Nova can create the correct path.

Verify the Configuration

To verify that things are setup correctly do the following:

Add a new image to Glance

$ wget http://cloud.fedoraproject.org/fedora-19.x86_64.qcow2
$ glance image-create --file fedora-19.x86_64.qcow2 --name fedora-19.x86_64.qcow2 --disk-format qcow2 --container-format bare --progress
[=============================>] 100%
+------------------+--------------------------------------+
| Property         | Value                                |
+------------------+--------------------------------------+
| checksum         | 9ff360edd3b3f1fc035205f63a58ec3e     |
| container_format | bare                                 |
| created_at       | 2013-08-26T20:23:12                  |
| deleted          | False                                |
| deleted_at       | None                                 |
| disk_format      | qcow2                                |
| id               | 461bf150-4d41-47be-967f-3b4dbafd7fa5 |
| is_public        | False                                |
| min_disk         | 0                                    |
| min_ram          | 0                                    |
| name             | fedora-19.x86_64.qcow2               |
| owner            | 96bd6038d1e4404e83ad12108cad7029     |
| protected        | False                                |
| size             | 237371392                            |
| status           | active                               |
| updated_at       | 2013-08-26T20:23:16                  |
+------------------+--------------------------------------+


Verify that Glance is exposing the direct URL information

$ glance –os-image-api-version 2 image-show 461bf150-4d41-47be-967f-3b4dbafd7fa5

...
| id | 461bf150-4d41-47be-967f-3b4dbafd7fa5 |
| locations | [{u'url': u'file:///opt/stack/data/glance/images/461bf150-4d41-47be-967f-3b4dbafd7fa5', u'metadata': {u'mountpoint': u'/', u'id': u'b9d69795-5951-4cb0-bb5c-29491e1e2daf'}}] |
...


Make sure that

Make sure that the metadata above matches what you put in the JSON file.

Boot the image

$ nova boot --image 461bf150-4d41-47be-967f-3b4dbafd7fa5 --flavor 2 testcopy
+--------------------------------------+--------------------------------------+
| Property                             | Value                                |
+--------------------------------------+--------------------------------------+
| OS-EXT-STS:task_state                | scheduling                           |
| image                                | fedora-19.x86_64.qcow2               |
| OS-EXT-STS:vm_state                  | building                             |
.....

Verify the copy in logs

The image should successfully boot. However we not only want to know it booted, we also want to know that it propagated with a direct file copy.  The easiest way to verify this is by looking nova compute’s logs.  Look for the string Successfully transferred using file.   Lines like the following should be found:


2013-08-26 19:39:38.170 INFO nova.image.download.file [-] Copied /opt/stack/data/glance/images/70746c77-5625-41ff-a3f8-a3dfb35d33e5 using <nova.image.download.file.FileTransfer object at 0x416f410>
2013-08-26 19:39:38.172 INFO nova.image.glance [req-20b0ce76-1f70-482b-a72b-382621e9c8f9 admin admin] Successfully transferred using file


If these lines are found, then congratulations, you just made your system more efficient.