Design document for attach/detach functionality

The goal of this design document is to initiate the discussion regarding the implementation of the attach/detach operations for Synnefo Volumes. We will first document what is the current state of Synnefo (0.16.1 as of writing this) regarding Volumes, explain some Ganeti issues that make the implementation a bit tricky and then provide a design draft.

Current state and shortcomings

Since 0.16, Synnefo has provided an API for Volumes that is compatible to the Openstack Cinder API (see Block Storage API Guide). Using this API, the user can create Volumes, update and list them. There is just a minor nitpick, that the user cannot create a Volume without specifying a VM that it will be attached to.

Moreover, the user currently has no way to detach a volume from a VM and attach it to another.

Ganeti issues

Ganeti 2.14 has added support for the attach/detach volume operations. However, these operations still have some limitations and caveats.

Limitations

  1. There is no gnt-volume command. All volume operations happen within the context of a VM modification, and more specifically from gnt-instance modify --disk .... This means that all volumes must be created with a VM as target and detached volumes can be referenced only for attach operations.
  2. The attach/detach operations work only within a Ganeti cluster. This means that there is no pure Ganeti way to detach a volume from a VM in gnt-cluster-1 and attach it to a VM in gnt-cluster-2.
  3. A volume can be attached to a VM if and only if that VM can access the volume data. This means that e.g. a DRBD volume can only be attached to DRBD VMs whose primary and secondary nodes match that of the volume.

Caveats

  1. The volume name that is generated by Ganeti and used by Synnefo follows roughly this form: UUID.disk_template.diskX. This means that during the disk’s lifetime, its name can change. Note that Ganeti can provide two names to the ExtStorage scripts:

    • VOL_NAME: A unique name that Ganeti generates for the volume.
    • VOL_CNAME: A unique name that the user (in our case Synnefo) creates for the volume.

    For more info, you can read the related Ganeti doc. Archipelago currently uses VOL_NAME as the name for its volumes.

  2. Ganeti does not support multiple disk templates for VMs.

  3. Ganeti does not pass user-provided parameters for Ext templates during the removal of a disk.

Design overview

In this section, we will propose a way to decouple the Create operation from the Attach operation and add a Detach operation. Also, we will suggest an altered Delete operation from the existing one.

Before showing the design behind the above operations, we will list a few compromises that are needed to make the attach/detach operations work:

  1. Since the user is not aware of Ganeti nodes and clusters, we must provide an Attach operation that will work Synnefo-wide. This is achievable only with Archipelago volumes, which are visible from all Ganeti clusters.

  2. In order to be able to reference the Archipelago volumes from different clusters, we have to resort in using the VOL_CNAME instead of VOL_NAME, since the first will remain the same throughout the Volume’s lifetime.

    For more info regarding this issue, you can consult the Backend names of existing volumes section.

  3. We must alter Ganeti in order to pass an option that will keep the disk data after the removal of a disk from the VM. The snf-ganeti package already has this option (--keep-disks), so we only need to fix a minor bug.

We will now proceed with the design of each action:

Create

The Create operation currently is tightly coupled with the Attach operation, and requires the presence of a server_id argument in order to create and attach a Volume to a VM at once. In order to decouple these two operations and maintain backwards compatibility, we can make this argument optional.

Therefore, a Create operation with no server_id provided will simply do the checks that it does now and stop at the step where it stores a Volume entry in the database. The actual volume will be created once the user attempts to attach it to an instance.

Attach

The Attach operation will have two targets, a VM id and a Volume id. This operation will continue from where Create left off, i.e. it will send a disk creation job (gnt-instance modify --disk -1:add,name=<VOL_CNAME>,reuse_data=True) to Ganeti. The reuse_data Ext parameter should inform Archipelago to not create a new volume but reuse an existing one. Note that there are two cases here:

  1. The volume has not been attached to an instance before: In this case, we will inform the ExtStorage script to create the Volume in Archipelago, using the VOL_CNAME as its name.
  2. The volume has been attached to an instance before: This means that the Volume has been initialized. In this case, we will inform the ExtStorage script to simply map the existing volume to the instance, using the VOL_CNAME as identifier.

Detach

The Detach operation will not use the Ganeti detach operation as one would expect, but the remove operation (gnt-instance modify --keep-disks --disk <VOL_CNAME>:remove). As mentioned above, the --keep-disks will keep the disk data intact. In Archipelago terms, the detach ExtStorage script will be called but not the remove script.

The rationale behind this choice is to avoid having duplicate references to the same volume from different clusters, since the remove operation must operate only in one. Also, a detached Ganeti volume cannot be destroyed from Ganeti (see limitation 1), therefore it must be done from Archipelago. In this case, we do not want any reference of this volume to exist in any Ganeti cluster.

Delete

The Delete operation will have two cases:

  1. The volume is attached to an instance: In this case, we will issue a remove operation (gnt-instance modify --disk <VOL_CNAME>:remove), as we are currently doing.
  2. The volume is detached: Although at this point the volume is not in any Ganeti config and can be safely removed using a vlmc command, we need a way to receive callbacks for this action and to make sure that the remove has succeeded in order to change quotas etc. We could extend the snf-dispatcher to support Archipelago as a backend, but after consideration, we decided that it would be best if we reused the existing logic and removed the disk through Ganeti. This means that a detached volume must be attached to a helper server and then be removed. The attachment to the helper server must be transparent to the user.

Implementation details

The above design has some practical issues which must be tackled in order to have a functional Synnefo installation with detachable volumes.

Backend names of existing volumes

The attach/detach feature cannot work out-of-the-box for existing Synnefo installations which have live Archipelago volumes. The reason is that the name of these volumes is the VOL_NAME ExtStorage parameter which cannot be used as it is not consistent across Ganeti clusters.

Preferably, we would like to change the name of the Archipelago volumes to match the one that is stored in the DB (VOL_CNAME). However, this is not easy to do, especially for live volumes. Thus, we suggest to do the opposite, i.e. read the Ganeti config of each Ganeti cluster, find all Archipelago volumes and store their Ganeti name in the Cyclades DB.

In order to do the above, we need to add a new field in the Volume model, since the backend_volume_uuid field is not an actual column in the DB, but a Python class property which has the following definition:

@property
def backend_volume_uuid(self):
    return u"%svol-%d" % (settings.BACKEND_PREFIX_ID, self.id)

We propose the following change in the Volume class:

legacy_backend_volume_uuid = models.CharField("Legacy volume UUID in backend",
                                              max_length=128, null=True)

@property
def backend_volume_uuid(self):
    return (self.legacy_backend_volume_uuid or
            u"%svol-%d" % (settings.BACKEND_PREFIX_ID, self.id))

With this change, we can:

  • Κeep the backend_volume_uuid interface intact and avoid refactoring the existing code,
  • Allow any new volume to use the existing naming scheme (VOL_CNAME),
  • ...and update the old ones so that backend_volume_uuid outputs the legacy name (VOL_NAME).

Note

The above change needs a migration script to run before the new Archipelago version is installed in Synnefo. This migration script will be similar to the add_unique_name_to_disks script.

Helper servers

In order to be able to delete a detached volume, there has to be a helper server instance in an accessible Ganeti cluster. This means that the administrator must create some helper servers, preferably one for every Ganeti cluster, using the following command:

snf-manage server-create ... --helper --backend-id <id>

Also, for security reasons, the helper servers should be in stopped state, which means that the administrator must use the following command for each server:

snf-manage server-modify ... --action=stop

To make the administrator’s life easier, the above can be wrapped in an snf-manage command.

API extensions

Synnefo’s current API implementation regarding Volumes is almost fully compatible with the OpenStack Cinder and Nova (os-volume_attachments) API. The only change that it needs to be marked as fully compatible is to lift the requirement of a server id during the creation of a Volume. The user will still be able to provide a server id, in order to retain the backwards compatibility, however it should no longer be necessary.