Docker for Azure persistent data volumes

What is Cloudstor?

Cloudstor is a modern volume plugin built by Docker. It comes pre-installed and pre-configured in Docker Swarms deployed on Docker for Azure. Docker swarm mode tasks and regular Docker containers can use a volume created with Cloudstor to mount a persistent data volume. The volume stays attached to the swarm tasks no matter which swarm node they get scheduled on or migrated to. Cloudstor relies on shared storage infrastructure provided by Azure (specifically File Storage shares exposed over SMB) to allow swarm tasks to create/mount their persistent volumes on any node in the swarm.

Note: Direct attached storage, which is used to satisfy very low latency / high IOPS requirements, is not yet supported.

You can share the same volume among tasks running the same service, or you can use a unique volume for each task.

Use Cloudstor

After initializing or joining a swarm on Docker for Azure, connect to any swarm manager using SSH. Verify that the CloudStor plugin is already installed and configured for the stack or resource group:

$ docker plugin ls

ID                  NAME                        DESCRIPTION                       ENABLED
f416c95c0dcc        cloudstor:azure             cloud storage plugin for Docker   true

The following examples show how to create swarm services that require data persistence using the --mount flag and specifying Cloudstor as the volume driver.

Share the same volume among tasks

If you specify a static value for the source option to the --mount flag, a single volume is shared among the tasks participating in the service.

Cloudstor volumes can be created to share access to persistent data across all tasks in a swarm service running in multiple nodes. Example:

$ docker service create \
  --replicas 5 \
  --name ping1 \
  --mount type=volume,volume-driver=cloudstor:azure,source=sharedvol1,destination=/shareddata \
  alpine ping docker.com

In this example, all replicas/tasks of the service ping1 share the same persistent volume sharedvol1 mounted at /shareddata path within the container. Docker takes care of interacting with the Cloudstor plugin to ensure that the common backing store is mounted on all nodes in the swarm where service tasks are scheduled. Your application needs to be designed to ensure that tasks do not write concurrently on the same file at the same time, to protect against data corruption.

You can verify that the same volume is shared among all the tasks by logging into one of the task containers, writing a file under /shareddata/, and logging into another task container to verify that the file is available there as well.

Use a unique volume per task

You can use a templatized notation with the docker service create CLI to create and mount a unique Cloudstor volume for each task in a swarm service.

It is possible to use the templatized notation to indicate to Docker Swarm that a unique Cloudstor volume be created and mounted for each replica/task of a service. This may be useful if the tasks write to the same file under the same path which may lead to corruption in case of shared storage. Example:

$ docker service create \
  --replicas 5 \
  --name ping2 \
  --mount type=volume,volume-driver=cloudstor:azure,source={{.Service.Name}}-{{.Task.Slot}}-vol,destination=/mydata \
  alpine ping docker.com

A unique volume is created and mounted for each task participating in the ping2 service. Each task mounts its own volume at /mydata/ and all files under that mountpoint are unique to the task mounting the volume.

If a task is rescheduled on a different node after the volume is created and mounted, Docker interacts with the Cloudstor plugin to create and mount the volume corresponding to the task on the new node for the task.

It is highly recommended that you use the .Task.Slot template to ensure that task N always gets access to volume N, no matter which node it is executing on/scheduled to.

Volume options

Cloudstor creates a new File Share in Azure File Storage for each volume and uses SMB to mount these File Shares. SMB has limited compatibility with generic Unix file ownership and permissions-related operations. Certain containers, such as jenkins and gitlab, define specific users and groups which perform different file operations. These types of workloads require the Cloudstor volume to be mounted with the UID/GID of the user specified in the Dockerfile or setup scripts. Cloudstor allows for this scenario and provides greater control over default file permissions by exposing the following volume options that map to SMB parameters used for mounting the backing file share.

This example sets uid to 1000 and share to sharedvol rather than a md5 hash:

$ docker service create \
  --replicas 5 \
  --name ping1 \
  --mount type=volume,volume-driver=cloudstor:azure,source=sharedvol1,destination=/shareddata,volume-opt=uid=1000,volume-opt=share=sharedvol \
  alpine ping docker.com

List or remove volumes created by Cloudstor

You can use docker volume ls on any node to enumerate all volumes created by Cloudstor across the swarm.

You can use docker volume rm [volume name] to remove a Cloudstor volume from any node. If you remove a volume from one node, make sure it is not being used by another active node, since those tasks/containers in another node will lose access to their data.

Use a different storage endpoint

If you need to use a different storage endpoint, such as core.cloudapi.de, you can specify it when you install the plugin:

$ docker plugin install --alias cloudstor:azure \
  --grant-all-permissions docker4x/cloudstor:17.06.1-ce-azure1  \
  CLOUD_PLATFORM=AZURE \
  AZURE_STORAGE_ACCOUNT_KEY="$SA_KEY" \
  AZURE_STORAGE_ACCOUNT="$SWARM_INFO_STORAGE_ACCOUNT" \
  AZURE_STORAGE_ENDPOINT="core.cloudapi.de" \
  DEBUG=1

By default, the Public Azure Storage endpoint is used.