convoy

command module
v0.4.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 23, 2016 License: Apache-2.0 Imports: 4 Imported by: 0

README

Convoy Build Status

Overview

Convoy is a Docker volume plugin for a variety of storage back-ends. It's designed to be a simple Docker volume plug-ins that supports vendor-specific extensions such as snapshots, backups and restore. It's written in Go and can be deployed as a standalone binary.

Convoy_DEMO

Why we need Convoy?

Docker has various drivers(aufs, device mapper, etc) for container's root image, but not for volumes. User can create volume through docker run -v volname, but it's disposable, cannot be easily reused for new containers or containers on the other hosts. For example, if you start a wordpress container with database, add some posts, remove the container, then the modified database would lost.

Before volume plugin, the only way to reuse the volume is using host bind mount feature of Docker, as docker run -v /host_path:/container_path, then maintain the content of the volume at /hostpath. You can also use --volume-from but that would require original container still exists on the same host.

Convoy used Docker volume plugin mechanism to provide persistent volume for Docker containers, and supports various of backends(e.g. device mapper, NFS, EBS) and more features like snapshot/backup/restore. So user would able to migrate the volumes between the hosts, share the same volume across the hosts, make scheduled snapshots of as well as recover to previous version of volume. It's much easier for user to manage data with Docker volumes with Convoy.

Backends supported by Convoy currently
  • Device Mapper
  • Virtual File System(VFS)/Network File System(NFS)
  • Amazon Elastic Block Store(EBS)

Quick Start Guide

First let's make sure we have Docker 1.8 or above running.

docker --version

If not, install the latest Docker daemon as follows:

curl -sSL https://get.docker.com/ | sh

Once we have made sure we have the right Docker daemon running, we can install and configure Convoy volume plugin as follows:

wget https://github.com/rancher/convoy/releases/download/v0.4.3/convoy.tar.gz
tar xvf convoy.tar.gz
sudo cp convoy/convoy convoy/convoy-pdata_tools /usr/local/bin/
sudo mkdir -p /etc/docker/plugins/
sudo bash -c 'echo "unix:///var/run/convoy/convoy.sock" > /etc/docker/plugins/convoy.spec'

We can use file-backed lookback device to test and demo Convoy Device Mapper driver. Loopback device, however, is known to be unstable and should not be used in production.

truncate -s 100G data.vol
truncate -s 1G metadata.vol
sudo losetup /dev/loop5 data.vol
sudo losetup /dev/loop6 metadata.vol

Once we have the data and metadata device setup, we can start the Convoy plugin daemon as follows:

sudo convoy daemon --drivers devicemapper --driver-opts dm.datadev=/dev/loop5 --driver-opts dm.metadatadev=/dev/loop6

We can create a Docker container with a convoy volume. As a test, we create a file called /vol1/foo in the convoy volume:

sudo docker run -v vol1:/vol1 --volume-driver=convoy ubuntu touch /vol1/foo

Next we take a snapshot of the convoy volume. We backup the snapshot to a local directory: (Backup to NFS share or S3 objectore is also supported.)

sudo convoy snapshot create vol1 --name snap1vol1
sudo mkdir -p /opt/convoy/
sudo convoy backup create snap1vol1 --dest vfs:///opt/convoy/

The convoy backup command returns a URL string representing backup dataset. You can use the same URL string to recover the volume to another host:

sudo convoy create res1 --backup <backup_url>

The following command creates a new container and mounts the recovered convoy volume into that container:

sudo docker run -v res1:/res1 --volume-driver=convoy ubuntu ls /res1/foo

You should see the recovered file /res1/foo.

Installation

Ensure you have Docker 1.8 or above installed.

Download latest version of convoy and unzip it. Put the binaries in a directory in the execution $PATH of sudo and root users (e.g. /usr/local/bin).

wget https://github.com/rancher/convoy/releases/download/v0.4.3/convoy.tar.gz
tar xvf convoy.tar.gz
sudo cp convoy/convoy convoy/convoy-pdata_tools /usr/local/bin/

Run the following commands to setup the Convoy volume plugin for Docker:

sudo mkdir -p /etc/docker/plugins/
sudo bash -c 'echo "unix:///var/run/convoy/convoy.sock" > /etc/docker/plugins/convoy.spec'

Start Convoy Daemon

You need to pass different arguments to convoy daemon depending on the choice of backend implementation.

Device Mapper

For production environment, it's recommended to attach a new empty block device to the host Convoy running on for using Device Mapper driver. Then you can make two partitions out of the device using dm_dev_partition.sh to get two block devices fitting for Device Mapper driver. See Device Mapper Partition Helper for more details.

Device Mapper requires two block devices to create storage pool for all volumes and snapshots. Assuming you have two devices created, one data device called /dev/convoy-vg/data and the other metadata device called /dev/convoy-vg/metadata. You run the following command to start the Convoy daemon:

sudo convoy daemon --drivers devicemapper --driver-opts dm.datadev=/dev/convoy-vg/data --driver-opts dm.metadatadev=/dev/convoy-vg/metadata
  • A default Device Mapper volume size is 100G. You can override it with the ---driver-opts dm.defaultvolumesize option.
  • You can take a look at here if you want to know how much storage need to be allocated for metadata device.
NFS

First, mount the NFS share to the root directory used to store volumes. Substitute <vfs_path> to the appropriate directory of your choice:

sudo mkdir <vfs_path>
sudo mount -t nfs <nfs_server>:/path <vfs_path>

The NFS-based Convoy daemon can be started as follows:

sudo convoy daemon --drivers vfs --driver-opts vfs.path=<vfs_path>
EBS

Make sure you're running on EC2 instance, and has already configured AWS credentials correctly.

sudo convoy daemon --drivers ebs

Volume Commands

Create a Volume

Volumes can be created using the convoy create command:

sudo convoy create volume_name
  • Device Mapper: Default volume size is 100G. --size option is supported.
  • EBS: Default volume size is 4G. --size and some other options are supported.

We can also create a volume using the docker run command. If the volume does not yet exist, a new volume will be created. Otherwise the existing volume will be used.

sudo docker run -it -v test_volume:/test --volume-driver=convoy ubuntu
Delete a Volume
sudo convoy delete <volume_name>

or

sudo docker rm -v <container_name>
  • NFS or EBS: The -r/--reference option instructs the convoy delete command to only delete the reference to the volume from the current host and leave the underlying files on NFS server or EBS volume unchanged. This is useful when the volume need to be reused later.
  • docker rm -v would be treated as convoy delete with -r/--reference.
  • If you use --rm with docker run, all the volumes associated with the container would be deleted in the same way as executing docker rm -v when exit. See Docker run reference for details.
List and Inspect a Volume
sudo convoy list
sudo convoy inspect vol1
Take Snapshot of a Volume
sudo convoy snapshot create vol1 --name snap1vol1
Delete a Snapshot
sudo convoy snapshot delete snap1vol1
  • Device Mapper: please make sure you keep the latest backed-up snapshot for the same volume available to enable incremental backup mechanism, since Convoy need it to calculate the differences between snapshots.
Backup a Snapshot
  • Device Mapper or VFS: We can backup a snapshot to S3 object store or an NFS mount/local directory:
sudo convoy backup create snap1vol1 --dest s3://backup-bucket@us-west-2/

or

sudo convoy backup create snap1vol1 --dest vfs:///opt/backup/

The backup operation returns a URL string that uniquely idenfied the backup dataset.

s3://backup-bucket@us-west-2/?backup=f98f9ea1-dd6e-4490-8212-6d50df1982ea\u0026volume=e0d386c5-6a24-446c-8111-1077d10356b0

If you're using S3, please make sure you have AWS credential ready either at ~/.aws/credentials or as environment variables, as described here. You may need to put credentials to /root/.aws/credentials or setup sudo environment variables in order to get S3 credential works.

  • EBS: --dest is not needed. Just do convoy backup create snap1vol1.
Restore a Volume from Backup
sudo convoy create res1 --backup <url>
  • EBS: Current host must be in the same region of backup to be restored.
Mount a Restored Volume into a Docker Container

We can use the standard docker run command to mount the restored volume into a Docker container:

sudo docker run -it -v res1:/res1 --volume-driver convoy ubuntu
Mount an NFS-Backed Volume on Multiple Servers

You can mount an NFS-backed volume on multiple servers. You can use the standard docker run command to mount an existing NFS-backed mount into a Docker container. For example, if you have already created an NFS-based volume vol1 on one host, you can run the following command to mount the existing vol1 volume into a new container:

sudo docker run -it -v vol1:/vol1 --volume-driver=convoy ubuntu

Support and Discussion

If you need any help with Convoy, please join us at either our forum or #rancher IRC channel.

Feel free to submit any bugs, issues, and feature requests to Convoy Issues.

Contribution

Contribution are welcome! Please take a look at Development Guide if you want to how to build Convoy from source or running test cases.

We love to hear new Convoy Driver ideas from you. Implemenations are most welcome! Please consider take a look at enhancement ideas if you want contribute.

And of course, bug fixes are always welcome!

References

Convoy Command Line Reference

Using Convoy with Docker

Driver Specific

Device Mapper

Amazon Elastic Block Store

Virtual File System/Network File System

Documentation

The Go Gopher

There is no documentation for this package.

Directories

Path Synopsis
Godeps
_workspace/src/code.google.com/p/go-uuid/uuid
The uuid package generates and inspects UUIDs.
The uuid package generates and inspects UUIDs.
_workspace/src/github.com/aws/aws-sdk-go/aws
Package aws provides core functionality for making requests to AWS services.
Package aws provides core functionality for making requests to AWS services.
_workspace/src/github.com/aws/aws-sdk-go/aws/awserr
Package awserr represents API error interface accessors for the SDK.
Package awserr represents API error interface accessors for the SDK.
_workspace/src/github.com/aws/aws-sdk-go/aws/credentials
Package credentials provides credential retrieval and management The Credentials is the primary method of getting access to and managing credentials Values.
Package credentials provides credential retrieval and management The Credentials is the primary method of getting access to and managing credentials Values.
_workspace/src/github.com/aws/aws-sdk-go/aws/credentials/stscreds
Package stscreds are credential Providers to retrieve STS AWS credentials.
Package stscreds are credential Providers to retrieve STS AWS credentials.
_workspace/src/github.com/aws/aws-sdk-go/aws/defaults
Package defaults is a collection of helpers to retrieve the SDK's default configuration and handlers.
Package defaults is a collection of helpers to retrieve the SDK's default configuration and handlers.
_workspace/src/github.com/aws/aws-sdk-go/aws/ec2metadata
Package ec2metadata provides the client for making API calls to the EC2 Metadata service.
Package ec2metadata provides the client for making API calls to the EC2 Metadata service.
_workspace/src/github.com/aws/aws-sdk-go/aws/session
Package session provides a way to create service clients with shared configuration and handlers.
Package session provides a way to create service clients with shared configuration and handlers.
_workspace/src/github.com/aws/aws-sdk-go/private/endpoints
Package endpoints validates regional endpoints for services.
Package endpoints validates regional endpoints for services.
_workspace/src/github.com/aws/aws-sdk-go/private/protocol/ec2query
Package ec2query provides serialisation of AWS EC2 requests and responses.
Package ec2query provides serialisation of AWS EC2 requests and responses.
_workspace/src/github.com/aws/aws-sdk-go/private/protocol/query
Package query provides serialisation of AWS query requests, and responses.
Package query provides serialisation of AWS query requests, and responses.
_workspace/src/github.com/aws/aws-sdk-go/private/protocol/rest
Package rest provides RESTful serialization of AWS requests and responses.
Package rest provides RESTful serialization of AWS requests and responses.
_workspace/src/github.com/aws/aws-sdk-go/private/protocol/restxml
Package restxml provides RESTful XML serialisation of AWS requests and responses.
Package restxml provides RESTful XML serialisation of AWS requests and responses.
_workspace/src/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil
Package xmlutil provides XML serialisation of AWS requests and responses.
Package xmlutil provides XML serialisation of AWS requests and responses.
_workspace/src/github.com/aws/aws-sdk-go/private/signer/v4
Package v4 implements signing for AWS V4 signer
Package v4 implements signing for AWS V4 signer
_workspace/src/github.com/aws/aws-sdk-go/service/ec2
Package ec2 provides a client for Amazon Elastic Compute Cloud.
Package ec2 provides a client for Amazon Elastic Compute Cloud.
_workspace/src/github.com/aws/aws-sdk-go/service/ec2/ec2iface
Package ec2iface provides an interface for the Amazon Elastic Compute Cloud.
Package ec2iface provides an interface for the Amazon Elastic Compute Cloud.
_workspace/src/github.com/aws/aws-sdk-go/service/s3
Package s3 provides a client for Amazon Simple Storage Service.
Package s3 provides a client for Amazon Simple Storage Service.
_workspace/src/github.com/aws/aws-sdk-go/service/s3/s3iface
Package s3iface provides an interface for the Amazon Simple Storage Service.
Package s3iface provides an interface for the Amazon Simple Storage Service.
_workspace/src/github.com/aws/aws-sdk-go/service/s3/s3manager
Package s3manager provides utilities to upload and download objects from S3 concurrently.
Package s3manager provides utilities to upload and download objects from S3 concurrently.
_workspace/src/github.com/aws/aws-sdk-go/service/s3/s3manager/s3manageriface
Package s3manageriface provides an interface for the s3manager package
Package s3manageriface provides an interface for the s3manager package
_workspace/src/github.com/codegangsta/cli
Package cli provides a minimal framework for creating and organizing command line Go applications.
Package cli provides a minimal framework for creating and organizing command line Go applications.
_workspace/src/github.com/docker/docker/pkg/truncindex
Package truncindex package provides a general 'index tree', used by Docker in order to be able to reference containers by only a few unambiguous characters of their id.
Package truncindex package provides a general 'index tree', used by Docker in order to be able to reference containers by only a few unambiguous characters of their id.
_workspace/src/github.com/go-ini/ini
Package ini provides INI file read and write functionality in Go.
Package ini provides INI file read and write functionality in Go.
_workspace/src/github.com/gorilla/context
Package context stores values shared during a request lifetime.
Package context stores values shared during a request lifetime.
_workspace/src/github.com/gorilla/mux
Package gorilla/mux implements a request router and dispatcher.
Package gorilla/mux implements a request router and dispatcher.
_workspace/src/github.com/gorilla/websocket
Package websocket implements the WebSocket protocol defined in RFC 6455.
Package websocket implements the WebSocket protocol defined in RFC 6455.
_workspace/src/github.com/gorilla/websocket/examples/autobahn
Command server is a test server for the Autobahn WebSockets Test Suite.
Command server is a test server for the Autobahn WebSockets Test Suite.
_workspace/src/github.com/mcuadros/go-version
Version normalizer and comparison library for go, heavy based on PHP version_compare function and Version comparsion libs from Composer (https://github.com/composer/composer) PHP project
Version normalizer and comparison library for go, heavy based on PHP version_compare function and Version comparsion libs from Composer (https://github.com/composer/composer) PHP project
_workspace/src/golang.org/x/sys/unix
Package unix contains an interface to the low-level operating system primitives.
Package unix contains an interface to the low-level operating system primitives.
_workspace/src/gopkg.in/check.v1
Package check is a rich testing extension for Go's testing package.
Package check is a rich testing extension for Go's testing package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL