distsync

STATUS: Kinda works. Fork it, improve it. DEAL WITH IT.

distsync

distsync is the best damn way to distribute tarballs from your build infrastructure to production servers. Unlike projects like syncthing, distsync is not intended for personal use, instead all options and design choices are optimized for servers.

This means distsync is only optimized to move your application tarball or docker export from your CI, to a group of servers, using public cloud object stores as it's primary storage backend.

Features

• Simple: Single command to upload from CI, and a daemon mode for servers.
• Encrypted: AEAD Encryption of both file contents and file names.
• Multi-Cloud: Supports both AWS S3 and Rackspace Cloud Files as storage backends.
• Pluggable: Contributions Welcome: New storage, encryption, and transfer plugins are welcome.

Usage

1. distsync setup Answer the prompts, it will create a ~/.distsync and ~/.distsyncd.
2. Copy ~/.distsync to your uploader (eg, Jenkins).
3. Copy ~/.distsyncd to your servers.
4. Run distsync daemon on servers.
5. distsync upload foo.tar.gz on your uploader.
6. Voilà! Your files are now on all your servers.

What does this do?

• distsync setup creates two identities with limited permissions. The first is for uploading, it allows distsync to upload to a single bucket. The second is for downloading which gives it permissions to watch for notifications, list, and download from the bucket.
• distsync upload encrypts the specified file, uploads it to s3, and notifies servers it is available.
• distsync daemon watches for notifications, and on a new file being available will download it to the local path using HTTPS from S3.

Configuration File Reference

The configuration file is in TOML syntax. When invoked as distsync daeomn, ~/.distsyncd is read by default. For all other invocations, ~/.distsync is read by default. All commands also take a -c path/to/conf argument to specify the path to the configuration file.

Example

SharedSecret = "<random-secret-here>"
StorageBucket = "distsync-503aa718-89cc-488c-ae82-0d8f6d08ed1c"
Encrypt = "AEAD_AES_128_CBC_HMAC_SHA_256"
Notify = "S3Poll"
Storage = "S3"

[Aws]
  Region = "us-east-1"
  AccessKey = "<access-key here>"
  SecretKey = "<secret-key here>"

Reference

SharedSecret

Default Value: None

Type: String

Details: A base64 encoded shared secret used to encrypt and HMAC all objects. Generally created by distsync setup.

StorageBucket

Default Value: None

Type: String

Details: Name of the bucket to use in the storage backend.

Encrypt

Default Value: AEAD_AES_128_CBC_HMAC_SHA_256

Type: Enum String

Details: Type of encryption and HMAC to use on objects. Must be one of:

• AEAD_AES_128_CBC_HMAC_SHA_256

Notify

Default Value: S3Poll

Type: Enum String

Details: Method to detect new files are available. Must be one of:

• S3Poll

Storage

Default Value: S3

Type: Enum String

Details: Storage backend used to upload and download files. Must be one of:

• S3
• CloudFilesb

Section: Aws

Credentials to use against AWS. The user associated with these credentials should be setup with AWS IAM to have limited privileges.

TODO: Document IAM policy that is created with distsync setup

Aws.Region

Default Value: us-east-1

Type: Enum String

Details: Region to use. Must be one of:

• ap-northeast-1
• ap-southeast-1
• ap-southeast-2
• cn-north-1
• eu-central-1
• eu-west-1
• sa-east-1
• us-east-1
• us-gov-west-1
• us-west-1
• us-west-2

Aws.AccessKey

Default Value: None

Type: String

Details: Access Key to use with AWS.

Aws.SecretKey

Default Value: None

Type: String

Details: Secret Key to use with AWS.

Section: Rackspace

Credentials to use against Rackspace. The user associated with these credentials should be setup with RBAC to limit permissions.

By default distsync setup creates two users:

• distsyncUpload${UUID}: API Key only user with the object-store:admin role. For use with distsync upload.
• distsyncDownload${UUID}: API Key only user with the object-store:observer role. For use with distsync daemon.

Rackspace.Region

Default Value: None

Type: Enum String

Details: Region to use. Must be one of:

• DFW
• HKG
• IAD
• ORD
• SYD

Rackspace.Username

Default Value: None

Type: String

Details: Username to use with Rackspace.

Rackspace.ApiKey

Default Value: None

Type: String

Details: API Key associated with the user, to use with Rackspace.

License

distsync was created by Paul Querna is licensed under the Apache Software License 2.0