conveyor

package module
v0.0.10 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 6, 2016 License: BSD-2-Clause Imports: 14 Imported by: 8

README

Conveyor

Conveyor builds Docker images. Fast.

How it works

  1. Conveyor receives a build request via a GitHub commit webhook.
  2. Conveyor builds and tags the resulting image with 3 tags: latest, the git commit sha and the git branch.
  3. It then pushes the image to the Docker registry and adds a commit status to the GitHub commit.

Installation

  1. Conveyor needs access to pull GitHub repositories. The easiest way to do this is to add a bot user to your organization and generate an ssh key for them. Once you've done that, create a new S3 bucket and upload id_rsa and id_rsa.pub to the root of the bucket.
  2. Create a GitHub access token with repo, admin:repo_hook scopes.
  3. Create a new CloudFormation stack using cloudformation.json in this repo.

Configuration

The server command has the following available options:

NAME:
   server - Run an http server to build Docker images whenever a push event happens on GitHub

USAGE:
   command server [command options] [arguments...]

OPTIONS:
   --port '8080'        Port to run the server on [$PORT]
   --github.token         GitHub API token to use when updating commit statuses on repositories. [$GITHUB_TOKEN]
   --github.secret        Shared secret used by GitHub to sign webhook payloads. This secret will be used to verify that the request came from GitHub. [$GITHUB_SECRET]
   --dry          Enable dry run mode. [$DRY]
   --builder.image 'remind101/conveyor-builder' A docker image to use to perform the build. [$BUILDER_IMAGE]
   --logger 'stdout://'       The logger to use. Available options are `stdout://`, or `s3://bucket`. [$LOGGER]
   

Performance

Conveyor is designed to be faster than alternative build systems like the Docker Hub or Quay. It does this by making the following tradeoffs.

  1. It uses the latest version of Docker (1.8), which has a number of performance improvements when building and pushing images.
  2. It pulls the last built image for the branch to maximize the number of layers that can be used from the cache.

Cache

By default, conveyor will pull the last built image for the branch. This isn't always desirable, so you can disable the initial docker pull by adding the following to the git commit description:

[docker nocache]

Scale Out

Conveyor supports two methods to scale out to multiple machines.

Docker Swarm

The first method to scale out Conveyor is to scale out using Docker Swarm. Using this method, Conveyor runs its builds across a cluster of Docker daemons. The advantage of using this method is that you don't need to provide a queue flag since Conveyor can use an in memory queue.

Queue

The recommended way to scale out is to scale out using a build queue. Using this method, you run the conveyor worker subcommand on a machine that hosts a local Docker daemon. The worker process will pull build requests off of the queue and perform the build. The conveyor server command can then run completely separate from the worker nodes.

Conveyor currently supports the following build queues:

  1. SQS
Slack Integration

Conveyor can optionally expose some management tasks via Slack slash commands.

Setup

  1. Add a new Slash command. I'd recommend using /conveyor as the command.
  2. Copy the token and provide it as the --slack.token flag.

Now, you can use Conveyor to automatically manage the GitHub webhook:

/conveyor setup org/repo

API (Soon)

Conveyor also sports a restful API for triggering builds. You can use this with tooling to, say for example, trigger a build before you deploy.

POST /builds

This endpoint will create a build and stream it's output back to the client.

Example Request

{
  "Repository": "remind101/acme-inc",
  "Sha": "827fecd2d36ebeaa2fd05aa8ef3eed1e56a8cd57"
}

Example Command

$ conveyor build remind101/acme-inc:827fecd2d36ebeaa2fd05aa8ef3eed1e56a8cd57
remind101/acme-inc@sha256:44be4f298f764a2a8eb7eecce5383d2b

Development

First, bootstrap the remind101/conveyor-builder image, SSH keys and docker config:

$ make bootstrap

Then start it up with docker-compose:

$ docker-compose up

If you want to test external GitHub webhooks, the easiest way to do that is using ngrok:

$ ngrok $(docker-machine ip default):8080

Then add a new push webhook to a repo, pointed at the ngrok URL. No secret is necessary unless you set GITHUB_SECRET in .env.

NOTE: If you're testing on a private repo, you need to make sure that you've added the generated SSH key to your github account. The generated SSH key can be found in ./builder/docker/data/.ssh/id_rsa.pub

Tests

To run the full test suite. Note that you need to run make bootstrap prior to running this:

$ godep go test ./...

To run only the unit tests (no need to run make bootstrap):

$ godep go test ./... -short

Documentation

Index

Constants

View Source
const (
	// DefaultTimeout is the default amount of time to wait for a build
	// to complete before cancelling it.
	DefaultTimeout = 20 * time.Minute
)

Variables

This section is empty.

Functions

This section is empty.

Types

type BuildQueue added in v0.0.10

type BuildQueue interface {
	// Push pushes the build request onto the queue.
	Push(context.Context, builder.BuildOptions) error

	// Subscribe starts sending build requests on the channel. This method
	// should not block.
	Subscribe(chan BuildRequest) error
}

BuildQueue represents a queue that can push build requests onto a queue, and also pop requests from the queue.

func NewBuildQueue added in v0.0.10

func NewBuildQueue(buffer int) BuildQueue

NewBuildQueue returns a new in memory BuildQueue.

type BuildRequest added in v0.0.10

type BuildRequest struct {
	builder.BuildOptions
	Ctx context.Context
}

BuildRequest adds a context.Context to build options.

type Builder added in v0.0.10

type Builder struct {

	// A Reporter to use to report errors.
	Reporter reporter.Reporter

	// Timeout controls how long to wait before canceling a build. A timeout
	// of 0 means no timeout.
	Timeout time.Duration
	// contains filtered or unexported fields
}

Builder is an implementation of the builder.Builder interface that adds timeouts and error reporting.

func NewBuilder added in v0.0.10

func NewBuilder(b builder.Builder) *Builder

NewBuilder returns a new Builder instance backed by b. It also wraps it with cancellation and closes the logs when the build finishes.

func (*Builder) Build added in v0.0.10

func (b *Builder) Build(ctx context.Context, w io.Writer, opts builder.BuildOptions) (image string, err error)

Build builds the image.

type SQSBuildQueue added in v0.0.10

type SQSBuildQueue struct {
	// QueueURL is the URL for the SQS queue.
	QueueURL string

	// Context is used to generate a context.Context when receiving a
	// message. The zero value is context.Background.
	Context func() context.Context

	// ErrHandler is called when there is an error against the AWS API's.
	// SQSBuildQueue will continue trying to pull messages from the if an
	// error occurs. The zero value logs the error.
	ErrHandler func(error)
	// contains filtered or unexported fields
}

SQSBuildQueue is an implementation of the BuildQueue interface backed by Amazon SQS.

func NewSQSBuildQueue added in v0.0.10

func NewSQSBuildQueue(config *aws.Config) *SQSBuildQueue

NewSQSBuildQueue returns a new SQSBuildQueue instance backed by a pre-configured sqs client.

func (*SQSBuildQueue) Push added in v0.0.10

func (q *SQSBuildQueue) Push(ctx context.Context, options builder.BuildOptions) error

func (*SQSBuildQueue) Subscribe added in v0.0.10

func (q *SQSBuildQueue) Subscribe(ch chan BuildRequest) error

Subscribe enters into a loop and sends BuildRequests to ch. This method blocks.

type Worker added in v0.0.10

type Worker struct {
	// Builder to use to build.
	builder.Builder

	// Logger to use to build a builder.Logger
	Logger logs.Logger
	// contains filtered or unexported fields
}

Worker pulls jobs off of a BuildQueue and performs the build.

func NewWorker added in v0.0.10

func NewWorker(options WorkerOptions) *Worker

NewWorker returns a new Worker instance and subscribes to receive build requests from the BuildQueue.

func (*Worker) Shutdown added in v0.0.10

func (w *Worker) Shutdown() error

Shutdown stops this worker for processing any build requests. If the Builder supports the Cancel method, this function will block until all currently processesing builds have been canceled.

func (*Worker) Start added in v0.0.10

func (w *Worker) Start() error

Start starts the worker consuming for the BuildQueue.

type WorkerOptions added in v0.0.10

type WorkerOptions struct {
	// Builder to use to perform the builds.
	Builder builder.Builder

	// BuildQueue to pull BuildRequests from.
	BuildRequests chan BuildRequest

	// Logger used to generate an io.Writer for each build.
	Logger logs.Logger
}

WorkerOptions are options passed when building a new Worker instance.

type Workers added in v0.0.10

type Workers []*Worker

Workers is a collection of workers.

func NewWorkerPool added in v0.0.10

func NewWorkerPool(num int, options WorkerOptions) (workers Workers)

NewWorkerPool returns a new set of Worker instances.

func (Workers) Shutdown added in v0.0.10

func (w Workers) Shutdown() error

Shutdown shuts down all of the workers in the pool.

func (Workers) Start added in v0.0.10

func (w Workers) Start()

Start starts all of the workers in the pool in their own goroutine.

Directories

Path Synopsis
Godeps
_workspace/src/code.google.com/p/go-uuid/uuid
The uuid package generates and inspects UUIDs.
The uuid package generates and inspects UUIDs.
_workspace/src/github.com/DataDog/datadog-go/statsd
Package statsd provides a Go dogstatsd client.
Package statsd provides a Go dogstatsd client.
_workspace/src/github.com/aws/aws-sdk-go/aws
Package aws provides core functionality for making requests to AWS services.
Package aws provides core functionality for making requests to AWS services.
_workspace/src/github.com/aws/aws-sdk-go/aws/awserr
Package awserr represents API error interface accessors for the SDK.
Package awserr represents API error interface accessors for the SDK.
_workspace/src/github.com/aws/aws-sdk-go/aws/credentials
Package credentials provides credential retrieval and management The Credentials is the primary method of getting access to and managing credentials Values.
Package credentials provides credential retrieval and management The Credentials is the primary method of getting access to and managing credentials Values.
_workspace/src/github.com/aws/aws-sdk-go/aws/credentials/stscreds
Package stscreds are credential Providers to retrieve STS AWS credentials.
Package stscreds are credential Providers to retrieve STS AWS credentials.
_workspace/src/github.com/aws/aws-sdk-go/internal/endpoints
Package endpoints validates regional endpoints for services.
Package endpoints validates regional endpoints for services.
_workspace/src/github.com/aws/aws-sdk-go/internal/protocol/json/jsonutil
Package jsonutil provides JSON serialisation of AWS requests and responses.
Package jsonutil provides JSON serialisation of AWS requests and responses.
_workspace/src/github.com/aws/aws-sdk-go/internal/protocol/jsonrpc
Package jsonrpc provides JSON RPC utilities for serialisation of AWS requests and responses.
Package jsonrpc provides JSON RPC utilities for serialisation of AWS requests and responses.
_workspace/src/github.com/aws/aws-sdk-go/internal/protocol/query
Package query provides serialisation of AWS query requests, and responses.
Package query provides serialisation of AWS query requests, and responses.
_workspace/src/github.com/aws/aws-sdk-go/internal/protocol/rest
Package rest provides RESTful serialization of AWS requests and responses.
Package rest provides RESTful serialization of AWS requests and responses.
_workspace/src/github.com/aws/aws-sdk-go/internal/protocol/restxml
Package restxml provides RESTful XML serialisation of AWS requests and responses.
Package restxml provides RESTful XML serialisation of AWS requests and responses.
_workspace/src/github.com/aws/aws-sdk-go/internal/protocol/xml/xmlutil
Package xmlutil provides XML serialisation of AWS requests and responses.
Package xmlutil provides XML serialisation of AWS requests and responses.
_workspace/src/github.com/aws/aws-sdk-go/internal/signer/v4
Package v4 implements signing for AWS V4 signer
Package v4 implements signing for AWS V4 signer
_workspace/src/github.com/aws/aws-sdk-go/service/cloudwatchlogs
Package cloudwatchlogs provides a client for Amazon CloudWatch Logs.
Package cloudwatchlogs provides a client for Amazon CloudWatch Logs.
_workspace/src/github.com/aws/aws-sdk-go/service/cloudwatchlogs/cloudwatchlogsiface
Package cloudwatchlogsiface provides an interface for the Amazon CloudWatch Logs.
Package cloudwatchlogsiface provides an interface for the Amazon CloudWatch Logs.
_workspace/src/github.com/aws/aws-sdk-go/service/s3
Package s3 provides a client for Amazon Simple Storage Service.
Package s3 provides a client for Amazon Simple Storage Service.
_workspace/src/github.com/aws/aws-sdk-go/service/s3/s3iface
Package s3iface provides an interface for the Amazon Simple Storage Service.
Package s3iface provides an interface for the Amazon Simple Storage Service.
_workspace/src/github.com/aws/aws-sdk-go/service/sqs
Package sqs provides a client for Amazon Simple Queue Service.
Package sqs provides a client for Amazon Simple Queue Service.
_workspace/src/github.com/aws/aws-sdk-go/service/sqs/sqsiface
Package sqsiface provides an interface for the Amazon Simple Queue Service.
Package sqsiface provides an interface for the Amazon Simple Queue Service.
_workspace/src/github.com/codegangsta/cli
Package cli provides a minimal framework for creating and organizing command line Go applications.
Package cli provides a minimal framework for creating and organizing command line Go applications.
_workspace/src/github.com/codegangsta/negroni
Package negroni is an idiomatic approach to web middleware in Go.
Package negroni is an idiomatic approach to web middleware in Go.
_workspace/src/github.com/ejholmes/hookshot
Package hookshot is a router that de-multiplexes and authorizes github webhooks.
Package hookshot is a router that de-multiplexes and authorizes github webhooks.
_workspace/src/github.com/ejholmes/hookshot/events
Package events containers types representing GitHub webhook payloads.
Package events containers types representing GitHub webhook payloads.
_workspace/src/github.com/ejholmes/hookshot/hooker
Package hooker can generate github webhooks.
Package hooker can generate github webhooks.
_workspace/src/github.com/fsouza/go-dockerclient
Package docker provides a client for the Docker remote API.
Package docker provides a client for the Docker remote API.
_workspace/src/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/parsers
Package parsers provides helper functions to parse and validate different type of string.
Package parsers provides helper functions to parse and validate different type of string.
_workspace/src/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/pools
Package pools provides a collection of pools which provide various data types with buffers.
Package pools provides a collection of pools which provide various data types with buffers.
_workspace/src/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/ulimit
Package ulimit provides structure and helper function to parse and represent resource limits (Rlimit and Ulimit, its human friendly version).
Package ulimit provides structure and helper function to parse and represent resource limits (Rlimit and Ulimit, its human friendly version).
_workspace/src/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/units
Package units provides helper function to parse and print size and time units in human-readable format.
Package units provides helper function to parse and print size and time units in human-readable format.
_workspace/src/github.com/fsouza/go-dockerclient/external/github.com/gorilla/context
Package context stores values shared during a request lifetime.
Package context stores values shared during a request lifetime.
_workspace/src/github.com/fsouza/go-dockerclient/external/github.com/gorilla/mux
Package gorilla/mux implements a request router and dispatcher.
Package gorilla/mux implements a request router and dispatcher.
_workspace/src/github.com/fsouza/go-dockerclient/testing
Package testing provides a fake implementation of the Docker API, useful for testing purpose.
Package testing provides a fake implementation of the Docker API, useful for testing purpose.
_workspace/src/github.com/google/go-github/github
Package github provides a client for using the GitHub API.
Package github provides a client for using the GitHub API.
_workspace/src/github.com/google/go-querystring/query
Package query implements encoding of structs into URL query parameters.
Package query implements encoding of structs into URL query parameters.
_workspace/src/github.com/gorilla/context
Package context stores values shared during a request lifetime.
Package context stores values shared during a request lifetime.
_workspace/src/github.com/gorilla/mux
Package gorilla/mux implements a request router and dispatcher.
Package gorilla/mux implements a request router and dispatcher.
_workspace/src/github.com/remind101/pkg/logger
package logger is a package that provides a structured logger that's context.Context aware.
package logger is a package that provides a structured logger that's context.Context aware.
_workspace/src/github.com/remind101/pkg/reporter
package reporter provides a context.Context aware abstraction for shuttling errors and panics to third partys.
package reporter provides a context.Context aware abstraction for shuttling errors and panics to third partys.
_workspace/src/github.com/remind101/pkg/reporter/hb2
package hb2 is a Go package for sending errors to Honeybadger using the official client library
package hb2 is a Go package for sending errors to Honeybadger using the official client library
_workspace/src/github.com/remind101/pkg/stream
Package stream provides types that make it easier to perform streaming io.
Package stream provides types that make it easier to perform streaming io.
_workspace/src/github.com/remind101/pkg/stream/http
Package http provides streaming implementations of various net/http types.
Package http provides streaming implementations of various net/http types.
_workspace/src/github.com/shirou/gopsutil/common
gopsutil is a port of psutil(http://pythonhosted.org/psutil/).
gopsutil is a port of psutil(http://pythonhosted.org/psutil/).
_workspace/src/github.com/stretchr/objx
objx - Go package for dealing with maps, slices, JSON and other data.
objx - Go package for dealing with maps, slices, JSON and other data.
_workspace/src/github.com/stretchr/testify/assert
Package assert provides a set of comprehensive testing tools for use with the normal Go testing system.
Package assert provides a set of comprehensive testing tools for use with the normal Go testing system.
_workspace/src/github.com/stretchr/testify/mock
Provides a system by which it is possible to mock your objects and verify calls are happening as expected.
Provides a system by which it is possible to mock your objects and verify calls are happening as expected.
_workspace/src/github.com/vaughan0/go-ini
Package ini provides functions for parsing INI configuration files.
Package ini provides functions for parsing INI configuration files.
_workspace/src/golang.org/x/net/context
Package context defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes.
Package context defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes.
_workspace/src/golang.org/x/oauth2
Package oauth2 provides support for making OAuth2 authorized and authenticated HTTP requests.
Package oauth2 provides support for making OAuth2 authorized and authenticated HTTP requests.
_workspace/src/golang.org/x/oauth2/clientcredentials
Package clientcredentials implements the OAuth2.0 "client credentials" token flow, also known as the "two-legged OAuth 2.0".
Package clientcredentials implements the OAuth2.0 "client credentials" token flow, also known as the "two-legged OAuth 2.0".
_workspace/src/golang.org/x/oauth2/facebook
Package facebook provides constants for using OAuth2 to access Facebook.
Package facebook provides constants for using OAuth2 to access Facebook.
_workspace/src/golang.org/x/oauth2/github
Package github provides constants for using OAuth2 to access Github.
Package github provides constants for using OAuth2 to access Github.
_workspace/src/golang.org/x/oauth2/google
Package google provides support for making OAuth2 authorized and authenticated HTTP requests to Google APIs.
Package google provides support for making OAuth2 authorized and authenticated HTTP requests to Google APIs.
_workspace/src/golang.org/x/oauth2/internal
Package internal contains support packages for oauth2 package.
Package internal contains support packages for oauth2 package.
_workspace/src/golang.org/x/oauth2/jws
Package jws provides encoding and decoding utilities for signed JWS messages.
Package jws provides encoding and decoding utilities for signed JWS messages.
_workspace/src/golang.org/x/oauth2/jwt
Package jwt implements the OAuth 2.0 JSON Web Token flow, commonly known as "two-legged OAuth 2.0".
Package jwt implements the OAuth 2.0 JSON Web Token flow, commonly known as "two-legged OAuth 2.0".
_workspace/src/golang.org/x/oauth2/linkedin
Package linkedin provides constants for using OAuth2 to access LinkedIn.
Package linkedin provides constants for using OAuth2 to access LinkedIn.
_workspace/src/golang.org/x/oauth2/odnoklassniki
Package odnoklassniki provides constants for using OAuth2 to access Odnoklassniki.
Package odnoklassniki provides constants for using OAuth2 to access Odnoklassniki.
_workspace/src/golang.org/x/oauth2/paypal
Package paypal provides constants for using OAuth2 to access PayPal.
Package paypal provides constants for using OAuth2 to access PayPal.
_workspace/src/golang.org/x/oauth2/vk
Package vk provides constants for using OAuth2 to access VK.com.
Package vk provides constants for using OAuth2 to access VK.com.
Package builder provides methods for building docker images from GitHub repositories.
Package builder provides methods for building docker images from GitHub repositories.
datadog
Package datadog provides middleware that will send events and timings for image builds to datadog.
Package datadog provides middleware that will send events and timings for image builds to datadog.
cmd
Package logs provides an interface and implementations for reading and writing streaming logs.
Package logs provides an interface and implementations for reading and writing streaming logs.
s3
api
Package slack provides an slash Handler for adding the Conveyor push webhook on the GitHub repo.
Package slack provides an slash Handler for adding the Conveyor push webhook on the GitHub repo.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL