baur

package module
v0.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 9, 2018 License: GPL-2.0 Imports: 20 Imported by: 0

README

baur

baur manages builds and artifacts in mono repositories.

When Git repositories contain only a single applications the CI/CD jobs can be triggered on every new commit. Monorepositories can contain hundreds of apps and it becomes inefficient and slow to building, test and deploy all applications on every commit. A solution is required to detect which applications have changed and run CI/CD tasks only for those.

baur will implement:

  • discovery of applications in a repository,
  • management to store and retrieve build artifacts for applications,
  • detect if build artifacts for an application version already exist or if it's need to be build

baur makes certain Assumptions:

  • the baur repository is part of a git repository,
  • an application directory only contains one application,
  • an application can be build by running a single command, a build has to produce 1 or more build artifacts

Build

To build baur run make.

Dependencies

  • The git command lines tools are used to retrieve information in a baur repository. The tools must be installed and be in one of the paths of the $PATH environment variable.

Configuration

  1. Setup your PostgreSQL baur database

  2. Create a baur.toml file in the root of your repository by running baur init in the repository root.

  3. Adapt the configuration files to your needs:

    • Add paths containing your applications to the application_dirs parameter.
    • set the command parameter in the Build section to the command that should be run to build your application directories
    • set the postgresql_url to a valid connection string to a database that will store information about builds, it's inputs and outputs. If you do not want to store the password of your database user in the baur.toml file. You can also put it in a ~/.pgpass file (https://www.postgresql.org/docs/9.3/static/libpq-pgpass.html).
    • create the tables in the database by running the SQL-script storage/postgres/migrations/0001.up.sql

  4. Run baur appinit in your application directories to create an .app.toml file. Every application that is build via baur must have an .app.toml file.

  5. Specify in your .app.toml files the inputs and outputs of builds.

Application configs (app.toml)
Build Inputs

To enable baur to reliably detect if an application needs to be rebuild, it tracks all influencing factors as build inputs. Things that can change the output of an build can be e.g.:

  • build flags change
  • source file change,
  • the build tools change (e.g. update to a newer gcc version),
  • a docker image changes that is used to build the application

It's important that the list of build inputs and outputs is complete. Otherwise it happens that baur won't rebuild an application despite it changed.

Build Inputs must be configured per application in the app.toml file with the following directives

[Build.Input.GitFiles]

It's the preferred way to specify input files:

  • it's faster for large directories then [Build.Input.Files],
  • it ignores untracked files like temporary build files that are in the repository and probably not affect the build result,

The baur repository has to be part of a checked out git repository to work.

It's paths parameter accepts a list git path patterns that are relative to the application directory. It only matches files that are tracked by the git repository, untracked files are ignored. Modified tracked files are considered.

[Build.Input.Files]

The section has a paths parameter that accepts a list of glob paths to source files.

To make it easier to track changes in the build environment it's advised to build application in docker containers and define the docker image as build input.

[[Build.Input.DockerImages]]

Specifies a docker image as build input. This can be for example the docker image in that the application is build. The docker image is specified by it's manifest digest. The manifest digest for a docker image can be retrieved with docker images --digests or docker inspect

[Build.Input.GolangSources]

Allows to add Golang applications as inputs. The paths parameters take a list of paths to directories relative to the application directory. In every directory .go files are discovered and the files they depend on. Imports in the .go files are evaluated, resolved to files and tracked as build inputs. Go test files and imports belong to the Golang stdlib are ignored.

To be able to resolve the imports either the GOPATH environment variable must be set correctly or alternatively the go_path parameter in the config section must be set the GOPATH. The go_path expects a path relative to the application directory.

Build Outputs

Build outputs are the results that are produced by a build. They can be described in the [Build.Output] section. Baur supports to upload build results to S3 and docker images to a remote docker repository. Authentication information for output repositories are read from environment variables. S3 configuration parameters are the same then for the aws CLI tool. See https://docs.aws.amazon.com/cli/latest/userguide/cli-environment.html. The credentials for the hub.docker.com registry can be specified by setting the DOCKER_USERNAME and DOCKER_PASSWORD environment variables. DOCKER_PASSWORD can be the cleartext password or a valid authentication token.

The dest_file parameter in the [Build.Output.File.S3Upload] sections and the tag parameter of [Build.Output.DockerImage.RegistryUpload] sections support variables in their values. The variables are replaced by baur during a run.

The following variables are supported:

  • $APPNAME - is replaced with the name of the application
  • $UUID - is replaced with a generated UUID
  • $GITCOMMIT - is replaced with the current Git commit ID. The .baur.toml file must be part of a git repository and the git command must be in one of the directories in the $PATH environment variable.

Examples

  • List all applications in the repository with their build status: baur ls --build-status
  • Build all applications with outstanding builds, upload their artifacts and records the results: baur build --upload
  • Show information about an application called currency-service: baur show currency-service
  • Show inputs of an application called claim-service with their digests: baur inputs --digest claim-server

Commands

baur verify

Verify can be used to check for inconsistencies in past builds. It can find builds that have the same totalinputdigest but produced different artifacts. This can either mean that the build output it not reproducible (same inputs don't produce the same output) or that the specified Inputs are not complete. The command only compares the digests of file outputs. Docker container digest always differ when the container is rebuild.

To analyze differences in build outputs the https://diffoscope.org/ tool can be handy.

Development

Create new Release
  1. Update the version number in the ver file and commit the change.
  2. Run make release to create the release tar.xz archives.
  3. Create a new git tag (follow the instructions printed by make release).
  4. Push the ver file change to the remote git repository.
  5. Create a new release on github.com and upload the binaries.

Documentation

Index

Constants

View Source 
const AppCfgFile = ".app.toml"

AppCfgFile contains the name of application configuration files

View Source 
const RepositoryCfgFile = ".baur.toml"

RepositoryCfgFile contains the name of the repository configuration file.

Variables

This section is empty.

Functions

func SortAppsByName 

func SortAppsByName(apps []*App)

SortAppsByName sorts the apps in the slice by Name

Types

type App 

type App struct {
	RelPath         string
	Path            string
	Name            string
	BuildCmd        string
	Repository      *Repository
	Outputs         []BuildOutput
	BuildInputPaths []BuildInputPathResolver
	// contains filtered or unexported fields
}

App represents an application

func NewApp 

func NewApp(repository *Repository, cfgPath string) (*App, error)

NewApp reads the configuration file and returns a new App

func (*App) BuildInputs 

func (a *App) BuildInputs() ([]BuildInput, error)

BuildInputs returns all BuildInputs. If the function is called the first time, the BuildInputPaths are resolved and stored. On following calls the stored BuildInputs are returned.

func (*App) String 

func (a *App) String() string

func (*App) TotalInputDigest 

func (a *App) TotalInputDigest() (digest.Digest, error)

TotalInputDigest returns the total input digest that is calculated over all input sources. The calculation is only done on the 1. call on following calls the stored digest is returned

type BuildInput 

type BuildInput interface {
	Digest() (digest.Digest, error)
	String() string
	URL() string
}

BuildInput represents an input object of an application build, can be source files, compiler binaries etc, everything that can influence the produced build output

type BuildInputPathResolver 

type BuildInputPathResolver interface {
	Resolve() ([]BuildInput, error)
}

BuildInputPathResolver is an interface to resolve abstract paths like file glob paths to concrete values (files)

type BuildOutput 

type BuildOutput interface {
	Exists() bool
	UploadJob() (upload.Job, error)
	Name() string
	String() string
	LocalPath() string
	UploadDestination() string
	Digest() (*digest.Digest, error)
	Size(*BuildOutputBackends) (int64, error)
}

BuildOutput is an interface for build artifacts

type BuildOutputBackends 

type BuildOutputBackends struct {
	DockerClt *docker.Client
}

BuildOutputBackends contains a list of backends that are required to interact with artifacts

type BuildStatus 

type BuildStatus int

BuildStatus indicates if build for a current application version exist 

const (

	// BuildStatusInputsUndefined inputs of the application are undefined,
	BuildStatusInputsUndefined BuildStatus
	// BuildStatusExist a build exist
	BuildStatusExist
	// BuildStatusOutstanding no build exist
	BuildStatusOutstanding
)

func GetBuildStatus 

func GetBuildStatus(storer storage.Storer, app *App) (BuildStatus, *storage.Build, error)

GetBuildStatus calculates the total input digest of the app and checks in the storage if a build for this input digest already exist. If the function returns BuildStatusExist the returned build pointer is valid otherwise it is nil.

func (BuildStatus) String 

func (b BuildStatus) String() string

type DockerArtifact 

type DockerArtifact struct {
	ImageIDFile string
	Tag         string
	Repository  string
}

DockerArtifact is a docker container artifact

func (*DockerArtifact) Digest 

func (d *DockerArtifact) Digest() (*digest.Digest, error)

Digest returns the image ID as Digest object

func (*DockerArtifact) Exists 

func (d *DockerArtifact) Exists() bool

Exists returns true if the ImageIDFile exists

func (*DockerArtifact) ImageID 

func (d *DockerArtifact) ImageID() (string, error)

ImageID reads the image from ImageIDFile

func (*DockerArtifact) LocalPath 

func (d *DockerArtifact) LocalPath() string

LocalPath returns the local path to the artifact

func (*DockerArtifact) Name 

func (d *DockerArtifact) Name() string

Name returns the docker repository name

func (*DockerArtifact) Size 

func (d *DockerArtifact) Size(b *BuildOutputBackends) (int64, error)

Size returns the size of the docker image in bytes

func (*DockerArtifact) String 

func (d *DockerArtifact) String() string

String returns the absolute path to the ImageID file

func (*DockerArtifact) UploadDestination 

func (d *DockerArtifact) UploadDestination() string

UploadDestination returns the upload destination

func (*DockerArtifact) UploadJob 

func (d *DockerArtifact) UploadJob() (upload.Job, error)

UploadJob returns a upload.DockerJob for the artifact

type DockerImageRef 

type DockerImageRef struct {
	Repository string
	Digest     string
}

DockerImageRef represents a DockerSource entry

func (*DockerImageRef) Resolve 

func (d *DockerImageRef) Resolve() ([]BuildInput, error)

Resolve resolves the DockerImageRef to a RemoteDockerImg instance

type File 

type File struct {
	// contains filtered or unexported fields
}

File represent a file

func NewFile 

func NewFile(repoRootPath, relPath string) *File

NewFile returns a new file

func (*File) Digest 

func (f *File) Digest() (digest.Digest, error)

Digest returns a digest of the file

func (*File) Path 

func (f *File) Path() string

Path returns it's absolute path

func (*File) RepoRelPath 

func (f *File) RepoRelPath() string

RepoRelPath returns the path relative to the baur repository

func (*File) String 

func (f *File) String() string

String returns it's string representation

func (*File) URL 

func (f *File) URL() string

URL returns an URL

type FileArtifact 

type FileArtifact struct {
	RelPath   string
	Path      string
	DestFile  string
	UploadURL string
}

FileArtifact is a file build artifact

func (*FileArtifact) Digest 

func (f *FileArtifact) Digest() (*digest.Digest, error)

Digest returns the file digest

func (*FileArtifact) Exists 

func (f *FileArtifact) Exists() bool

Exists returns true if the artifact exist

func (*FileArtifact) LocalPath 

func (f *FileArtifact) LocalPath() string

LocalPath returns the local path to the artifact

func (*FileArtifact) Name 

func (f *FileArtifact) Name() string

Name returns the path to the artifact relatively to application dir

func (*FileArtifact) Size 

func (f *FileArtifact) Size(_ *BuildOutputBackends) (int64, error)

Size returns the size of the file in bytes

func (*FileArtifact) String 

func (f *FileArtifact) String() string

String returns the String representation

func (*FileArtifact) UploadDestination 

func (f *FileArtifact) UploadDestination() string

UploadDestination returns the upload destination

func (*FileArtifact) UploadJob 

func (f *FileArtifact) UploadJob() (upload.Job, error)

UploadJob returns a upload.DockerJob for the artifact

type FileGlobPath 

type FileGlobPath struct {
	// contains filtered or unexported fields
}

FileGlobPath is Source file of an application represented by a glob path

func NewFileGlobPath 

func NewFileGlobPath(repositoryRootPath, relAppPath, glob string) *FileGlobPath

NewFileGlobPath returns a new FileGlobPath object

func (*FileGlobPath) Resolve 

func (f *FileGlobPath) Resolve() ([]BuildInput, error)

Resolve returns a list of files that are matching the glob path of the FileGlobPath

func (*FileGlobPath) String 

func (f *FileGlobPath) String() string

String returns the glob path

type GitPaths 

type GitPaths struct {
	// contains filtered or unexported fields
}

GitPaths resolves multiple git filepath patterns to paths in the filesystem.

func NewGitPaths 

func NewGitPaths(repositoryRootPath, relAppPath string, gitPaths []string) *GitPaths

NewGitPaths returns a new GitPaths

func (*GitPaths) Resolve 

func (g *GitPaths) Resolve() ([]BuildInput, error)

Resolve returns a list of files that are matching it's path

func (*GitPaths) String 

func (g *GitPaths) String() string

String returns the paths, separated by ", "

type GoSrcDirs 

type GoSrcDirs struct {
	// contains filtered or unexported fields
}

GoSrcDirs resolves Golang source files in directories to files including resolving all imports to files

func NewGoSrcDirs 

func NewGoSrcDirs(repositoryRootPath, relAppPath, gopath string, paths []string) *GoSrcDirs

NewGoSrcDirs returns a GoSrcDirs

func (*GoSrcDirs) Resolve 

func (g *GoSrcDirs) Resolve() ([]BuildInput, error)

Resolve returns list of Go src files

func (*GoSrcDirs) String 

func (g *GoSrcDirs) String() string

String returns the path

type RemoteDockerImg 

type RemoteDockerImg struct {
	// contains filtered or unexported fields
}

RemoteDockerImg represents a docker image in a docker repository

func NewRemoteDockerImg 

func NewRemoteDockerImg(repository, digest string) *RemoteDockerImg

NewRemoteDockerImg creates a RemoteDockerImg

func (*RemoteDockerImg) Digest 

func (r *RemoteDockerImg) Digest() (digest.Digest, error)

Digest returns the image digest

func (*RemoteDockerImg) String 

func (r *RemoteDockerImg) String() string

String returns a string representation

func (*RemoteDockerImg) URL 

func (r *RemoteDockerImg) URL() string

URL returns it's URL

type Repository 

type Repository struct {
	Path            string
	CfgPath         string
	AppSearchDirs   []string
	SearchDepth     int
	DefaultBuildCmd string

	PSQLURL string
	// contains filtered or unexported fields
}

Repository represents an repository containing applications

func FindRepository 

func FindRepository() (*Repository, error)

FindRepository searches for a repository config file in the current directory and all it's parents. If a repository config file is found it returns a Repository

func NewRepository 

func NewRepository(cfgPath string) (*Repository, error)

NewRepository reads the configuration file and returns a Repository

func (*Repository) AppByDir 

func (r *Repository) AppByDir(appDir string) (*App, error)

AppByDir reads an application config file from the direcory and returns an App

func (*Repository) AppByName 

func (r *Repository) AppByName(name string) (*App, error)

AppByName searches for an App with the given name in the repository and returns it. If none is found os.ErrNotExist is returned.

func (*Repository) FindApps 

func (r *Repository) FindApps() ([]*App, error)

FindApps searches for application config files in the AppSearchDirs of the repository and returns all found apps

func (*Repository) GitCommitID 

func (r *Repository) GitCommitID() (string, error)

GitCommitID returns the Git commit ID in the baur repository root

func (*Repository) GitWorkTreeIsDirty 

func (r *Repository) GitWorkTreeIsDirty() (bool, error)

GitWorkTreeIsDirty returns true if the git repository contains untracked changes

Source Files

View all Source files

Directories

Path Synopsis
seq
Package seq provides a sequential builder.
 Package seq provides a sequential builder.
cmd
baur
sha384
Package golang determines all Go Source files that are imported by a Go-Files in a directory.
 Package golang determines all Go Source files that are imported by a Go-Files in a directory.
postgres
testutils
fstest
Package fstest provides test utilties to operate with files and directories
 Package fstest provides test utilties to operate with files and directories
strtest
Package strtest provides test utilties to operate with strings
 Package strtest provides test utilties to operate with strings
seq
Package seq implements a simple Sequential Uploader.
 Package seq implements a simple Sequential Uploader.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.