baur

package module
v0.20.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 10, 2020 License: GPL-2.0 Imports: 19 Imported by: 2

README

baur CircleCI Go Report Card

Content

About

baur is a build management tool for Git based monolithic repositories.

Developers specify configuration in a TOML file:

  • what the inputs for the build process are
  • which command must be run to build the application
  • which outputs are generated by the build
  • where the output artifacts should be uploaded to

baur detects which applications need to be built by calculating a digest of the application's build inputs and comparing them with digests from previous builds. If a build with the same build input digest was done previously, a build is not necessary. If the build input digest differs from the stored ones, baur runs a user-specified command to build the application.

Quickstart

This chapter describes a quick way to setup baur for experimenting with it without using the Example Repository.

For setting it up in a production environment see the chapter Production Setup.

Installation

Install baur by downloading a release archive from the Release Page and extracting baur into your $PATH.

Setup

baur uses a PostgreSQL database to store information about builds, the quickest way to setup a PostgreSQL for local testing is with docker:

docker run -p 5432:5432 -e POSTGRES_DB=baur postgres:latest

Afterwards your are ready to create your baur repository configuration.

In the root directory of your Git repository run:

baur init repo

Next, follow the printed steps to create the database and application config files.

First Steps

Some commands to start with are:

command action
baur ls apps List applications in the repository with their build status
baur build Build all applications with pending builds, upload their artifacts and records the result
baur ls all List recorded builds
baur show currency-service Show information about an application called "currency-service"
baur ls inputs --digests shop-api Show inputs with their digests of an application called "shop-api"

To get more information about a command pass the --help parameter to baur.

Key Features:

  • Detecting Changed Applications The inputs of applications are specified in the .app.toml config file for each application. baur calculates a SHA384 digest for all inputs and stores the digest in the database when an application was built and its artifacts uploaded (baur build). The digest is used to detect if a previous build for the same input files exists. If a build exist, the application does not need to be rebuilt, otherwise a build is done. This allows specific applications to be run through a CI pipeline that changed in a given commit. This approach also prevents applications from unnecessarily being rebuilt if commits are reverted in the Git repository.

  • Artifact Upload to S3 and Docker Registries baur supports uploading built File artifacts to S3 buckets and produced docker images to docker registries.

  • Managing Applications baur can be used as management tool in monorepositories to list applications and find their locations.

  • CI Optimized: baur is aimed to be run in CI environments and allows to print relevant output in CSV format to be easily parsed by scripts.

  • Build Statistics: The data that baur stores in its PostgreSQL database enables the graphing of statistics about builds such as which application changes most, which produces the biggest build artifacts, which build runs the longest.

Why?

Monorepositories come with new challenges in CI environments. When a Git repository contains only a single application, every commit can trigger the whole CI workflow of builds, checks, tests and deployments. This is not effective in Monorepositories when a repository can contain hundreds of different applications. Running the whole CI flow for all applications on every commit takes a lot of time and wastes resources. Therefore the CI environment has to detect which applications changed to only run those through the CI flow.

When all build inputs per applications are isolated in directories and CI artifacts are always produced for the reference branches, the git-history can be used to determine which files changed. Simple Shell-Scripts or the mbt build tool can be used for it.

When applications in the monorepository share libraries (Protobuf or other files), standard solutions are not sufficient anymore. Full-fledged build tools like Bazel and pants exist to address those issues in Monorepositories but they come with complex configurations and complex usage. Developers have to get used to defining the build steps in those tools instead of relying on their more favorite build tools.

baur solves these problems by concentrating on tracking build inputs and build outputs while enabling to use the build tool of your choice.

Documentation

Documentation is available in the wiki.

Example Repository

You can find an example monorepository that uses baur at: https://github.com/simplesurance/baur-example. Please follow the quickstart guide for the example repository.

Grafana Dashboard

Grafana baur Dashboard

The dashboard is available at: https://grafana.com/dashboards/8835

Project Status

baur is used in production CI setups since the first version. It's not considered as API-Stable yet, interface breaking changes can happen in any release.

We are happy to receive Pull Requests, Feature Requests and Bug Reports to further improve baur.

Documentation

Index

Constants

View Source 
const AppCfgFile = ".app.toml"

AppCfgFile contains the name of application configuration files

View Source 
const RepositoryCfgFile = ".baur.toml"

RepositoryCfgFile contains the name of the repository configuration file.

Variables

This section is empty.

Functions

func SortAppsByName 

func SortAppsByName(apps []*App)

SortAppsByName sorts the apps in the slice by Name

Types

type App 

type App struct {
	RelPath    string
	Path       string
	Name       string
	BuildCmd   string
	Repository *Repository
	Outputs    []BuildOutput

	UnresolvedInputs []*cfg.BuildInput
	// contains filtered or unexported fields
}

App represents an application

func NewApp 

func NewApp(repository *Repository, cfgPath string) (*App, error)

NewApp reads the configuration file and returns a new App

func (*App) BuildInputs 

func (a *App) BuildInputs() ([]*File, error)

BuildInputs resolves all build inputs of the app. The BuildInputs are deduplicates before they are returned. If one more resolved path does not match a file an error is generated. If not build inputs are defined, an empty slice and no error is returned. If the function is called the first time, the BuildInputPaths are resolved and stored. On following calls the stored BuildInputs are returned.

func (*App) HasBuildInputs added in v0.11.1 

func (a *App) HasBuildInputs() bool

HasBuildInputs returns true if BuildInputs are defined for the app

func (*App) String 

func (a *App) String() string

String returns the string representation of an app

func (*App) TotalInputDigest 

func (a *App) TotalInputDigest() (digest.Digest, error)

TotalInputDigest returns the total input digest that is calculated over all input sources. The calculation is only done on the 1. call on following calls the stored digest is returned

type BuildOutput 

type BuildOutput interface {
	Exists() bool
	UploadJob() (scheduler.Job, error)
	Name() string
	String() string
	LocalPath() string
	UploadDestination() string
	Digest() (*digest.Digest, error)
	Size(*BuildOutputBackends) (int64, error)
	Type() string
}

BuildOutput is an interface for build artifacts

type BuildOutputBackends 

type BuildOutputBackends struct {
	DockerClt *docker.Client
}

BuildOutputBackends contains a list of backends that are required to interact with artifacts

type BuildStatus 

type BuildStatus int

BuildStatus indicates if build for a current application version exist 

const (

	// BuildStatusInputsUndefined inputs of the application are undefined,
	BuildStatusInputsUndefined BuildStatus
	// BuildStatusBuildCommandUndefined build.command of the application is undefined,
	BuildStatusBuildCommandUndefined
	// BuildStatusExist a build exist
	BuildStatusExist
	// BuildStatusPending no build exist
	BuildStatusPending
)

func GetBuildStatus 

func GetBuildStatus(storer storage.Storer, app *App) (BuildStatus, *storage.BuildWithDuration, error)

GetBuildStatus calculates the total input digest of the app and checks in the storage if a build for this input digest already exist. If the function returns BuildStatusExist the returned build pointer is valid otherwise it is nil.

func (BuildStatus) String 

func (b BuildStatus) String() string

type DockerArtifact 

type DockerArtifact struct {
	ImageIDFile string
	Tag         string
	Repository  string
	Registry    string
}

DockerArtifact is a docker container artifact

func (*DockerArtifact) Digest 

func (d *DockerArtifact) Digest() (*digest.Digest, error)

Digest returns the image ID as Digest object

func (*DockerArtifact) Exists 

func (d *DockerArtifact) Exists() bool

Exists returns true if the ImageIDFile exists

func (*DockerArtifact) ImageID 

func (d *DockerArtifact) ImageID() (string, error)

ImageID reads the image from ImageIDFile

func (*DockerArtifact) LocalPath 

func (d *DockerArtifact) LocalPath() string

LocalPath returns the local path to the artifact

func (*DockerArtifact) Name 

func (d *DockerArtifact) Name() string

Name returns the docker repository name

func (*DockerArtifact) Size 

func (d *DockerArtifact) Size(b *BuildOutputBackends) (int64, error)

Size returns the size of the docker image in bytes

func (*DockerArtifact) String 

func (d *DockerArtifact) String() string

String returns the absolute path to the ImageID file

func (*DockerArtifact) Type added in v0.11.1 

func (d *DockerArtifact) Type() string

Type returns "docker"

func (*DockerArtifact) UploadDestination 

func (d *DockerArtifact) UploadDestination() string

UploadDestination returns the upload destination

func (*DockerArtifact) UploadJob 

func (d *DockerArtifact) UploadJob() (scheduler.Job, error)

UploadJob returns a upload.DockerJob for the artifact

type File 

type File struct {
	// contains filtered or unexported fields
}

File represent a file

func NewFile 

func NewFile(repoRootPath, relPath string) *File

NewFile returns a new file

func (*File) Digest 

func (f *File) Digest() (digest.Digest, error)

Digest returns a digest of the file

func (*File) Path 

func (f *File) Path() string

Path returns it's absolute path

func (*File) RepoRelPath 

func (f *File) RepoRelPath() string

RepoRelPath returns the path relative to the baur repository

func (*File) String 

func (f *File) String() string

String returns it's string representation

type FileArtifact 

type FileArtifact struct {
	RelPath   string
	Path      string
	DestFile  string
	UploadURL string
	// contains filtered or unexported fields
}

FileArtifact is a file build artifact

func (*FileArtifact) Digest 

func (f *FileArtifact) Digest() (*digest.Digest, error)

Digest returns the file digest

func (*FileArtifact) Exists 

func (f *FileArtifact) Exists() bool

Exists returns true if the artifact exist

func (*FileArtifact) LocalPath 

func (f *FileArtifact) LocalPath() string

LocalPath returns the local path to the artifact

func (*FileArtifact) Name 

func (f *FileArtifact) Name() string

Name returns the path to the artifact relatively to application dir

func (*FileArtifact) Size 

func (f *FileArtifact) Size(_ *BuildOutputBackends) (int64, error)

Size returns the size of the file in bytes

func (*FileArtifact) String 

func (f *FileArtifact) String() string

String returns the String representation

func (*FileArtifact) Type added in v0.11.1 

func (f *FileArtifact) Type() string

Type returns "File"

func (*FileArtifact) UploadDestination 

func (f *FileArtifact) UploadDestination() string

UploadDestination returns the upload destination

func (*FileArtifact) UploadJob 

func (f *FileArtifact) UploadJob() (scheduler.Job, error)

UploadJob returns a upload.DockerJob for the artifact

type RemoteDockerImg 

type RemoteDockerImg struct {
	// contains filtered or unexported fields
}

RemoteDockerImg represents a docker image in a docker repository

func NewRemoteDockerImg 

func NewRemoteDockerImg(repository, digest string) *RemoteDockerImg

NewRemoteDockerImg creates a RemoteDockerImg

func (*RemoteDockerImg) Digest 

func (r *RemoteDockerImg) Digest() (digest.Digest, error)

Digest returns the image digest

func (*RemoteDockerImg) String 

func (r *RemoteDockerImg) String() string

String returns a string representation

func (*RemoteDockerImg) URI added in v0.11.1 

func (r *RemoteDockerImg) URI() string

URI returns <repository>/<path>@:<digest>

type Repository 

type Repository struct {
	Path          string
	CfgPath       string
	AppSearchDirs []string
	SearchDepth   int

	PSQLURL string
	// contains filtered or unexported fields
}

Repository represents an repository containing applications

func FindRepository 

func FindRepository(dir string) (*Repository, error)

FindRepository searches for a repository config file. The search starts in the passed directory and traverses the parent directory down to '/'. The first found repository configuration file is returned.

func FindRepositoryCwd added in v0.11.1 

func FindRepositoryCwd() (*Repository, error)

FindRepositoryCwd searches for a repository config file in the current directory and all it's parents. If a repository config file is found it returns a Repository

func NewRepository 

func NewRepository(cfgPath string) (*Repository, error)

NewRepository reads the configuration file and returns a Repository

func (*Repository) AppByDir 

func (r *Repository) AppByDir(appDir string) (*App, error)

AppByDir reads an application config file from the direcory and returns an App

func (*Repository) AppByName 

func (r *Repository) AppByName(name string) (*App, error)

AppByName searches for an App with the given name in the repository and returns it. If none is found os.ErrNotExist is returned.

func (*Repository) FindApps 

func (r *Repository) FindApps() ([]*App, error)

FindApps searches for application config files in the AppSearchDirs of the repository and returns all found apps

func (*Repository) GitCommitID 

func (r *Repository) GitCommitID() (string, error)

GitCommitID returns the Git commit ID in the baur repository root

func (*Repository) GitWorkTreeIsDirty 

func (r *Repository) GitWorkTreeIsDirty() (bool, error)

GitWorkTreeIsDirty returns true if the git repository contains untracked changes

Source Files

View all Source files

Directories

Path Synopsis
seq
Package seq provides a sequential builder.
 Package seq provides a sequential builder.
cmd
baur
sha384
Package exec runs external commands
 Package exec runs external commands
Package format outputs data in formatted table structures
 Package format outputs data in formatted table structures
csv
table
internal
command
command/flag
gitpath
glob
gosource
postgres
testutils
fstest
Package fstest provides test utilties to operate with files and directories
 Package fstest provides test utilties to operate with files and directories
strtest
Package strtest provides test utilties to operate with strings
 Package strtest provides test utilties to operate with strings
upload
docker
filecopy
s3
scheduler
scheduler/seq
Package seq implements a simple Sequential Uploader.
 Package seq implements a simple Sequential Uploader.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.