Documentation ¶
Overview ¶
Package baur implements an incremental task runner.
A directory and child-directories containing baur configuration files is called a repository. The root of the repository contains the RepositoryCfgFile. The root and each child directory can contain an application with an AppCfgFile.
Each AppCfgFile can define one or more tasks. A task consist of a Command to run, Input definitions and optionally Output definitions. An Input is a file or string that when it changes, it changes the result of the executed command. An output is a something that is created when the task is executed. Outputs define remote upload locations to which they should be uploaded too. The results of task runs together with the state of the inputs are recorded in a storage. The state of inputs is tracked by calculating a digest for all inputs of a task. With the records about past task runs in the storage and the digest of the inputs of a task, baur evaluates if a task has been run with the same inputs in the past already. If it has not, it's executions is pending.
Basic Workflow ¶
- Loader: Locate and load a repository configuration file, discover applications and load and parse their configuration files.
- TaskStatusEvaluator: Query the storage and determine which applications have not been run before with their current inputs.
- TaskRunner: Execute the commands for the tasks that should be run.
- Uploader: Uploads the outputs that the tasks produced.
- StoreRun: Record the task executions the uploaded outputs in the database.
Index ¶
- Constants
- Variables
- func FindRepositoryCfg(dir string) (string, error)
- func SortAppsByName(apps []*App)
- func SortTasksByID(tasks []*Task)
- func StoreRun(ctx context.Context, storer storage.Storer, vcsState vcs.StateFetcher, ...) (int, error)
- type App
- type CfgUpgrader
- type DiffType
- type DockerImgUploader
- type DockerInfoClient
- type FileCopyUploader
- type Input
- type InputDiff
- type InputFile
- type InputResolver
- type InputString
- type Inputs
- type Loader
- type Logger
- type Output
- type OutputDockerImage
- type OutputFile
- func (f *OutputFile) AbsPath() string
- func (f *OutputFile) CalcDigest() (*digest.Digest, error)
- func (f *OutputFile) Digest() (*digest.Digest, error)
- func (f *OutputFile) Exists() (bool, error)
- func (f *OutputFile) Name() string
- func (f *OutputFile) SizeBytes() (uint64, error)
- func (f *OutputFile) String() string
- func (f *OutputFile) Type() OutputType
- type OutputType
- type Repository
- type RunResult
- type S3Uploader
- type Task
- type TaskRunner
- type TaskStatus
- type TaskStatusEvaluator
- type UploadInfo
- type UploadInfoDocker
- type UploadInfoFileCopy
- type UploadInfoS3
- type UploadMethod
- type UploadResult
- type UploadResultFn
- type UploadStartFn
- type Uploader
Constants ¶
const AppCfgFile = ".app.toml"
AppCfgFile is the name of application configuration files.
const RepositoryCfgFile = ".baur.toml"
RepositoryCfgFile is the name of the repository configuration file.
Variables ¶
var ErrTaskRunSkipped = errors.New("task run skipped")
ErrTaskRunSkipped is returned when a task run was skipped instead of executed.
Functions ¶
func FindRepositoryCfg ¶
FindRepositoryCfg searches for a repository config file. The search starts in dir and traverses the parent directory down to the root. It returns the path to the first found repository configuration file.
func SortAppsByName ¶
func SortAppsByName(apps []*App)
SortAppsByName sorts the slice by application names.
func SortTasksByID ¶
func SortTasksByID(tasks []*Task)
SortTasksByID sorts the tasks slice by task IDs.
Types ¶
type App ¶
type App struct { RelPath string Path string Name string // contains filtered or unexported fields }
App represents an Application.
type CfgUpgrader ¶
type CfgUpgrader struct {
// contains filtered or unexported fields
}
CfgUpgrader converts baur configurations files from a previous format to the current one.
func NewCfgUpgrader ¶
func NewCfgUpgrader(repositoryRootDir string) *CfgUpgrader
NewCfgUpgrader returns an new CfgUpgrader to upgrade configuration files in repositoryRootDir.
func (*CfgUpgrader) Upgrade ¶
func (u *CfgUpgrader) Upgrade() error
Upgrade upgrades all baur configuration files. Of all changed files a backup copy is created with the same filename and a ".bak" suffix. Each upgraded configuration file is validated by running the responsible validate() method from the cfg package.
type DockerImgUploader ¶
type DockerImgUploader interface {
Upload(image, registryAddr, repository, tag string) (string, error)
}
DockerImgUploader is an interface for uploading docker images to a docker registry.
type DockerInfoClient ¶
type DockerInfoClient interface { // Size returns the size of an image in bytes SizeBytes(imageID string) (int64, error) // Exists returns true and no error if an image with the image ID imageID exists locally. Exists(imageID string) (bool, error) }
DockerInfoClient is an interface fo retrieving information about a docker image.
type FileCopyUploader ¶
FileCopyUploader is an interface for copying files from one directory to another.
type Input ¶
Input represents an input
func AsInputStrings ¶
AsInputStrings returns InputStrings for all elements in strs.
type InputDiff ¶
InputDiff represents the result of a DiffInputs operation.
func DiffInputs ¶
DiffInputs returns the differences between two sets of Inputs. The Input.String() is used as the key to identify each Input.
type InputFile ¶
type InputFile struct {
// contains filtered or unexported fields
}
InputFile represent a file.
func NewInputFile ¶
NewInputFile returns a new input file
func (*InputFile) CalcDigest ¶
CalcDigest calculates the digest of the file. The Digest is the sha384 sum of the repoRelPath and the content of the file.
func (*InputFile) Digest ¶
Digest returns the previous calculated digest. If the digest wasn't calculated yet, CalcDigest() is called.
type InputResolver ¶
type InputResolver struct {
// contains filtered or unexported fields
}
InputResolver resolves input definitions of a task to concrete files.
func NewCachingInputResolver ¶
func NewCachingInputResolver() *InputResolver
NewCachingInputResolver returns an InputResolver that caches resolver results.
func (*InputResolver) Resolve ¶
func (i *InputResolver) Resolve(ctx context.Context, repositoryDir string, task *Task) ([]Input, error)
Resolve resolves the input definition of the task to concrete Files. If an input definition does not resolve to >=1 paths, an error is returned. The resolved Files are deduplicated.
type InputString ¶
type InputString struct {
// contains filtered or unexported fields
}
InputString represents a string
func NewInputString ¶
func NewInputString(val string) *InputString
NewInputString returns a new InputString
func (*InputString) Digest ¶
func (i *InputString) Digest() (*digest.Digest, error)
Digest returns the previous calculated digest. If the digest wasn't calculated yet, CalcDigest() is called and it's return values are returned.
func (*InputString) String ¶
func (i *InputString) String() string
String returns it's string representation (string:VAL)
func (*InputString) Value ¶
func (i *InputString) Value() string
Value returns the string that the input represents.
type Inputs ¶
type Inputs struct {
// contains filtered or unexported fields
}
Inputs are resolved Inputs of a task.
type Loader ¶
type Loader struct {
// contains filtered or unexported fields
}
Loader discovers and instantiates apps and tasks.
func NewLoader ¶
func NewLoader(repoCfg *cfg.Repository, gitCommitIDFunc func() (string, error), logger Logger) (*Loader, error)
NewLoader instantiates a Loader. When an app config is loaded the DefaultResolvers are applied on the content before they are merged with their includes. The gitCommitIDFunc is used as config resolved to resolve $GITCOMMIT variables.
func (*Loader) LoadApps ¶
LoadApps loads the apps that match the passed specifiers. Valid specifiers are: - application directory path - <APP-NAME> - '*' If no specifier is passed all apps are returned. If multiple specifiers match the same app, it's only returned 1x in the returned slice.
func (*Loader) LoadTasks ¶
LoadTasks loads the tasks of apps that match the passed specifier. Specifier format is (<APP-SPEC>[.<TASK-SPEC>])|PATH <APP-SPEC> is:
- <APP-NAME> or
- '*'
<TASK-SPEC> is:
- Task Name or
- '*'
If no specifier is passed all tasks of all apps are returned. If multiple specifiers match the same task, it's only returned 1x in the returned slice.
type Output ¶
type Output interface { Name() string String() string Exists() (bool, error) SizeBytes() (uint64, error) Digest() (*digest.Digest, error) Type() OutputType }
Output is an interface for an output that is produced by a task run.
func OutputsFromTask ¶
func OutputsFromTask(dockerClient DockerInfoClient, task *Task) ([]Output, error)
OutputsFromTask returns the Outputs that running the task produces. If the outputs do not exist, the function might fail.
type OutputDockerImage ¶
type OutputDockerImage struct { ImageID string UploadDestinations []*UploadInfoDocker // contains filtered or unexported fields }
OutputDockerImage is a docker image artifact.
func NewOutputDockerImageFromIIDFile ¶
func NewOutputDockerImageFromIIDFile( dockerClient DockerInfoClient, name, iidfile string, uploadDest []*UploadInfoDocker, ) (*OutputDockerImage, error)
NewOutputDockerImageFromIIDFile instantiates a new OutputDockerImage. name only acts as an identifier for the image. iidfile is the path to a file containing the ID of the docker image. uploadDest describes where the image should be uploaded to.
func (*OutputDockerImage) Digest ¶
func (d *OutputDockerImage) Digest() (*digest.Digest, error)
Digest returns the imageID as a digest object. The method always returns a nil error.
func (*OutputDockerImage) Exists ¶
func (d *OutputDockerImage) Exists() (bool, error)
func (*OutputDockerImage) Name ¶
func (d *OutputDockerImage) Name() string
func (*OutputDockerImage) SizeBytes ¶
func (d *OutputDockerImage) SizeBytes() (uint64, error)
func (*OutputDockerImage) String ¶
func (d *OutputDockerImage) String() string
func (*OutputDockerImage) Type ¶
func (d *OutputDockerImage) Type() OutputType
type OutputFile ¶
type OutputFile struct { UploadsS3 []*UploadInfoS3 UploadsFilecopy []*UploadInfoFileCopy // contains filtered or unexported fields }
OutputFile is a file created by a task run.
func NewOutputFile ¶
func NewOutputFile(name, absPath string, s3uploads []*UploadInfoS3, filecopyUploads []*UploadInfoFileCopy) *OutputFile
func (*OutputFile) AbsPath ¶
func (f *OutputFile) AbsPath() string
func (*OutputFile) CalcDigest ¶
func (f *OutputFile) CalcDigest() (*digest.Digest, error)
CalcDigest calculates the digest of the file. The Digest is the sha384 sum of the content of the file.
func (*OutputFile) Digest ¶
func (f *OutputFile) Digest() (*digest.Digest, error)
Digest returns the previous calculated digest. If the digest wasn't calculated yet, CalcDigest() is called.
func (*OutputFile) Exists ¶
func (f *OutputFile) Exists() (bool, error)
func (*OutputFile) Name ¶
func (f *OutputFile) Name() string
func (*OutputFile) SizeBytes ¶
func (f *OutputFile) SizeBytes() (uint64, error)
func (*OutputFile) String ¶
func (f *OutputFile) String() string
func (*OutputFile) Type ¶
func (f *OutputFile) Type() OutputType
type OutputType ¶
type OutputType int
const ( DockerOutput OutputType = iota FileOutput )
func (OutputType) String ¶
func (o OutputType) String() string
type Repository ¶
type Repository struct { Path string CfgPath string Cfg *cfg.Repository SearchDepth int }
Repository represents a baur repository.
func NewRepository ¶
func NewRepository(cfgPath string) (*Repository, error)
NewRepository parses the repository configuration file cfgPath and returns a Repository.
type S3Uploader ¶
S3Uploader is an interface for uploading files to AWS S3 buckets.
type Task ¶
type Task struct { RepositoryRoot string Directory string AppName string Name string Command []string UnresolvedInputs *cfg.Input Outputs *cfg.Output CfgFilepaths []string }
Task is a an execution step belonging to an app. A task has a set of Inputs that produce a set of outputs by executing it's Command.
func (*Task) HasOutputs ¶
HasOutputs returns true if outputs are defined for the task
type TaskRunner ¶
type TaskRunner struct {
// contains filtered or unexported fields
}
TaskRunner executes the command of a task.
func NewTaskRunner ¶
func NewTaskRunner() *TaskRunner
func (*TaskRunner) Run ¶
func (t *TaskRunner) Run(task *Task) (*RunResult, error)
Run executes the command of a task and returns the execution result. The output of the commands are logged with debug log level.
func (*TaskRunner) SkipRuns ¶ added in v2.1.0
func (t *TaskRunner) SkipRuns(enabled bool)
SkipRuns can be enabled to skip all further executions of tasks by Run().
func (*TaskRunner) SkipRunsIsEnabled ¶ added in v2.1.0
func (t *TaskRunner) SkipRunsIsEnabled() bool
SkipRunsIsEnabled returns true if SkipRuns is enabled.
type TaskStatus ¶
type TaskStatus int
TaskStatus describes the status of the task
const ( TaskStatusUndefined TaskStatus TaskStatusRunExist TaskStatusExecutionPending )
func (TaskStatus) String ¶
func (b TaskStatus) String() string
type TaskStatusEvaluator ¶
type TaskStatusEvaluator struct {
// contains filtered or unexported fields
}
TaskStatusEvaluator determines if a task already run with the same set of inputs in the past.
func NewTaskStatusEvaluator ¶
func NewTaskStatusEvaluator( repositoryDir string, store storage.Storer, inputResolver *InputResolver, inputStr []string, lookupInputStr string, ) *TaskStatusEvaluator
NewTaskStatusEvaluator returns a new TaskSNewTaskStatusEvaluator.
func (*TaskStatusEvaluator) Status ¶
func (t *TaskStatusEvaluator) Status(ctx context.Context, task *Task) (TaskStatus, *Inputs, *storage.TaskRunWithID, error)
Status resolves the inputs of the task, calculates the total input digest and checks in the storage if a run record for the task and total input digest already exist. If TaskStatusExecutionPending is returned, the returned TaskRunWithID is nil.
type UploadInfo ¶
type UploadInfo interface {
String() string
}
type UploadInfoDocker ¶
type UploadInfoDocker struct {
*cfg.DockerImageRegistryUpload
}
func (*UploadInfoDocker) String ¶
func (d *UploadInfoDocker) String() string
type UploadInfoFileCopy ¶
func (*UploadInfoFileCopy) String ¶
func (f *UploadInfoFileCopy) String() string
type UploadInfoS3 ¶
func (*UploadInfoS3) String ¶
func (s *UploadInfoS3) String() string
type UploadMethod ¶
type UploadMethod int
const ( UploadMethodS3 UploadMethod = iota UploadMethodFilecopy UploadMethodDocker )
type UploadResult ¶
type UploadResult struct { // Output is the output that was uploaded. Output Output URL string Start time.Time Stop time.Time Method UploadMethod }
UploadResult is the result of an upload operation.
type UploadResultFn ¶
type UploadResultFn func(Output, *UploadResult)
UploadResultFn is a function that is called after an upload finishes.
type UploadStartFn ¶
type UploadStartFn func(Output, UploadInfo)
UploadStartFn is a function that is called before the upload operation starts.
type Uploader ¶
type Uploader struct {
// contains filtered or unexported fields
}
Uploader uploads outputs, produced by task run, to remote locations.
func NewUploader ¶
func NewUploader(dockerClient DockerImgUploader, s3client S3Uploader, filecopyUploader FileCopyUploader) *Uploader
func (*Uploader) Upload ¶
func (u *Uploader) Upload(output Output, uploadStartCb UploadStartFn, resultCb UploadResultFn) error
Upload uploads an output to remote locations. Output must be a *OutputDockerImage or *OutputFile type storing or more upload locations. Immediately before the upload starts uploadStartCb is called, when the upload finished resultCb is called.
Source Files ¶
- app.go
- baur.go
- cfgupgrade.go
- diff.go
- input.go
- inputfile.go
- inputresolver.go
- inputresolver_cache.go
- inputs.go
- inputstring.go
- loader.go
- loader_specifiers.go
- output.go
- outputdockerimage.go
- outputfile.go
- repository.go
- storage.go
- task.go
- taskrunner.go
- taskstatus.go
- taskstatusevaluator.go
- uploader.go
- uploadinfo.go