biome

package
v0.7.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 30, 2021 License: Apache-2.0 Imports: 23 Imported by: 0

README

Build Environment Reference

WIP specification. This information should eventually live in end-user reference documentation.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Environment Variables

Each target SHALL run in a separate, isolated environment. For example, a target may run in a Docker container or a chroot jail. A target MAY run on a different host from the build runner. At a minimum, the following environment variables MUST be set for commands running in POSIX environments:

  • HOME MUST be set to the path of an readable and writable directory. This SHOULD NOT be the same as the user's actual HOME directory to keep builds reproducible.
  • LOGNAME and USER MUST be set to the name of the POSIX user running the command (not the runner of yb).
  • PATH MUST NOT be empty.
  • TZ MUST be set to UTC0.
  • LANG SHOULD be set to C.UTF-8 if the environment supports it. Otherwise, LANG MUST be set to C.
  • One of LC_ALL or LC_CTYPE MUST be set to C-like locale category whose charmap is UTF-8. If LC_CTYPE is set, LC_ALL MUST NOT be set.
Examples of Locale Settings

For Linux systems:

LANG=C.UTF-8
LC_ALL=C.UTF-8

For macOS systems:

LANG=C
LC_CTYPE=UTF-8
Further Reading
  • POSIX.1-2017 Environment Variables describes the meaning of the standard environment variables.
  • PEP 538 documents Python's process for bootstrapping a UTF-8 locale with rationale and platform-specific caveats.

Expected Userspace

The build environment that a target's commands run in MUST include the standard POSIX utilities. yb also depends on the following utilities being available:

  • python on non-Linux to fill in readlink --canonicalize-existing behavior
  • readlink on Linux
  • tar
  • unzip

Documentation

Overview

Package biome provides an API for interacting with build environments.

Index

Constants

View Source
const (
	Linux   = "linux"
	MacOS   = "darwin"
	Windows = "windows"
)

Operating systems. Values are based off GOOS.

View Source
const (
	Intel64 = "amd64"
	Intel32 = "386"
	ARM64   = "arm64"
)

CPU Architectures. Values are based off GOARCH.

View Source
const BindMount = "bind"

BindMount is the docker.HostMount.Type for a bind mount.

View Source
const (
	TiniURL = "https://github.com/krallin/tini/releases/download/v0.19.0/tini-amd64"
)

Variables

View Source
var ErrUnsupported = errors.New("unsupported operation")

ErrUnsupported indicates that a request operation cannot be performed, because it is unsupported.

TODO(light): Replace with errors.ErrUnsupported when https://golang.org/issue/41198 is resolved.

Functions

func AbsPath

func AbsPath(bio Biome, path string) string

AbsPath returns an absolute representation of path. If the path is not absolute it will be joined with the package directory to turn it into an absolute path. The absolute path name for a given file is not guaranteed to be unique. AbsPath calls Clean on the result.

func CleanPath added in v0.6.0

func CleanPath(bio Biome, path string) string

CleanPath returns the shortest path name equivalent to path by purely lexical processing. It uses the same algorithm as path/filepath.Clean.

func EvalSymlinks(ctx context.Context, bio Biome, path string) (string, error)

EvalSymlinks returns the path name after the evaluation of any symbolic links. Paths are resolved relative to the package directory. EvalSymlinks calls Clean on the result. If the path does not exist, EvalSymlinks returns an error.

If the biome has a method `EvalSymlinks(ctx context.Context, path string) (string, error)`, that will be used. If it does not or the method returns ErrUnsupported, EvalSymlinks will Run an appropriate fallback in the biome.

func MkdirAll

func MkdirAll(ctx context.Context, bio Biome, path string) error

MkdirAll creates a directory named path, along with any necessary parents, and returns nil, or else returns an error.

If the biome has a method `MkdirAll(ctx context.Context, path string) error`, that will be used. If it does not or the method returns ErrUnsupported, MkdirAll will Run an appropriate fallback in the biome.

func WriteFile

func WriteFile(ctx context.Context, bio Biome, path string, src io.Reader) error

WriteFile copies a file to the biome. Paths are resolved relative to the package directory.

If the biome has a method `WriteFile(ctx context.Context, path string, src io.Reader) error`, that will be used. If it does not or the method returns ErrUnsupported, WriteFile will Run an appropriate fallback in the biome.

Types

type Biome

type Biome interface {
	// Describe returns information about the execution environment.
	// The caller must not modify the returned Descriptor.
	Describe() *Descriptor

	// Dirs returns paths to special directories.
	// The caller must not modify the returned Dirs.
	Dirs() *Dirs

	// Run runs a program specified by the given invocation and waits for
	// it to complete. Run must not modify any fields in the Invocation or
	// retain them after Run returns.
	Run(ctx context.Context, invoke *Invocation) error

	// JoinPath joins any number of path elements into a single path.
	// The result is cleaned as if by path/filepath.Clean, however, if the
	// argument list is empty or all its elements are empty, JoinPath
	// returns an empty string.
	JoinPath(elem ...string) string

	// IsAbsPath reports whether the path is absolute.
	IsAbsPath(path string) bool
}

A Biome is an environment that a target is built or run in. Implementations must be safe to use from multiple goroutines.

type BiomeCloser

type BiomeCloser interface {
	Biome
	io.Closer
}

BiomeCloser is the interface for biomes that have resources that need to be cleaned up after use.

The behavior of any biome methods called after or concurrently with Close is undefined.

func NopCloser

func NopCloser(bio Biome) BiomeCloser

NopCloser returns a BiomeCloser with a no-op Close method wrapping the provided biome.

func WithClose

func WithClose(bio BiomeCloser, closeFunc func() error) BiomeCloser

WithClose returns a new biome that wraps another biome to call the given function at the beginning of Close, before the underlying biome's Close method is called. If the function returns an error, it will be returned from Close, but the underlying biome's Close method will still be called.

type Container added in v0.5.0

type Container struct {
	// contains filtered or unexported fields
}

Container is a biome that executes processes inside a Docker container.

func CreateContainer added in v0.5.0

func CreateContainer(ctx context.Context, client *docker.Client, opts *ContainerOptions) (_ *Container, err error)

CreateContainer starts a new Docker container. It is the caller's responsibility to call Close on the container.

func (*Container) Close added in v0.5.0

func (c *Container) Close() error

Close stops and removes the container.

func (*Container) Describe added in v0.5.0

func (c *Container) Describe() *Descriptor

Describe returns the Docker daemon's operating system and architecture information.

func (*Container) Dirs added in v0.5.0

func (c *Container) Dirs() *Dirs

Dirs returns special directories.

func (*Container) IsAbsPath added in v0.5.0

func (c *Container) IsAbsPath(path string) bool

IsAbsPath calls path.IsAbs.

func (*Container) JoinPath added in v0.5.0

func (c *Container) JoinPath(elem ...string) string

JoinPath calls path.Join.

func (*Container) MkdirAll added in v0.5.0

func (c *Container) MkdirAll(ctx context.Context, path string) error

MkdirAll ensures the given directory and any parent directories exist in the container.

func (*Container) Run added in v0.5.0

func (c *Container) Run(ctx context.Context, invoke *Invocation) error

Run runs a process in the container and waits for it to exit.

func (*Container) WriteFile added in v0.5.0

func (c *Container) WriteFile(ctx context.Context, path string, src io.Reader) error

WriteFile writes a file to the given path in the container.

type ContainerOptions added in v0.5.0

type ContainerOptions struct {
	// PackageDir is the path on the host where the package is located.
	// Must not be empty and the directory must exist.
	PackageDir string
	// HomeDir is the path on the host to use as the biome's home directory.
	// Must not be empty and the directory must exist.
	HomeDir string
	// TiniExe is a stream of TiniURL's contents. Must be non-nil.
	TiniExe io.Reader

	// PullOutput is where any `docker pull` progress output is sent to.
	// If nil, this output is discarded.
	PullOutput io.Writer
	// Definition optionally specifies details about the container to create.
	Definition *narwhal.ContainerDefinition
	// NetworkID is a Docker network ID to connect the container to, if not empty.
	NetworkID string
}

ContainerOptions holds parameters for CreateContainer.

type Descriptor

type Descriptor struct {
	OS   string
	Arch string
}

A Descriptor describes various facets of a biome.

func DockerDescriptor added in v0.5.0

func DockerDescriptor(ctx context.Context, client *docker.Client) (*Descriptor, error)

DockerDescriptor returns a descriptor for Docker daemon's container environment.

func (*Descriptor) Equal added in v0.6.0

func (desc *Descriptor) Equal(desc2 *Descriptor) bool

Equal reports whether two descriptors are equivalent.

type Dirs

type Dirs struct {
	// Package is the absolute path of the package directory.
	// Biomes guarantee this directory exists.
	Package string

	// Home is the absolute path of the build HOME directory.
	// Biomes guarantee this directory exists.
	Home string

	// Tools is the absolute path of a directory where helper tools can be
	// installed. It is not shared with other biomes. It may have to be
	// created.
	Tools string
}

Dirs holds paths to special directories in a Context.

type EnvBiome

type EnvBiome struct {
	Biome
	Env Environment
}

EnvBiome wraps a biome to add a base environment to any run commands.

func (EnvBiome) Close

func (eb EnvBiome) Close() error

Close calls eb.Biome.Close if such a method exists or returns nil if not present.

func (eb EnvBiome) EvalSymlinks(ctx context.Context, path string) (string, error)

EvalSymlinks calls eb.Context.EvalSymlinks or returns ErrUnsupported if not present.

func (EnvBiome) MkdirAll

func (eb EnvBiome) MkdirAll(ctx context.Context, path string) error

MkdirAll calls eb.Context.MkdirAll or returns ErrUnsupported if not present.

func (EnvBiome) Run

func (eb EnvBiome) Run(ctx context.Context, invoke *Invocation) error

Run runs a command with eb.Env as a base environment with invoke.Env merged in.

func (EnvBiome) WriteFile

func (eb EnvBiome) WriteFile(ctx context.Context, path string, src io.Reader) error

WriteFile calls eb.Context.WriteFile or returns ErrUnsupported if not present.

type Environment

type Environment struct {
	// Vars is a mapping of variables.
	Vars map[string]string
	// PrependPath is a list of paths to prepend to PATH.
	PrependPath []string
	// AppendPath is a list of paths to append to PATH.
	AppendPath []string
}

Environment holds environment variables. The zero value is an empty environment.

func (Environment) IsEmpty

func (env Environment) IsEmpty() bool

IsEmpty reports whether env contains no variables.

func (Environment) Merge

func (env Environment) Merge(env2 Environment) Environment

Merge returns a new environment that merges env2 into env.

func (Environment) String

func (env Environment) String() string

String formats the environment variables one per line sorted by variable name. This output is for debugging purposes only and should not be depended upon.

type ExecPrefix

type ExecPrefix struct {
	Biome
	PrependArgv []string
}

ExecPrefix intercepts calls to Run and prepends elements to the Argv slice. This can be used to invoke tools with a wrapping command like `time` or `sudo`.

func (ExecPrefix) Close

func (ep ExecPrefix) Close() error

Close calls ep.Biome.Close if such a method exists or returns nil if not present.

func (ep ExecPrefix) EvalSymlinks(ctx context.Context, path string) (string, error)

EvalSymlinks calls ep.Context.EvalSymlinks or returns ErrUnsupported if not present.

func (ExecPrefix) MkdirAll

func (ep ExecPrefix) MkdirAll(ctx context.Context, path string) error

MkdirAll calls ep.Context.MkdirAll or returns ErrUnsupported if not present.

func (ExecPrefix) Run

func (ep ExecPrefix) Run(ctx context.Context, invoke *Invocation) error

Run calls ep.Biome.Run with an invocation whose Argv is the result of appending invoke.Argv to ep.PrependArgv.Argv.

func (ExecPrefix) WriteFile

func (ep ExecPrefix) WriteFile(ctx context.Context, path string, src io.Reader) error

WriteFile calls ep.Context.WriteFile or returns ErrUnsupported if not present.

type Fake

type Fake struct {
	// Separator is the path separator character. If NUL, then slash '/' is used.
	Separator rune

	// Descriptor is the descriptor that will be returned by Describe.
	Descriptor Descriptor

	// DirsResult is what will be returned by Dirs.
	DirsResult Dirs

	// RunFunc is called to handle the Run method.
	RunFunc func(context.Context, *Invocation) error
}

Fake is a biome that operates in-memory. It uses POSIX-style paths, but permits any character to be used as the separator.

func (*Fake) Close

func (f *Fake) Close() error

Close does nothing and returns nil.

func (*Fake) Describe

func (f *Fake) Describe() *Descriptor

Describe returns f.Descriptor.

func (*Fake) Dirs

func (f *Fake) Dirs() *Dirs

Dirs returns f.DirsResult.

func (*Fake) IsAbsPath

func (f *Fake) IsAbsPath(path string) bool

IsAbsPath reports whether the path is absolute.

func (*Fake) JoinPath

func (f *Fake) JoinPath(elem ...string) string

JoinPath joins any number of path elements into a single path.

func (*Fake) Run

func (f *Fake) Run(ctx context.Context, invoke *Invocation) error

Run calls f.RunFunc. It returns an error if f.RunFunc is nil.

type Invocation

type Invocation struct {
	// Argv is the argument list. Argv[0] is the name of the program to execute.
	Argv []string

	// Dir is the directory to execute the program in. Paths are resolved relative to
	// the package directory. If empty, then it will be executed in the package
	// directory. It is separated by the biome's path separator.
	Dir string

	// Env specifies additional environment variables to send to the program.
	// The biome may provide additional environment variables to the program, but
	// will not override the provided environment variables.
	Env Environment

	// Stdin specifies the program's standard input.
	// If Stdin is nil, the program reads from the null device.
	Stdin io.Reader

	// Interactive indicates whether the program will be surfaced to the user
	// interactively. This usually indicates that the program should be given
	// a pseudo-TTY as input.
	Interactive bool

	// Stdout and Stderr specify the program's standard output and error.
	// If either is nil, Run connects the corresponding file descriptor to the
	// null device.
	//
	// If Stdout and Stderr are the same writer, and have a type that can
	// be compared with ==, at most one goroutine at a time will call Write.
	Stdout io.Writer
	Stderr io.Writer
}

An Invocation holds the parameters for a single command.

type Local

type Local struct {
	// PackageDir is the absolute path to a directory containing the source files
	// of the package.
	PackageDir string

	// HomeDir is the absolute path to a directory that should be used as HOME.
	// This directory may or may not exist. It SHOULD NOT be the user's actual
	// home directory. It's meant for storing configuration and intermediate files
	// that any build tools need.
	HomeDir string
}

Local is a biome that executes processes in a directory on the local machine.

func (Local) Close

func (l Local) Close() error

Close does nothing and returns nil.

func (Local) Describe

func (l Local) Describe() *Descriptor

Describe returns the values of GOOS/GOARCH.

func (Local) Dirs

func (l Local) Dirs() *Dirs

Dirs returns special directories.

func (l Local) EvalSymlinks(ctx context.Context, path string) (string, error)

EvalSymlinks calls filepath.EvalSymlinks.

func (Local) IsAbsPath

func (l Local) IsAbsPath(path string) bool

IsAbsPath calls filepath.IsAbs.

func (Local) JoinPath

func (l Local) JoinPath(elem ...string) string

JoinPath calls filepath.Join.

func (Local) MkdirAll

func (l Local) MkdirAll(ctx context.Context, path string) error

MkdirAll calls os.MkdirAll(path, 0777).

func (Local) Run

func (l Local) Run(ctx context.Context, invoke *Invocation) error

Run runs a subprocess and waits for it to exit.

func (Local) WriteFile

func (l Local) WriteFile(ctx context.Context, path string, src io.Reader) error

WriteFile writes the data from src to the gven path with the mode 0666.

Directories

Path Synopsis
Package replay provides a biome that records all events to an underlying biome and then allows replaying those command invocations later.
Package replay provides a biome that records all events to an underlying biome and then allows replaying those command invocations later.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL