luci: go.chromium.org/luci/cipd/client/cipd/ensure Index | Files

package ensure

import "go.chromium.org/luci/cipd/client/cipd/ensure"

Package ensure contains methods and types for interacting with the 'ensure file format'.

The format is used by the CIPD client to describe the desired state of a CIPD installation. This states can be asserted with the CIPD client 'ensure' command. The state is essentialy a list of packages, their versions and their installation subdirectories ("subdirs").

Format Description

The file is line-oriented. All statements fit on a single line.

A line can be blank, a comment, a setting, a package, or a directive.

Comments

A comment begins with a # and goes to the end of the line. It is ignored. Example:

# This is a comment. It has no effect.

Settings

A setting looks like `$name value`. Settings are global and can only be set once per file. The following settings are allowed:

- `$ServiceURL <url>` is the url for the CIPD service. It can be used in
  lieu of the -service-url command line parameter.
- `$VerifiedPlatform <platform> [ <platform>]*` allows the manifest to
  specify a list of ${platform} expansions to use for verification.
  Multiple $VerifiedPlatform directives can be specified, and will
  accumulate.
- `$ParanoidMode NotParanoid|CheckPresence` controls how paranoid CIPD
  should be when checking the state of already installed packages.
  `NotParanoid` (default) indicates that CIPD should trust that no one is
  messing with the installation root directory. `CheckPresence` indicates
  CIPD should verify all files that are supposed to be already installed
  are indeed present (this incurs additional overhead).
- `$ResolvedVersions <filename>` is a path (either relative to the ensure
  file or absolute) to a file that contains "frozen" instance IDs (hashes)
  of all packages for all verified platforms, resolved at the time the
  "frozen" file was generated by `cipd ensure-file-resolve`. The CIPD
  client will use these instance IDs (instead of contacting the backend)
  when installing packages in `cipd ensure`. Using this file allows to
  cryptographically bind the intended state of the deployment to a git
  commit, since all hashes of all packages become part of the git commit
  object. Use this if you want to reduce trust in the CIPD backend and root
  in the git server instead.

Package Definitions

A package line looks like `<package_template> <version>`. Package templates are CIPD package names, with optional expansion parameters `${os}`, `${arch}`, and `${platform}`. These placeholders can appear anywhere in the package template except for the first letter. All other characters in the template are taken verbatim.

${os} will expand to one of the following, based on the value of this client's runtime.GOOS value:

* windows
* mac
* linux

${arch} will expand to one of the following, based on the value of this client's runtime.GOARCH value:

* 386
* amd64
* armv6l

Since these two often appear together, the convenience placeholder `${platform}` expands to the equivalent of `${os}-${arch}`.

All of these paramters also support the syntax ${var=possible,values}. What this means is that the package line will be expanded if, and only if, var equals one of the possible values. If that var does not match a possible value, the line is ignored. This allows you to do, e.g.:

path/to/package/${os=windows}  windows_release
path/to/package/${os=linux}    linux_release
# no version for mac

path/to/posix/tool/${os=mac,linux}  some_tag:value
# no version for windows

Directives

A directive looks like `@name value`. Directives are 'sticky' and apply until the next same-name directive. The following directive names are allowed:

- `@Subdir` allows you to change the subdirectory that subsequent packages
  are installed to.
  * The subdir value is always relative to the root of the CIPD
    installation (the directory containing the .cipd folder).
  * The value of Subdir before any @Subdir directives is "", or the root
    of the CIPD installation.
  * You can reset the directory back to root by doing `@Subdir` by itself,
    without a value.
  * @Subdir directives also support expansion parameters `${os}`, `${arch}`
    and `${platform}`. Using a subdir expansion like `${param=val}` will
    cause that subdirectory, and any packages in it, to only exist if the
    param matches one of the values.

Example

Here is an example ensure file which demonstrates all the various features.

# This is an ensure file!
$ServiceURL https://chrome-infra-packages.appspot.com/
$ParanoidMode CheckPresence
$ResolvedVersions cipd_lock.versions

# This is the CIPD client itself
infra/tools/cipd/${os}-${arch}  latest

@Subdir python
python/wheels/pip                     version:8.1.2
# use the convenience placeholder
python/wheels/coverage/${platform}    version:4.1

@Subdir infra/support
infra/some/other/package deadbeefdeadbeefdeadbeefdeadbeefdeadbeef

# only exists on windows machines
@Subdir support/${os=windows}-${arch}
some/support/package  latest
some/other/support/package  latest

# Always exists, but the directory changes based on the os
@Subdir platform/${os}
a/platform/package latest

Index

Package Files

doc.go file.go item_parsers.go loading.go package_def.go versions_file.go

type File Uses

type File struct {
    ServiceURL       string
    ParanoidMode     deployer.ParanoidMode
    ResolvedVersions string

    PackagesBySubdir map[string]PackageSlice
    VerifyPlatforms  []template.Platform
}

File is an in-process representation of the 'ensure file' format.

func LoadEnsureFile Uses

func LoadEnsureFile(path string) (*File, error)

LoadEnsureFile loads the ensure file from the given path or stdin (if 'path' is "-").

If the ensure file has $ResolvedVersions directive, the returned File will have ResolvedVersions field set to an absolute path to the resolved versions file. Its presence or correctness is not checked.

func ParseFile Uses

func ParseFile(r io.Reader) (*File, error)

ParseFile parses an ensure file from the given reader. See the package docs for the format of this file.

This file will contain unresolved template strings for package names as well as unpinned package versions. Use File.Resolve() to obtain resolved+pinned versions of these.

func (*File) Resolve Uses

func (f *File) Resolve(rslv VersionResolver, expander template.Expander) (*ResolvedFile, error)

Resolve takes the current unresolved File and expands all package templates using the provided expander (usually template.DefaultExpander()), and also resolves all versions with the provided VersionResolver, calling it concurrently from multiple goroutines.

Returns either a single error (if something is wrong with the ensure file), or a multi-error with all resolution errors, sorted by definition line numbers.

func (*File) Serialize Uses

func (f *File) Serialize(w io.Writer) error

Serialize writes the File to an io.Writer in canonical order.

type PackageDef Uses

type PackageDef struct {
    PackageTemplate   string
    UnresolvedVersion string

    // LineNo is set while parsing an ensure file by the ParseFile method. It is
    // used by File.Resolve to give additional context if an error occurs.
    LineNo int
}

PackageDef defines a package line parsed out of an ensure file.

func (*PackageDef) Expand Uses

func (p *PackageDef) Expand(expander template.Expander) (pkg string, err error)

Expand expands the package name template and checks that resulting package name and version are syntactically correct.

May return template.ErrSkipTemplate is this package definition should be skipped given the current expansion variables values.

func (PackageDef) String Uses

func (p PackageDef) String() string

type PackageSlice Uses

type PackageSlice []PackageDef

PackageSlice is a sortable slice of PackageDef

func (PackageSlice) Len Uses

func (ps PackageSlice) Len() int

func (PackageSlice) Less Uses

func (ps PackageSlice) Less(i, j int) bool

func (PackageSlice) Swap Uses

func (ps PackageSlice) Swap(i, j int)

type ResolvedFile Uses

type ResolvedFile struct {
    ServiceURL   string
    ParanoidMode deployer.ParanoidMode

    PackagesBySubdir common.PinSliceBySubdir
}

ResolvedFile only contains valid, fully-resolved information and is the result of calling File.Resolve.

func (*ResolvedFile) Serialize Uses

func (f *ResolvedFile) Serialize(w io.Writer) error

Serialize writes the ResolvedFile to an io.Writer in canonical order.

type VersionResolver Uses

type VersionResolver func(pkg, vers string) (common.Pin, error)

VersionResolver transforms a {PackageName, Version} tuple (corresponding to the given `def`) into a resolved pin.

- `pkg` is guaranteed to pass common.ValidatePackageName
- `vers` is guaranteed to pass common.ValidateInstanceVersion

VersionResolver should expect to be called concurrently from multiple goroutines.

type VersionsFile Uses

type VersionsFile map[unresolvedVer]string

VersionsFile contains a mapping "(package name, version) -> instance ID" used to resolve versions (instead of backend calls) for ensure files that have $ResolvedVersions directive.

In serialized form it is represented as a set of triples separated by one or more new lines (there must be no new lines inside the triple):

""" # Comments are allowed, they are skipped (not even considered as '\n'). <package name> <version> <resolved instance ID>

<package name> <version> <resolved instance ID> """

Leading and trailing whitespace on a line is ignored.

In the canonical serialization triples are ordered by (package, version).

VersionsFile is safe for read-only concurrent use. Concurrent modifications should be protected by a lock. This is just a map in disguise.

func ParseVersionsFile Uses

func ParseVersionsFile(r io.Reader) (VersionsFile, error)

ParseVersionsFile parses previously serialized versions file.

func (VersionsFile) AddVersion Uses

func (v VersionsFile) AddVersion(pkg, ver, iid string) error

AddVersion adds (or overrides) an instance ID mapped to the given version.

Returns an error if any of the arguments is invalid.

If 'ver' is already an instance ID, just checks if it is equal to 'iid' and silently doesn't modify the map.

func (VersionsFile) Equal Uses

func (v VersionsFile) Equal(a VersionsFile) bool

Equal returns true if version files have same entries.

func (VersionsFile) ResolveVersion Uses

func (v VersionsFile) ResolveVersion(pkg, ver string) (common.Pin, error)

ResolveVersion returns a pin matching the given version or an error if such version is not in the map.

If 'ver' is already an instance ID, returns it right away.

func (VersionsFile) Serialize Uses

func (v VersionsFile) Serialize(w io.Writer) error

Serialize writes the VersionsFile to an io.Writer in canonical order.

Package ensure imports 16 packages (graph) and is imported by 6 packages. Updated 2018-12-17. Refresh now. Tools for package owners.