luci: Index | Files

package local

import ""


Package Files

builder.go deployer.go file_attrs_posix.go files.go fs.go fs_posix.go json_descs.go pkgdef.go reader.go testing.go


const (
    // SiteServiceDir is a name of the directory inside an installation root
    // reserved for cipd stuff.
    SiteServiceDir = ".cipd"

    // PackageServiceDir is a name of the directory inside the package
    // reserved for cipd stuff.
    PackageServiceDir = ".cipdpkg"

    // ManifestName is a full name of the manifest file inside the package.
    ManifestName = PackageServiceDir + "/manifest.json"

    // ManifestFormatVersion is a version to write to the manifest file.
    ManifestFormatVersion = "1.1"


var ErrHashMismatch = errors.New("package hash mismatch")

ErrHashMismatch is an error when package hash doesn't match.

func BuildInstance Uses

func BuildInstance(ctx context.Context, opts BuildInstanceOptions) error

BuildInstance builds a new package instance.

If build an instance of package named opts.PackageName by archiving input files (passed via opts.Input).

The final binary is written to opts.Output. Some output may be written even if BuildInstance eventually returns an error.

func EnsureFile Uses

func EnsureFile(ctx context.Context, fs FileSystem, path string, content io.Reader) error

EnsureFile creates a file with the given content. It will create full directory path to the file if necessary.

func ExtractFiles Uses

func ExtractFiles(ctx context.Context, files []File, dest Destination, withManifest ManifestMode) error

ExtractFiles extracts all given files into a destination, with a progress report.

This function has intimate understanding of specifics of CIPD packages, such as manifest files and .cipdpkg/* directory, thus it should be used only when extracting files from CIPD packages.

If withManifest is WithManifest, the manifest file is required to be among 'files'. It will be extended with information about extracted files and placed into the destination.

If withManifest is WithoutManifest, the function will fail if the manifest is among 'files' (as a precaution against unintended override of manifests).

func ExtractFilesTxn Uses

func ExtractFilesTxn(ctx context.Context, files []File, dest TransactionalDestination, withManifest ManifestMode) (err error)

ExtractFilesTxn is like ExtractFiles, but it also opens and closes the transaction over TransactionalDestination object.

It guarantees that if extraction fails for some reason, there'll be no garbage laying around.

func IsCorruptionError Uses

func IsCorruptionError(err error) bool

IsCorruptionError returns true iff err indicates corruption.

func ValidateInstallMode Uses

func ValidateInstallMode(mode InstallMode) error

ValidateInstallMode returns non nil if install mode is invalid.

Valid modes are: "" (client will pick platform default), "copy" (aka InstallModeCopy), "symlink" (aka InstallModeSymlink).

type BuildInstanceOptions Uses

type BuildInstanceOptions struct {
    // Input is a list of files to add to the package.
    Input []File

    // Output is where to write the package file to.
    Output io.Writer

    // PackageName is name of the package being built, e.g. 'infra/tools/cipd'.
    PackageName string

    // VersionFile is slash separated path where to drop JSON with version info.
    VersionFile string

    // InstallMode defines how to install the package: "copy" or "symlink".
    InstallMode InstallMode

    // CompressionLevel defines deflate compression level in range [0-9].
    CompressionLevel int
    // contains filtered or unexported fields

BuildInstanceOptions defines options for BuildInstance function.

type CreateFileOptions Uses

type CreateFileOptions struct {
    // Executable makes the file executable.
    Executable bool
    // Writable makes the file user-writable.
    Writable bool
    // ModTime, when non-zero, sets the mtime of the file.
    ModTime time.Time
    // WinAttrs is used on Windows.
    WinAttrs WinAttrs

CreateFileOptions provides arguments to Destination.CreateFile().

type DeployedPackage Uses

type DeployedPackage struct {
    Deployed    bool        // true if the package is deployed (perhaps partially)
    Pin         common.Pin  // the currently installed pin
    Subdir      string      // the site subdirectory where the package is installed
    Manifest    *Manifest   // instance's manifest, if available
    InstallMode InstallMode // validated install mode, if available

    // ToRedeploy is a list of files that needs to be reextracted from the
    // original package and relinked into the site root.
    ToRedeploy []string

    // ToRelink is a list of files that needs to be relinked into the site root.
    // They are already present in the .cipd/* guts, so there's no need to fetch
    // the original package to get them.
    ToRelink []string
    // contains filtered or unexported fields

DeployedPackage represents a state of the deployed (or partially deployed) package, as returned by CheckDeployed.

ToRedeploy is populated only when CheckDeployed is called in a paranoid mode, and the package needs repairs.

type Deployer Uses

type Deployer interface {
    // DeployInstance installs an instance of a package into the given subdir of
    // the root.
    // It unpacks the package into <base>/.cipd/pkgs/*, and rearranges
    // symlinks to point to unpacked files. It tries to make it as "atomic" as
    // possible. Returns information about the deployed instance.
    // Due to a historical bug, if inst contains any files which are intended to
    // be deployed to `.cipd/*`, they will not be extracted and you'll see
    // warnings logged.
    DeployInstance(ctx context.Context, subdir string, inst PackageInstance) (common.Pin, error)

    // CheckDeployed checks whether a given package is deployed at the given
    // subdir.
    // Returns an error if it can't check the package state for some reason.
    // Otherwise returns the state of the package. In particular, if the package
    // is not deployed, returns DeployedPackage{Deployed: false}.
    // Depending on the paranoia mode will also verify that package's files are
    // correctly installed into the site root and will return a list of files
    // that needs to be redeployed (as part of DeployedPackage).
    // If manifest is set to WithManifest, will also fetch and return the instance
    // manifest and install mode. This is optional, since not all callers need it,
    // and it is pretty heavy operation. Any paranoid mode implies WithManifest
    // too.
    CheckDeployed(ctx context.Context, subdir, packageName string, paranoia common.ParanoidMode, manifest ManifestMode) (*DeployedPackage, error)

    // FindDeployed returns a list of packages deployed to a site root.
    // It just does a shallow examination of the metadata directory, without
    // paranoid checks that all installed packages are free from corruption.
    FindDeployed(ctx context.Context) (out common.PinSliceBySubdir, err error)

    // RemoveDeployed deletes a package from a subdir given its name.
    RemoveDeployed(ctx context.Context, subdir, packageName string) error

    // RepairDeployed attempts to restore broken deployed instance.
    // Use CheckDeployed first to figure out what parts of the package need
    // repairs.
    // 'pin' indicates an instances that is supposed to be installed in the given
    // subdir. If there's no such package there or its version is different from
    // the one specified in the pin, returns an error.
    RepairDeployed(ctx context.Context, subdir string, pin common.Pin, params RepairParams) error

    // TempFile returns os.File located in <base>/.cipd/tmp/*.
    // The file is open for reading and writing.
    TempFile(ctx context.Context, prefix string) (*os.File, error)

    // CleanupTrash attempts to remove stale files.
    // This is a best effort operation. Errors are logged (either at Debug or
    // Warning level, depending on severity of the trash state).
    CleanupTrash(ctx context.Context)

Deployer knows how to unzip and place packages into site root directory.

func NewDeployer Uses

func NewDeployer(root string) Deployer

NewDeployer return default Deployer implementation.

type Description Uses

type Description struct {
    Subdir      string `json:"subdir,omitempty"`
    PackageName string `json:"package_name,omitempty"`

Description defines the structure of the description.json file located at .cipd/pkgs/<foo>/description.json.

type Destination Uses

type Destination interface {
    // CreateFile opens a writer to extract some package file to.
    CreateFile(ctx context.Context, name string, opts CreateFileOptions) (io.WriteCloser, error)
    // CreateSymlink creates a symlink (with absolute or relative target).
    CreateSymlink(ctx context.Context, name string, target string) error

Destination knows how to create files and symlink when extracting a package.

All paths are slash separated and relative to the destination root.

'CreateFile' and 'CreateSymlink' can be called concurrently.

func ExistingDestination Uses

func ExistingDestination(dest string, fs FileSystem) Destination

ExistingDestination returns an object that knows how to create files in an existing directory in the file system.

Will use a provided FileSystem object to operate on files if given, otherwise uses a default one. If FileSystem is provided, dest must be in a subdirectory of the given FileSystem root.

Note that the returned object doesn't support transactional semantics, since transactionally writing a bunch of files into an existing directory is hard.

See NewDestination for writing files into a completely new directory. This method supports transactions.

type File Uses

type File interface {
    // Name returns slash separated file path relative to a package root
    // For example "dir/dir/file".
    Name() string

    // Size returns size of the file. 0 for symlinks.
    Size() uint64

    // Executable returns true if the file is executable.
    // Only used for Linux\Mac archives. false for symlinks.
    Executable() bool

    // Writable returns true if the file is user-writable.
    Writable() bool

    // ModTime returns modification time of the file. Zero value means no mtime is
    // recorded.
    ModTime() time.Time

    // Symlink returns true if the file is a symlink.
    Symlink() bool

    // SymlinkTarget return a path the symlink is pointing to.
    SymlinkTarget() (string, error)

    // WinAttrs returns the windows attributes, if any.
    WinAttrs() WinAttrs

    // Open opens the regular file for reading.
    // Returns error for symlink files.
    Open() (io.ReadCloser, error)

File defines a single file to be added or extracted from a package.

All paths are slash separated (including symlink targets).

func NewTestFile Uses

func NewTestFile(name string, data string, opts TestFileOpts) File

NewTestFile returns File implementation (Symlink == false) backed by a fake in-memory data. It is useful in unit tests.

func NewTestSymlink(name string, target string) File

NewTestSymlink returns File implementation (Symlink == true) backed by a fake in-memory data. It is useful in unit tests.

func NewWinTestFile Uses

func NewWinTestFile(name string, data string, attrs WinAttrs) File

NewWinTestFile returns a File implementation (Symlink == false, Executable == false) backed by a fake in-memory data with windows attributes. It is useful in unit tests.

func ScanFileSystem Uses

func ScanFileSystem(dir string, root string, exclude ScanFilter, scanOpts ScanOptions) ([]File, error)

ScanFileSystem returns all files in some file system directory in an alphabetical order. It returns only files, skipping directory entries (i.e. empty directories are completely invisible).

ScanFileSystem follows symbolic links which have a target in <root>/<SiteServiceDir> (i.e. the .cipd folder). Other symlinks will will be preserved as symlinks (see Symlink() method of File interface). Symlinks with absolute targets inside of the root will be converted to relative targets inside of the root.

It scans "dir" path, returning File objects that have paths relative to "root". It skips files and directories for which 'exclude(absolute path)' returns true. It also will always skip <root>/<SiteServiceDir>.

func WrapFile Uses

func WrapFile(abs string, root string, fileInfo *os.FileInfo, scanOpts ScanOptions) (File, error)

WrapFile constructs File object for some file in the file system, specified by its native absolute path 'abs' (subpath of 'root', also specified as a native absolute path). Returned File object has path relative to 'root'. If fileInfo is given, it will be used to grab file mode and size, otherwise os.Lstat will be used to get it. Recognizes symlinks.

type FileInfo Uses

type FileInfo struct {
    // Name is slash separated file path relative to a package root.
    Name string `json:"name"`

    // Size is a size of the file. 0 for symlinks.
    Size uint64 `json:"size"`

    // Executable is true if the file is executable.
    // Only used for Linux\Mac archives. False for symlinks.
    Executable bool `json:"executable,omitempty"`

    // Writable is true if the file is user-writable.
    Writable bool `json:"writable,omitempty"`

    // ModTime is Unix timestamp with modification time of the file as it is set
    // inside CIPD package.
    // May be 0 if the package was built without preserving the modification
    // times.
    ModTime int64 `json:"modtime,omitempty"`

    // WinAttrs is a string representation of extra windows file attributes.
    // Only used for Win archives.
    WinAttrs string `json:"win_attrs,omitempty"`

    // Symlink is a path the symlink points to or "" if the file is not a symlink.
    Symlink string `json:"symlink,omitempty"`

FileInfo is JSON-ish struct with info extracted from File interface.

type FileSystem Uses

type FileSystem interface {
    // Root returns absolute path to a directory FileSystem operates in.
    // All FS actions are restricted to this directory.
    Root() string

    // CwdRelToAbs converts a path relative to cwd to an absolute one.
    // If also verifies the path is under the root path of the FileSystem object.
    // If passed path is already absolute, just checks that it's under the root.
    CwdRelToAbs(path string) (string, error)

    // RootRelToAbs converts a path relative to Root() to an absolute one.
    // It verifies the path is under the root path of the FileSystem object.
    // If passed path is already absolute, just checks that it's under the root.
    RootRelToAbs(path string) (string, error)

    // OpenFile opens a file and returns its file handle.
    // Files opened with OpenFile can be safely manipulated by other
    // FileSystem functions.
    // This differs from os.Open notably on Windows, where OpenFile ensures that
    // files are open with FILE_SHARE_DELETE permisson to enable them to be
    // atomically renamed without contention.
    OpenFile(path string) (*os.File, error)

    // Stat returns a FileInfo describing the named file, following symlinks.
    Stat(ctx context.Context, path string) (os.FileInfo, error)

    // Lstat returns a FileInfo describing the named file, not following symlinks.
    Lstat(ctx context.Context, path string) (os.FileInfo, error)

    // EnsureDirectory creates a directory at given native path.
    // Does nothing it the path already exists. It takes an absolute path or
    // a path relative to the current working directory and always returns
    // absolute path.
    EnsureDirectory(ctx context.Context, path string) (string, error)

    // EnsureSymlink creates a symlink pointing to a target.
    // It will create full directory path to the symlink if necessary.
    EnsureSymlink(ctx context.Context, path string, target string) error

    // EnsureFile creates a file and calls the function to write file content.
    // It will create full directory path to the file if necessary.
    EnsureFile(ctx context.Context, path string, write func(*os.File) error) error

    // EnsureFileGone removes a file, logging the errors (if any).
    // Missing file is not an error.
    EnsureFileGone(ctx context.Context, path string) error

    // EnsureDirectoryGone recursively removes a directory.
    EnsureDirectoryGone(ctx context.Context, path string) error

    // Renames oldpath to newpath.
    // If newpath already exists (be it a file or a directory), removes it first.
    // If oldpath is a symlink, it's moved as is (e.g. as a symlink).
    Replace(ctx context.Context, oldpath, newpath string) error

    // CleanupTrash attempts to remove all files that ended up in the trash.
    // This is a best effort operation. Errors are logged (either at Debug or
    // Warning level, depending on severity of the trash state).
    CleanupTrash(ctx context.Context)

FileSystem abstracts operations that touch single file system subpath.

All functions operate in terms of native file paths. It exists mostly to hide differences between file system semantic on Windows and Linux\Mac.

func NewFileSystem Uses

func NewFileSystem(root, trash string) FileSystem

NewFileSystem returns default FileSystem implementation.

It operates with files under a given path. All methods accept absolute paths or paths relative to current working directory. FileSystem will ensure they are under 'root' directory.

It can also accept a path to a directory to put "trash" into: files that can't be removed because there are some processes keeping lock on them. This is useful on Windows when replacing running executables. The trash directory must be on the same disk as the root directory.

It 'trash' is empty string, the trash directory will be created under 'root'.

type InstallMode Uses

type InstallMode string

InstallMode defines how to install a package.

const (
    // InstallModeSymlink is default (for backward compatibility).
    // In this mode all files are extracted to .cipd/*/... and then symlinked to
    // the site root directory. Version switch happens atomically.
    InstallModeSymlink InstallMode = "symlink"

    // InstallModeCopy is used when files should be directly copied.
    // This mode is always used on Windows (and can be optionally) used on
    // other OSes. If installation is aborted midway, the package may end up
    // in inconsistent state.
    InstallModeCopy InstallMode = "copy"

func (*InstallMode) Set Uses

func (m *InstallMode) Set(value string) error

Set is called by 'flag' package when parsing command line options.

func (InstallMode) String Uses

func (m InstallMode) String() string

String is needed to conform to flag.Value interface.

type InstanceFile Uses

type InstanceFile interface {

    // Close is a bit non-standard, and can be used to indicate to the storage
    // (filesystem and/or cache) layer that this instance is actually bad. The
    // storage layer can then evict/revoke, etc. the bad file.
    Close(ctx context.Context, corrupt bool) error

InstanceFile is an underlying data file for a PackageInstance.

type Manifest Uses

type Manifest struct {
    FormatVersion string      `json:"format_version"`
    PackageName   string      `json:"package_name"`
    VersionFile   string      `json:"version_file,omitempty"` // where to put JSON with info about deployed package
    InstallMode   InstallMode `json:"install_mode,omitempty"` // how to install: "copy" or "symlink"
    Files         []FileInfo  `json:"files,omitempty"`        // present only in deployed manifest

Manifest defines structure of manifest.json file.

type ManifestMode Uses

type ManifestMode bool

ManifestMode is used to indicate presence of absence of manifest when calling various functions.

Just to improve code readability, since Func(..., WithManifest) is less cryptic than Func(..., true).

const (
    // WithoutManifest indicates the function should skip manifest.
    WithoutManifest ManifestMode = false
    // WithManifest indicates the function should handle manifest.
    WithManifest ManifestMode = true

type OpenInstanceOpts Uses

type OpenInstanceOpts struct {
    // VerificationMode specifies what to do with the hash of the instance file.
    // Passing VerifyHash instructs OpenPackage to calculate the hash of the
    // instance file using the hash algorithm matching InstanceID, and then
    // compare the resulting digest to InstanceID, failing on mismatch with
    // ErrHashMismatch. HashAlgo is ignored in this case and should be 0.
    // Passing SkipHashVerification instructs OpenPackage to unconditionally
    // trust the given InstanceID. HashAlgo is also ignored in this case and
    // should be 0.
    // Passing CalculateHash instructs OpenPackage to calculate the hash of the
    // instance file using the given HashAlgo, and use the resulting digest
    // as an instance ID of the new PackageInstance object. InstanceID is ignored
    // in this case and should be "".
    VerificationMode VerificationMode

    // InstanceID encodes the expected hash of the instance file.
    // May be empty. See the comment for VerificationMode for more details.
    InstanceID string

    // HashAlgo specifies what hashing algorithm to use for computing instance ID.
    // May be empty. See the comment for VerificationMode for more details.
    HashAlgo api.HashAlgo

OpenInstanceOpts is passed to OpenInstance and OpenInstanceFile.

type PackageChunkDef Uses

type PackageChunkDef struct {
    // Dir is a directory to add to the package (recursively).
    Dir string

    // File is a single file to add to the package.
    File string

    // VersionFile defines where to drop JSON file with package version.
    VersionFile string `yaml:"version_file"`

    // Exclude is a list of regexp patterns to exclude when scanning a directory.
    Exclude []string

PackageChunkDef represents one entry in 'data' section of package definition.

It is either a single file, or a recursively scanned directory (with optional list of regexps for files to skip).

type PackageDef Uses

type PackageDef struct {
    // Package defines a name of the package.
    Package string

    // Root defines where to search for files. It may either be an absolute path,
    // or it may be a path relative to the package file itself. If omitted, it
    // defaults to "." (i.e., the same directory as the package file)
    Root string

    // InstallMode defines how to deploy the package file: "copy" or "symlink".
    InstallMode InstallMode `yaml:"install_mode"`

    // PreserveModTime instructs CIPD to preserve the mtime of the files.
    PreserveModTime bool `yaml:"preserve_mtime"`

    // PreserveWritable instructs CIPD to preserve the user-writable permission
    // mode on the files.
    PreserveWritable bool `yaml:"preserve_writable"`

    // Data describes what is deployed with the package.
    Data []PackageChunkDef

PackageDef defines how exactly to build a package.

It specified what files to put into it, how to name them, how to name the package itself, etc. It is loaded from *.yaml file.

func LoadPackageDef Uses

func LoadPackageDef(r io.Reader, vars map[string]string) (PackageDef, error)

LoadPackageDef loads package definition from a YAML source code.

It substitutes %{...} strings in the definition with corresponding values from 'vars' map.

func (*PackageDef) FindFiles Uses

func (def *PackageDef) FindFiles(cwd string) ([]File, error)

FindFiles scans files system and returns files to be added to the package.

It uses a path to package definition file directory ('cwd' argument) to find a root of the package.

func (*PackageDef) VersionFile Uses

func (def *PackageDef) VersionFile() string

VersionFile defines where to drop JSON file with package version.

type PackageInstance Uses

type PackageInstance interface {
    // Pin identifies package name and concreted instance ID of this package file.
    Pin() common.Pin

    // Files returns a list of files to deploy with the package.
    Files() []File

    // DataReader returns reader that reads raw package data.
    DataReader() io.ReadSeeker

PackageInstance represents a binary CIPD package file (with manifest inside).

func OpenInstance Uses

func OpenInstance(ctx context.Context, r InstanceFile, opts OpenInstanceOpts) (PackageInstance, error)

OpenInstance prepares the package for extraction.

func OpenInstanceFile Uses

func OpenInstanceFile(ctx context.Context, path string, opts OpenInstanceOpts) (inst PackageInstance, closer func() error, err error)

OpenInstanceFile opens a package instance file on disk.

The caller of this function must call closer() if err != nil to close the underlying file.

type RepairParams Uses

type RepairParams struct {
    // Instance holds the original package data.
    // Must be present if ToRedeploy is not empty. Otherwise not used.
    Instance PackageInstance

    // ToRedeploy is a list of files that needs to be extracted from the instance
    // and relinked into the site root.
    ToRedeploy []string

    // ToRelink is a list of files that just needs to be relinked into the site
    // root.
    ToRelink []string

RepairsParams is passed to RepairDeployed.

type ScanFilter Uses

type ScanFilter func(abs string) bool

ScanFilter is predicate used by ScanFileSystem to decide what files to skip.

type ScanOptions Uses

type ScanOptions struct {
    // PreserveModTime when true saves the file's modification time.
    PreserveModTime bool
    // PreserveWritable when true saves the file's writable bit for either user,
    // group or world (mask 0222), and converts it to user-only writable bit (mask
    // 0200).
    PreserveWritable bool

ScanOptions specify which file properties to preserve in the archive.

type TestFileOpts Uses

type TestFileOpts struct {
    Executable bool
    Writable   bool
    ModTime    time.Time

TestFileOpts holds options for NewTestFile method. Used in unittests.

type TransactionalDestination Uses

type TransactionalDestination interface {

    // Begin initiates a new write transaction.
    Begin(ctx context.Context) error
    // End finalizes package extraction (commit or rollback, based on success).
    End(ctx context.Context, success bool) error

TransactionalDestination is a destination that supports transactions.

It provides 'Begin' and 'End' methods: all calls to 'CreateFile' and 'CreateSymlink' should happen between them. No changes are really applied until End(true) is called. A call to End(false) discards any pending changes.

func NewDestination Uses

func NewDestination(dest string, fs FileSystem) TransactionalDestination

NewDestination returns a destination in the file system (a new directory) to extract a package to.

Will use a provided FileSystem object to operate on files if given, otherwise use a default one. If FileSystem is provided, dir must be in a subdirectory of the given FileSystem root.

type VerificationMode Uses

type VerificationMode int

VerificationMode defines whether to verify hash or not.

const (
    // VerifyHash instructs OpenPackage to calculate the hash of the package file
    // and compare it to the given InstanceID.
    VerifyHash VerificationMode = 0

    // SkipHashVerification instructs OpenPackage to skip the hash verification
    // and trust that the given InstanceID matches the package.
    SkipHashVerification VerificationMode = 1

    // CalculateHash instructs OpenPackage to calculate the hash of the package
    // file using the given hash algo, and use it as a new instance ID.
    CalculateHash VerificationMode = 2

type VersionFile Uses

type VersionFile struct {
    PackageName string `json:"package_name"`
    InstanceID  string `json:"instance_id"`

VersionFile describes JSON file with package version information that's deployed to a path specified in 'version_file' attribute of the manifest.

type WinAttrs Uses

type WinAttrs uint32

WinAttrs represents the extra file attributes for windows.

const (
    WinAttrHidden WinAttrs = 0x2
    WinAttrSystem WinAttrs = 0x4

    WinAttrsAll WinAttrs = WinAttrHidden | WinAttrSystem

These are the valid WinAttrs values. They may be OR'd together to form a mask. These match the windows GetFileAttributes values.

func (WinAttrs) String Uses

func (w WinAttrs) String() string

Package local imports 31 packages (graph) and is imported by 6 packages. Updated 2018-08-21. Refresh now. Tools for package owners.