local

package
v0.0.0-...-678bb0e Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 8, 2017 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Index

Constants

View Source
const SiteServiceDir = ".cipd"

SiteServiceDir is a name of the directory inside an installation root reserved for cipd stuff.

Variables

View Source
var ErrHashMismatch = errors.New("package hash mismatch")

Functions

func BuildInstance

func BuildInstance(ctx context.Context, opts BuildInstanceOptions) error

BuildInstance builds a new package instance.

If build an instance of package named opts.PackageName by archiving input files (passed via opts.Input).

The final binary is written to opts.Output. Some output may be written even if BuildInstance eventually returns an error.

func EnsureFile

func EnsureFile(ctx context.Context, fs FileSystem, path string, content io.Reader) error

EnsureFile creates a file with the given content. It will create full directory path to the file if necessary.

func ExtractInstance

func ExtractInstance(ctx context.Context, inst PackageInstance, dest Destination, exclude ExtractFilter) error

ExtractInstance extracts all files from a package instance into a destination.

If exclude != nil, it will be used to filter the contents of the PackageInstance before extraction.

func IsCorruptionError

func IsCorruptionError(err error) bool

IsCorruptionError returns true iff err indicates corruption.

func ValidateInstallMode

func ValidateInstallMode(mode InstallMode) error

ValidateInstallMode returns non nil if install mode is invalid.

Valid modes are: "" (client will pick platform default), "copy" (aka InstallModeCopy), "symlink" (aka InstallModeSymlink).

Types

type BuildInstanceOptions

type BuildInstanceOptions struct {
	// Input is a list of files to add to the package.
	Input []File

	// Output is where to write the package file to.
	Output io.Writer

	// PackageName is name of the package being built, e.g. 'infra/tools/cipd'.
	PackageName string

	// VersionFile is slash separated path where to drop JSON with version info.
	VersionFile string

	// InstallMode defines how to install the package: "copy" or "symlink".
	InstallMode InstallMode

	// CompressionLevel defines deflate compression level in range [0-9].
	CompressionLevel int
}

BuildInstanceOptions defines options for BuildInstance function.

type Deployer

type Deployer interface {
	// DeployInstance installs an instance of a package into the given subdir of
	// the root.
	//
	// It unpacks the package into <base>/.cipd/pkgs/*, and rearranges
	// symlinks to point to unpacked files. It tries to make it as "atomic" as
	// possible. Returns information about the deployed instance.
	//
	// Due to a historical bug, if inst contains any files which are intended to
	// be deployed to `.cipd/*`, they will not be extracted and you'll see
	// warnings logged.
	DeployInstance(ctx context.Context, subdir string, inst PackageInstance) (common.Pin, error)

	// CheckDeployed checks whether a given package is deployed at the given
	// subdir.
	//
	// It returns information about installed version (or error if not installed).
	CheckDeployed(ctx context.Context, subdir, packageName string) (common.Pin, error)

	// FindDeployed returns a list of packages deployed to a site root.
	FindDeployed(ctx context.Context) (out common.PinSliceBySubdir, err error)

	// RemoveDeployed deletes a package from a subdir given its name.
	RemoveDeployed(ctx context.Context, subdir, packageName string) error

	// TempFile returns os.File located in <base>/.cipd/tmp/*.
	//
	// The file is open for reading and writing.
	TempFile(ctx context.Context, prefix string) (*os.File, error)

	// CleanupTrash attemps to remove stale files.
	//
	// May return errors if some files are still locked, this is fine.
	CleanupTrash(ctx context.Context) error
}

Deployer knows how to unzip and place packages into site root directory.

func NewDeployer

func NewDeployer(root string) Deployer

NewDeployer return default Deployer implementation.

type Description

type Description struct {
	Subdir      string `json:"subdir,omitempty"`
	PackageName string `json:"package_name,omitempty"`
}

Description defines the structure of the description.json file located at .cipd/pkgs/<foo>/description.json.

type Destination

type Destination interface {
	// Begin initiates a new write transaction. Called before first CreateFile.
	Begin(ctx context.Context) error

	// CreateFile opens a writer to extract some package file to.
	CreateFile(ctx context.Context, name string, executable bool, winAttrs WinAttrs) (io.WriteCloser, error)

	// CreateSymlink creates a symlink (with absolute or relative target).
	//
	// 'name' must be a slash separated path relative to the destination root.
	CreateSymlink(ctx context.Context, name string, target string) error

	// End finalizes package extraction (commit or rollback, based on success).
	End(ctx context.Context, success bool) error
}

Destination knows how to create files when extracting a package.

It supports transactional semantic by providing 'Begin' and 'End' methods. No changes should be applied until End(true) is called. A call to End(false) should discard any pending changes. All paths are slash separated.

While between 'Begin' and 'End', 'CreateFile' and 'CreateSymlink' can be called concurrently.

func NewFileSystemDestination

func NewFileSystemDestination(dir string, fs FileSystem) Destination

NewFileSystemDestination returns a destination in the file system (directory) to extract a package to.

Will use a provided FileSystem object to operate on files if given, otherwise use a default one. If FileSystem is provided, dir must be in a subdirectory of the given FileSystem root.

type ExtractFilter

type ExtractFilter func(f File) bool

ExtractFilter is predicate used by ExtractInstance to exclude files from extraction and the manifest.json. The function will be given the name of the file inside the instance zip (so, the path relative to the extraction destination).

type File

type File interface {
	// Name returns slash separated file path relative to a package root
	//
	// For example "dir/dir/file".
	Name() string

	// Size returns size of the file. 0 for symlinks.
	Size() uint64

	// Executable returns true if the file is executable.
	//
	// Only used for Linux\Mac archives. false for symlinks.
	Executable() bool

	// Symlink returns true if the file is a symlink.
	Symlink() bool

	// SymlinkTarget return a path the symlink is pointing to.
	SymlinkTarget() (string, error)

	// WinAttrs returns the windows attributes, if any.
	WinAttrs() WinAttrs

	// Open opens the regular file for reading.
	//
	// Returns error for symlink files.
	Open() (io.ReadCloser, error)
}

File defines a single file to be added or extracted from a package.

All paths are slash separated (including symlink targets).

func NewTestFile

func NewTestFile(name string, data string, executable bool) File

NewTestFile returns File implementation (Symlink == false) backed by a fake in-memory data. It is useful in unit tests.

func NewTestSymlink(name string, target string) File

NewTestSymlink returns File implementation (Symlink == true) backed by a fake in-memory data. It is useful in unit tests.

func NewWinTestFile

func NewWinTestFile(name string, data string, attrs WinAttrs) File

NewWinTestFile returns a File implementation (Symlink == false, Executable == false) backed by a fake in-memory data with windows attributes. It is useful in unit tests.

func ScanFileSystem

func ScanFileSystem(dir string, root string, exclude ScanFilter) ([]File, error)

ScanFileSystem returns all files in some file system directory in an alphabetical order. It returns only files, skipping directory entries (i.e. empty directories are completely invisible).

ScanFileSystem follows symbolic links which have a target in <root>/<SiteServiceDir> (i.e. the .cipd folder). Other symlinks will will be preserved as symlinks (see Symlink() method of File interface). Symlinks with absolute targets inside of the root will be converted to relative targets inside of the root.

It scans "dir" path, returning File objects that have paths relative to "root". It skips files and directories for which 'exclude(absolute path)' returns true. It also will always skip <root>/<SiteServiceDir>.

func WrapFile

func WrapFile(abs string, root string, fileInfo *os.FileInfo) (File, error)

WrapFile constructs File object for some file in the file system, specified by its native absolute path 'abs' (subpath of 'root', also specified as a native absolute path). Returned File object has path relative to 'root'. If fileInfo is given, it will be used to grab file mode and size, otherwise os.Lstat will be used to get it. Recognizes symlinks.

type FileInfo

type FileInfo struct {
	// Name is slash separated file path relative to a package root.
	Name string `json:"name"`

	// Size is a size of the file. 0 for symlinks.
	Size uint64 `json:"size"`

	// Executable is true if the file is executable.
	//
	// Only used for Linux\Mac archives. False for symlinks.
	Executable bool `json:"executable,omitempty"`

	// WinAttrs is a string representation of extra windows file attributes.
	//
	// Only used for Win archives.
	WinAttrs string `json:"win_attrs,omitempty"`

	// Symlink is a path the symlink points to or "" if the file is not a symlink.
	Symlink string `json:"symlink,omitempty"`
}

FileInfo is JSON-ish struct with info extracted from File interface.

type FileSystem

type FileSystem interface {
	// Root returns absolute path to a directory FileSystem operates in.
	//
	// All FS actions are restricted to this directory.
	Root() string

	// CwdRelToAbs converts a path relative to cwd to an absolute one.
	//
	// If also verifies the path is under the root path of the FileSystem object.
	// If passed path is already absolute, just checks that it's under the root.
	CwdRelToAbs(path string) (string, error)

	// RootRelToAbs converts a path relative to Root() to an absolute one.
	//
	// It verifies the path is under the root path of the FileSystem object.
	// If passed path is already absolute, just checks that it's under the root.
	RootRelToAbs(path string) (string, error)

	// EnsureDirectory creates a directory at given native path.
	//
	// Does nothing it the path already exists. It takes an absolute path or
	// a path relative to the current working directory and always returns
	// absolute path.
	EnsureDirectory(ctx context.Context, path string) (string, error)

	// EnsureSymlink creates a symlink pointing to a target.
	//
	// It will create full directory path to the symlink if necessary.
	EnsureSymlink(ctx context.Context, path string, target string) error

	// EnsureFile creates a file and calls the function to write file content.
	//
	// It will create full directory path to the file if necessary.
	EnsureFile(ctx context.Context, path string, write func(*os.File) error) error

	// EnsureFileGone removes a file, logging the errors (if any).
	//
	// Missing file is not an error.
	EnsureFileGone(ctx context.Context, path string) error

	// EnsureDirectoryGone recursively removes a directory.
	EnsureDirectoryGone(ctx context.Context, path string) error

	// Renames oldpath to newpath.
	//
	// If newpath already exists (be it a file or a directory), removes it first.
	// If oldpath is a symlink, it's moved as is (e.g. as a symlink).
	Replace(ctx context.Context, oldpath, newpath string) error

	// CleanupTrash attempts to remove all files that ended up in the trash.
	//
	// This is best effort operation.
	CleanupTrash(ctx context.Context) error
}

FileSystem abstracts operations that touch single file system subpath.

All functions operate in terms of native file paths. It exists mostly to hide differences between file system semantic on Windows and Linux\Mac.

func NewFileSystem

func NewFileSystem(root, trash string) FileSystem

NewFileSystem returns default FileSystem implementation.

It operates with files under a given path. All methods accept absolute paths or paths relative to current working directory. FileSystem will ensure they are under 'root' directory.

It can also accept a path to a directory to put "trash" into: files that can't be removed because there are some processes keeping lock on them. This is useful on Windows when replacing running executables. The trash directory must be on the same disk as the root directory.

It 'trash' is empty string, the trash directory will be created under 'root'.

type InstallMode

type InstallMode string

InstallMode defines how to install a package.

const (
	// InstallModeSymlink is default (for backward compatibility).
	//
	// In this mode all files are extracted to .cipd/*/... and then symlinked to
	// the site root directory. Version switch happens atomically.
	InstallModeSymlink InstallMode = "symlink"

	// InstallModeCopy is used when files should be directly copied.
	//
	// This mode is always used on Windows (and can be optionally) used on
	// other OSes. If installation is aborted midway, the package may end up
	// in inconsistent state.
	InstallModeCopy InstallMode = "copy"
)

func (*InstallMode) Set

func (m *InstallMode) Set(value string) error

Set is called by 'flag' package when parsing command line options.

func (InstallMode) String

func (m InstallMode) String() string

String is needed to conform to flag.Value interface.

type InstanceFile

type InstanceFile interface {
	io.ReadSeeker

	// Close is a bit non-standard, and can be used to indicate to the storage
	// (filesystem and/or cache) layer that this instance is actaully bad. The
	// storage layer can then evict/revoke, etc. the bad file.
	Close(ctx context.Context, corrupt bool) error
}

InstanceFile is an underlying data file for a PackageInstance.

type Manifest

type Manifest struct {
	FormatVersion string      `json:"format_version"`
	PackageName   string      `json:"package_name"`
	VersionFile   string      `json:"version_file,omitempty"` // where to put JSON with info about deployed package
	InstallMode   InstallMode `json:"install_mode,omitempty"` // how to install: "copy" or "symlink"
	Files         []FileInfo  `json:"files,omitempty"`        // present only in deployed manifest
}

Manifest defines structure of manifest.json file.

type PackageChunkDef

type PackageChunkDef struct {
	// Dir is a directory to add to the package (recursively).
	Dir string

	// File is a single file to add to the package.
	File string

	// VersionFile defines where to drop JSON file with package version.
	VersionFile string `yaml:"version_file"`

	// Exclude is a list of regexp patterns to exclude when scanning a directory.
	Exclude []string
}

PackageChunkDef represents one entry in 'data' section of package definition.

It is either a single file, or a recursively scanned directory (with optional list of regexps for files to skip).

type PackageDef

type PackageDef struct {
	// Package defines a name of the package.
	Package string

	// Root defines where to search for files. It may either be an absolute path,
	// or it may be a path relative to the package file itself. If omitted, it
	// defaults to "." (i.e., the same directory as the package file)
	Root string

	// InstallMode defines how to deploy the package file: "copy" or "symlink".
	InstallMode InstallMode `yaml:"install_mode"`

	// Data describes what is deployed with the package.
	Data []PackageChunkDef
}

PackageDef defines how exactly to build a package.

It specified what files to put into it, how to name them, how to name the package itself, etc. It is loaded from *.yaml file.

func LoadPackageDef

func LoadPackageDef(r io.Reader, vars map[string]string) (PackageDef, error)

LoadPackageDef loads package definition from a YAML source code.

It substitutes %{...} strings in the definition with corresponding values from 'vars' map.

func (*PackageDef) FindFiles

func (def *PackageDef) FindFiles(cwd string) ([]File, error)

FindFiles scans files system and returns files to be added to the package.

It uses a path to package definition file directory ('cwd' argument) to find a root of the package.

func (*PackageDef) VersionFile

func (def *PackageDef) VersionFile() string

VersionFile defines where to drop JSON file with package version.

type PackageInstance

type PackageInstance interface {
	// Pin identifies package name and concreted instance ID of this package file.
	Pin() common.Pin

	// Files returns a list of files to deploy with the package.
	Files() []File

	// DataReader returns reader that reads raw package data.
	DataReader() io.ReadSeeker
}

PackageInstance represents a binary CIPD package file (with manifest inside).

func OpenInstance

func OpenInstance(ctx context.Context, r InstanceFile, instanceID string, v VerificationMode) (PackageInstance, error)

OpenInstance prepares the package for extraction.

If instanceID is an empty string, OpenInstance will calculate the hash of the package and use it as InstanceID (regardless of verification mode).

If instanceID is not empty and verification mode is VerifyHash, OpenInstance will check that package data matches the given instanceID. It skips this check if verification mode is SkipHashVerification.

func OpenInstanceFile

func OpenInstanceFile(ctx context.Context, path string, instanceID string, v VerificationMode) (inst PackageInstance, closer func() error, err error)

OpenInstanceFile opens a package instance file on disk.

The caller of this function must call closer() if err != nil to close the underlying file.

type ScanFilter

type ScanFilter func(abs string) bool

ScanFilter is predicate used by ScanFileSystem to decide what files to skip.

type VerificationMode

type VerificationMode int

VerificationMode is defines whether to verify hash or not.

const (
	// VerifyHash instructs OpenPackage to calculate hash of the package and
	// compare it to the given instanceID.
	VerifyHash VerificationMode = 0

	// SkipVerification instructs OpenPackage to skip the hash verification and
	// trust that the given instanceID matches the package.
	SkipHashVerification VerificationMode = 1
)

type VersionFile

type VersionFile struct {
	PackageName string `json:"package_name"`
	InstanceID  string `json:"instance_id"`
}

VersionFile describes JSON file with package version information that's deployed to a path specified in 'version_file' attribute of the manifest.

type WinAttrs

type WinAttrs uint32

WinAttrs represents the extra file attributes for windows.

const (
	WinAttrHidden WinAttrs = 0x2
	WinAttrSystem WinAttrs = 0x4

	WinAttrsAll WinAttrs = WinAttrHidden | WinAttrSystem
)

These are the valid WinAttrs values. They may be OR'd together to form a mask. These match the windows GetFileAttributes values.

func (WinAttrs) String

func (w WinAttrs) String() string

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL