luci: Index | Files

package processing

import ""

Package processing contains code related to post-registration instance processing.


Package Files

client_extractor.go doc.go processor.go reader.go


const ClientExtractorProcID = "cipd_client_binary:v1"

ClientExtractorProcID is identifier of ClientExtractor processor.

func GetClientBinaryName Uses

func GetClientBinaryName(pkg string) string

GetClientBinaryName returns name of CIPD binary inside the package.

Either 'cipd' or 'cipd.exe'.

func GetClientPackage Uses

func GetClientPackage(platform string) (string, error)

GetClientPackage returns the name of the client package for CIPD client for the given platform.

Returns an error if the platform name is invalid.

func IsClientPackage Uses

func IsClientPackage(pkg string) bool

IsClientPackage returns true if the given package stores a CIPD client.

type ClientExtractor Uses

type ClientExtractor struct {
    CAS cas.StorageServer
    // contains filtered or unexported fields

ClientExtractor is a processor that extracts CIPD client binary from CIPD client packages (infra/tools/cipd/...) and stores it in the CAS, so it can be fetched directly.

This is needed to support CIPD client bootstrap using e.g. 'curl'.

func (*ClientExtractor) Applicable Uses

func (e *ClientExtractor) Applicable(inst *model.Instance) bool

Applicable is part of Processor interface.

func (*ClientExtractor) ID Uses

func (e *ClientExtractor) ID() string

ID is part of Processor interface.

func (*ClientExtractor) Run Uses

func (e *ClientExtractor) Run(ctx context.Context, inst *model.Instance, pkg *PackageReader) (res Result, err error)

Run is part of Processor interface.

type ClientExtractorResult Uses

type ClientExtractorResult struct {
    ClientBinary struct {
        Size int64 `json:"size"`

        // Algo used to name the extracted file, matches the client package algo.
        HashAlgo   string `json:"hash_algo"`   // cas.HashAlgo enum serialized to string
        HashDigest string `json:"hash_digest"` // as hex string

        // AllHashDigests are hex digests of the extracted file calculated using all
        // algos known to the server at the time the file was uploaded.
        // Keys are cas.HashAlgo enum values as strings ('SHA1', 'SHA256', ...).
        // If empty (for old records), only supported algo is HashAlgo from above
        // (which for old records is always SHA1).
        AllHashDigests map[string]string `json:"all_hash_digests"`
    } `json:"client_binary"`

ClientExtractorResult is stored in JSON form as a result of ClientExtractor execution.

Compatible with Python version of the backend.

If format of this struct changes in a non backward compatible way, the version number in ClientExtractorProcID should change too.

func GetClientExtractorResult Uses

func GetClientExtractorResult(c context.Context, inst *api.Instance) (*ClientExtractorResult, error)

GetClientExtractorResult returns results of client extractor processor.

They contain a reference to the unpacked CIPD binary object in the Google Storage.


(result, nil) on success.
(nil, datastore.ErrNoSuchEntity) if results are not available.
(nil, transient-tagged error) on retrieval errors.
(nil, non-transient-tagged error) if the client extractor failed.

func (*ClientExtractorResult) ObjectRefAliases Uses

func (r *ClientExtractorResult) ObjectRefAliases() []*api.ObjectRef

ObjectRefAliases is list of ObjectRefs calculated using all hash algos known to the server when the client binary was extracted.

Additionally all algos not understood by the server right NOW are skipped too. This may arise if the server was rolled back, but some files have already been uploaded with a newer algo.

func (*ClientExtractorResult) ToObjectRef Uses

func (r *ClientExtractorResult) ToObjectRef() (*api.ObjectRef, error)

ToObjectRef returns a reference to the extracted client binary in CAS.

The returned ObjectRef is validated to be syntactically correct already.

type PackageReader Uses

type PackageReader struct {
    // contains filtered or unexported fields

PackageReader knows how to extract files from CIPD packages.

CIPD packages are actually zip archives, but we don't want to expose it everywhere.

func NewPackageReader Uses

func NewPackageReader(r io.ReaderAt, size int64) (*PackageReader, error)

NewPackageReader opens the package by reading its directory.

func (*PackageReader) Open Uses

func (p *PackageReader) Open(path string) (io.ReadCloser, int64, error)

Open opens some file inside the package for reading.

Returns the ReadCloser and the uncompressed file size.

type Processor Uses

type Processor interface {
    // ID is unique identifier of this processor used store processing results in
    // the datastore.
    ID() string

    // Applicable returns true if this processor should be applied to an instance.
    Applicable(inst *model.Instance) bool

    // Run executes the processing on the package instance.
    // Returns either a result, or a transient error. All fatal errors should be
    // communicated through the result.
    // Must be idempotent. The processor may be called multiple times when
    // retrying task queue tasks.
    Run(ctx context.Context, inst *model.Instance, pkg *PackageReader) (Result, error)

Processor runs some post-processing step on a package instance after it has been uploaded.

type Result Uses

type Result struct {
    Result interface{} // JSON-serializable summary extracted by the processor
    Err    error       // if non-nil, contains the fatal error message

Result is returned by processors.

It is either some JSON-serializable data or a fatal error.

Package processing imports 18 packages (graph) and is imported by 1 packages. Updated 2021-01-26. Refresh now. Tools for package owners.