luci: go.chromium.org/luci/client/archiver Index | Files

package archiver

import "go.chromium.org/luci/client/archiver"

Package archiver implements the pipeline to efficiently archive file sets to an isolated server as fast as possible.

Index

Package Files

archiver.go checker.go directory.go doc.go tar_archiver.go tarring_archiver.go upload_tracker.go uploader.go

type Archiver Uses

type Archiver struct {
    // contains filtered or unexported fields
}

Archiver is an high level interface to an isolatedclient.Client.

Uses a 4 stages pipeline, each doing work concurrently:

- Deduplicating similar requests or known server hot cache hits.
- Hashing files.
- Batched cache hit lookups on the server.
- Uploading cache misses.

func New Uses

func New(ctx context.Context, c *isolatedclient.Client, out io.Writer) *Archiver

New returns a thread-safe Archiver instance.

If not nil, out will contain tty-oriented progress information.

ctx will be used for logging.

func (*Archiver) Cancel Uses

func (a *Archiver) Cancel(reason error)

Cancel implements common.Canceler

func (*Archiver) CancelationReason Uses

func (a *Archiver) CancelationReason() error

CancelationReason implements common.Canceler

func (*Archiver) Channel Uses

func (a *Archiver) Channel() <-chan error

Channel implements common.Canceler

func (*Archiver) Close Uses

func (a *Archiver) Close() error

Close waits for all pending files to be done. If an error occured during processing, it is returned.

func (*Archiver) Push Uses

func (a *Archiver) Push(displayName string, source isolatedclient.Source, priority int64) *PendingItem

Push schedules item upload to the isolate server. Smaller priority value means earlier processing.

func (*Archiver) PushFile Uses

func (a *Archiver) PushFile(displayName, path string, priority int64) *PendingItem

PushFile schedules file upload to the isolate server. Smaller priority value means earlier processing.

func (*Archiver) Stats Uses

func (a *Archiver) Stats() *Stats

Stats returns a copy of the statistics.

type BundlingChecker Uses

type BundlingChecker struct {
    Hit, Miss CountBytes
    // contains filtered or unexported fields
}

BundlingChecker uses the isolatedclient.Client to check whether items are available on the server. BundlingChecker methods are safe to call concurrently.

func NewChecker Uses

func NewChecker(ctx context.Context, client *isolatedclient.Client, maxConcurrent int) *BundlingChecker

NewChecker creates a new Checker with the given isolated client. maxConcurrent controls maximum number of check requests to be in-flight at once. The provided context is used to make all requests to the isolate server.

func (*BundlingChecker) AddItem Uses

func (c *BundlingChecker) AddItem(item *Item, isolated bool, callback CheckerCallback)

AddItem adds the given item to the checker for testing, and invokes the provided callback asynchronously. The isolated param indicates whether the given item represents a JSON isolated manifest (as opposed to a regular file). In the case of an error, the callback may never be invoked.

func (*BundlingChecker) Close Uses

func (c *BundlingChecker) Close() error

Close shuts down the checker, blocking until all pending items have been checked with the server. Close returns the first error encountered during the checking process, if any. After Close has returned, Checker is guaranteed to no longer invoke any previously-provided callback.

func (*BundlingChecker) PresumeExists Uses

func (c *BundlingChecker) PresumeExists(item *Item)

PresumeExists causes the Checker to report that item exists on the server.

type Checker Uses

type Checker interface {
    AddItem(item *Item, isolated bool, callback CheckerCallback)
    PresumeExists(item *Item)
    Close() error
}

A Checker checks whether items are available on the server. It has a single implementation, *BundlingChecker. See BundlingChecker for method documentation.

type CheckerCallback Uses

type CheckerCallback func(*Item, *isolatedclient.PushState)

CheckerCallback is the callback used by Checker to indicate whether a file is present on the isolate server. If the item not present, the callback will be include the PushState necessary to upload it. Otherwise, the PushState will be nil.

type ConcurrentUploader Uses

type ConcurrentUploader struct {
    // contains filtered or unexported fields
}

ConcurrentUploader uses an isolatedclient.Client to upload items to the server. All methods are safe for concurrent use.

func NewUploader Uses

func NewUploader(ctx context.Context, client *isolatedclient.Client, maxConcurrent int) *ConcurrentUploader

NewUploader creates a new ConcurrentUploader with the given isolated client. maxConcurrent controls maximum number of uploads to be in-flight at once. The provided context is used to make all requests to the isolate server.

func (*ConcurrentUploader) Close Uses

func (u *ConcurrentUploader) Close() error

Close waits for any pending uploads (and associated done callbacks) to complete, and returns the first encountered error if any. Uploader cannot be used once it is closed.

func (*ConcurrentUploader) Upload Uses

func (u *ConcurrentUploader) Upload(name string, src isolatedclient.Source, ps *isolatedclient.PushState, done func())

Upload uploads an item from an isolated.Source. Upload does not block. If not-nil, the done func will be invoked on upload completion (both success and failure).

func (*ConcurrentUploader) UploadBytes Uses

func (u *ConcurrentUploader) UploadBytes(name string, b []byte, ps *isolatedclient.PushState, done func())

UploadBytes uploads an item held in-memory. UploadBytes does not block. If not-nil, the done func will be invoked on upload completion (both success and failure). The provided byte slice b must not be modified until the upload is completed. TODO(djd): Consider using Upload directly and deleting UploadBytes.

func (*ConcurrentUploader) UploadFile Uses

func (u *ConcurrentUploader) UploadFile(item *Item, ps *isolatedclient.PushState, done func())

UploadFile uploads a file from disk. UploadFile does not block. If not-nil, the done func will be invoked on upload completion (both success and failure). TODO(djd): Consider using Upload directly and deleting UploadFile.

type CountBytes Uses

type CountBytes struct {
    // contains filtered or unexported fields
}

CountBytes aggregates a count of files and the number of bytes in them.

func (*CountBytes) Bytes Uses

func (cb *CountBytes) Bytes() int64

Bytes returns total byte count.

func (*CountBytes) Count Uses

func (cb *CountBytes) Count() int

Count returns the file count.

type IsolatedSummary Uses

type IsolatedSummary struct {
    // Name is the base name an isolated file with any extension stripped
    Name   string
    Digest isolated.HexDigest
}

IsolatedSummary contains an isolate name and its digest.

type Item Uses

type Item struct {
    Path    string
    RelPath string
    Size    int64
    Mode    os.FileMode

    Digest isolated.HexDigest
}

Item represents a file or symlink referenced by an isolate file.

type PendingItem Uses

type PendingItem struct {
    // Immutable.
    DisplayName string // Name to use to qualify this item
    // contains filtered or unexported fields
}

PendingItem is an item being processed.

It is caried over from pipeline stage to stage to do processing on it.

func PushDirectory Uses

func PushDirectory(a *Archiver, root string, relDir string, blacklist []string) *PendingItem

PushDirectory walks a directory at root and creates a .isolated file.

It walks the directories synchronously, then returns a *PendingItem to signal when the background work is completed. The PendingItem is signaled once all files are hashed. In particular, the *PendingItem is signaled before server side cache lookups and upload is completed. Use archiver.Close() to wait for completion.

relDir is a relative directory to offset relative paths against in the generated .isolated file.

blacklist is a list of globs of files to ignore.

func (*PendingItem) Digest Uses

func (i *PendingItem) Digest() isolated.HexDigest

Digest returns the calculated digest once calculated, empty otherwise.

func (*PendingItem) Error Uses

func (i *PendingItem) Error() error

Error returns any error that occurred for this item if any.

func (*PendingItem) SetErr Uses

func (i *PendingItem) SetErr(err error)

SetErr forcibly set an item as failed. Normally not used by callers.

func (*PendingItem) WaitForHashed Uses

func (i *PendingItem) WaitForHashed()

WaitForHashed hangs until the item hash is known.

type Stats Uses

type Stats struct {
    Hits   []units.Size  // Bytes; each item is immutable.
    Pushed []*UploadStat // Misses; each item is immutable.
}

Stats is statistics from the Archiver.

func (*Stats) TotalBytesHits Uses

func (s *Stats) TotalBytesHits() units.Size

TotalBytesHits is the number of bytes not uploaded due to cache hits on the server.

func (*Stats) TotalBytesPushed Uses

func (s *Stats) TotalBytesPushed() units.Size

TotalBytesPushed returns the sum of bytes uploaded.

func (*Stats) TotalHits Uses

func (s *Stats) TotalHits() int

TotalHits is the number of cache hits on the server.

func (*Stats) TotalMisses Uses

func (s *Stats) TotalMisses() int

TotalMisses returns the number of cache misses on the server.

type TarringArchiver Uses

type TarringArchiver struct {
    // contains filtered or unexported fields
}

TarringArchiver archives the files specified by an isolate file to the server, Small files are combining into tar archives before uploading.

func NewTarringArchiver Uses

func NewTarringArchiver(checker Checker, uploader Uploader) *TarringArchiver

NewTarringArchiver constructs a TarringArchiver.

func (*TarringArchiver) Archive Uses

func (ta *TarringArchiver) Archive(deps []string, rootDir string, isol *isolated.Isolated, blacklist []string, isolated string) (IsolatedSummary, error)

Archive uploads a single isolate.

type UploadStat Uses

type UploadStat struct {
    Duration time.Duration
    Size     units.Size
    Name     string
}

UploadStat is the statistic for a single upload.

type UploadTracker Uses

type UploadTracker struct {
    // contains filtered or unexported fields
}

UploadTracker uploads and keeps track of files.

func NewUploadTracker Uses

func NewUploadTracker(checker Checker, uploader Uploader, isol *isolated.Isolated) *UploadTracker

NewUploadTracker constructs an UploadTracker. It tracks uploaded files in isol.Files.

func (*UploadTracker) Files Uses

func (ut *UploadTracker) Files() map[string]isolated.File

Files returns the files which have been uploaded. Note: files may not have completed uploading until the tracker's Checker and Uploader have been closed.

func (*UploadTracker) Finalize Uses

func (ut *UploadTracker) Finalize(isolatedPath string) (IsolatedSummary, error)

Finalize creates and uploads the isolate JSON at the isolatePath, and closes the checker and uploader. It returns the isolate name and digest. Finalize should only be called after UploadDeps.

func (*UploadTracker) UploadDeps Uses

func (ut *UploadTracker) UploadDeps(parts partitionedDeps) error

UploadDeps uploads all of the items in parts.

type Uploader Uses

type Uploader interface {
    Close() error
    Upload(name string, src isolatedclient.Source, ps *isolatedclient.PushState, done func())
    UploadBytes(name string, b []byte, ps *isolatedclient.PushState, done func())
    UploadFile(item *Item, ps *isolatedclient.PushState, done func())
}

Uploader uploads items to the server. It has a single implementation, *ConcurrentUploader. See ConcurrentUploader for method documentation.

Package archiver imports 27 packages (graph) and is imported by 9 packages. Updated 2018-12-14. Refresh now. Tools for package owners.