Documentation
¶
Overview ¶
Package archiver implements the pipeline to efficiently archive file sets to an isolated server as fast as possible.
Index ¶
- func PushDirectory(a *Archiver, root, relDir string) (error, map[*PendingItem]PushedDirectoryItem, map[string]string)
- type Archiver
- type BundlingChecker
- type Checker
- type CheckerCallback
- type ConcurrentUploader
- func (u *ConcurrentUploader) Close() error
- func (u *ConcurrentUploader) Upload(name string, src isolatedclient.Source, ps *isolatedclient.PushState, ...)
- func (u *ConcurrentUploader) UploadBytes(name string, b []byte, ps *isolatedclient.PushState, done func())
- func (u *ConcurrentUploader) UploadFile(item *Item, ps *isolatedclient.PushState, done func())
- type CountBytes
- type IsolatedSummary
- type Item
- type PendingItem
- type PushedDirectoryItem
- type Stats
- type TarringArchiver
- type TarringArgs
- type UploadStat
- type UploadTracker
- type Uploader
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func PushDirectory ¶
func PushDirectory(a *Archiver, root, relDir string) (error, map[*PendingItem]PushedDirectoryItem, map[string]string)
PushDirectory walks a directory at root and pushes the files in it. relDir is a relative directory to offset relative paths against the root directory. For symlink that points to another file under the same root (called *in-tree*), the symlink is preserved. Otherwise, the symlink is followed (*out-of-tree*), and is pushed as an ordinary file.
It does not return until all the pushed items are hashed, which is done asynchornously.
The returned files are in a map from its relative path to PushedDirectoryItem. The returned in-tree symlinks are in a map from its relative path to the relative path it points to.
Types ¶
type Archiver ¶
type Archiver struct {
// contains filtered or unexported fields
}
Archiver is an high level interface to an isolatedclient.Client.
Uses a 4 stages pipeline, each doing work concurrently:
- Deduplicating similar requests or known server hot cache hits.
- Hashing files.
- Batched cache hit lookups on the server.
- Uploading cache misses.
func New ¶
New returns a thread-safe Archiver instance.
If not nil, out will contain tty-oriented progress information.
ctx will be used for logging.
func (*Archiver) Close ¶
Close waits for all pending files to be done. If an error occurred during processing, it is returned.
func (*Archiver) Push ¶
func (a *Archiver) Push(displayName string, source isolatedclient.Source, priority int64) *PendingItem
Push schedules item upload to the isolate server. Smaller priority value means earlier processing.
type BundlingChecker ¶
type BundlingChecker struct {
Hit, Miss CountBytes
// contains filtered or unexported fields
}
BundlingChecker uses the isolatedclient.Client to check whether items are available on the server. BundlingChecker methods are safe to call concurrently.
func NewChecker ¶
func NewChecker(ctx context.Context, client *isolatedclient.Client, maxConcurrent int) *BundlingChecker
NewChecker creates a new Checker with the given isolated client. maxConcurrent controls maximum number of check requests to be in-flight at once. The provided context is used to make all requests to the isolate server.
func (*BundlingChecker) AddItem ¶
func (c *BundlingChecker) AddItem(item *Item, isolated bool, callback CheckerCallback)
AddItem adds the given item to the checker for testing, and invokes the provided callback asynchronously. The isolated param indicates whether the given item represents a JSON isolated manifest (as opposed to a regular file). In the case of an error, the callback may never be invoked.
func (*BundlingChecker) Close ¶
func (c *BundlingChecker) Close() error
Close shuts down the checker, blocking until all pending items have been checked with the server. Close returns the first error encountered during the checking process, if any. After Close has returned, Checker is guaranteed to no longer invoke any previously-provided callback.
func (*BundlingChecker) Hash ¶
func (c *BundlingChecker) Hash() crypto.Hash
Hash returns the hashing algorithm used by this checker.
func (*BundlingChecker) PresumeExists ¶
func (c *BundlingChecker) PresumeExists(item *Item)
PresumeExists causes the Checker to report that item exists on the server.
type Checker ¶
type Checker interface {
Close() error
AddItem(item *Item, isolated bool, callback CheckerCallback)
PresumeExists(item *Item)
Hash() crypto.Hash
}
A Checker checks whether items are available on the server. It has a single implementation, *BundlingChecker. See BundlingChecker for method documentation.
type CheckerCallback ¶
type CheckerCallback func(*Item, *isolatedclient.PushState)
CheckerCallback is the callback used by Checker to indicate whether a file is present on the isolate server. If the item not present, the callback will be include the PushState necessary to upload it. Otherwise, the PushState will be nil.
type ConcurrentUploader ¶
type ConcurrentUploader struct {
// contains filtered or unexported fields
}
ConcurrentUploader uses an isolatedclient.Client to upload items to the server. All methods are safe for concurrent use.
func NewUploader ¶
func NewUploader(ctx context.Context, client *isolatedclient.Client, maxConcurrent int) *ConcurrentUploader
NewUploader creates a new ConcurrentUploader with the given isolated client. maxConcurrent controls maximum number of uploads to be in-flight at once. The provided context is used to make all requests to the isolate server.
func (*ConcurrentUploader) Close ¶
func (u *ConcurrentUploader) Close() error
Close waits for any pending uploads (and associated done callbacks) to complete, and returns the first encountered error if any. Uploader cannot be used once it is closed.
func (*ConcurrentUploader) Upload ¶
func (u *ConcurrentUploader) Upload(name string, src isolatedclient.Source, ps *isolatedclient.PushState, done func())
Upload uploads an item from an isolated.Source. Upload does not block. If not-nil, the done func will be invoked on upload completion (both success and failure).
func (*ConcurrentUploader) UploadBytes ¶
func (u *ConcurrentUploader) UploadBytes(name string, b []byte, ps *isolatedclient.PushState, done func())
UploadBytes uploads an item held in-memory. UploadBytes does not block. If not-nil, the done func will be invoked on upload completion (both success and failure). The provided byte slice b must not be modified until the upload is completed. TODO(djd): Consider using Upload directly and deleting UploadBytes.
func (*ConcurrentUploader) UploadFile ¶
func (u *ConcurrentUploader) UploadFile(item *Item, ps *isolatedclient.PushState, done func())
UploadFile uploads a file from disk. UploadFile does not block. If not-nil, the done func will be invoked on upload completion (both success and failure). TODO(djd): Consider using Upload directly and deleting UploadFile.
type CountBytes ¶
type CountBytes struct {
// contains filtered or unexported fields
}
CountBytes aggregates a count of files and the number of bytes in them.
type IsolatedSummary ¶
type IsolatedSummary struct {
// Name is the base name an isolated file with any extension stripped
Name string
Digest isolated.HexDigest
}
IsolatedSummary contains an isolate name and its digest.
type Item ¶
type Item struct {
Path string
RelPath string
Size int64
Mode os.FileMode
Digest isolated.HexDigest
}
Item represents a file or symlink referenced by an isolate file.
type PendingItem ¶
type PendingItem struct {
// Immutable.
DisplayName string // Name to use to qualify this item
// contains filtered or unexported fields
}
PendingItem is an item being processed.
It is caried over from pipeline stage to stage to do processing on it.
func (*PendingItem) Digest ¶
func (i *PendingItem) Digest() isolated.HexDigest
Digest returns the calculated digest once calculated, empty otherwise.
func (*PendingItem) Error ¶
func (i *PendingItem) Error() error
Error returns any error that occurred for this item if any.
func (*PendingItem) SetErr ¶
func (i *PendingItem) SetErr(err error)
SetErr forcibly set an item as failed. Normally not used by callers.
func (*PendingItem) WaitForHashed ¶
func (i *PendingItem) WaitForHashed()
WaitForHashed hangs until the item hash is known.
type PushedDirectoryItem ¶
type PushedDirectoryItem struct {
// Absolute path to the item.
FullPath string
// Relativie path to the item within the directory.
RelPath string
// FileInfo of the item.
Info os.FileInfo
}
PushedDirectoryItem represents a file within the directory being pushed.
type Stats ¶
type Stats struct {
Hits []units.Size // Bytes; each item is immutable.
Pushed []*UploadStat // Misses; each item is immutable.
}
Stats is statistics from the Archiver.
func (*Stats) PackedHits ¶
PackedHits returns the size of hit items in packed format.
func (*Stats) PackedMisses ¶
PackedMisses returns size of missed items in packed format.
func (*Stats) TotalBytesHits ¶
TotalBytesHits is the number of bytes not uploaded due to cache hits on the server.
func (*Stats) TotalBytesPushed ¶
TotalBytesPushed returns the sum of bytes uploaded.
func (*Stats) TotalMisses ¶
TotalMisses returns the number of cache misses on the server.
type TarringArchiver ¶
type TarringArchiver struct {
// contains filtered or unexported fields
}
TarringArchiver archives the files specified by an isolate file to the server, Small files are combining into tar archives before uploading.
func NewTarringArchiver ¶
func NewTarringArchiver(checker Checker, uploader Uploader) *TarringArchiver
NewTarringArchiver constructs a TarringArchiver.
func (*TarringArchiver) Archive ¶
func (ta *TarringArchiver) Archive(args *TarringArgs) (IsolatedSummary, error)
Archive uploads a single isolate.
type TarringArgs ¶
type TarringArgs struct {
Deps []string
RootDir string
IgnoredPathRe string
Isolated string
Isol *isolated.Isolated
}
TarringArgs wraps all the args for TarringArchiver.Archive().
type UploadStat ¶
UploadStat is the statistic for a single upload.
type UploadTracker ¶
type UploadTracker struct {
// contains filtered or unexported fields
}
UploadTracker uploads and keeps track of files.
func (*UploadTracker) Files ¶
func (ut *UploadTracker) Files() map[string]isolated.File
Files returns the files which have been uploaded. Note: files may not have completed uploading until the tracker's Checker and Uploader have been closed.
func (*UploadTracker) Finalize ¶
func (ut *UploadTracker) Finalize(isolatedPath string) (IsolatedSummary, error)
Finalize creates and uploads the isolate JSON at the isolatePath, and closes the checker and uploader. It returns the isolate name and digest. Finalize should only be called after UploadDeps.
func (*UploadTracker) UploadDeps ¶
func (ut *UploadTracker) UploadDeps(parts partitionedDeps) error
UploadDeps uploads all of the items in parts.
type Uploader ¶
type Uploader interface {
Close() error
Upload(name string, src isolatedclient.Source, ps *isolatedclient.PushState, done func())
UploadBytes(name string, b []byte, ps *isolatedclient.PushState, done func())
UploadFile(item *Item, ps *isolatedclient.PushState, done func())
}
Uploader uploads items to the server. It has a single implementation, *ConcurrentUploader. See ConcurrentUploader for method documentation.
Source Files