Documentation
¶
Index ¶
- func NewCASDirectory(directoryContext CASDirectoryContext, digestFunction digest.Function, ...) (virtual.Directory, virtual.HandleResolver)
- func NewDigestParsingDirectory(digestFunction digest.Function, lookupFunc DigestLookupFunc) virtual.Directory
- func NewInstanceNameParsingDirectory(handleAllocation virtual.StatelessHandleAllocation, ...) virtual.Directory
- type BazelOutputServiceDirectory
- func (d *BazelOutputServiceDirectory) BatchStat(ctx context.Context, request *bazeloutputservice.BatchStatRequest) (*bazeloutputservice.BatchStatResponse, error)
- func (d *BazelOutputServiceDirectory) Clean(ctx context.Context, request *bazeloutputservice.CleanRequest) (*bazeloutputservice.CleanResponse, error)
- func (d *BazelOutputServiceDirectory) FinalizeArtifacts(ctx context.Context, request *bazeloutputservice.FinalizeArtifactsRequest) (*bazeloutputservice.FinalizeArtifactsResponse, error)
- func (d *BazelOutputServiceDirectory) FinalizeBuild(ctx context.Context, request *bazeloutputservice.FinalizeBuildRequest) (*bazeloutputservice.FinalizeBuildResponse, error)
- func (d *BazelOutputServiceDirectory) StageArtifacts(ctx context.Context, request *bazeloutputservice.StageArtifactsRequest) (*bazeloutputservice.StageArtifactsResponse, error)
- func (d *BazelOutputServiceDirectory) StartBuild(ctx context.Context, request *bazeloutputservice.StartBuildRequest) (*bazeloutputservice.StartBuildResponse, error)
- func (d *BazelOutputServiceDirectory) VirtualGetAttributes(ctx context.Context, requested virtual.AttributesMask, ...)
- func (d *BazelOutputServiceDirectory) VirtualLookup(ctx context.Context, name path.Component, requested virtual.AttributesMask, ...) (virtual.DirectoryChild, virtual.Status)
- func (d *BazelOutputServiceDirectory) VirtualOpenChild(ctx context.Context, name path.Component, shareAccess virtual.ShareMask, ...) (virtual.Leaf, virtual.AttributesMask, virtual.ChangeInfo, virtual.Status)
- func (d *BazelOutputServiceDirectory) VirtualReadDir(ctx context.Context, firstCookie uint64, requested virtual.AttributesMask, ...) virtual.Status
- type CASDirectoryContext
- type CASDirectoryFactory
- type CommandFileFactory
- type DigestLookupFunc
- type InstanceNameLookupFunc
- type OutputPath
- type OutputPathFactory
- func NewInMemoryOutputPathFactory(filePool filesystem.FilePool, symlinkFactory virtual.SymlinkFactory, ...) OutputPathFactory
- func NewLocalFileUploadingOutputPathFactory(base OutputPathFactory, contentAddressableStorage blobstore.BlobAccess, ...) OutputPathFactory
- func NewPersistentOutputPathFactory(base OutputPathFactory, store outputpathpersistency.Store, clock clock.Clock, ...) OutputPathFactory
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewCASDirectory ¶
func NewCASDirectory(directoryContext CASDirectoryContext, digestFunction digest.Function, handleAllocation virtual.ResolvableHandleAllocation, sizeBytes uint64) (virtual.Directory, virtual.HandleResolver)
NewCASDirectory creates an immutable directory that is backed by a Directory message stored in the Content Addressable Storage (CAS). In order to load the Directory message and to instantiate inodes for any of its children, calls are made into a CASDirectoryContext object.
TODO: Reimplement this on top of cas.DirectoryWalker.
func NewDigestParsingDirectory ¶
func NewDigestParsingDirectory(digestFunction digest.Function, lookupFunc DigestLookupFunc) virtual.Directory
NewDigestParsingDirectory creates a directory that can be exposed through FUSE that parses filenames as REv2 digests.
Though REv2 uses ByteStream paths that encode digests in the form of {hash}/{sizeBytes}, this type uses the format {hash}-{sizeBytes}. This is done to ensure every resulting file only uses a single inode, as opposed to two.
func NewInstanceNameParsingDirectory ¶
func NewInstanceNameParsingDirectory(handleAllocation virtual.StatelessHandleAllocation, lookupFuncs map[path.Component]InstanceNameLookupFunc) virtual.Directory
NewInstanceNameParsingDirectory creates a directory that can be exposed through FUSE that parses REv2 instance names.
Because instance names are path-like and can contain a variable number of pathname components (zero or more), the path must be terminated with one of the REv2 reserved keywords ("blobs", "uploads", etc.). For each of these reserved keywords, a separate callback can be provided to complete the parsing.
Types ¶
type BazelOutputServiceDirectory ¶
type BazelOutputServiceDirectory struct {
virtual.ReadOnlyDirectory
// contains filtered or unexported fields
}
BazelOutputServiceDirectory is FUSE directory that acts as the top-level directory for Bazel Output Service. The Bazel Output Service can be used by build clients to efficiently populate a directory with build outputs.
In addition to acting as a FUSE directory, this type also implements a gRPC server for the Bazel Output Service. This gRPC service can be used to start and finalize builds, but also to perform bulk creation and stat() operations.
This implementation of the Bazel Output Service is relatively simple:
- There is no persistency of build information across restarts.
- No snapshotting of completed builds takes place, meaning that only the results of the latest build of a given output base are exposed.
- Every output path is backed by an InMemoryPrepopulatedDirectory, meaning that memory usage may be high.
- No automatic garbage collection of old output paths is performed.
This implementation should eventually be extended to address the issues listed above.
func NewBazelOutputServiceDirectory ¶
func NewBazelOutputServiceDirectory(handleAllocator virtual.StatefulHandleAllocator, outputPathFactory OutputPathFactory, bareContentAddressableStorage, retryingContentAddressableStorage blobstore.BlobAccess, directoryFetcher re_cas.DirectoryFetcher, symlinkFactory virtual.SymlinkFactory, maximumTreeSizeBytes int64) *BazelOutputServiceDirectory
NewBazelOutputServiceDirectory creates a new instance of BazelOutputServiceDirectory.
func (*BazelOutputServiceDirectory) BatchStat ¶
func (d *BazelOutputServiceDirectory) BatchStat(ctx context.Context, request *bazeloutputservice.BatchStatRequest) (*bazeloutputservice.BatchStatResponse, error)
BatchStat can be called by a build client to obtain the status of files and directories.
Calling this method over gRPC may be far more efficient than obtaining this information through the FUSE file system, as batching significantly reduces the amount of context switching. It also prevents the computation of digests for files for which the digest is already known.
func (*BazelOutputServiceDirectory) Clean ¶
func (d *BazelOutputServiceDirectory) Clean(ctx context.Context, request *bazeloutputservice.CleanRequest) (*bazeloutputservice.CleanResponse, error)
Clean all build outputs associated with a single output base.
func (*BazelOutputServiceDirectory) FinalizeArtifacts ¶
func (d *BazelOutputServiceDirectory) FinalizeArtifacts(ctx context.Context, request *bazeloutputservice.FinalizeArtifactsRequest) (*bazeloutputservice.FinalizeArtifactsResponse, error)
FinalizeArtifacts can be called by a build client to indicate that files are no longer expected to be modified by the build client. If the file is modified after finalization, it may be reported through InitialOutputPathContents.
func (*BazelOutputServiceDirectory) FinalizeBuild ¶
func (d *BazelOutputServiceDirectory) FinalizeBuild(ctx context.Context, request *bazeloutputservice.FinalizeBuildRequest) (*bazeloutputservice.FinalizeBuildResponse, error)
FinalizeBuild can be called by a build client to indicate the current build has completed. This prevents successive StageArtifacts() and BatchStat() calls from being processed.
func (*BazelOutputServiceDirectory) StageArtifacts ¶
func (d *BazelOutputServiceDirectory) StageArtifacts(ctx context.Context, request *bazeloutputservice.StageArtifactsRequest) (*bazeloutputservice.StageArtifactsResponse, error)
StageArtifacts can be called by a build client to create files and directories.
Because files and directories are provided in the form of REv2 digests, this implementation is capable of creating files and directories whose contents get loaded from the Content Addressable Storage lazily.
func (*BazelOutputServiceDirectory) StartBuild ¶
func (d *BazelOutputServiceDirectory) StartBuild(ctx context.Context, request *bazeloutputservice.StartBuildRequest) (*bazeloutputservice.StartBuildResponse, error)
StartBuild is called by a build client to indicate that a new build in a given output base is starting.
func (*BazelOutputServiceDirectory) VirtualGetAttributes ¶
func (d *BazelOutputServiceDirectory) VirtualGetAttributes(ctx context.Context, requested virtual.AttributesMask, attributes *virtual.Attributes)
VirtualGetAttributes returns the attributes of the root directory of the Bazel Output Service.
func (*BazelOutputServiceDirectory) VirtualLookup ¶
func (d *BazelOutputServiceDirectory) VirtualLookup(ctx context.Context, name path.Component, requested virtual.AttributesMask, out *virtual.Attributes) (virtual.DirectoryChild, virtual.Status)
VirtualLookup can be used to look up the root directory of an output path for a given output base.
func (*BazelOutputServiceDirectory) VirtualOpenChild ¶
func (d *BazelOutputServiceDirectory) VirtualOpenChild(ctx context.Context, name path.Component, shareAccess virtual.ShareMask, createAttributes *virtual.Attributes, existingOptions *virtual.OpenExistingOptions, requested virtual.AttributesMask, openedFileAttributes *virtual.Attributes) (virtual.Leaf, virtual.AttributesMask, virtual.ChangeInfo, virtual.Status)
VirtualOpenChild can be used to open or create a file in the root directory of the Bazel Output Service. Because this directory never contains any files, this function is guaranteed to fail.
func (*BazelOutputServiceDirectory) VirtualReadDir ¶
func (d *BazelOutputServiceDirectory) VirtualReadDir(ctx context.Context, firstCookie uint64, requested virtual.AttributesMask, reporter virtual.DirectoryEntryReporter) virtual.Status
VirtualReadDir returns a list of all the output paths managed by this Bazel Output Service.
type CASDirectoryContext ¶
type CASDirectoryContext interface {
CASDirectoryFactory
virtual.CASFileFactory
GetDirectoryContents() (*remoteexecution.Directory, virtual.Status)
LogError(err error)
}
CASDirectoryContext contains all of the methods that the directory created by NewCASDirectory uses to obtain its contents and instantiate inodes for its children.
type CASDirectoryFactory ¶
CASDirectoryFactory is a factory type for directories whose contents correspond with an object stored in the Content Addressable Storage (CAS).
func NewDecomposedCASDirectoryFactory ¶
func NewDecomposedCASDirectoryFactory(ctx context.Context, casFileFactory re_vfs.CASFileFactory, directoryFetcher re_cas.DirectoryFetcher, handleAllocation re_vfs.StatelessHandleAllocation, errorLogger util.ErrorLogger) CASDirectoryFactory
NewDecomposedCASDirectoryFactory creates a CASDirectoryFactory that is capable of creating directories that are backed by individual REv2 Directory messages that are stored in a Content Addressable Storage (CAS).
func NewTreeCASDirectoryFactory ¶
func NewTreeCASDirectoryFactory(ctx context.Context, casFileFactory re_vfs.CASFileFactory, directoryFetcher re_cas.DirectoryFetcher, handleAllocation re_vfs.StatelessHandleAllocation, errorLogger util.ErrorLogger) CASDirectoryFactory
NewTreeCASDirectoryFactory creates a CASDirectoryFactory that is capable of creating directories that correspond to the root directory of REv2 Tree messages messages that are stored in a Content Addressable Storage (CAS).
type CommandFileFactory ¶
type CommandFileFactory interface {
LookupFile(blobDigest digest.Digest) (virtual.Leaf, virtual.Status)
}
CommandFileFactory is a factory type for virtual files that contain shell scripts that launch build actions according to the contents of an REv2 Command message. This makes it easy to run a build action locally.
func NewBlobAccessCommandFileFactory ¶
func NewBlobAccessCommandFileFactory(ctx context.Context, contentAddressableStorage blobstore.BlobAccess, maximumMessageSizeBytes int, errorLogger util.ErrorLogger) CommandFileFactory
NewBlobAccessCommandFileFactory creates a new CommandFileFactory that loads REv2 Command messages from the provided Content Addressable Storage, turning them into shell scripts capable of launching build actions.
func NewHandleAllocatingCommandFileFactory ¶
func NewHandleAllocatingCommandFileFactory(base CommandFileFactory, allocation virtual.StatelessHandleAllocation) CommandFileFactory
NewHandleAllocatingCommandFileFactory creates a decorator for CommandFileFactory that creates shell scripts for launching build actions that have a resolvable handle associated with them. This gives every shell script its own inode number.
type DigestLookupFunc ¶
DigestLookupFunc is called by directories created using NewDigestParsingDirectory to complete the parsing of a digest. The callback can yield a directory or file corresponding with the digest.
type InstanceNameLookupFunc ¶
type InstanceNameLookupFunc func(instanceName digest.InstanceName) virtual.Directory
InstanceNameLookupFunc is called by directories created using NewInstanceNameParsingDirectory to complete the parsing of an instance name. The callback can yield a directory corresponding to the instance name.
type OutputPath ¶
type OutputPath interface {
virtual.PrepopulatedDirectory
// FinalizeBuild() is called at the end of every build.
// Implementations of OutputPath may use this method to persist
// state.
FinalizeBuild(ctx context.Context, digestFunction digest.Function)
}
OutputPath is a directory where build results may be stored. Simple implementations may store these results in memory, while more complex ones use local or networked storage.
type OutputPathFactory ¶
type OutputPathFactory interface {
// StartInitialBuild() is called when a build is started that
// uses an output base ID that hasn't been observed before, or
// was cleaned previously.
StartInitialBuild(outputBaseID path.Component, casFileFactory virtual.CASFileFactory, digestFunction digest.Function, errorLogger util.ErrorLogger) OutputPath
// Clean() is called when the BazelOutputServiceDirectory
// service is instructed to clean an output path that is not yet
// managed by the BazelOutputServiceDirectory. This may occur if
// a 'bazel clean' is invoked right after startup. In that case
// the output path may not yet be attached, though we should
// clean up any persistent data associated with it.
//
// Output paths that have already been returned by
// StartInitialBuild() will be cleaned by
// BazelOutputServiceDirectory by calling
// PrepopulatedDirectory.RemoveAllChildren(true).
Clean(outputBaseID path.Component) error
}
OutputPathFactory is an interface that is invoked by BazelOutputServiceDirectory to manage individual directories where build results may be stored.
func NewInMemoryOutputPathFactory ¶
func NewInMemoryOutputPathFactory(filePool filesystem.FilePool, symlinkFactory virtual.SymlinkFactory, handleAllocator virtual.StatefulHandleAllocator, initialContentsSorter virtual.Sorter, clock clock.Clock) OutputPathFactory
NewInMemoryOutputPathFactory creates an OutputPathFactory that simply creates output paths that store all of their data in memory.
func NewLocalFileUploadingOutputPathFactory ¶
func NewLocalFileUploadingOutputPathFactory(base OutputPathFactory, contentAddressableStorage blobstore.BlobAccess, errorLogger util.ErrorLogger, concurrency *semaphore.Weighted) OutputPathFactory
NewLocalFileUploadingOutputPathFactory creates a decorator for OutputPathFactory that at the end of every build traverses the full output path and uploads any files into the Content Addressable Storage (CAS) that are not present remotely.
This decorator can be used to ensure that PersistentOutputPathFactory is capable of persisting and restoring all files in the output path, even if they were at no point uploaded by Bazel.
func NewPersistentOutputPathFactory ¶
func NewPersistentOutputPathFactory(base OutputPathFactory, store outputpathpersistency.Store, clock clock.Clock, errorLogger util.ErrorLogger, symlinkFactory virtual.SymlinkFactory) OutputPathFactory
NewPersistentOutputPathFactory creates a decorator for OutputPathFactory that persists the contents of an OutputPath to disk after every build. When an OutputPath is created, it will attempt to reload the state from disk.
Source Files
¶
- bazel_output_service_directory.go
- blob_access_command_file_factory.go
- cas_directory.go
- cas_directory_factory.go
- command_file_factory.go
- decomposed_cas_directory_factory.go
- digest_parsing_directory.go
- handle_allocating_command_file_factory.go
- in_memory_output_path_factory.go
- instance_name_parsing_directory.go
- local_file_uploading_output_path_factory.go
- non_iterable_directory.go
- output_path_factory.go
- persistent_output_path_factory.go
- tree_cas_directory_factory.go