Documentation
¶
Index ¶
- func NewLazyDirectory(directoryOpener DirectoryOpener) filesystem.Directory
- type DirectoryOpener
- type FilePool
- func NewBlockDeviceBackedFilePool(blockDevice blockdevice.BlockDevice, sectorAllocator SectorAllocator, ...) FilePool
- func NewDirectoryBackedFilePool(directory filesystem.Directory) FilePool
- func NewFilePoolFromConfiguration(configuration *pb.FilePoolConfiguration) (FilePool, error)
- func NewMetricsFilePool(base FilePool) FilePool
- func NewQuotaEnforcingFilePool(base FilePool, maximumFileCount, maximumTotalSize int64) FilePool
- type SectorAllocator
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewLazyDirectory ¶
func NewLazyDirectory(directoryOpener DirectoryOpener) filesystem.Directory
NewLazyDirectory creates a directory handle that forwards all calls to a directory that is created on demand. The primary use case for this adapter is for the FUSE-based runner.
A runner process may get started before the worker is able to create its FUSE mount point. This would cause the runner to obtain a handle to the build directory underneath the FUSE mount, causing builds to fail due to missing input files.
Relatedly, if the worker would start before the runner, but end up crashing/restarting, the runner would still have a directory handle pointing to a stale FUSE mount.
This wrapper prevents these problems by ensuring that we never hold on to a file descriptor to the build directory.
Types ¶
type DirectoryOpener ¶
type DirectoryOpener func() (filesystem.DirectoryCloser, error)
DirectoryOpener is a callback that is used by LazyDirectory to open the underlying directory on demand.
type FilePool ¶
type FilePool interface {
NewFile() (filesystem.FileReadWriter, error)
}
FilePool is an allocator for temporary files. Files are created by calling NewFile(). They are automatically removed by calling Close().
File handles returned by NewFile() are not thread-safe. Additional locking needs to be done at higher levels to permit safe concurrent access.
var EmptyFilePool FilePool = emptyFilePool{}
EmptyFilePool is a FilePool that does not permit the creation of new files. It is used as the default FilePool for the root of the worker's FUSE file system to disallow the creation of files not bound to a specific build action.
var InMemoryFilePool FilePool = inMemoryFilePool{}
InMemoryFilePool is a FilePool that stores all data in memory.
func NewBlockDeviceBackedFilePool ¶
func NewBlockDeviceBackedFilePool(blockDevice blockdevice.BlockDevice, sectorAllocator SectorAllocator, sectorSizeBytes int) FilePool
NewBlockDeviceBackedFilePool creates a FilePool that stores all temporary file contents directly on a block device. Using a block device tends to be faster than using a directory on a file system, for the reason that no metadata (e.g., a directory hierarchy and inode attributes) needs to be stored.
func NewDirectoryBackedFilePool ¶
func NewDirectoryBackedFilePool(directory filesystem.Directory) FilePool
NewDirectoryBackedFilePool creates a FilePool that stores all data written to files into a single directory on disk. Files stored in the underlying directory are simply identified by an incrementing number.
As many files may exist at a given point in time, this implementation does not keep any backing files open. This would exhaust the worker's file descriptor table. Files are opened on demand.
TODO: Maybe use an eviction.Set to keep a small number of files open?
func NewFilePoolFromConfiguration ¶
func NewFilePoolFromConfiguration(configuration *pb.FilePoolConfiguration) (FilePool, error)
NewFilePoolFromConfiguration constructs a FilePool based on parameters provided in a configuration file.
func NewMetricsFilePool ¶
NewMetricsFilePool creates a decorator for FilePool that exposes Prometheus metrics on how many files are created and closed.
func NewQuotaEnforcingFilePool ¶
NewQuotaEnforcingFilePool creates a FilePool that enforces disk quotas. It limits how many files may be extracted from an underlying FilePool, while also limiting the total size of all files that are extracted. Space is reclaimed by either truncating files or closing them.
type SectorAllocator ¶
type SectorAllocator interface {
// Allocate a contiguous range of sectors.
//
// Under high utilization, it may not be possible to allocate
// all space contiguously. In that case, this function returns
// fewer sectors than requested. Repeated calls to this function
// are necessary to request the desired amount of space, albeit
// fragmented.
//
// Sector numbers handed out by this function start at one.
// Zero can be used by the user of this interface for special
// purposes (e.g., sparse files).
AllocateContiguous(maximum int) (uint32, int, error)
// Free a contiguous range of sectors. It is invalid to call
// this function with the first sector number being zero.
FreeContiguous(first uint32, count int)
// Free a potentially fragmented list of sectors. Elements with
// value zero are ignored.
FreeList(sectors []uint32)
}
SectorAllocator is used by BlockDeviceBackedFilePool to allocate space on the block device that is needed to store files.
func NewBitmapSectorAllocator ¶
func NewBitmapSectorAllocator(sectorCount uint32) SectorAllocator
NewBitmapSectorAllocator creates a SectorAllocator that stores information on which sectors are allocated in a bitmap. Sectors are allocated by sequentially scanning the bitmap, continuing where previous calls left off.
Due to its simplicity, this allocator would be prone to heavy fragmentation if used for general purpose file systems. For FilePool this is not a problem, because files have short lifetimes. Fragmentation disappears entirely when a worker goes idle, causing the FilePool to go empty.
Source Files
Directories