fsutil

package
v0.0.0-...-ba09d25 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 29, 2021 License: Apache-2.0, MIT Imports: 21 Imported by: 0

README

This package provides utilities for implementing virtual filesystem objects.

[TOC]

Page cache

CachingInodeOperations implements a page cache for files that cannot use the host page cache. Normally these are files that store their data in a remote filesystem. This also applies to files that are accessed on a platform that does not support directly memory mapping host file descriptors (e.g. the ptrace platform).

An CachingInodeOperations buffers regions of a single file into memory. It is owned by an fs.Inode, the in-memory representation of a file (all open file descriptors are backed by an fs.Inode). The fs.Inode provides operations for reading memory into an CachingInodeOperations, to represent the contents of the file in-memory, and for writing memory out, to relieve memory pressure on the kernel and to synchronize in-memory changes to filesystems.

An CachingInodeOperations enables readable and/or writable memory access to file content. Files can be mapped shared or private, see mmap(2). When a file is mapped shared, changes to the file via write(2) and truncate(2) are reflected in the shared memory region. Conversely, when the shared memory region is modified, changes to the file are visible via read(2). Multiple shared mappings of the same file are coherent with each other. This is consistent with Linux.

When a file is mapped private, updates to the mapped memory are not visible to other memory mappings. Updates to the mapped memory are also not reflected in the file content as seen by read(2). If the file is changed after a private mapping is created, for instance by write(2), the change to the file may or may not be reflected in the private mapping. This is consistent with Linux.

An CachingInodeOperations keeps track of ranges of memory that were modified (or "dirtied"). When the file is explicitly synced via fsync(2), only the dirty ranges are written out to the filesystem. Any error returned indicates a failure to write all dirty memory of an CachingInodeOperations to the filesystem. In this case the filesystem may be in an inconsistent state. The same operation can be performed on the shared memory itself using msync(2). If neither fsync(2) nor msync(2) is performed, then the dirty memory is written out in accordance with the CachingInodeOperations eviction strategy (see below) and there is no guarantee that memory will be written out successfully in full.

Memory allocation and eviction

An CachingInodeOperations implements the following allocation and eviction strategy:

  • Memory is allocated and brought up to date with the contents of a file when a region of mapped memory is accessed (or "faulted on").

  • Dirty memory is written out to filesystems when an fsync(2) or msync(2) operation is performed on a memory mapped file, for all memory mapped files when saved, and/or when there are no longer any memory mappings of a range of a file, see munmap(2). As the latter implies, in the absence of a panic or SIGKILL, dirty memory is written out for all memory mapped files when an application exits.

  • Memory is freed when there are no longer any memory mappings of a range of a file (e.g. when an application exits). This behavior is consistent with Linux for shared memory that has been locked via mlock(2).

Notably, memory is not allocated for read(2) or write(2) operations. This means that reads and writes to the file are only accelerated by an CachingInodeOperations if the file being read or written has been memory mapped and if the shared memory has been accessed at the region being read or written. This diverges from Linux which buffers memory into a page cache on read(2) proactively (i.e. readahead) and delays writing it out to filesystems on write(2) (i.e. writeback). The absence of these optimizations is not visible to applications beyond less than optimal performance when repeatedly reading and/or writing to same region of a file. See Future Work for plans to implement these optimizations.

Additionally, memory held by CachingInodeOperationss is currently unbounded in size. An CachingInodeOperations does not write out dirty memory and free it under system memory pressure. This can cause pathological memory usage.

When memory is written back, an CachingInodeOperations may write regions of shared memory that were never modified. This is due to the strategy of minimizing page faults (see below) and handling only a subset of memory write faults. In the absence of an application or sentry crash, it is guaranteed that if a region of shared memory was written to, it is written back to a filesystem.

Life of a shared memory mapping

A file is memory mapped via mmap(2). For example, if A is an address, an application may execute:

mmap(A, 0x1000, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);

This creates a shared mapping of fd that reflects 4k of the contents of fd starting at offset 0, accessible at address A. This in turn creates a virtual memory area region ("vma") which indicates that [A, A+0x1000) is now a valid address range for this application to access.

At this point, memory has not been allocated in the file's CachingInodeOperations. It is also the case that the address range [A, A+0x1000) has not been mapped on the host on behalf of the application. If the application then tries to modify 8 bytes of the shared memory:

char buffer[] = "aaaaaaaa";
memcpy(A, buffer, 8);

The host then sends a SIGSEGV to the sentry because the address range [A, A+8) is not mapped on the host. The SIGSEGV indicates that the memory was accessed writable. The sentry looks up the vma associated with [A, A+8), finds the file that was mapped and its CachingInodeOperations. It then calls CachingInodeOperations.Translate which allocates memory to back [A, A+8). It may choose to allocate more memory (i.e. do "readahead") to minimize subsequent faults.

Memory that is allocated comes from a host tmpfs file (see pgalloc.MemoryFile). The host tmpfs file memory is brought up to date with the contents of the mapped file on its filesystem. The region of the host tmpfs file that reflects the mapped file is then mapped into the host address space of the application so that subsequent memory accesses do not repeatedly generate a SIGSEGV.

The range that was allocated, including any extra memory allocation to minimize faults, is marked dirty due to the write fault. This overcounts dirty memory if the extra memory allocated is never modified.

To make the scenario more interesting, imagine that this application spawns another process and maps the same file in the exact same way:

mmap(A, 0x1000, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);

Imagine that this process then tries to modify the file again but with only 4 bytes:

char buffer[] = "bbbb";
memcpy(A, buffer, 4);

Since the first process has already mapped and accessed the same region of the file writable, CachingInodeOperations.Translate is called but returns the memory that has already been allocated rather than allocating new memory. The address range [A, A+0x1000) reflects the same cached view of the file as the first process sees. For example, reading 8 bytes from the file from either process via read(2) starting at offset 0 returns a consistent "bbbbaaaa".

When this process no longer needs the shared memory, it may do:

munmap(A, 0x1000);

At this point, the modified memory cached by the CachingInodeOperations is not written back to the file because it is still in use by the first process that mapped it. When the first process also does:

munmap(A, 0x1000);

Then the last memory mapping of the file at the range [0, 0x1000) is gone. The file's CachingInodeOperations then starts writing back memory marked dirty to the file on its filesystem. Once writing completes, regardless of whether it was successful, the CachingInodeOperations frees the memory cached at the range [0, 0x1000).

Subsequent read(2) or write(2) operations on the file go directly to the filesystem since there no longer exists memory for it in its CachingInodeOperations.

Future Work

Page cache

The sentry does not yet implement the readahead and writeback optimizations for read(2) and write(2) respectively. To do so, on read(2) and/or write(2) the sentry must ensure that memory is allocated in a page cache to read or write into. However, the sentry cannot boundlessly allocate memory. If it did, the host would eventually OOM-kill the sentry+application process. This means that the sentry must implement a page cache memory allocation strategy that is bounded by a global user or container imposed limit. When this limit is approached, the sentry must decide from which page cache memory should be freed so that it can allocate more memory. If it makes a poor decision, the sentry may end up freeing and re-allocating memory to back regions of files that are frequently used, nullifying the optimization (and in some cases causing worse performance due to the overhead of memory allocation and general management). This is a form of "cache thrashing".

In Linux, much research has been done to select and implement a lightweight but optimal page cache eviction algorithm. Linux makes use of hardware page bits to keep track of whether memory has been accessed. The sentry does not have direct access to hardware. Implementing a similarly lightweight and optimal page cache eviction algorithm will need to either introduce a kernel interface to obtain these page bits or find a suitable alternative proxy for access events.

In Linux, readahead happens by default but is not always ideal. For instance, for files that are not read sequentially, it would be more ideal to simply read from only those regions of the file rather than to optimistically cache some number of bytes ahead of the read (up to 2MB in Linux) if the bytes cached won't be accessed. Linux implements the fadvise64(2) system call for applications to specify that a range of a file will not be accessed sequentially. The advice bit FADV_RANDOM turns off the readahead optimization for the given range in the given file. However fadvise64 is rarely used by applications so Linux implements a readahead backoff strategy if reads are not sequential. To ensure that application performance is not degraded, the sentry must implement a similar backoff strategy.

Documentation

Overview

Package fsutil provides utilities for implementing fs.InodeOperations and fs.FileOperations:

- For embeddable utilities, see inode.go and file.go.

  • For fs.Inodes that require a page cache to be memory mapped, see inode_cache.go.

- For anon fs.Inodes, see anon.go.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func GenericConfigureMMap

func GenericConfigureMMap(file *fs.File, m memmap.Mappable, opts *memmap.MMapOpts) error

GenericConfigureMMap implements fs.FileOperations.ConfigureMMap for most filesystems that support memory mapping.

func SeekWithDirCursor

func SeekWithDirCursor(ctx context.Context, file *fs.File, whence fs.SeekWhence, offset int64, dirCursor *string) (int64, error)

SeekWithDirCursor is used to implement fs.FileOperations.Seek. If dirCursor is not nil and the seek was on a directory, the cursor will be updated.

Currently only seeking to 0 on a directory is supported.

FIXME(b/33075855): Lift directory seeking limitations.

func SyncDirty

func SyncDirty(ctx context.Context, mr memmap.MappableRange, cache *FileRangeSet, dirty *DirtySet, max uint64, mem memmap.File, writeAt func(ctx context.Context, srcs safemem.BlockSeq, offset uint64) (uint64, error)) error

SyncDirty passes pages in the range mr that are stored in cache and identified as dirty to writeAt, updating dirty to reflect successful writes. If writeAt returns a successful partial write, SyncDirty will call it repeatedly until all bytes have been written. max is the true size of the cached object; offsets beyond max will not be passed to writeAt, even if they are marked dirty.

func SyncDirtyAll

func SyncDirtyAll(ctx context.Context, cache *FileRangeSet, dirty *DirtySet, max uint64, mem memmap.File, writeAt func(ctx context.Context, srcs safemem.BlockSeq, offset uint64) (uint64, error)) error

SyncDirtyAll passes all pages stored in cache identified as dirty to writeAt, updating dirty to reflect successful writes. If writeAt returns a successful partial write, SyncDirtyAll will call it repeatedly until all bytes have been written. max is the true size of the cached object; offsets beyond max will not be passed to writeAt, even if they are marked dirty.

Types

type CachedFileObject

type CachedFileObject interface {
	// ReadToBlocksAt reads up to dsts.NumBytes() bytes from the file to dsts,
	// starting at offset, and returns the number of bytes read. ReadToBlocksAt
	// may return a partial read without an error.
	ReadToBlocksAt(ctx context.Context, dsts safemem.BlockSeq, offset uint64) (uint64, error)

	// WriteFromBlocksAt writes up to srcs.NumBytes() bytes from srcs to the
	// file, starting at offset, and returns the number of bytes written.
	// WriteFromBlocksAt may return a partial write without an error.
	WriteFromBlocksAt(ctx context.Context, srcs safemem.BlockSeq, offset uint64) (uint64, error)

	// SetMaskedAttributes sets the attributes in attr that are true in
	// mask on the backing file. If the mask contains only ATime or MTime
	// and the CachedFileObject has an FD to the file, then this operation
	// is a noop unless forceSetTimestamps is true. This avoids an extra
	// RPC to the gofer in the open-read/write-close case, when the
	// timestamps on the file will be updated by the host kernel for us.
	//
	// SetMaskedAttributes may be called at any point, regardless of whether
	// the file was opened.
	SetMaskedAttributes(ctx context.Context, mask fs.AttrMask, attr fs.UnstableAttr, forceSetTimestamps bool) error

	// Allocate allows the caller to reserve disk space for the inode.
	// It's equivalent to fallocate(2) with 'mode=0'.
	Allocate(ctx context.Context, offset int64, length int64) error

	// Sync instructs the remote filesystem to sync the file to stable storage.
	Sync(ctx context.Context) error

	// FD returns a host file descriptor. If it is possible for
	// CachingInodeOperations.AddMapping to have ever been called with writable
	// = true, the FD must have been opened O_RDWR; otherwise, it may have been
	// opened O_RDONLY or O_RDWR. (mmap unconditionally requires that mapped
	// files are readable.) If no host file descriptor is available, FD returns
	// a negative number.
	//
	// For any given CachedFileObject, if FD() ever succeeds (returns a
	// non-negative number), it must always succeed.
	//
	// FD is called iff the file has been memory mapped. This implies that
	// the file was opened (see fs.InodeOperations.GetFile).
	FD() int
}

CachedFileObject is a file that may require caching.

type CachingInodeOperations

type CachingInodeOperations struct {
	// contains filtered or unexported fields
}

CachingInodeOperations caches the metadata and content of a CachedFileObject. It implements a subset of InodeOperations. As a utility it can be used to implement the full set of InodeOperations. Generally it should not be embedded to avoid unexpected inherited behavior.

CachingInodeOperations implements Mappable for the CachedFileObject:

  • If CachedFileObject.FD returns a value >= 0 then the file descriptor will be memory mapped on the host.
  • Otherwise, the contents of CachedFileObject are buffered into memory managed by the CachingInodeOperations.

Implementations of FileOperations for a CachedFileObject must read and write through CachingInodeOperations using Read and Write respectively.

Implementations of InodeOperations.WriteOut must call Sync to write out in-memory modifications of data and metadata to the CachedFileObject.

+stateify savable

func NewCachingInodeOperations

func NewCachingInodeOperations(ctx context.Context, backingFile CachedFileObject, uattr fs.UnstableAttr, opts CachingInodeOperationsOptions) *CachingInodeOperations

NewCachingInodeOperations returns a new CachingInodeOperations backed by a CachedFileObject and its initial unstable attributes.

func (*CachingInodeOperations) AddMapping

func (c *CachingInodeOperations) AddMapping(ctx context.Context, ms memmap.MappingSpace, ar hostarch.AddrRange, offset uint64, writable bool) error

AddMapping implements memmap.Mappable.AddMapping.

func (*CachingInodeOperations) Allocate

func (c *CachingInodeOperations) Allocate(ctx context.Context, offset, length int64) error

Allocate implements fs.InodeOperations.Allocate.

func (*CachingInodeOperations) CopyMapping

func (c *CachingInodeOperations) CopyMapping(ctx context.Context, ms memmap.MappingSpace, srcAR, dstAR hostarch.AddrRange, offset uint64, writable bool) error

CopyMapping implements memmap.Mappable.CopyMapping.

func (c *CachingInodeOperations) DecLinks(ctx context.Context)

DecLinks decreases the link count and updates cached modification time.

func (*CachingInodeOperations) DecRef

func (c *CachingInodeOperations) DecRef(fr memmap.FileRange)

DecRef implements memmap.File.DecRef. This is used when we directly map an underlying host fd and CachingInodeOperations is used as the memmap.File during translation.

func (*CachingInodeOperations) Evict

Evict implements pgalloc.EvictableMemoryUser.Evict.

func (*CachingInodeOperations) FD

func (c *CachingInodeOperations) FD() int

FD implements memmap.File.FD. This is used when we directly map an underlying host fd and CachingInodeOperations is used as the memmap.File during translation.

func (c *CachingInodeOperations) IncLinks(ctx context.Context)

IncLinks increases the link count and updates cached modification time.

func (*CachingInodeOperations) IncRef

func (c *CachingInodeOperations) IncRef(fr memmap.FileRange)

IncRef implements memmap.File.IncRef. This is used when we directly map an underlying host fd and CachingInodeOperations is used as the memmap.File during translation.

func (*CachingInodeOperations) InvalidateUnsavable

func (c *CachingInodeOperations) InvalidateUnsavable(ctx context.Context) error

InvalidateUnsavable implements memmap.Mappable.InvalidateUnsavable.

func (*CachingInodeOperations) MapInternal

MapInternal implements memmap.File.MapInternal. This is used when we directly map an underlying host fd and CachingInodeOperations is used as the memmap.File during translation.

func (*CachingInodeOperations) NotifyChangeFD

func (c *CachingInodeOperations) NotifyChangeFD() error

NotifyChangeFD must be called after the file description represented by CachedFileObject.FD() changes.

func (*CachingInodeOperations) Read

func (c *CachingInodeOperations) Read(ctx context.Context, file *fs.File, dst usermem.IOSequence, offset int64) (int64, error)

Read reads from frames and otherwise directly from the backing file into dst starting at offset until dst is full, EOF is reached, or an error is encountered.

Read may partially fill dst and return a nil error.

func (*CachingInodeOperations) Release

func (c *CachingInodeOperations) Release()

Release implements fs.InodeOperations.Release.

func (*CachingInodeOperations) RemoveMapping

func (c *CachingInodeOperations) RemoveMapping(ctx context.Context, ms memmap.MappingSpace, ar hostarch.AddrRange, offset uint64, writable bool)

RemoveMapping implements memmap.Mappable.RemoveMapping.

func (*CachingInodeOperations) SetOwner

func (c *CachingInodeOperations) SetOwner(ctx context.Context, inode *fs.Inode, owner fs.FileOwner) error

SetOwner implements fs.InodeOperations.SetOwner.

func (*CachingInodeOperations) SetPermissions

func (c *CachingInodeOperations) SetPermissions(ctx context.Context, inode *fs.Inode, perms fs.FilePermissions) bool

SetPermissions implements fs.InodeOperations.SetPermissions.

func (*CachingInodeOperations) SetTimestamps

func (c *CachingInodeOperations) SetTimestamps(ctx context.Context, inode *fs.Inode, ts fs.TimeSpec) error

SetTimestamps implements fs.InodeOperations.SetTimestamps.

func (*CachingInodeOperations) TouchAccessTime

func (c *CachingInodeOperations) TouchAccessTime(ctx context.Context, inode *fs.Inode)

TouchAccessTime updates the cached access time in-place to the current time. It does not update status change time in-place. See mm/filemap.c:do_generic_file_read -> include/linux/h:file_accessed.

func (*CachingInodeOperations) TouchModificationAndStatusChangeTime

func (c *CachingInodeOperations) TouchModificationAndStatusChangeTime(ctx context.Context)

TouchModificationAndStatusChangeTime updates the cached modification and status change times in-place to the current time.

func (*CachingInodeOperations) TouchStatusChangeTime

func (c *CachingInodeOperations) TouchStatusChangeTime(ctx context.Context)

TouchStatusChangeTime updates the cached status change time in-place to the current time.

func (*CachingInodeOperations) Translate

func (c *CachingInodeOperations) Translate(ctx context.Context, required, optional memmap.MappableRange, at hostarch.AccessType) ([]memmap.Translation, error)

Translate implements memmap.Mappable.Translate.

func (*CachingInodeOperations) Truncate

func (c *CachingInodeOperations) Truncate(ctx context.Context, inode *fs.Inode, size int64) error

Truncate implements fs.InodeOperations.Truncate.

func (*CachingInodeOperations) UnstableAttr

func (c *CachingInodeOperations) UnstableAttr(ctx context.Context, inode *fs.Inode) (fs.UnstableAttr, error)

UnstableAttr implements fs.InodeOperations.UnstableAttr.

func (*CachingInodeOperations) UpdateUnstable

func (c *CachingInodeOperations) UpdateUnstable(attr fs.UnstableAttr)

UpdateUnstable updates the cached unstable attributes. Only non-dirty attributes are updated.

func (*CachingInodeOperations) Write

func (c *CachingInodeOperations) Write(ctx context.Context, src usermem.IOSequence, offset int64) (int64, error)

Write writes to frames and otherwise directly to the backing file from src starting at offset and until src is empty or an error is encountered.

If Write partially fills src, a non-nil error is returned.

func (*CachingInodeOperations) WriteDirtyPagesAndAttrs

func (c *CachingInodeOperations) WriteDirtyPagesAndAttrs(ctx context.Context, inode *fs.Inode) error

WriteDirtyPagesAndAttrs will write the dirty pages and attributes to the gofer without calling Fsync on the remote file.

func (*CachingInodeOperations) WriteOut

func (c *CachingInodeOperations) WriteOut(ctx context.Context, inode *fs.Inode) error

WriteOut implements fs.InodeOperations.WriteOut.

type CachingInodeOperationsOptions

type CachingInodeOperationsOptions struct {
	// If ForcePageCache is true, use the sentry page cache even if a host file
	// descriptor is available.
	ForcePageCache bool

	// If LimitHostFDTranslation is true, apply maxFillRange() constraints to
	// host file descriptor mappings returned by
	// CachingInodeOperations.Translate().
	LimitHostFDTranslation bool
}

CachingInodeOperationsOptions configures a CachingInodeOperations.

+stateify savable

type DirFileOperations

DirFileOperations implements most of fs.FileOperations for directories, except for Readdir and UnstableAttr which the embedding type must implement.

func (*DirFileOperations) Read

Read implements fs.FileOperations.Read

func (*DirFileOperations) Write

Write implements fs.FileOperations.Write.

type DirtyInfo

type DirtyInfo struct {
	// Keep is true if the represented offset is concurrently writable, such
	// that writing the data for that offset back to the source does not
	// guarantee that the offset is clean (since it may be concurrently
	// rewritten after the writeback).
	Keep bool
}

DirtyInfo is the value type of DirtySet, and represents information about a Mappable offset that is dirty (the cached data for that offset is newer than its source).

+stateify savable

type FileGenericSeek

type FileGenericSeek struct{}

FileGenericSeek implements fs.FileOperations.Seek for files that use a generic seek implementation.

func (FileGenericSeek) Seek

func (FileGenericSeek) Seek(ctx context.Context, file *fs.File, whence fs.SeekWhence, offset int64) (int64, error)

Seek implements fs.FileOperations.Seek.

type FileNoFsync

type FileNoFsync struct{}

FileNoFsync implements fs.FileOperations.Fsync for files that don't support syncing.

func (FileNoFsync) Fsync

Fsync implements fs.FileOperations.Fsync.

type FileNoIoctl

type FileNoIoctl struct{}

FileNoIoctl implements fs.FileOperations.Ioctl for files that don't implement the ioctl syscall.

func (FileNoIoctl) Ioctl

Ioctl implements fs.FileOperations.Ioctl.

type FileNoMMap

type FileNoMMap struct{}

FileNoMMap implements fs.FileOperations.Mappable for files that cannot be memory mapped.

func (FileNoMMap) ConfigureMMap

func (FileNoMMap) ConfigureMMap(context.Context, *fs.File, *memmap.MMapOpts) error

ConfigureMMap implements fs.FileOperations.ConfigureMMap.

type FileNoRead

type FileNoRead struct{}

FileNoRead implements fs.FileOperations.Read to return EINVAL.

func (FileNoRead) Read

Read implements fs.FileOperations.Read.

type FileNoSeek

type FileNoSeek struct{}

FileNoSeek implements fs.FileOperations.Seek to return EINVAL.

func (FileNoSeek) Seek

Seek implements fs.FileOperations.Seek.

type FileNoSplice

type FileNoSplice struct{}

FileNoSplice implements fs.FileOperations.ReadFrom and fs.FileOperations.WriteTo for files that don't support splice.

func (FileNoSplice) ReadFrom

ReadFrom implements fs.FileOperations.ReadFrom.

func (FileNoSplice) WriteTo

WriteTo implements fs.FileOperations.WriteTo.

type FileNoWrite

type FileNoWrite struct{}

FileNoWrite implements fs.FileOperations.Write to return EINVAL.

func (FileNoWrite) Write

Write implements fs.FileOperations.Write.

type FileNoopFlush

type FileNoopFlush struct{}

FileNoopFlush implements fs.FileOperations.Flush as a no-op.

func (FileNoopFlush) Flush

Flush implements fs.FileOperations.Flush.

type FileNoopFsync

type FileNoopFsync struct{}

FileNoopFsync implements fs.FileOperations.Fsync for files that don't need to synced.

func (FileNoopFsync) Fsync

Fsync implements fs.FileOperations.Fsync.

type FileNoopRead

type FileNoopRead struct{}

FileNoopRead implement fs.FileOperations.Read as a noop.

func (FileNoopRead) Read

Read implements fs.FileOperations.Read.

type FileNoopRelease

type FileNoopRelease struct{}

FileNoopRelease implements fs.FileOperations.Release for files that have no resources to release.

func (FileNoopRelease) Release

func (FileNoopRelease) Release(context.Context)

Release is a no-op.

type FileNoopWrite

type FileNoopWrite struct{}

FileNoopWrite implements fs.FileOperations.Write as a noop.

func (FileNoopWrite) Write

Write implements fs.FileOperations.Write.

type FileNotDirReaddir

type FileNotDirReaddir struct{}

FileNotDirReaddir implements fs.FileOperations.Readdir for non-directories.

func (FileNotDirReaddir) Readdir

Readdir implements fs.FileOperations.FileNotDirReaddir.

type FilePipeSeek

type FilePipeSeek struct{}

FilePipeSeek implements fs.FileOperations.Seek and can be used for files that behave like pipes (seeking is not supported).

func (FilePipeSeek) Seek

Seek implements fs.FileOperations.Seek.

type FileRangeSetFunctions

type FileRangeSetFunctions struct{}

FileRangeSetFunctions implements segment.Functions for FileRangeSet.

func (FileRangeSetFunctions) ClearValue

func (FileRangeSetFunctions) ClearValue(_ *uint64)

ClearValue implements segment.Functions.ClearValue.

func (FileRangeSetFunctions) MaxKey

func (FileRangeSetFunctions) MaxKey() uint64

MaxKey implements segment.Functions.MaxKey.

func (FileRangeSetFunctions) Merge

func (FileRangeSetFunctions) Merge(mr1 memmap.MappableRange, frstart1 uint64, _ memmap.MappableRange, frstart2 uint64) (uint64, bool)

Merge implements segment.Functions.Merge.

func (FileRangeSetFunctions) MinKey

func (FileRangeSetFunctions) MinKey() uint64

MinKey implements segment.Functions.MinKey.

func (FileRangeSetFunctions) Split

func (FileRangeSetFunctions) Split(mr memmap.MappableRange, frstart uint64, split uint64) (uint64, uint64)

Split implements segment.Functions.Split.

type FileStaticContentReader

type FileStaticContentReader struct {
	// contains filtered or unexported fields
}

FileStaticContentReader is a helper to implement fs.FileOperations.Read with static content.

+stateify savable

func NewFileStaticContentReader

func NewFileStaticContentReader(b []byte) FileStaticContentReader

NewFileStaticContentReader initializes a FileStaticContentReader with the given content.

func (*FileStaticContentReader) Read

func (scr *FileStaticContentReader) Read(ctx context.Context, _ *fs.File, dst usermem.IOSequence, offset int64) (int64, error)

Read implements fs.FileOperations.Read.

type FileUseInodeUnstableAttr

type FileUseInodeUnstableAttr struct{}

FileUseInodeUnstableAttr implements fs.FileOperations.UnstableAttr by calling InodeOperations.UnstableAttr.

func (FileUseInodeUnstableAttr) UnstableAttr

func (FileUseInodeUnstableAttr) UnstableAttr(ctx context.Context, file *fs.File) (fs.UnstableAttr, error)

UnstableAttr implements fs.FileOperations.UnstableAttr.

type FileZeroSeek

type FileZeroSeek struct{}

FileZeroSeek implements fs.FileOperations.Seek for files that maintain a constant zero-value offset and require a no-op Seek.

func (FileZeroSeek) Seek

Seek implements fs.FileOperations.Seek.

type FrameRefSetFunctions

type FrameRefSetFunctions struct{}

FrameRefSetFunctions implements segment.Functions for FrameRefSet.

func (FrameRefSetFunctions) ClearValue

func (FrameRefSetFunctions) ClearValue(val *uint64)

ClearValue implements segment.Functions.ClearValue.

func (FrameRefSetFunctions) MaxKey

func (FrameRefSetFunctions) MaxKey() uint64

MaxKey implements segment.Functions.MaxKey.

func (FrameRefSetFunctions) Merge

Merge implements segment.Functions.Merge.

func (FrameRefSetFunctions) MinKey

func (FrameRefSetFunctions) MinKey() uint64

MinKey implements segment.Functions.MinKey.

func (FrameRefSetFunctions) Split

Split implements segment.Functions.Split.

type HostFileMapper

type HostFileMapper struct {
	// contains filtered or unexported fields
}

HostFileMapper caches mappings of an arbitrary host file descriptor. It is used by implementations of memmap.Mappable that represent a host file descriptor.

+stateify savable

func NewHostFileMapper

func NewHostFileMapper() *HostFileMapper

NewHostFileMapper returns an initialized HostFileMapper allocated on the heap with no references or cached mappings.

func (*HostFileMapper) DecRefOn

func (f *HostFileMapper) DecRefOn(mr memmap.MappableRange)

DecRefOn decrements the reference count on all offsets in mr.

Preconditions: * mr.Length() != 0. * mr.Start and mr.End must be page-aligned.

func (*HostFileMapper) IncRefOn

func (f *HostFileMapper) IncRefOn(mr memmap.MappableRange)

IncRefOn increments the reference count on all offsets in mr.

Preconditions: * mr.Length() != 0. * mr.Start and mr.End must be page-aligned.

func (*HostFileMapper) Init

func (f *HostFileMapper) Init()

Init must be called on zero-value HostFileMappers before first use.

func (*HostFileMapper) IsInited

func (f *HostFileMapper) IsInited() bool

IsInited returns true if f.Init() has been called. This is used when restoring a checkpoint that contains a HostFileMapper that may or may not have been initialized.

func (*HostFileMapper) MapInternal

func (f *HostFileMapper) MapInternal(fr memmap.FileRange, fd int, write bool) (safemem.BlockSeq, error)

MapInternal returns a mapping of offsets in fr from fd. The returned safemem.BlockSeq is valid as long as at least one reference is held on all offsets in fr or until the next call to UnmapAll.

Preconditions: The caller must hold a reference on all offsets in fr.

func (*HostFileMapper) RegenerateMappings

func (f *HostFileMapper) RegenerateMappings(fd int) error

RegenerateMappings must be called when the file description mapped by f changes, to replace existing mappings of the previous file description.

func (*HostFileMapper) UnmapAll

func (f *HostFileMapper) UnmapAll()

UnmapAll unmaps all cached mappings. Callers are responsible for synchronization with mappings returned by previous calls to MapInternal.

type HostMappable

type HostMappable struct {
	// contains filtered or unexported fields
}

HostMappable implements memmap.Mappable and memmap.File over a CachedFileObject.

Lock order (compare the lock order model in mm/mm.go):

truncateMu ("fs locks")
  mu ("memmap.Mappable locks not taken by Translate")
    ("memmap.File locks")
	     backingFile ("CachedFileObject locks")

+stateify savable

func NewHostMappable

func NewHostMappable(backingFile CachedFileObject) *HostMappable

NewHostMappable creates a new mappable that maps directly to host FD.

func (*HostMappable) AddMapping

func (h *HostMappable) AddMapping(ctx context.Context, ms memmap.MappingSpace, ar hostarch.AddrRange, offset uint64, writable bool) error

AddMapping implements memmap.Mappable.AddMapping.

func (*HostMappable) Allocate

func (h *HostMappable) Allocate(ctx context.Context, offset int64, length int64) error

Allocate reserves space in the backing file.

func (*HostMappable) CopyMapping

func (h *HostMappable) CopyMapping(ctx context.Context, ms memmap.MappingSpace, srcAR, dstAR hostarch.AddrRange, offset uint64, writable bool) error

CopyMapping implements memmap.Mappable.CopyMapping.

func (*HostMappable) DecRef

func (h *HostMappable) DecRef(fr memmap.FileRange)

DecRef implements memmap.File.DecRef.

func (*HostMappable) FD

func (h *HostMappable) FD() int

FD implements memmap.File.FD.

func (*HostMappable) IncRef

func (h *HostMappable) IncRef(fr memmap.FileRange)

IncRef implements memmap.File.IncRef.

func (*HostMappable) InvalidateUnsavable

func (h *HostMappable) InvalidateUnsavable(_ context.Context) error

InvalidateUnsavable implements memmap.Mappable.InvalidateUnsavable.

func (*HostMappable) MapInternal

MapInternal implements memmap.File.MapInternal.

func (*HostMappable) NotifyChangeFD

func (h *HostMappable) NotifyChangeFD() error

NotifyChangeFD must be called after the file description represented by CachedFileObject.FD() changes.

func (*HostMappable) RemoveMapping

func (h *HostMappable) RemoveMapping(ctx context.Context, ms memmap.MappingSpace, ar hostarch.AddrRange, offset uint64, writable bool)

RemoveMapping implements memmap.Mappable.RemoveMapping.

func (*HostMappable) Translate

func (h *HostMappable) Translate(ctx context.Context, required, optional memmap.MappableRange, at hostarch.AccessType) ([]memmap.Translation, error)

Translate implements memmap.Mappable.Translate.

func (*HostMappable) Truncate

func (h *HostMappable) Truncate(ctx context.Context, newSize int64, uattr fs.UnstableAttr) error

Truncate truncates the file, invalidating any mapping that may have been removed after the size change.

Truncation and writes are synchronized to prevent races where writes make the file grow between truncation and invalidation below:

T1: Calls SetMaskedAttributes and stalls
T2: Appends to file causing it to grow
T2: Writes to mapped pages and COW happens
T1: Continues and wronly invalidates the page mapped in step above.

func (*HostMappable) Write

func (h *HostMappable) Write(ctx context.Context, src usermem.IOSequence, offset int64, uattr fs.UnstableAttr) (int64, error)

Write writes to the file backing this mappable.

type InodeDenyWriteChecker

type InodeDenyWriteChecker struct{}

InodeDenyWriteChecker implements fs.InodeOperations.Check which denies all write operations.

func (InodeDenyWriteChecker) Check

func (InodeDenyWriteChecker) Check(ctx context.Context, inode *fs.Inode, p fs.PermMask) bool

Check implements fs.InodeOperations.Check.

type InodeGenericChecker

type InodeGenericChecker struct{}

InodeGenericChecker implements fs.InodeOperations.Check with a generic implementation.

func (InodeGenericChecker) Check

func (InodeGenericChecker) Check(ctx context.Context, inode *fs.Inode, p fs.PermMask) bool

Check implements fs.InodeOperations.Check.

type InodeIsDirAllocate

type InodeIsDirAllocate struct{}

InodeIsDirAllocate implements fs.InodeOperations.Allocate for directories.

func (InodeIsDirAllocate) Allocate

func (InodeIsDirAllocate) Allocate(_ context.Context, _ *fs.Inode, _, _ int64) error

Allocate implements fs.InodeOperations.Allocate.

type InodeIsDirTruncate

type InodeIsDirTruncate struct{}

InodeIsDirTruncate implements fs.InodeOperations.Truncate for directories.

func (InodeIsDirTruncate) Truncate

Truncate implements fs.InodeOperations.Truncate.

type InodeNoExtendedAttributes

type InodeNoExtendedAttributes struct{}

InodeNoExtendedAttributes can be used by Inodes that do not support extended attributes.

func (InodeNoExtendedAttributes) GetXattr

GetXattr implements fs.InodeOperations.GetXattr.

func (InodeNoExtendedAttributes) ListXattr

func (InodeNoExtendedAttributes) ListXattr(context.Context, *fs.Inode, uint64) (map[string]struct{}, error)

ListXattr implements fs.InodeOperations.ListXattr.

func (InodeNoExtendedAttributes) RemoveXattr

RemoveXattr implements fs.InodeOperations.RemoveXattr.

func (InodeNoExtendedAttributes) SetXattr

SetXattr implements fs.InodeOperations.SetXattr.

type InodeNoStatFS

type InodeNoStatFS struct{}

InodeNoStatFS implement StatFS by retuning ENOSYS.

func (InodeNoStatFS) StatFS

StatFS implements fs.InodeOperations.StatFS.

type InodeNoopAllocate

type InodeNoopAllocate struct{}

InodeNoopAllocate implements fs.InodeOperations.Allocate as a noop.

func (InodeNoopAllocate) Allocate

func (InodeNoopAllocate) Allocate(_ context.Context, _ *fs.Inode, _, _ int64) error

Allocate implements fs.InodeOperations.Allocate.

type InodeNoopRelease

type InodeNoopRelease struct{}

InodeNoopRelease implements fs.InodeOperations.Release as a noop.

func (InodeNoopRelease) Release

func (InodeNoopRelease) Release(context.Context)

Release implements fs.InodeOperations.Release.

type InodeNoopTruncate

type InodeNoopTruncate struct{}

InodeNoopTruncate implements fs.InodeOperations.Truncate as a noop.

func (InodeNoopTruncate) Truncate

Truncate implements fs.InodeOperations.Truncate.

type InodeNoopWriteOut

type InodeNoopWriteOut struct{}

InodeNoopWriteOut is a no-op implementation of fs.InodeOperations.WriteOut.

func (InodeNoopWriteOut) WriteOut

WriteOut is a no-op.

type InodeNotAllocatable

type InodeNotAllocatable struct{}

InodeNotAllocatable can be used by Inodes that do not support Allocate().

func (InodeNotAllocatable) Allocate

func (InodeNotAllocatable) Allocate(_ context.Context, _ *fs.Inode, _, _ int64) error

Allocate implements fs.InodeOperations.Allocate.

type InodeNotDirectory

type InodeNotDirectory struct{}

InodeNotDirectory can be used by Inodes that are not directories.

func (InodeNotDirectory) Bind

Bind implements fs.InodeOperations.Bind.

func (InodeNotDirectory) Create

Create implements fs.InodeOperations.Create.

func (InodeNotDirectory) CreateDirectory

CreateDirectory implements fs.InodeOperations.CreateDirectory.

func (InodeNotDirectory) CreateFifo

CreateFifo implements fs.InodeOperations.CreateFifo.

func (InodeNotDirectory) CreateHardLink(context.Context, *fs.Inode, *fs.Inode, string) error

CreateHardLink implements fs.InodeOperations.CreateHardLink.

CreateLink implements fs.InodeOperations.CreateLink.

func (InodeNotDirectory) Lookup

Lookup implements fs.InodeOperations.Lookup.

func (InodeNotDirectory) Remove

Remove implements fs.InodeOperations.Remove.

func (InodeNotDirectory) RemoveDirectory

func (InodeNotDirectory) RemoveDirectory(context.Context, *fs.Inode, string) error

RemoveDirectory implements fs.InodeOperations.RemoveDirectory.

func (InodeNotDirectory) Rename

Rename implements fs.FileOperations.Rename.

type InodeNotMappable

type InodeNotMappable struct{}

InodeNotMappable returns a nil memmap.Mappable.

func (InodeNotMappable) Mappable

Mappable implements fs.InodeOperations.Mappable.

type InodeNotOpenable

type InodeNotOpenable struct{}

InodeNotOpenable can be used by Inodes that cannot be opened.

func (InodeNotOpenable) GetFile

GetFile implements fs.InodeOperations.GetFile.

type InodeNotRenameable

type InodeNotRenameable struct{}

InodeNotRenameable can be used by Inodes that cannot be truncated.

func (InodeNotRenameable) Rename

Rename implements fs.InodeOperations.Rename.

type InodeNotSocket

type InodeNotSocket struct{}

InodeNotSocket can be used by Inodes that are not sockets.

func (InodeNotSocket) BoundEndpoint

BoundEndpoint implements fs.InodeOperations.BoundEndpoint.

type InodeNotSymlink struct{}

InodeNotSymlink can be used by Inodes that are not symlinks.

Getlink implements fs.InodeOperations.Getlink.

Readlink implements fs.InodeOperations.Readlink.

type InodeNotTruncatable

type InodeNotTruncatable struct{}

InodeNotTruncatable can be used by Inodes that cannot be truncated.

func (InodeNotTruncatable) Truncate

Truncate implements fs.InodeOperations.Truncate.

type InodeNotVirtual

type InodeNotVirtual struct{}

InodeNotVirtual can be used by Inodes that are not virtual.

func (InodeNotVirtual) IsVirtual

func (InodeNotVirtual) IsVirtual() bool

IsVirtual implements fs.InodeOperations.IsVirtual.

type InodeSimpleAttributes

type InodeSimpleAttributes struct {
	// contains filtered or unexported fields
}

InodeSimpleAttributes implements methods for updating in-memory unstable attributes.

+stateify savable

func NewInodeSimpleAttributes

func NewInodeSimpleAttributes(ctx context.Context, owner fs.FileOwner, perms fs.FilePermissions, typ uint64) InodeSimpleAttributes

NewInodeSimpleAttributes returns a new InodeSimpleAttributes with the given owner and permissions, and all timestamps set to the current time.

func NewInodeSimpleAttributesWithUnstable

func NewInodeSimpleAttributesWithUnstable(uattr fs.UnstableAttr, typ uint64) InodeSimpleAttributes

NewInodeSimpleAttributesWithUnstable returns a new InodeSimpleAttributes with the given unstable attributes.

func (i *InodeSimpleAttributes) AddLink()

AddLink implements fs.InodeOperations.AddLink.

func (i *InodeSimpleAttributes) DropLink()

DropLink implements fs.InodeOperations.DropLink.

func (*InodeSimpleAttributes) NotifyAccess

func (i *InodeSimpleAttributes) NotifyAccess(ctx context.Context)

NotifyAccess updates the access time.

func (*InodeSimpleAttributes) NotifyModification

func (i *InodeSimpleAttributes) NotifyModification(ctx context.Context)

NotifyModification updates the modification time.

func (*InodeSimpleAttributes) NotifyModificationAndStatusChange

func (i *InodeSimpleAttributes) NotifyModificationAndStatusChange(ctx context.Context)

NotifyModificationAndStatusChange updates the modification and status change times.

func (*InodeSimpleAttributes) NotifyStatusChange

func (i *InodeSimpleAttributes) NotifyStatusChange(ctx context.Context)

NotifyStatusChange updates the status change time.

func (*InodeSimpleAttributes) SetOwner

func (i *InodeSimpleAttributes) SetOwner(ctx context.Context, _ *fs.Inode, owner fs.FileOwner) error

SetOwner implements fs.InodeOperations.SetOwner.

func (*InodeSimpleAttributes) SetPermissions

func (i *InodeSimpleAttributes) SetPermissions(ctx context.Context, _ *fs.Inode, p fs.FilePermissions) bool

SetPermissions implements fs.InodeOperations.SetPermissions.

func (*InodeSimpleAttributes) SetTimestamps

func (i *InodeSimpleAttributes) SetTimestamps(ctx context.Context, _ *fs.Inode, ts fs.TimeSpec) error

SetTimestamps implements fs.InodeOperations.SetTimestamps.

func (*InodeSimpleAttributes) StatFS

StatFS implements fs.InodeOperations.StatFS.

func (*InodeSimpleAttributes) UnstableAttr

func (i *InodeSimpleAttributes) UnstableAttr(ctx context.Context, _ *fs.Inode) (fs.UnstableAttr, error)

UnstableAttr implements fs.InodeOperations.UnstableAttr.

type InodeSimpleExtendedAttributes

type InodeSimpleExtendedAttributes struct {
	// contains filtered or unexported fields
}

InodeSimpleExtendedAttributes implements fs.InodeOperations.{Get,Set,List}Xattr.

+stateify savable

func (*InodeSimpleExtendedAttributes) GetXattr

GetXattr implements fs.InodeOperations.GetXattr.

func (*InodeSimpleExtendedAttributes) ListXattr

func (i *InodeSimpleExtendedAttributes) ListXattr(context.Context, *fs.Inode, uint64) (map[string]struct{}, error)

ListXattr implements fs.InodeOperations.ListXattr.

func (*InodeSimpleExtendedAttributes) RemoveXattr

func (i *InodeSimpleExtendedAttributes) RemoveXattr(_ context.Context, _ *fs.Inode, name string) error

RemoveXattr implements fs.InodeOperations.RemoveXattr.

func (*InodeSimpleExtendedAttributes) SetXattr

func (i *InodeSimpleExtendedAttributes) SetXattr(_ context.Context, _ *fs.Inode, name, value string, flags uint32) error

SetXattr implements fs.InodeOperations.SetXattr.

type InodeStaticFileGetter

type InodeStaticFileGetter struct {
	Contents []byte
}

InodeStaticFileGetter implements GetFile for a file with static contents.

+stateify savable

func (*InodeStaticFileGetter) GetFile

func (i *InodeStaticFileGetter) GetFile(ctx context.Context, dirent *fs.Dirent, flags fs.FileFlags) (*fs.File, error)

GetFile implements fs.InodeOperations.GetFile.

type InodeVirtual

type InodeVirtual struct{}

InodeVirtual can be used by Inodes that are virtual.

func (InodeVirtual) IsVirtual

func (InodeVirtual) IsVirtual() bool

IsVirtual implements fs.InodeOperations.IsVirtual.

type NoReadWriteFile

type NoReadWriteFile struct {
	waiter.AlwaysReady       `state:"nosave"`
	FileGenericSeek          `state:"nosave"`
	FileNoIoctl              `state:"nosave"`
	FileNoMMap               `state:"nosave"`
	FileNoopFsync            `state:"nosave"`
	FileNoopFlush            `state:"nosave"`
	FileNoopRelease          `state:"nosave"`
	FileNoRead               `state:"nosave"`
	FileNoWrite              `state:"nosave"`
	FileNotDirReaddir        `state:"nosave"`
	FileUseInodeUnstableAttr `state:"nosave"`
	FileNoSplice             `state:"nosave"`
}

NoReadWriteFile is a file that does not support reading or writing.

+stateify savable

type NoReadWriteFileInode

type NoReadWriteFileInode struct {
	InodeGenericChecker       `state:"nosave"`
	InodeNoExtendedAttributes `state:"nosave"`
	InodeNoopRelease          `state:"nosave"`
	InodeNoopWriteOut         `state:"nosave"`
	InodeNotAllocatable       `state:"nosave"`
	InodeNotDirectory         `state:"nosave"`
	InodeNotMappable          `state:"nosave"`
	InodeNotSocket            `state:"nosave"`
	InodeNotSymlink           `state:"nosave"`
	InodeNotTruncatable       `state:"nosave"`
	InodeNotVirtual           `state:"nosave"`

	InodeSimpleAttributes
}

NoReadWriteFileInode is an implementation of InodeOperations that supports opening files that are not readable or writeable.

+stateify savable

func NewNoReadWriteFileInode

func NewNoReadWriteFileInode(ctx context.Context, owner fs.FileOwner, perms fs.FilePermissions, typ uint64) *NoReadWriteFileInode

NewNoReadWriteFileInode returns a new NoReadWriteFileInode.

func (*NoReadWriteFileInode) GetFile

func (*NoReadWriteFileInode) GetFile(ctx context.Context, dirent *fs.Dirent, flags fs.FileFlags) (*fs.File, error)

GetFile implements fs.InodeOperations.GetFile.

type SimpleFileInode

type SimpleFileInode struct {
	InodeGenericChecker       `state:"nosave"`
	InodeNoExtendedAttributes `state:"nosave"`
	InodeNoopRelease          `state:"nosave"`
	InodeNoopWriteOut         `state:"nosave"`
	InodeNotAllocatable       `state:"nosave"`
	InodeNotDirectory         `state:"nosave"`
	InodeNotMappable          `state:"nosave"`
	InodeNotOpenable          `state:"nosave"`
	InodeNotSocket            `state:"nosave"`
	InodeNotSymlink           `state:"nosave"`
	InodeNotTruncatable       `state:"nosave"`
	InodeNotVirtual           `state:"nosave"`

	InodeSimpleAttributes
}

SimpleFileInode is a simple implementation of InodeOperations.

+stateify savable

func NewSimpleFileInode

func NewSimpleFileInode(ctx context.Context, owner fs.FileOwner, perms fs.FilePermissions, typ uint64) *SimpleFileInode

NewSimpleFileInode returns a new SimpleFileInode.

type StaticDirFileOperations

type StaticDirFileOperations struct {
	DirFileOperations        `state:"nosave"`
	FileUseInodeUnstableAttr `state:"nosave"`
	// contains filtered or unexported fields
}

StaticDirFileOperations implements fs.FileOperations for directories with static children.

+stateify savable

func NewStaticDirFileOperations

func NewStaticDirFileOperations(dentries *fs.SortedDentryMap) *StaticDirFileOperations

NewStaticDirFileOperations returns a new StaticDirFileOperations that will iterate the given denty map.

func (*StaticDirFileOperations) IterateDir

func (sdfo *StaticDirFileOperations) IterateDir(ctx context.Context, d *fs.Dirent, dirCtx *fs.DirCtx, offset int) (int, error)

IterateDir implements DirIterator.IterateDir.

func (*StaticDirFileOperations) Readdir

func (sdfo *StaticDirFileOperations) Readdir(ctx context.Context, file *fs.File, serializer fs.DentrySerializer) (int64, error)

Readdir implements fs.FileOperations.Readdir.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL