buf: github.com/bufbuild/buf/internal/pkg/storage Index | Files | Directories

package storage

import "github.com/bufbuild/buf/internal/pkg/storage"

Package storage implements a simple storage abstraction.

This is meant to abstract filesystem calls, as well as be a wrapper for in-memory or remote storage. It also provides a smaller attack vector as implementations can do verifications as to what is accessed and what is not.

Index

Package Files

bucket.go copy.go diff.go errors.go map.go mapper.go matcher.go multi.go storage.go util.go

Variables

var (
    // ErrClosed is the error returned if a bucket or object is already closed.
    ErrClosed = errors.New("already closed")
    // ErrSetExternalPathUnsupported is the error returned if a bucket does not support SetExternalPath.
    ErrSetExternalPathUnsupported = errors.New("setting the external path is unsupported for this bucket")
)

func AllPaths Uses

func AllPaths(ctx context.Context, readBucket ReadBucket, prefix string) ([]string, error)

AllPaths walks the bucket and gets all the paths.

func Copy Uses

func Copy(
    ctx context.Context,
    from ReadBucket,
    to WriteBucket,
    options ...CopyOption,
) (int, error)

Copy copies the bucket at from to the bucket at to.

Copies done concurrently. Returns the number of files copied.

func CopyReader Uses

func CopyReader(
    ctx context.Context,
    writeBucket WriteBucket,
    reader io.Reader,
    path string,
) (retErr error)

CopyReader copies the contents of the Reader into the WriteBucket at the path.

func Diff Uses

func Diff(
    ctx context.Context,
    one ReadBucket,
    two ReadBucket,
    oneBucketName string,
    twoBucketName string,
) ([]byte, error)

Diff does a diff of the ReadBuckets.

func Exists Uses

func Exists(ctx context.Context, readBucket ReadBucket, path string) (bool, error)

Exists returns true if the path exists, false otherwise.

Returns error on system error.

func IsExistsMultipleLocations Uses

func IsExistsMultipleLocations(err error) bool

IsExistsMultipleLocations returns true if the error is for a path existing in multiple locations.

func IsNotExist Uses

func IsNotExist(err error) bool

IsNotExist returns true for a error that is for a path not existing.

func NewErrExistsMultipleLocations Uses

func NewErrExistsMultipleLocations(path string, externalPaths ...string) error

NewErrExistsMultipleLocations returns a new error if a path exists in multiple locations.

func NewErrNotExist Uses

func NewErrNotExist(path string) error

NewErrNotExist returns a new error for a path not existing.

func PutPath Uses

func PutPath(ctx context.Context, writeBucket WriteBucket, path string, data []byte) (retErr error)

PutPath puts the data at the path.

func ReadPath Uses

func ReadPath(ctx context.Context, readBucket ReadBucket, path string) (_ []byte, retErr error)

ReadPath is analogous to ioutil.ReadFile.

Returns an error that fufills IsNotExist if the path does not exist.

func WalkReadObjects Uses

func WalkReadObjects(
    ctx context.Context,
    readBucket ReadBucket,
    prefix string,
    f func(ReadObject) error,
) error

WalkReadObjects walks the bucket and calls get on each, closing the resulting ReadObjectCloser when done.

type CopyOption Uses

type CopyOption func(*copyOptions)

CopyOption is an option for Copy.

func CopyWithExternalPaths Uses

func CopyWithExternalPaths() CopyOption

CopyWithExternalPaths returns a new CopyOption that says to copy external paths.

The to WriteBucket must support setting external paths.

type Mapper Uses

type Mapper interface {
    // Map maps the path to the full path.
    //
    // The path is expected to be normalized and validated.
    // The returned path is expected to be normalized and validated.
    // If the path cannot be mapped, this returns false.
    MapPath(path string) (string, bool)
    // Map maps the prefix to the full prefix.
    //
    // The path is expected to be normalized and validated.
    // The returned path is expected to be normalized and validated.
    // If the path cannot be mapped, this returns false.
    MapPrefix(prefix string) (string, bool)
    // UnmapFullPath maps the full path to the path.
    //
    // Returns false if the full path does not apply.
    // The path is expected to be normalized and validated.
    // The returned path is expected to be normalized and validated.
    UnmapFullPath(fullPath string) (string, bool, error)
    // contains filtered or unexported methods
}

Mapper is a path mapper.

This will cause a Bucket to operate as if the Mapper has all paths mapped.

func MapChain Uses

func MapChain(mappers ...Mapper) Mapper

MapChain chains the mappers.

If any mapper does not match, this stops checking Mappers and returns an empty path and false. This is as opposed to MatchAnd, that runs every Matcher and returns the path regardless.

If the Mappers are empty, a no-op Mapper is returned. If there is more than one Mapper, the Mappers are called in order for UnmapFullPath, with the order reversed for MapPath and MapPrefix.

That is, order these assuming you are starting with a full path and working to a path.

func MapOnPrefix Uses

func MapOnPrefix(prefix string) Mapper

MapOnPrefix returns a Mapper that will map the Bucket as if it was created on the given prefix.

The prefix is expected to be normalized and validated.

type Matcher Uses

type Matcher interface {
    Mapper
    // contains filtered or unexported methods
}

Matcher is a path matcher.

This will cause a Bucket to operate as if it only contains matching paths.

func MatchAnd Uses

func MatchAnd(matchers ...Matcher) Matcher

MatchAnd returns an And of the Matchers.

func MatchNot Uses

func MatchNot(matcher Matcher) Matcher

MatchNot returns an Not of the Matcher.

func MatchOr Uses

func MatchOr(matchers ...Matcher) Matcher

MatchOr returns an Or of the Matchers.

func MatchPathContained Uses

func MatchPathContained(containingDir string) Matcher

MatchPathContained returns a Matcher for the directory that matches on paths by contained by containingDir.

func MatchPathEqual Uses

func MatchPathEqual(equalPath string) Matcher

MatchPathEqual returns a Matcher for the path.

func MatchPathEqualOrContained Uses

func MatchPathEqualOrContained(equalOrContainingPath string) Matcher

MatchPathEqualOrContained returns a Matcher for the path that matches on paths equal or contained by equalOrContainingPath.

func MatchPathExt Uses

func MatchPathExt(ext string) Matcher

MatchPathExt returns a Matcher for the extension.

type ObjectInfo Uses

type ObjectInfo interface {
    // Size is the size of the object.
    Size() uint32
    // Path is the path of the object.
    //
    // This will always correspond to a path within the Bucket. For sub-buckets, this is the sub-path, but the
    // external path will include the sub-bucket path.
    //
    // This path will always be normalized, validated, and non-empty.
    Path() string
    // ExternalPath is the path that identifies the object externally.
    //
    // This path is not necessarily a file path, and should only be used to
    // uniquely identify this file as compared to other assets, to for display
    // to users.
    //
    // The path will be unnormalized, if it is a file path.
    // The path will never be empty. If a given implementation has no external path, this falls back to path.
    //
    // Example:
    //   Directory: /foo/bar
    //   Path: baz/bat.proto
    //   ExternalPath: /foo/bar/baz/bat.proto
    //
    // Example:
    //   Directory: .
    //   Path: baz/bat.proto
    //   ExternalPath: baz/bat.proto
    //
    // Example:
    //   S3 Bucket: https://s3.amazonaws.com/foo
    //   Path: baz/bat.proto
    //   ExternalPath: s3://foo/baz/bat.proto
    ExternalPath() string
}

ObjectInfo contains object info.

type ReadBucket Uses

type ReadBucket interface {
    // Get gets the path.
    //
    // Returns ErrNotExist if the path does not exist, other error
    // if there is a system error.
    Get(ctx context.Context, path string) (ReadObjectCloser, error)
    // Stat gets info in the object.
    //
    // Returns ErrNotExist if the path does not exist, other error
    // if there is a system error.
    Stat(ctx context.Context, path string) (ObjectInfo, error)
    // Walk walks the bucket with the prefix, calling f on each path.
    // If the prefix doesn't exist, this is a no-op.
    //
    // Note that foo/barbaz will not be called for foo/bar, but will
    // be called for foo/bar/baz.
    //
    // All paths given to f are normalized and validated.
    // If f returns error, Walk will stop short and return this error.
    // Returns other error on system error.
    Walk(ctx context.Context, prefix string, f func(ObjectInfo) error) error
}

ReadBucket is a simple read-only bucket.

All paths are regular files - Buckets do not handle directories. All paths must be relative. All paths are cleaned and ToSlash'ed by each function. Paths must not jump the bucket context, that is after clean, they cannot contain "..".

func MapReadBucket Uses

func MapReadBucket(readBucket ReadBucket, mappers ...Mapper) ReadBucket

MapReadBucket maps the ReadBucket.

If the Mappers are empty, the original ReadBucket is returned. If there is more than one Mapper, the Mappers are called in order for UnmapFullPath, with the order reversed for MapPath and MapPrefix.

That is, order these assuming you are starting with a full path and working to a path.

func MultiReadBucket Uses

func MultiReadBucket(readBuckets ...ReadBucket) ReadBucket

MultiReadBucket takes the union of the ReadBuckets.

If no readBuckets are given, this returns a no-op ReadBucket. If one readBucket is given, this returns the original ReadBucket. Otherwise, this returns a ReadBucket that will get from all buckets.

This expects and validates that no paths overlap between the ReadBuckets. This assumes that buckets are logically unique.

type ReadBucketCloser Uses

type ReadBucketCloser interface {
    io.Closer
    ReadBucket
}

ReadBucketCloser is a read-only bucket that must be closed.

func NopReadBucketCloser Uses

func NopReadBucketCloser(readBucket ReadBucket) ReadBucketCloser

NopReadBucketCloser returns a ReadBucketCloser for the ReadBucket.

type ReadObject Uses

type ReadObject interface {
    ObjectInfo
    io.Reader
}

ReadObject is an object read from a bucket.

type ReadObjectCloser Uses

type ReadObjectCloser interface {
    ReadObject
    io.Closer
}

ReadObjectCloser is a ReadObject with a closer.

It must be closed when done.

type ReadWriteBucket Uses

type ReadWriteBucket interface {
    ReadBucket
    WriteBucket
}

ReadWriteBucket is a simple read/write bucket.

func MapReadWriteBucket Uses

func MapReadWriteBucket(readWriteBucket ReadWriteBucket, mappers ...Mapper) ReadWriteBucket

MapReadWriteBucket maps the ReadWriteBucket.

If the Mappers are empty, the original ReadWriteBucket is returned. If there is more than one Mapper, the Mappers are called in order for UnmapFullPath, with the order reversed for MapPath and MapPrefix.

That is, order these assuming you are starting with a full path and working to a path.

type ReadWriteBucketCloser Uses

type ReadWriteBucketCloser interface {
    io.Closer
    ReadWriteBucket
}

ReadWriteBucketCloser is a read/write bucket that must be closed.

func NopReadWriteBucketCloser Uses

func NopReadWriteBucketCloser(readWriteBucket ReadWriteBucket) ReadWriteBucketCloser

NopReadWriteBucketCloser returns a ReadWriteBucketCloser for the ReadWriteBucket.

type WriteBucket Uses

type WriteBucket interface {
    // Put returns a WriteObjectCloser to write to the path.
    //
    // The path is truncated on close.
    //
    // Returns error on system error.
    Put(ctx context.Context, path string) (WriteObjectCloser, error)
    // Delete deletes the object at the path.
    //
    // Returns ErrNotExist if the path does not exist, other error
    // if there is a system error.
    Delete(ctx context.Context, path string) error
    // SetExternalPathSupported returns true if SetExternalPath is supported.
    //
    // For example, in-memory buckets may choose to return true so that object sources
    // are preserved, but filesystem buckets may choose to return false as they have
    // their own external paths.
    SetExternalPathSupported() bool
}

WriteBucket is a write-only bucket.

func MapWriteBucket Uses

func MapWriteBucket(writeBucket WriteBucket, mappers ...Mapper) WriteBucket

MapWriteBucket maps the WriteBucket.

If the Mappers are empty, the original WriteBucket is returned. If there is more than one Mapper, the Mappers are called in order for UnmapFullPath, with the order reversed for MapPath and MapPrefix.

That is, order these assuming you are starting with a full path and working to a path.

If a path that does not match is called for Put, an error is returned.

type WriteBucketCloser Uses

type WriteBucketCloser interface {
    io.Closer
    WriteBucket
}

WriteBucketCloser is a write-only bucket that must be closed.

func NopWriteBucketCloser Uses

func NopWriteBucketCloser(writeBucket WriteBucket) WriteBucketCloser

NopWriteBucketCloser returns a WriteBucketCloser for the WriteBucket.

type WriteObject Uses

type WriteObject interface {
    io.Writer

    // ExternalPath attempts to explicitly set the external path for the new object.
    //
    // If SetExternalPathSupported returns false, this returns error.
    SetExternalPath(externalPath string) error
}

WriteObject object written to a bucket.

type WriteObjectCloser Uses

type WriteObjectCloser interface {
    WriteObject
    io.Closer
}

WriteObjectCloser is a WriteObject with a closer.

It must be closed when done.

Directories

PathSynopsis
internal/storagetestingPackage storagetesting implements testing utilities and integration tests for storage.
storagearchivePackage storagearchive implements archive utilities.
storagememPackage storagemem implements an in-memory storage Bucket.
storageosPackage storageos implements an os-backed storage Bucket.
storagetestingPackage storagetesting implements testing utilities and integration tests for storage.
storageutilPackage storageutil provides helpers for storage implementations.

Package storage imports 14 packages (graph) and is imported by 19 packages. Updated 2020-10-24. Refresh now. Tools for package owners.