storage

package
v0.52.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 19, 2021 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Overview

Package storage implements a simple storage abstraction.

This is meant to abstract filesystem calls, as well as be a wrapper for in-memory or remote storage. It also provides a smaller attack vector as implementations can do verifications as to what is accessed and what is not.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrClosed is the error returned if a bucket or object is already closed.
	ErrClosed = errors.New("already closed")
	// ErrSetExternalPathUnsupported is the error returned if a bucket does not support SetExternalPath.
	ErrSetExternalPathUnsupported = errors.New("setting the external path is unsupported for this bucket")
)

Functions

func AllPaths added in v0.19.0 

func AllPaths(ctx context.Context, readBucket ReadBucket, prefix string) ([]string, error)

AllPaths walks the bucket and gets all the paths.

func Copy added in v0.14.0 

func Copy(
	ctx context.Context,
	from ReadBucket,
	to WriteBucket,
	options ...CopyOption,
) (int, error)

Copy copies the bucket at from to the bucket at to.

Copies done concurrently. Returns the number of files copied.

func CopyReadObject added in v0.30.1 

func CopyReadObject(
	ctx context.Context,
	writeBucket WriteBucket,
	readObject ReadObject,
	options ...CopyOption,
) (retErr error)

CopyReadObject copies the contents of the ReadObject into the WriteBucket at the path.

func CopyReader added in v0.25.0 

func CopyReader(
	ctx context.Context,
	writeBucket WriteBucket,
	reader io.Reader,
	path string,
) (retErr error)

CopyReader copies the contents of the Reader into the WriteBucket at the path.

func Diff added in v0.19.0 

func Diff(
	ctx context.Context,
	writer io.Writer,
	one ReadBucket,
	two ReadBucket,
	options ...DiffOption,
) error

Diff writes a diff of the ReadBuckets to the Writer.

func DiffBytes added in v0.30.1 

func DiffBytes(
	ctx context.Context,
	one ReadBucket,
	two ReadBucket,
	options ...DiffOption,
) ([]byte, error)

DiffBytes does a diff of the ReadBuckets.

func Exists added in v0.14.0 

func Exists(ctx context.Context, readBucket ReadBucket, path string) (bool, error)

Exists returns true if the path exists, false otherwise.

Returns error on system error.

func IsEmpty added in v0.42.0 

func IsEmpty(ctx context.Context, readBucket ReadBucket, prefix string) (bool, error)

IsEmpty returns true if the bucket is empty under the prefix.

A prefix of "" or "." will check if the entire bucket is empty.

func IsExistsMultipleLocations added in v0.19.0 

func IsExistsMultipleLocations(err error) bool

IsExistsMultipleLocations returns true if the error is for a path existing in multiple locations.

func IsNotExist 

func IsNotExist(err error) bool

IsNotExist returns true for a error that is for a path not existing.

func NewErrExistsMultipleLocations added in v0.19.0 

func NewErrExistsMultipleLocations(path string, externalPaths ...string) error

NewErrExistsMultipleLocations returns a new error if a path exists in multiple locations.

func NewErrNotExist 

func NewErrNotExist(path string) error

NewErrNotExist returns a new error for a path not existing.

func PutPath added in v0.20.2 

func PutPath(ctx context.Context, writeBucket WriteBucket, path string, data []byte) (retErr error)

PutPath puts the data at the path.

func ReadPath added in v0.14.0 

func ReadPath(ctx context.Context, readBucket ReadBucket, path string) (_ []byte, retErr error)

ReadPath is analogous to os.ReadFile.

Returns an error that fufills IsNotExist if the path does not exist.

func WalkReadObjects added in v0.19.0 

func WalkReadObjects(
	ctx context.Context,
	readBucket ReadBucket,
	prefix string,
	f func(ReadObject) error,
) error

WalkReadObjects walks the bucket and calls get on each, closing the resulting ReadObjectCloser when done.

Types

type CopyOption added in v0.19.0 

type CopyOption func(*copyOptions)

CopyOption is an option for Copy.

func CopyWithExternalPaths added in v0.19.0 

func CopyWithExternalPaths() CopyOption

CopyWithExternalPaths returns a new CopyOption that says to copy external paths.

The to WriteBucket must support setting external paths.

type DiffOption added in v0.30.1 

type DiffOption func(*diffOptions)

DiffOption is an option for Diff.

func DiffWithExternalPathPrefixes added in v0.30.1 

func DiffWithExternalPathPrefixes(
	oneExternalPathPrefix string,
	twoExternalPathPrefix string,
) DiffOption

DiffWithExternalPathPrefixes returns a new DiffOption that sets the external path prefixes for the buckets.

If a file is in one bucket but not the other, it will be assumed that the file begins with the given prefix, and this prefix should be substituted for the other prefix.

For example, if diffing the directories "test/a" and "test/b", use "test/a/" and "test/b/", and a file that is in one with path "test/a/foo.txt" will be shown as not existing as "test/b/foo.txt" in two.

Note that the prefixes are directly concatenated, so "/" should be included generally.

This option has no effect if DiffWithExternalPaths is not set. This option is not required if the prefixes are equal.

func DiffWithExternalPaths added in v0.30.1 

func DiffWithExternalPaths() DiffOption

DiffWithExternalPaths returns a new DiffOption that prints diffs with external paths instead of paths.

func DiffWithSuppressCommands added in v0.30.1 

func DiffWithSuppressCommands() DiffOption

DiffWithSuppressCommands returns a new DiffOption that suppresses printing of commands.

func DiffWithSuppressTimestamps added in v0.30.1 

func DiffWithSuppressTimestamps() DiffOption

DiffWithSuppressCommands returns a new DiffOption that suppresses printing of timestamps.

type Mapper added in v0.19.0 

type Mapper interface {
	// Map maps the path to the full path.
	//
	// The path is expected to be normalized and validated.
	// The returned path is expected to be normalized and validated.
	// If the path cannot be mapped, this returns false.
	MapPath(path string) (string, bool)
	// Map maps the prefix to the full prefix.
	//
	// The path is expected to be normalized and validated.
	// The returned path is expected to be normalized and validated.
	// If the path cannot be mapped, this returns false.
	MapPrefix(prefix string) (string, bool)
	// UnmapFullPath maps the full path to the path.
	//
	// Returns false if the full path does not apply.
	// The path is expected to be normalized and validated.
	// The returned path is expected to be normalized and validated.
	UnmapFullPath(fullPath string) (string, bool, error)
	// contains filtered or unexported methods
}

Mapper is a path mapper.

This will cause a Bucket to operate as if the Mapper has all paths mapped.

func MapChain added in v0.19.0 

func MapChain(mappers ...Mapper) Mapper

MapChain chains the mappers.

If any mapper does not match, this stops checking Mappers and returns an empty path and false. This is as opposed to MatchAnd, that runs every Matcher and returns the path regardless.

If the Mappers are empty, a no-op Mapper is returned. If there is more than one Mapper, the Mappers are called in order for UnmapFullPath, with the order reversed for MapPath and MapPrefix.

That is, order these assuming you are starting with a full path and working to a path.

func MapOnPrefix added in v0.19.0 

func MapOnPrefix(prefix string) Mapper

MapOnPrefix returns a Mapper that will map the Bucket as if it was created on the given prefix.

The prefix is expected to be normalized and validated.

type Matcher added in v0.19.0 

type Matcher interface {
	Mapper
	// contains filtered or unexported methods
}

Matcher is a path matcher.

This will cause a Bucket to operate as if it only contains matching paths.

func MatchAnd added in v0.19.0 

func MatchAnd(matchers ...Matcher) Matcher

MatchAnd returns an And of the Matchers.

func MatchNot added in v0.19.0 

func MatchNot(matcher Matcher) Matcher

MatchNot returns an Not of the Matcher.

func MatchOr added in v0.19.0 

func MatchOr(matchers ...Matcher) Matcher

MatchOr returns an Or of the Matchers.

func MatchPathContained added in v0.19.0 

func MatchPathContained(containingDir string) Matcher

MatchPathContained returns a Matcher for the directory that matches on paths by contained by containingDir.

func MatchPathEqual added in v0.19.0 

func MatchPathEqual(equalPath string) Matcher

MatchPathEqual returns a Matcher for the path.

func MatchPathEqualOrContained added in v0.19.0 

func MatchPathEqualOrContained(equalOrContainingPath string) Matcher

MatchPathEqualOrContained returns a Matcher for the path that matches on paths equal or contained by equalOrContainingPath.

func MatchPathExt added in v0.19.0 

func MatchPathExt(ext string) Matcher

MatchPathExt returns a Matcher for the extension.

type ObjectInfo 

type ObjectInfo interface {
	// Path is the path of the object.
	//
	// This will always correspond to a path within the Bucket. For sub-buckets, this is the sub-path, but the
	// external path will include the sub-bucket path.
	//
	// This path will always be normalized, validated, and non-empty.
	Path() string
	// ExternalPath is the path that identifies the object externally.
	//
	// This path is not necessarily a file path, and should only be used to
	// uniquely identify this file as compared to other assets, to for display
	// to users.
	//
	// The path will be unnormalized, if it is a file path.
	// The path will never be empty. If a given implementation has no external path, this falls back to path.
	//
	// Example:
	//   Directory: /foo/bar
	//   Path: baz/bat.proto
	//   ExternalPath: /foo/bar/baz/bat.proto
	//
	// Example:
	//   Directory: .
	//   Path: baz/bat.proto
	//   ExternalPath: baz/bat.proto
	//
	// Example:
	//   S3 Bucket: https://s3.amazonaws.com/foo
	//   Path: baz/bat.proto
	//   ExternalPath: s3://foo/baz/bat.proto
	ExternalPath() string
}

ObjectInfo contains object info.

type ReadBucket 

type ReadBucket interface {
	// Get gets the path.
	//
	// Returns ErrNotExist if the path does not exist, other error
	// if there is a system error.
	Get(ctx context.Context, path string) (ReadObjectCloser, error)
	// Stat gets info in the object.
	//
	// Returns ErrNotExist if the path does not exist, other error
	// if there is a system error.
	Stat(ctx context.Context, path string) (ObjectInfo, error)
	// Walk walks the bucket with the prefix, calling f on each path.
	// If the prefix doesn't exist, this is a no-op.
	//
	// Note that foo/barbaz will not be called for foo/bar, but will
	// be called for foo/bar/baz.
	//
	// All paths given to f are normalized and validated.
	// If f returns error, Walk will stop short and return this error.
	// Returns other error on system error.
	Walk(ctx context.Context, prefix string, f func(ObjectInfo) error) error
}

ReadBucket is a simple read-only bucket.

All paths are regular files - Buckets do not handle directories. All paths must be relative. All paths are cleaned and ToSlash'ed by each function. Paths must not jump the bucket context, that is after clean, they cannot contain "..".

func MapReadBucket added in v0.21.0 

func MapReadBucket(readBucket ReadBucket, mappers ...Mapper) ReadBucket

MapReadBucket maps the ReadBucket.

If the Mappers are empty, the original ReadBucket is returned. If there is more than one Mapper, the Mappers are called in order for UnmapFullPath, with the order reversed for MapPath and MapPrefix.

That is, order these assuming you are starting with a full path and working to a path.

func MultiReadBucket added in v0.21.0 

func MultiReadBucket(readBuckets ...ReadBucket) ReadBucket

MultiReadBucket takes the union of the ReadBuckets.

If no readBuckets are given, this returns a no-op ReadBucket. If one readBucket is given, this returns the original ReadBucket. Otherwise, this returns a ReadBucket that will get from all buckets.

This expects and validates that no paths overlap between the ReadBuckets. This assumes that buckets are logically unique.

func MultiReadBucketSkipMultipleLocations added in v0.44.0 

func MultiReadBucketSkipMultipleLocations(readBuckets ...ReadBucket) ReadBucket

MultiReadBucketSkipMultipleLocations takes the union of the ReadBuckets.

If no readBuckets are given, this returns a no-op ReadBucket. If one readBucket is given, this returns the original ReadBucket. Otherwise, this returns a ReadBucket that will get from all buckets.

This allows overlap between the ReadBuckets. If there are overlapping paths, the first ReadBucket given with the path is used.

func NoExternalPathReadBucket added in v0.50.0 

func NoExternalPathReadBucket(readBucket ReadBucket) ReadBucket

NoExternalPathReadBucket disables the external paths for the ReadBucket.

This results in ExternalPath just calling Path.

type ReadBucketCloser added in v0.9.0 

type ReadBucketCloser interface {
	io.Closer
	ReadBucket
}

ReadBucketCloser is a read-only bucket that must be closed.

func NopReadBucketCloser added in v0.19.0 

func NopReadBucketCloser(readBucket ReadBucket) ReadBucketCloser

NopReadBucketCloser returns a ReadBucketCloser for the ReadBucket.

type ReadObject 

type ReadObject interface {
	ObjectInfo
	io.Reader
}

ReadObject is an object read from a bucket.

type ReadObjectCloser added in v0.19.0 

type ReadObjectCloser interface {
	ReadObject
	io.Closer
}

ReadObjectCloser is a ReadObject with a closer.

It must be closed when done.

type ReadWriteBucket added in v0.9.0 

type ReadWriteBucket interface {
	ReadBucket
	WriteBucket
}

ReadWriteBucket is a simple read/write bucket.

func MapReadWriteBucket added in v0.21.0 

func MapReadWriteBucket(readWriteBucket ReadWriteBucket, mappers ...Mapper) ReadWriteBucket

MapReadWriteBucket maps the ReadWriteBucket.

If the Mappers are empty, the original ReadWriteBucket is returned. If there is more than one Mapper, the Mappers are called in order for UnmapFullPath, with the order reversed for MapPath and MapPrefix.

That is, order these assuming you are starting with a full path and working to a path.

type ReadWriteBucketCloser added in v0.9.0 

type ReadWriteBucketCloser interface {
	io.Closer
	ReadWriteBucket
}

ReadWriteBucketCloser is a read/write bucket that must be closed.

func NopReadWriteBucketCloser added in v0.19.0 

func NopReadWriteBucketCloser(readWriteBucket ReadWriteBucket) ReadWriteBucketCloser

NopReadWriteBucketCloser returns a ReadWriteBucketCloser for the ReadWriteBucket.

type WriteBucket added in v0.19.0 

type WriteBucket interface {
	// Put returns a WriteObjectCloser to write to the path.
	//
	// The path is truncated on close.
	//
	// Returns error on system error.
	Put(ctx context.Context, path string) (WriteObjectCloser, error)
	// Delete deletes the object at the path.
	//
	// Returns ErrNotExist if the path does not exist, other error
	// if there is a system error.
	Delete(ctx context.Context, path string) error
	// DeleteAll deletes all objects with the prefix.
	// If the prefix doesn't exist, this is a no-op.
	//
	// Note that the prefix is used as a filepath prefix, and
	// NOT a string prefix. For example, the prefix "foo/bar"
	// will delete "foo/bar/baz", but NOT "foo/barbaz".
	DeleteAll(ctx context.Context, prefix string) error
	// SetExternalPathSupported returns true if SetExternalPath is supported.
	//
	// For example, in-memory buckets may choose to return true so that object sources
	// are preserved, but filesystem buckets may choose to return false as they have
	// their own external paths.
	SetExternalPathSupported() bool
}

WriteBucket is a write-only bucket.

func MapWriteBucket added in v0.21.0 

func MapWriteBucket(writeBucket WriteBucket, mappers ...Mapper) WriteBucket

MapWriteBucket maps the WriteBucket.

If the Mappers are empty, the original WriteBucket is returned. If there is more than one Mapper, the Mappers are called in order for UnmapFullPath, with the order reversed for MapPath and MapPrefix.

That is, order these assuming you are starting with a full path and working to a path.

If a path that does not match is called for Put, an error is returned.

type WriteBucketCloser added in v0.19.0 

type WriteBucketCloser interface {
	io.Closer
	WriteBucket
}

WriteBucketCloser is a write-only bucket that must be closed.

func NopWriteBucketCloser added in v0.19.0 

func NopWriteBucketCloser(writeBucket WriteBucket) WriteBucketCloser

NopWriteBucketCloser returns a WriteBucketCloser for the WriteBucket.

type WriteObject 

type WriteObject interface {
	io.Writer

	// ExternalPath attempts to explicitly set the external path for the new object.
	//
	// If SetExternalPathSupported returns false, this returns error.
	SetExternalPath(externalPath string) error
}

WriteObject object written to a bucket.

type WriteObjectCloser added in v0.19.0 

type WriteObjectCloser interface {
	WriteObject
	io.Closer
}

WriteObjectCloser is a WriteObject with a closer.

It must be closed when done.

Source Files

View all Source files

Directories

Path Synopsis
cmd
ddiff
Package main implements the ddiff command that diffs two directories.
 Package main implements the ddiff command that diffs two directories.
storage-go-data
Package storagearchive implements archive utilities.
 Package storagearchive implements archive utilities.
Package storagemem implements an in-memory storage Bucket.
 Package storagemem implements an in-memory storage Bucket.
Package storageos implements an os-backed storage Bucket.
 Package storageos implements an os-backed storage Bucket.
Package storagetesting implements testing utilities and integration tests for storage.
 Package storagetesting implements testing utilities and integration tests for storage.
Package storageutil provides helpers for storage implementations.
 Package storageutil provides helpers for storage implementations.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.