vfs

package module
v0.0.14 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 27, 2019 License: Apache-2.0 Imports: 16 Imported by: 1

README

vfs Travis-CI Go Report Card GoDoc Sourcegraph Coverage

Another virtual filesystem abstraction for go.

Actually I've used and written already quite a few abstraction layers in a variety of languages and because each one get's better than the predecessor and Go lacks a vfs which fulfills our needs, a new one has to see the light of the day.

Design goals of our vfs implementation are a clear and well designed API which is not only easy to use but also to implement. At the other hand a simple API usually comes with a compromise in either expressiveness or usability and our VFS makes no exemptions: as usual our API is based on the experiences we made and is therefore highly opinionated, which is something your hear everywhere in the go ecosystem. Therefore we only optimized the use cases we had in mind and not every possible scenario. So if you require an API change, please don't be disappointed if we may reject it, even if it is a perfect solution in a very specific scenario.

Status

This library is still in alpha and it's API has not been stabilized yet. As soon as this happens, there will be no incompatible structrual API changes anymore. However the CTS profiles will be updated and refined over time.

Available implementations

FilesystemDataProvider

import github.com/worldiety/vfs

CTS Check Result
Empty
Write any
Read any
Write and Read
Rename
Attributes
Close

Documentation

Overview

Package vfs provides the API specification and basic tools for virtual filesystems.

Index

Constants

View Source
const (
	// Introduced for completeness. A non-nil VFSError is not allowed to return this. Details is always nil.
	EOK = 0

	// Operation not permitted
	//
	// Caution, definition changed from posix meaning: This error differs from EACCES (permission denied) in the way
	// that the the entire vfs is not accessible and needs to be reauthenticated. A typical situation is that
	// the realm of a user has changed, e.g. by revoking an access token.
	//
	// The details are implementation specific.
	EPERM = 1

	// No such file or directory / resource not found
	//
	// A resource like a folder or file may get removed at any time. Certain non-cached operations will therefore fail.
	//
	// The details contain a []string array which includes all affected path Entries.
	ENOENT = 2

	// No such process
	//
	// A useful indicator if an implementation needs a process as counterpart because of IPC communication.
	//
	// The details are implementation specific.
	ESRCH = 3

	// Interrupted system call
	//
	// Used to indicate that an operation has been cancelled.
	//
	// The details are implementation specific.
	EINTR = 4

	// I/O error
	//
	// Channels always have errors like timeouts, pulled ethernet cables, ssl failures etc. Usually these errors are related
	// to OSI-layer 1-4 but may also reach up to layer 5-6 (e.g. https and ssl failures). Note that protocols do not always fit unambiguously into the OSI model.
	// Interrupts are mapped to EINTR and timeouts ETIMEDOUT.
	//
	// The details are implementation specific.
	EIO = 5

	// No such device or address
	//
	// Usually indicates a configuration problem or a physical problem.
	//
	// The details are implementation specific.
	ENXIO = 6

	// Bad file number or descriptor
	//
	// Indicates that a resource has been closed but accessed or that an open mode (e.g. read) does not match an
	// operation (e.g. write)
	//
	// The details contain a []string array which includes all affected path Entries.
	EBADF = 9

	// No child processes
	//
	// If an implementation spawns new process and a child process manipulation fails.
	//
	// The details are implementation specific.
	ECHILD = 10

	// Try again
	//
	// The data service behind the channel (which is working) is temporarily unavailable.
	// You may want to retry it later again. Usually this may be used by servers which are currently in maintenance mode.
	// You may also want to map throttle errors (e.g. to many requests) to this.
	//
	// The details contain a non-nil UnavailableDetails interface.
	EAGAIN = 11

	// Out of memory
	//
	// A requested operation failed or would fail because of insufficient memory resources. This may also be due to
	// configured artificial limit.
	//
	// The details are implementation specific.
	ENOMEM = 12

	// Permission denied
	//
	// Right management is complex. There are usually different rights for the current user to read and write data in various locations.
	// This message usually signals that the general access to the datasource is possible, but not for the requested operation. Examples:
	//
	//    * A local filesystem has usually system files, which cannot be read or write
	//    * A remote filesystem has usually provides folders or files which are read only
	//
	// The details contain a []string array which includes all affected path Entries.
	EACCES = 13

	// Block device required
	//
	// An implementation may require a block device and not a regular file.
	//
	// The details are implementation specific.
	ENOTBLK = 15

	// Device or resource busy
	//
	// A resource is accessed which cannot be shared (or has to many shares) and the operation cannot succeed.
	// This usually happens also to regular filesystems when deleting a parent folder but you continue using a child.
	//
	// The details contain a []string array which includes all affected path Entries.
	EBUSY = 16

	// File exists
	//
	// An operation expected that it has to create a new file and that it is definitely an error that it already exists.
	//
	// The details contain a []string array which includes all affected path Entries.
	EEXIST = 17

	// Cross-device link
	//
	// This is usually used in two situation: A local file system with different mount points and creating links
	// between file systems or doing the same across vfs mounted filesystems.
	//
	// The details contain a []string array which includes all affected path Entries.
	EXDEV = 18

	// No such device
	//
	// A specific device is required, which is not available.
	//
	// The details are implementation specific.
	ENODEV = 19

	// Not a directory
	//
	// An operation was requested which required a directory but found that it's not.
	//
	// The details contain a []string array which includes all affected path Entries.
	ENOTDIR = 20

	// Is a directory
	//
	// An operation was requested which required a file but found that it's a directory.
	//
	// The details contain a []string array which includes all affected path Entries.
	EISDIR = 21

	// Invalid argument
	//
	// A generic code to indicate one or more invalid parameters.
	//
	// The details are implementation specific.
	EINVAL = 22

	// File table overflow / to many open files
	//
	// The resources of the entire system to keep files open, are depleted. See also EMFILE, which is more common.
	//
	// The details contain a []string array which includes all affected path Entries.
	ENFILE = 23

	// Too many open files
	//
	// The configured limit of your process or filesystem has been reached. See also ENFILE.
	//
	// The details contain a []string array which includes all affected path Entries.
	EMFILE = 24

	// File too large
	//
	// The system may impose limits to the maximum supported file size.
	//
	// The details contain LimitDetails
	EFBIG = 27

	// No space left on device
	//
	// There is not enough free space. See also EDQUOT or EFBIG to distinguish other related conditions.
	//
	// The details contain LimitDetails
	ENOSPC = 28

	// Read-only file system
	//
	// Returned to distinguish intrinsic read-only property from a permission problem.
	//
	// The details are implementation specific.
	EROFS = 30

	// Resource deadlock would occur
	//
	// You are lucky and the system detected that your operation would result in a deadlock, like using folded
	// transactions.
	// The details are implementation specific.
	EDEADLK = 35

	// File name too long
	//
	// The identifier for a file, directory, hostname or similar is too long.
	//
	// The details contain a []string array which includes all affected path Entries.
	ENAMETOOLONG = 36

	// No record locks available
	//
	// A resource may be locked, so that others are not allowed to read, write, delete or move a resource. It depends on the
	// data source which operations are not allowed. For example it may still be allowed to read, but not to delete it.
	// To continue, you need to acquire a lock properly.
	//
	// The details are implementation specific.
	ENOLCK = 37

	// Function not implemented / operation not supported
	//
	// A persistent implementation detail of a filesystem, that it does not implement a specific operation.
	//
	// The details are implementation specific.
	ENOSYS = 38

	// Directory not empty
	//
	// An operation expected an empty directory but found it's not. Caution: in contrast to the posix specification,
	// deleting a non-empty directory is not allowed to fail because it is empty.
	//
	//
	ENOTEMPTY = 39

	// Too many symbolic links encountered / cycle detected
	//
	// When using links, a cycle can be constructed which may cause infinite loops.
	//
	// The details are implementation specific.
	ELOOP = 40

	// No data available
	//
	// The server responded but contained no data
	//
	// The details are implementation specific.
	ENODATA = 61

	// Object is remote
	//
	// Caution, this is defined entirely different: The object cannot be accessed because the system works in offline
	// mode and the object is only available on remote.
	//
	// The details are implementation specific.
	EREMOTE = 66

	// Communication error on send
	//
	// A corruption or error has been detected while sending or receiving, e.g. because something went wrong on the
	// line or has been tampered.
	//
	// The details are implementation specific.
	ECOMM = 70

	// Protocol error
	//
	// The implementation expected a certain behavior of the backend (e.g. a network server response) but the protocol failed. This is likely an implementation failure at the client side (assuming that the server is always right).
	// One could argue that it would be better to throw and die but the experience shows that assertions in backends are usually violated in corner cases, so dieing is not really helpful for the user.
	//
	// Inspect the details, which are implementation specific.
	EPROTO = 71

	// Value too large for defined data type
	//
	// In various situations, an implementation may force a maximum size of a data structure. This usually happens in
	// case of corruptions or attacks.
	//
	// Inspect the details, which are implementation specific.
	EOVERFLOW = 75

	// Id not unique on network
	//
	// There are several operations, especially for creating resources or renaming them which may fail due to uniqueness
	// constraints.
	//
	// The details contain a []string array which includes all affected path Entries.
	ENOTUNIQ = 76

	// Illegal byte sequence
	//
	// This may be returned, if there is a format error, e.g. because an UTF-8 sequence is not UTF-8 or a string
	// contains a null byte or a jpeg is truncated.
	//
	// Inspect the details, which are implementation specific.
	EILSEQ = 84

	// Protocol not available
	//
	// The client requested a specific protocol or version but the server rejects the connection because it won't support
	// that protocol anymore. Usually this indicates, that the client must be updated.
	//
	// The details are implementation specific.
	ENOPROTOOPT = 92

	// Protocol not supported
	//
	// Especially for network bindings (or also container formats). The channel works, however the protocol implementation
	// detected that it is incompatible with the service (e.g. the remote side is newer than the client side and is not
	// backwards compatible).
	//
	// The details are implementation specific.
	EPROTONOSUPPORT = 93

	// Address already in use
	//
	// Usually returned, if a socket server is spawned but another server is already running on that port.
	//
	// The details are implementation specific.
	EADDRINUSE = 98

	// Cannot assign requested address
	//
	// Usually returned, if a socket server should be bound to a host name or ip, which is not available.
	//
	// The details are implementation specific.
	EADDRNOTAVAIL = 99

	// Network is down
	//
	// Your network is gone and your user should plugin the cable or disable airplane mode.
	//
	// The details are implementation specific.
	ENETDOWN = 100

	// Network is unreachable
	//
	// A part of your network is gone and your user should plugin the cable to the switch.
	//
	// The details are implementation specific.
	ENETUNREACH = 101

	// Network dropped connection because of reset
	//
	// The network has a hickup, probably just try again
	//
	// The details are implementation specific.
	ENETRESET = 102

	// Software caused connection abort
	//
	// The connection was aborted by intention. You likely want to check your implementation. A reason may be
	// that a server wants to force some properties like SSL on you.
	//
	// The details are implementation specific.
	ECONNABORTED = 103

	// Connection reset by peer
	//
	// Your connection died, e.g. because the remote is rebooting, probably just try again later.
	ECONNRESET = 104

	// Connection timed out
	//
	// There was some kind of a time out.
	//
	// The details are implementation specific.
	ETIMEDOUT = 110

	// Connection refused
	//
	// Typically a host rejected the connection, because the service is not available.
	//
	// The details are implementation specific.
	ECONNREFUSED = 111

	// Host is down
	//
	// The remote host is down.
	//
	// The details are implementation specific.
	EHOSTDOWN = 112

	// No route to host
	//
	// The entire remote host is not reachable.
	//
	// The details are implementation specific.
	EHOSTUNREACH = 113

	// Operation already in progress
	//
	// An implementation may be clever and reject operations which are already in progress.
	//
	// The details are implementation specific.
	EALREADY = 114

	// Remote I/O error
	//
	// Used to indicate that the remote server detected an internal problem, like a crash (e.g. http 500)
	EREMOTEIO = 121

	// Quota exceeded
	//
	// A users storage quota has been exceeded. This may be caused by other resources, like transfer volume or throughput.
	//
	// Details contain LimitDetails
	EDQUOT = 122

	// End Of File
	EOF = 248

	// Invalid transaction
	//
	// The transaction is invalid, e.g. because it has been closed already.
	ETXINVALID = 249

	// Invalid iterator
	//
	// This error is returned as soon as the underlying iterator has been invalidated, which means that it has been
	// corrupted and can no longer be used. Iterators without snapshot support have to detect situations where the
	// consistency of already returned Entries is violated, however in such cases you may try to revalidate it to
	// get a new instance (not specified). Once EITINVALID has been returned, the iterator cannot be used
	// anymore. It signals an unfixable error which cannot be resolved by repetition.
	// Example situations:
	//
	//   * read A, B, C then adding D, E, F : getSize increases from 3 to 6 while iterating => all is good,
	//     everything is still consistent
	//   * read A, B, C then removing A, B, C and adding D, E, F: getSize is identical, but all items are entirely
	//     different. Because the consumer of the iterator cannot detect this situation, the iterator is => invalid
	//   * read A, B, C then removing A: getSize is different, but the consumer cannot know what happened, the iterator
	//     is => invalid
	//   * read A, jump, read C then removing B: getSize is different and jumping to B would return C again, the
	//     iterator is => invalid
	//   * tricky: jump, read Z, Y, X, remove Y => invalid
	//
	// => The generic rule is: deletion, insertion or changing order will always result in an invalid iterator. The only allowed modification is appending at the end. Jumping to the end may result in an inefficient entire deserialization of the list (depends on the implementation). If this is an issue, consider to reformulate your query using sorting criteria instead.
	EITINVALID = 250

	// Invalid isolation level
	//
	// Everything is executed in a transaction. If the level is unsupported this kind is returned, instead of executing it using the wrong level. The only exception of this rule is the usage of IsolationLevel#None
	// where the implementation is free to choose.
	//
	// The details are implementation specific.
	EINISOL = 251

	// Account expired
	//
	// The account used for login was marked as expired by the backend. This generally is used
	// to indicate some kind of user action outside the scope is required to fix this error,
	// like visiting the service's website or contacting it's admin.
	//
	// Inspect the details, which are implementation specific.
	EAEXP = 252

	// Mount point not found
	//
	// A MountPointNotFoundError is only used by the MountableFileSystem to indicate that the given path cannot be
	// associated with a mounted FileSystem. Check your prefix.
	//
	// The details contain a []string array which includes all affected path Entries.
	ENOMP = 253

	// Unsupported attributes
	//
	// Typically returned by FileSystem#ReadAttrs and FileSystem#WriteAttrs whenever a type
	// has been given which is not supported by the actual FileSystem implementation.
	//
	// The details are implementation specific.
	EUNATTR = 254

	// unknown error
	//
	// The unknown error must be used, if a return value with colliding or unknown meaning is transported, e.g. if
	// an implementation simply returns an unchecked http status code.
	//
	// The details contain an int of the unchecked original error code.
	EUNKOWN = 255
)

Reserved vfs status codes are between 0 and 255 (range of uint8). You can use any other values outside of this range for custom status codes. Many codes are intentionally equal to the posix specification and share most of semantics but not all, so be aware of the details. See also https://www.gnu.org/software/libc/manual/html_node/Error-Codes.html. Also a lot of codes have been tagged with unspecified, because the meaning is either not clear in the specification or because it does seem to bring any advantage to the vfs specification.

View Source
const EventBeforeBucketRead = "BeforeBucketRead"
View Source
const EventBeforeDelete = "BeforeDelete"
View Source
const EventBeforeHardLink = "BeforeHardLink"
View Source
const EventBeforeMkBucket = "BeforeMkBucket"
View Source
const EventBeforeOpen = "BeforeOpen"
View Source
const EventBeforeReadAttrs = "BeforeReadAttrs"
View Source
const EventBeforeSymLink = "BeforeSymLink"
View Source
const PathSeparator = "/"

The PathSeparator is always / and platform independent

Variables

This section is empty.

Functions

func Copy added in v0.0.4

func Copy(src string, dst string, options *CopyOptions) error

Copy performs a copy from src to dst. Dst is always removed and replaced with the contents of src. The copy options can be nil and can be used to get detailed information on the progress. The implementation tries to use RefLink if possible.

func Delete added in v0.0.2

func Delete(path string) error

Delete a path entry and all contained children. It is not considered an error to delete a non-existing resource. Delegates to Default()#Delete.

func Equals added in v0.0.7

func Equals(a Entry, b Entry) bool

Equals returns true if the values defined by Entry are equal. However it does not inspect or check other fields or values, especially not Sys()

func IsErr added in v0.0.7

func IsErr(err error, statusCode int) bool

IsErr inspects the wrapped hierarchy for a specific statusCode

func MkDirs added in v0.0.2

func MkDirs(path string) error

MkDirs tries to create the given path hierarchy. If path already denotes a directory nothing happens. If any path segment already refers a file, an error must be returned. Delegates to Default()#MkDirs.

func NewErr added in v0.0.7

func NewErr() errBuilder

func Read added in v0.0.2

func Read(path string) (io.ReadCloser, error)

Read opens the given resource for reading. May optionally also implement os.Seeker. If called on a directory UnsupportedOperationError is returned. Delegates to Default()#Open.

func ReadAll

func ReadAll(path string) ([]byte, error)

ReadAll loads the entire resource into memory. Only use it, if you know that it fits into memory

func Rename added in v0.0.2

func Rename(oldPath string, newPath string) error

Rename moves a file from the old to the new path. If oldPath does not exist, ResourceNotFoundError is returned. If newPath exists, it will be replaced. Delegates to Default()#Rename.

func SetDefault added in v0.0.2

func SetDefault(provider FileSystem)

SetDefault updates the default data provider. See also #Default()

func Stat

func Stat(path string) (os.FileInfo, error)

Stat emulates a standard library file info contract. See also #ReadAttrs() which allows a bit more control on how the call is made.

func StatusText added in v0.0.7

func StatusText(code int) string

StatusText returns th

func UnportableCharacter added in v0.0.7

func UnportableCharacter(str string) int

UnportableCharacter checks the given string for unsafe characters and returns the first index of occurrence or -1. This is important to exchange file names across different implementations, like windows, macos or linux. In general the following characters are considered unsafe *?:[]"<>|(){}&'!\;$ and chars <= 0x1F. As a developer you should check and avoid file path segments to contain any of these characters, especially because / or ? would clash with the path and fork separator. If the string is not a valid utf8 sequence, 0 is returned.

func Walk

func Walk(path string, each WalkClosure) error

Walk recursively goes down the entire path hierarchy starting at the given path

func Write added in v0.0.2

func Write(path string) (io.WriteCloser, error)

Write opens the given resource for writing. Removes and recreates the file. May optionally also implement os.Seeker. If elements of the path do not exist, they are created implicitly. Delegates to Default()#Open.

func WriteAll

func WriteAll(path string, data []byte) (int, error)

WriteAll just puts the given data into the path

Types

type AbsMapEntry added in v0.0.7

type AbsMapEntry map[string]interface{}

deprecated

func (AbsMapEntry) IsDir added in v0.0.7

func (a AbsMapEntry) IsDir() bool

func (AbsMapEntry) Name added in v0.0.7

func (a AbsMapEntry) Name() string

func (AbsMapEntry) Size added in v0.0.7

func (a AbsMapEntry) Size() int64

func (AbsMapEntry) Sys added in v0.0.7

func (a AbsMapEntry) Sys() interface{}

type AbstractFileSystem added in v0.0.7

type AbstractFileSystem struct {
	FConnect func(ctx context.Context, options interface{}) (interface{}, error)

	FDisconnect func(ctx context.Context) error

	FAddListener func(ctx context.Context, path string, listener ResourceListener) (int, error)

	FRemoveListener func(ctx context.Context, handle int) error

	FBegin func(ctx context.Context, options interface{}) (context.Context, error)

	FCommit func(ctx context.Context) error

	FRollback func(ctx context.Context) error

	FOpen func(ctx context.Context, path string, flag int, options interface{}) (Blob, error)

	FDelete func(ctx context.Context, path string) error

	FReadAttrs func(ctx context.Context, path string, options interface{}) (Entry, error)

	FReadForks func(ctx context.Context, path string) ([]string, error)

	FWriteAttrs func(ctx context.Context, path string, src interface{}) (Entry, error)

	FReadBucket func(ctx context.Context, path string, options interface{}) (ResultSet, error)

	FInvoke func(ctx context.Context, endpoint string, args ...interface{}) (interface{}, error)

	FMkBucket func(ctx context.Context, path string, options interface{}) error

	FRename func(ctx context.Context, oldPath string, newPath string) error

	FSymLink func(ctx context.Context, oldPath string, newPath string) error

	FHardLink func(ctx context.Context, oldPath string, newPath string) error

	FRefLink func(ctx context.Context, oldPath string, newPath string) error

	FClose func() error

	FString func() string

	FFireEvent func(ctx context.Context, path string, event interface{}) error
}

An AbstractFileSystem can be embedded to bootstrap a new implementation faster. One can even embed an uninitialized pointer type, which will return ENOSYS on each method call.

func (*AbstractFileSystem) AddListener added in v0.0.7

func (v *AbstractFileSystem) AddListener(ctx context.Context, path string, listener ResourceListener) (int, error)

func (*AbstractFileSystem) Begin added in v0.0.7

func (v *AbstractFileSystem) Begin(ctx context.Context, path string, options interface{}) (context.Context, error)

func (*AbstractFileSystem) Close added in v0.0.7

func (v *AbstractFileSystem) Close() error

func (*AbstractFileSystem) Commit added in v0.0.7

func (v *AbstractFileSystem) Commit(ctx context.Context) error

func (*AbstractFileSystem) Connect added in v0.0.7

func (v *AbstractFileSystem) Connect(ctx context.Context, path string, options interface{}) (interface{}, error)

func (*AbstractFileSystem) Delete added in v0.0.7

func (v *AbstractFileSystem) Delete(ctx context.Context, path string) error

func (*AbstractFileSystem) Disconnect added in v0.0.7

func (v *AbstractFileSystem) Disconnect(ctx context.Context, path string) error

func (*AbstractFileSystem) FireEvent added in v0.0.7

func (v *AbstractFileSystem) FireEvent(ctx context.Context, path string, event interface{}) error
func (v *AbstractFileSystem) HardLink(ctx context.Context, oldPath string, newPath string) error

func (*AbstractFileSystem) Invoke added in v0.0.7

func (v *AbstractFileSystem) Invoke(ctx context.Context, endpoint string, args ...interface{}) (interface{}, error)

func (*AbstractFileSystem) MkBucket added in v0.0.7

func (v *AbstractFileSystem) MkBucket(ctx context.Context, path string, options interface{}) error

func (*AbstractFileSystem) Open added in v0.0.7

func (v *AbstractFileSystem) Open(ctx context.Context, path string, flag int, options interface{}) (Blob, error)

func (*AbstractFileSystem) ReadAttrs added in v0.0.7

func (v *AbstractFileSystem) ReadAttrs(ctx context.Context, path string, options interface{}) (Entry, error)

func (*AbstractFileSystem) ReadBucket added in v0.0.7

func (v *AbstractFileSystem) ReadBucket(ctx context.Context, path string, options interface{}) (ResultSet, error)

func (*AbstractFileSystem) ReadForks added in v0.0.7

func (v *AbstractFileSystem) ReadForks(ctx context.Context, path string) ([]string, error)
func (v *AbstractFileSystem) RefLink(ctx context.Context, oldPath string, newPath string) error

func (*AbstractFileSystem) RemoveListener added in v0.0.7

func (v *AbstractFileSystem) RemoveListener(ctx context.Context, handle int) error

func (*AbstractFileSystem) Rename added in v0.0.7

func (v *AbstractFileSystem) Rename(ctx context.Context, oldPath string, newPath string) error

func (*AbstractFileSystem) Rollback added in v0.0.7

func (v *AbstractFileSystem) Rollback(ctx context.Context) error

func (*AbstractFileSystem) String added in v0.0.7

func (v *AbstractFileSystem) String() string
func (v *AbstractFileSystem) SymLink(ctx context.Context, oldPath string, newPath string) error

func (*AbstractFileSystem) WriteAttrs added in v0.0.7

func (v *AbstractFileSystem) WriteAttrs(ctx context.Context, path string, src interface{}) (Entry, error)

type AttrList added in v0.0.7

type AttrList struct {
	List
}

type Blob added in v0.0.7

type Blob interface {
	// ReadAt reads len(b) bytes from the File starting at byte offset off. It returns the number of bytes read and
	// the error, if any. ReadAt always returns a non-nil error when n < len(b). At end of file, that error is io.EOF.
	//
	// This method is explicitly thread safe, as long no overlapping
	// writes or reads to the given offset are happening. This is
	// especially useful in multi threaded scenarios to perform
	// concurrent changes on a single opened resource (see POSIX pread).
	ReadAt(b []byte, off int64) (n int, err error)
	io.Reader

	// WriteAt writes len(b) bytes to the File starting at byte offset off.
	// It returns the number of bytes written and an error, if any.
	// WriteAt returns a non-nil error when n != len(b).
	//
	// This method is explicitly thread safe, as long no overlapping
	// writes or reads to the given offset are happening. This is
	// especially useful in multi threaded scenarios to perform
	// concurrent changes on a single opened resource (see POSIX pwrite).
	WriteAt(b []byte, off int64) (n int, err error)
	io.Writer
	io.Seeker
	io.Closer
}

A Blob is an abstract accessor to read or write bytes. In general, not all methods are supported, either due to the way the resources have been opened or because of the underlying implementation. In such cases the affected method will always return an *UnsupportedOperationError.

type BlobAdapter added in v0.0.8

type BlobAdapter struct {
	// Delegate can be anything like io.ReaderAt, io.WriterAt, io.Writer, io.Closer, io.Reader and io.Seeker
	// in all combinations.
	Delegate interface{}
}

A BlobAdapter is used to wrap something like io.Reader into a Blob, whose other methods (e.g. other than Read) will simply return ENOSYS.

func (*BlobAdapter) Close added in v0.0.8

func (d *BlobAdapter) Close() error

func (*BlobAdapter) Read added in v0.0.8

func (d *BlobAdapter) Read(p []byte) (n int, err error)

func (*BlobAdapter) ReadAt added in v0.0.8

func (d *BlobAdapter) ReadAt(b []byte, off int64) (n int, err error)

func (*BlobAdapter) Seek added in v0.0.8

func (d *BlobAdapter) Seek(offset int64, whence int) (int64, error)

func (*BlobAdapter) Write added in v0.0.8

func (d *BlobAdapter) Write(p []byte) (n int, err error)

func (*BlobAdapter) WriteAt added in v0.0.8

func (d *BlobAdapter) WriteAt(b []byte, off int64) (n int, err error)

type BlobBuilder added in v0.0.7

type BlobBuilder struct {
	// contains filtered or unexported fields
}

func (*BlobBuilder) Add added in v0.0.7

func (b *BlobBuilder) Add() *Builder

func (*BlobBuilder) MatchAlso added in v0.0.7

func (b *BlobBuilder) MatchAlso(pattern string) *BlobBuilder

Match defines a pattern which is matched against a path and applies the defined data transformation rules

func (*BlobBuilder) OnDelete added in v0.0.7

func (b *BlobBuilder) OnDelete(delete func(context.Context, Path) error) *BlobBuilder

func (*BlobBuilder) OnOpen added in v0.0.7

func (b *BlobBuilder) OnOpen(open func(context.Context, Path, int, interface{}) (Blob, error)) *BlobBuilder

func (*BlobBuilder) OnRead added in v0.0.7

func (b *BlobBuilder) OnRead(open func(context.Context, Path) (io.Reader, error)) *BlobBuilder

func (*BlobBuilder) OnWrite added in v0.0.7

func (b *BlobBuilder) OnWrite(open func(context.Context, Path) (io.Writer, error)) *BlobBuilder

type BucketBuilder added in v0.0.7

type BucketBuilder struct {
	// contains filtered or unexported fields
}

== The BucketBuilder helps to specify the data transformation for a buckets content or listing

func (*BucketBuilder) Add added in v0.0.7

func (b *BucketBuilder) Add() *Builder

func (*BucketBuilder) MatchAlso added in v0.0.7

func (b *BucketBuilder) MatchAlso(pattern string) *BucketBuilder

Match defines a pattern which is matched against a path and applies the defined data transformation rules

func (*BucketBuilder) OnDelete added in v0.0.7

func (b *BucketBuilder) OnDelete(delete func(context.Context, Path) error) *BucketBuilder

func (*BucketBuilder) OnList added in v0.0.7

func (b *BucketBuilder) OnList(transformation func(Path) ([]*DefaultEntry, error)) *BucketBuilder

OnList configures the generic call to ReadBucket, which is either nil, *DefaultEntry or map[string]interface{}. In any other case ReadBucket will return map[string]interface{} with the 3 fields n,s and b which contains name, size and the isBucket flag.

type Builder added in v0.0.7

type Builder struct {
	// contains filtered or unexported fields
}

The Builder is used to create a VFS from scratch in a simpler way. A list of included batteries:

  • Supports a lot of events for read/write/delete/update etc. using string constants (Event*)
  • Implementations may always provide a Size() but may return -1 or be incorrect
  • Optimized reads in ReadAttrs if args is map[string]interface{}
  • Each undefined method will return ENOSYS error
  • Listeners can be used to intercept operations (before semantic)

func (*Builder) Create added in v0.0.7

func (b *Builder) Create() FileSystem

func (*Builder) Delete added in v0.0.7

func (b *Builder) Delete(f func(ctx context.Context, path Path) error) *Builder

Delete has lowest priority, after all blob and bucket matches have been checked

func (*Builder) Details added in v0.0.7

func (b *Builder) Details(name string, majorVersion int, minorVersion int, microVersion int) *Builder

Details sets the name of the VFS

func (b *Builder) Hardlink(f func(ctx context.Context, oldPath Path, newPath Path) error) *Builder

func (*Builder) MatchBlob added in v0.0.7

func (b *Builder) MatchBlob(pattern string) *BlobBuilder

func (*Builder) MatchBucket added in v0.0.7

func (b *Builder) MatchBucket(pattern string) *BucketBuilder

func (*Builder) MkBucket added in v0.0.7

func (b *Builder) MkBucket(f func(ctx context.Context, path Path, options interface{}) error) *Builder

func (*Builder) ReadEntryAttrs added in v0.0.7

func (b *Builder) ReadEntryAttrs(f func(ctx context.Context, path Path, dst *DefaultEntry) error) *Builder

func (*Builder) Reset added in v0.0.7

func (b *Builder) Reset()

Reset throws the internal state away

func (b *Builder) Symlink(f func(ctx context.Context, oldPath Path, newPath Path) error) *Builder

type ChRoot added in v0.0.4

type ChRoot struct {
	// The Prefix which is added before delegating
	Prefix Path
	// The Delegate to call with the prefixed path
	Delegate FileSystem
}

A ChRoot is a filesystem which is basically a poor man's chroot which just adds a prefix to all endpoints and delegates all calls. A security note: the path is normalized before prefixing, so that path based attacks using .. are not possible.

func (*ChRoot) AddListener added in v0.0.7

func (f *ChRoot) AddListener(ctx context.Context, path string, listener ResourceListener) (handle int, err error)

func (*ChRoot) Begin added in v0.0.7

func (f *ChRoot) Begin(ctx context.Context, path string, options interface{}) (context.Context, error)

func (*ChRoot) Close added in v0.0.4

func (f *ChRoot) Close() error

func (*ChRoot) Commit added in v0.0.7

func (f *ChRoot) Commit(ctx context.Context) error

func (*ChRoot) Connect added in v0.0.7

func (f *ChRoot) Connect(ctx context.Context, path string, options interface{}) (interface{}, error)

func (*ChRoot) Delete added in v0.0.4

func (f *ChRoot) Delete(ctx context.Context, path string) error

func (*ChRoot) Disconnect added in v0.0.7

func (f *ChRoot) Disconnect(ctx context.Context, path string) error

func (*ChRoot) FireEvent added in v0.0.7

func (f *ChRoot) FireEvent(ctx context.Context, path string, event interface{}) error
func (f *ChRoot) HardLink(ctx context.Context, oldPath string, newPath string) error

func (*ChRoot) Invoke added in v0.0.7

func (f *ChRoot) Invoke(ctx context.Context, endpoint string, args ...interface{}) (interface{}, error)

func (*ChRoot) MkBucket added in v0.0.7

func (f *ChRoot) MkBucket(ctx context.Context, path string, options interface{}) error

func (*ChRoot) Open added in v0.0.4

func (f *ChRoot) Open(ctx context.Context, path string, flag int, options interface{}) (Blob, error)

func (*ChRoot) ReadAttrs added in v0.0.4

func (f *ChRoot) ReadAttrs(ctx context.Context, path string, args interface{}) (Entry, error)

func (*ChRoot) ReadBucket added in v0.0.7

func (f *ChRoot) ReadBucket(ctx context.Context, path string, options interface{}) (ResultSet, error)

func (*ChRoot) ReadForks added in v0.0.7

func (f *ChRoot) ReadForks(ctx context.Context, path string) ([]string, error)
func (f *ChRoot) RefLink(ctx context.Context, oldPath string, newPath string) error

func (*ChRoot) RemoveListener added in v0.0.7

func (f *ChRoot) RemoveListener(ctx context.Context, handle int) error

func (*ChRoot) Rename added in v0.0.4

func (f *ChRoot) Rename(ctx context.Context, oldPath string, newPath string) error

func (*ChRoot) Resolve added in v0.0.4

func (f *ChRoot) Resolve(path string) string

Resolve normalizes the given Path and inserts the prefix. We normalize our path, before adding the prefix to avoid breaking out of our root

func (*ChRoot) Rollback added in v0.0.7

func (f *ChRoot) Rollback(ctx context.Context) error

func (*ChRoot) String added in v0.0.7

func (f *ChRoot) String() string
func (f *ChRoot) SymLink(ctx context.Context, oldPath string, newPath string) error

func (*ChRoot) WriteAttrs added in v0.0.4

func (f *ChRoot) WriteAttrs(ctx context.Context, path string, src interface{}) (Entry, error)

type CopyOptions added in v0.0.4

type CopyOptions struct {

	// OnScan is called while scanning the source
	OnScan func(obj string, objects int64, bytes int64)

	// OnCopied is called after each transferred object.
	OnCopied func(obj string, objectsTransferred int64, bytesTransferred int64)

	// OnProgress is called for each file which is progress of being copied
	OnProgress func(src string, dst string, bytes int64, size int64)

	// OnError is called if an error occurs. If an error is returned, the process is stopped and the returned error is returned.
	OnError func(object string, err error) error
	// contains filtered or unexported fields
}

CopyOptions is used to define the process of copying.

func (*CopyOptions) Cancel added in v0.0.4

func (o *CopyOptions) Cancel()

Cancel is used to signal an interruption

func (*CopyOptions) IsCancelled added in v0.0.4

func (o *CopyOptions) IsCancelled() bool

IsCancelled checks if the copy process has been cancelled

type DefaultEntry added in v0.0.8

type DefaultEntry struct {
	Id       string      // Id must be at least unique per bucket
	IsBucket bool        // IsBucket denotes the directory or folder flag
	Length   int64       // Length in bytes, if unknown set to -1
	Data     interface{} // Data is the original payload, if any, otherwise nil
}

DefaultEntry is a minimal type, useful to create simple VFS implementations. It may make sense to create a custom type which just fulfills the contract and avoids some GC pressure.

func (*DefaultEntry) IsDir added in v0.0.8

func (a *DefaultEntry) IsDir() bool

IsDir returns the IsBucket flag

func (*DefaultEntry) Name added in v0.0.8

func (a *DefaultEntry) Name() string

Name returns the (unique) Id

func (*DefaultEntry) Size added in v0.0.8

func (a *DefaultEntry) Size() int64

Size returns the Length value

func (*DefaultEntry) Sys added in v0.0.8

func (a *DefaultEntry) Sys() interface{}

Sys returns any internal or implementation specific payload, which may be nil.

type DefaultError added in v0.0.7

type DefaultError struct {
	Message        string
	Code           int
	CausedBy       error
	DetailsPayload interface{}
}

func NewENOSYS added in v0.0.8

func NewENOSYS(msg string, who interface{}) *DefaultError

NewENOSYS is a helper function to create an error which signals that an implementation is not available. The msg should indicate what is not implemented and who can be used to give a type hint for further inspection. If used correctly

func (*DefaultError) Details added in v0.0.7

func (e *DefaultError) Details() interface{}

func (*DefaultError) Error added in v0.0.7

func (e *DefaultError) Error() string

func (*DefaultError) StatusCode added in v0.0.7

func (e *DefaultError) StatusCode() int

func (*DefaultError) Unwrap added in v0.0.7

func (e *DefaultError) Unwrap() error

type DefaultResultSet added in v0.0.8

type DefaultResultSet struct {
	Entries []*DefaultEntry
}

DefaultResultSet is a minimal type, useful to create simple VFS implementation. However you should usually provide a custom implementation to give access to the raw data (see #Sys()), e.g. the original parsed JSON data structures.

func (*DefaultResultSet) Len added in v0.0.8

func (r *DefaultResultSet) Len() int

Len always returns len(Entries)

func (*DefaultResultSet) Next added in v0.0.8

func (r *DefaultResultSet) Next(ctx context.Context) error

Next always returns EOF

func (*DefaultResultSet) Pages added in v0.0.8

func (r *DefaultResultSet) Pages() int64

Pages always returns 1

func (*DefaultResultSet) ReadAttrs added in v0.0.8

func (r *DefaultResultSet) ReadAttrs(idx int, args interface{}) Entry

func (*DefaultResultSet) Sys added in v0.0.8

func (r *DefaultResultSet) Sys() interface{}

Sys always returns []*DefaultEntry

func (*DefaultResultSet) Total added in v0.0.8

func (r *DefaultResultSet) Total() int64

Total always returns Len

type Entry added in v0.0.7

type Entry interface {
	// Id returns the unique (at least per bucket) id of an entry
	Name() string

	// IsDir is the folder or bucket flag. This flag is an indicator if it makes sense to query contents
	// with #ReadBucket()
	IsDir() bool

	// Sys returns the implementation specific payload. This can be anything, e.g. a map[string]interface{} or
	// a distinct struct.
	Sys() interface{}
}

Entry is the contract which each implementation needs to support. It actually allows a named navigation. Intentionally it is a subset of os.FileInfo

func ReadAttrs added in v0.0.2

func ReadAttrs(path string, args interface{}) (Entry, error)

ReadAttrs reads Attributes. Every implementation must support ResourceInfo. Delegates to Default()#ReadAttrs.

func ReadBucket added in v0.0.7

func ReadBucket(path string) ([]Entry, error)

ReadBucket is a utility method to simply list a directory by querying all result set pages.

func WriteAttrs added in v0.0.2

func WriteAttrs(path string, src interface{}) (Entry, error)

WriteAttrs writes Attributes. This is an optional implementation and may simply return UnsupportedOperationError. Delegates to Default()#WriteAttrs.

type Error added in v0.0.7

type Error interface {
	error

	// Unwrap returns the next error in the error chain.
	// If there is no next error, Unwrap returns nil.
	Unwrap() error

	// StatusCode returns a code which specifies the kind of error in more details. You may also want to
	// inspect #Details() to get more insight.
	StatusCode() int

	// Details may contain more information about
	Details() interface{}
}

An Error incorporates the go 2 draft Wrapper interface and a status/error code and details. The error codes are largely inspired by the posix libc but are different in detail.

type Fields added in v0.0.7

type Fields = map[string]interface{}

type FileSystem added in v0.0.4

type FileSystem interface {
	// Connect may perform an authentication and authorization based on the given properties.
	// This method changes the internal state of the file system. Implementations may change the properties, so
	// that a refresh token can be returned. Some implementations may support distinct connections
	// per bucket. So a workflow may be as follows:
	//
	//  props := onLoadInstanceState() // load properties from somewhere
	//  err := vfs.Connect(context.Background(), "/", props) // try to connect
	//  if err == nil {
	//     return vfs // everything was fine, exit
	//  }
	//
	//  // connection failed, so fill in custom credentials (you need to read the documentation)
	//  props["user"] = "john.doe@mycompany.com"
	//  props["pwd"] = "1234"
	//  err = vfs.Connect(context.Background(), "/", props)  // reconnect
	//  if err == nil { // props may contain refresh token, a session id or anything arbitrary
	//	    delete(props, "user")  // you may want to keep the user to autofill
	//      delete(props, "pwd")  // but remove credentials which are worth protecting
	//      onSaveInstanceState(props) // save whatever else has been inserted into the properties
	//  }
	//
	// Implementations may reject this operation permanently with ENOSYS error.
	Connect(ctx context.Context, path string, options interface{}) (interface{}, error)

	// Disconnect terminates the internal state of authentication and authorization, e.g. by destroying a refresh
	// token or a session at the remote side.
	// Implementations may reject this operation permanently with ENOSYS error.
	Disconnect(ctx context.Context, path string) error

	// FireEvent may notify all registered listeners. Depending on the implementation, the behavior in case of
	// errors is undefined. However it is recommended, that an error of a listener will short circuit the invocations
	// of other listeners and return the error early. Also it is recommended that an implementation should evaluate
	// events according to their pre/post semantic and respect that error, so that a listener can cancel an entire
	// operation which is just returned by the calling method.
	//
	// Implementations may reject this operation permanently with ENOSYS error.
	FireEvent(ctx context.Context, path string, event interface{}) error

	// AddListener a ResourceListener to get notified about changes to resources. It returns a handle to unregister.
	// We use a handle to keep the api sleek and to avoid a discussion about the equality of interfaces.
	// Implementations may reject this operation permanently with ENOSYS error.
	AddListener(ctx context.Context, path string, listener ResourceListener) (handle int, err error)

	// RemoveListener removes an already registered listener. It is not an error to remove an unregistered
	// or invalid listener.
	// Implementations may reject this operation permanently with ENOSYS error.
	RemoveListener(ctx context.Context, handle int) error

	// Begin starts a transaction, so that all following method calls are interpreted in the context of the running
	// transaction. The options argument is a map of key/value primitives suitable for json serialization.
	// An implementation shall be as thread safe as possible and should support concurrent read/write
	// on any operation. The context is modified WithValue and used to track the transaction state.
	// If an implementation supports transactions and begin/commit/rollback cycle is not used, the transactional
	// behavior is not defined, which may be e.g. none at all, every operation in a single transaction or committed
	// within a time slot or anything else. However it is guaranteed that an implementation which supports
	// transaction must not fail because the transactional api has not been used. Some implementations
	// may support transaction for sub buckets, otherwise the path must be root (/).
	//
	// Implementations may reject this operation permanently with ENOSYS error. When used with the wrong arguments,
	// e.g. with an unsupported isolation level EINISOL is returned.
	Begin(ctx context.Context, path string, options interface{}) (context.Context, error)

	// Commit applies a running transaction. See also #Begin() for details.
	// Implementations may reject this operation permanently with ENOSYS error. Returns ETXINVALID if no transaction
	// is pending.
	Commit(ctx context.Context) error

	// Rollback does not apply the current state of the transaction and reverts all changes.
	// Implementations may reject this operation permanently with ENOSYS error. Returns ETXINVALID if no transaction
	// is pending.
	Rollback(ctx context.Context) error

	// Open is the general read or write call. It opens the named resource with specified flags (O_RDONLY etc.).
	// The type of options is implementation specific and may be e.g. something like os.FileMode to declare permissions.
	// If successful, methods on the returned File can be used for I/O.
	// If there is an error, the trace will also contain a *PathError.
	// Implementations have to create parent directories, if those do not exist. However if any existing
	// path segment denotes already a resource, the resource is not deleted and an error is returned instead.
	//
	// Resource Forks or Alternate Data Streams (or e.g. thumbnails from online resources) shall be addressed
	// using a colon (:). Example: /myfolder/test.png/thumb-jpg:720p. Note that the : is also considered to be an unportable
	// and unsafe character but indeed it should never make it into a real local path. See also #ReadForks().
	//
	// Do not forget to close the resource, to avoid any leak.
	// Implementations may reject this operation permanently with ENOSYS error.
	Open(ctx context.Context, path string, flags int, options interface{}) (Blob, error)

	// Deletes a path entry and all contained children. It is not considered an error to delete a non-existing resource.
	// This non-posix behavior is introduced to guarantee two things:
	//   * the implementation shall ensure that races have more consistent effects
	//   * descending the tree and collecting all children is an expensive procedure and often unnecessary, especially
	//     in relational databases with foreign key constraints.
	// Implementations may reject this operation permanently with ENOSYS error.
	Delete(ctx context.Context, path string) error

	// ReadAttrs reads arbitrary or extended attributes into an implementation specific destination.
	// This method may returns additional meta data
	// about the resource like size, last modified, permissions, ACLs or even EXIF data. This allows structured
	// information to pass out without going through a serialization process using the fork logic.
	// args may contain query options (like selected fields) but is also used to recycle objects to
	// avoid heap allocations, but this depends on the actual implementation.
	// Implementations may reject this operation permanently with ENOSYS error.
	ReadAttrs(ctx context.Context, path string, args interface{}) (Entry, error)

	// ReadForks reads all available named resource forks or alternate data streams. Any path object may have
	// an arbitrary amount of forks. A file object always has the unnamed fork, which is not included in the
	// returned list. To access a fork, concat the fork name to the regular file name with a colon.
	//
	// Why using the colon?
	//  * the MacOS convention (../forkName/rsrc) cannot distinguish between a relative path and a named fork
	//  * at least a single platform uses this convention (windows), Posix does not support it. Small things are represented
	//    in extended attributes (ReadAttrs)
	//  * the colon is not allowed on MacOS and Windows and also discouraged on Linux for filenames, because it conflicts
	//    e.g. with the path separator. Also most online sources do not allow it for compatibility reasons, besides
	//    google drive, which literally allows anything.
	//  * easy to be read by humans
	//  * seems to be the less of two evils in anticipated use cases within this api
	//
	// Example:
	//
	//   forks, err := vfs.ReadForks(context.Background(), "image.jpg") // forks may contain things like thumbnails/720p
	//   _, _ = vfs.Open(context.Background, os.O_RDONLY, 0, "image.jpg") // opens the unamed (original) data stream
	//   _, _ = vfs.Open(context.Background, os.O_RDONLY, 0, "image.jpg:thumbnails/720p") // opens a thumbnail by a named stream
	//  _ = vfs.Delete(context.Background, "image.jpg:
	//
	// Implementations may reject this operation permanently with ENOSYS error.
	ReadForks(ctx context.Context, path string) ([]string, error)

	// WriteAttrs inserts or updates properties or extended attributes e.g. with key/value primitives,
	// suitable for json serialization. Optionally an implementation may return an Entry, e.g. if the id or name
	// has changed or if the information comes for free.
	// So keep in mind, that even if no error is returned, the entry may still be nil.
	//
	// Implementations may reject this operation permanently with ENOSYS error.
	WriteAttrs(ctx context.Context, path string, src interface{}) (Entry, error)

	// ReadBucket reads the contents of a directory. If path is not a bucket, an ENOENT is returned.
	// options can be arbitrary primitives and should be json serializable.
	// At least nil and empty options must be supported, otherwise an EUNATTR can be returned.
	// Using the options, the query to retrieve the directory contents can be optimized,
	// like required fields, sorting or page size, if not
	// already passed through the path and its potential fork or query path.
	// Implementations may support additional parameters like sorting or page sizes but should not be appended
	// to the path (uri style), as long as they do not change the actual result set. Options which act like
	// a filter should always map to a distinct path, to avoid confusion or merge conflicts of caching layers on top.
	//
	// Conventionally the colon path /: has a special meaning, because it lists hidden endpoints, which
	// are not otherwise reachable. These endpoints do not make sense to be inspected in a hierarchy. One reason
	// could be, that they always require a set of options as method arguments. There is currently no way
	// of inspecting the arguments programmatically but if you know what you are doing (read the documentation)
	// you can peek through the abstraction but avoid to publish a concrete contract (which may be either something
	// you want to avoid or which you are favoring). See also #Invoke()
	//
	// Implementations may reject this operation permanently with ENOSYS error.
	ReadBucket(ctx context.Context, path string, options interface{}) (ResultSet, error)

	// Invoke is a peephole in the specification to call arbitrary endpoints which are not related to a filesystem
	// but share the same internal state, like the authorization. You need to use type assertions and to
	// consult the documentation to access the concrete
	// type, because it may be a json object, even a http response or an io.Reader, just anything.
	//
	// To inspect the available endpoints you can use /: with #ReadBucket(). Example:
	//
	//   // endpoints contains things like "fullTextSearch"
	//   endpoints, _ := vfs.ReadBucket(context.Background(), "/:", nil)
	//
	//   // perform a custom endpoint query with a json like argument object
	//   args := Options{}
	//   args["text"] = "hello world"
	//   args["sortBy"] = "asc"
	//   args["since"] = "2018.05.14"
	//   args["pageSize"] = 1000
	//   res, err := vfs.Invoke(context.Background(), "fullTextSearch", args) // res may contain a map[string]interface{}
	//
	//   // something which is not serializable at all
	//   reader := getSomeReader()
	//   writer, err := vfs.Invoke(context.Background(), "transfer", reader) // res contains a writer to append stuff
	//   writer.(io.Writer).Write([]byte("EOF"))
	//
	//   // or even more method like
	//   isShared := true
	//   listOfImages := getImageList()
	//   _, err := vfs.Invoke(context.Background(), "createAlbum", "my album title", isShared, listOfImages)
	//
	// Implementations may reject this operation permanently with ENOSYS error.
	Invoke(ctx context.Context, endpoint string, args ...interface{}) (interface{}, error)

	// MkBucket tries to create the given path hierarchy. If path already denotes a directory nothing happens. If any path
	// segment already refers a resource, an error must be returned. The type of options is implementation specific
	// and may be e.g. something like os.FileMode to declare permissions
	MkBucket(ctx context.Context, path string, options interface{}) error

	// Rename moves a file from the old to the new path. If oldPath does not exist, ResourceNotFoundError is returned.
	// If newPath exists, it will be replaced.
	Rename(ctx context.Context, oldPath string, newPath string) error

	// SymLink tries to create a soft link or an alias for an existing resource. This is usually just a special
	// reference which contains the oldPath entry which is resolved if required. The actual resource becomes unavailable
	// as soon as the original path is removed, which will cause the symbolic link to become invalid or to disappear.
	// Implementations may support this for buckets or blobs differently and will return ENOTDIR or EISDIR to
	// give insight. If newPath already exists EEXIST is returned.
	// Implementations may reject this operation permanently with ENOSYS error.
	SymLink(ctx context.Context, oldPath string, newPath string) error

	// HardLink tries to create a new named entry for an existing resource. Changes to one of both named Entries
	// are reflected vice versa, however due to eventual consistency it may take some time to become visible.
	// To remove the resource, one has to remove all named Entries.
	// Implementations may support this for buckets or blobs differently and will return ENOTDIR or EISDIR to
	// give insight. If newPath already exists EEXIST is returned.
	// Implementations may reject this operation permanently with ENOSYS error.
	HardLink(ctx context.Context, oldPath string, newPath string) error

	// RefLink tries to perform a copy from old to new using the most efficient possible way, e.g. by using
	// reference links. Implementations may reject this operation permanently with ENOSYS error. If oldPath and/or
	// newPath refer to buckets and the backend does not support that operations for buckets, EISDIR error is returned
	// or vice versa ENOTDIR if only buckets are supported.
	RefLink(ctx context.Context, oldPath string, newPath string) error

	// Close when Done to release resources. Subsequent calls have no effect and do not report additional errors.
	// An implementation may reject closing while still in process, so future calls may be necessary.
	Close() error

	// String returns a name or description of this VFS
	String() string
}

The FileSystem interface is the core contract to provide access to hierarchical structures using a compound key logic. This is an abstract of way of the design thinking behind a filesystem.

Design decisions

There are the following opinionated decisions:

  • It is an Interface, because it cannot be expected to have a reasonable code reusage between implementations but we need a common behavior. There is a builder to create implementation in a simple way.

  • It contains both read and write contracts, because a distinction between read-only and write-only and filesystems with both capabilities are edge cases. Mostly there will be implementations which provides each combination due to their permission handling.

  • It is not specified, if a FileSystem is thread safe. However every implementation should be as thread safe as possible, similar to the POSIX filesystem specification.

  • The entire VFS specification is completely based on (non recursive) interfaces and can be implemented without any dependencies. The builder, the package level functions and default implementations do not belong to the specification, however you are encouraged to use them. If you use the builder, you can be sure to get at least a commonly supported contract.

  • It makes heavy usage of interface{}, which always requires heap allocations and type assertions. However it is not possible to define a common contract for all. Even the os.Filemode makes no sense for most online implementations. Using map[string]interface{} is even worse because it prevents the usage of real types and would undermine the type system entirely. So at the end using interface{} is the most go-ish we could do here. This will probably also not change with the introduction of generics, because they cannot help us with a generic (and perhaps "meaningless" contract).

var LocalFileSystem FileSystem

func Default added in v0.0.2

func Default() FileSystem

Default returns the root data provider. By default this is a vfs.LocalFileSystem. Consider to reconfigure it to a vfs.MountableFileSystem which allows arbitrary prefixes (also called mount points). Use it to configure and setup a virtualized filesystem structure for your app.

Best practice

  • Mount your static app data into /assets
  • Implement variant and localization data at mount time not runtime, e.g. /assets contains the data for a specific client with a specific locale instead of a manual lookup e.g. in /assets/customer/DE_de. Keep your code clean.
  • Mount your user specific data into something like /media/local and /media/ftp and /media/gphotos etc.

type LimitDetails added in v0.0.7

type LimitDetails interface {
	// A specific message, useful for the user
	UserMessage() string
	// the required minimal value
	Min() int64
	// the actual used resources
	Used() int64
	// the maximum available resources
	Max() int64
}

LimitDetails represents a contract to access a min, current and max occupation of a resource.

type List added in v0.0.7

type List struct {
	// contains filtered or unexported fields
}

A List covers an interface slice

func (*List) Add added in v0.0.7

func (l *List) Add(v interface{})

func (*List) Size added in v0.0.7

func (l *List) Size() int

type MountableFileSystem added in v0.0.4

type MountableFileSystem struct {
	// contains filtered or unexported fields
}

A MountableFileSystem contains only other DataProviders mounted under a path. Mounting cross paths is not supported.

Example

If you have /my/dir/provider0 and mount /my/dir/provider0/some/dir/provider1 the existing provider0 will be removed.

func (*MountableFileSystem) AddListener added in v0.0.7

func (p *MountableFileSystem) AddListener(ctx context.Context, path string, listener ResourceListener) (handle int, err error)

func (*MountableFileSystem) Begin added in v0.0.7

func (p *MountableFileSystem) Begin(ctx context.Context, path string, options interface{}) (context.Context, error)

func (*MountableFileSystem) Close added in v0.0.4

func (p *MountableFileSystem) Close() error

Close does nothing.

func (*MountableFileSystem) Commit added in v0.0.7

func (p *MountableFileSystem) Commit(ctx context.Context) error

func (*MountableFileSystem) Connect added in v0.0.7

func (p *MountableFileSystem) Connect(ctx context.Context, path string, options interface{}) (interface{}, error)

func (*MountableFileSystem) Delete added in v0.0.4

func (p *MountableFileSystem) Delete(ctx context.Context, path string) error

func (*MountableFileSystem) Disconnect added in v0.0.7

func (p *MountableFileSystem) Disconnect(ctx context.Context, path string) error

Disconnect tries to dispatch the call to a mounted vfs. In any case, the vfs (leaf) is removed from the tree, even if disconnect has returned an error. If you need a different behavior, you can use #Resolve() to grab the actual vfs instance.

func (*MountableFileSystem) FireEvent added in v0.0.7

func (p *MountableFileSystem) FireEvent(ctx context.Context, path string, event interface{}) error
func (p *MountableFileSystem) HardLink(ctx context.Context, oldPath string, newPath string) error

func (*MountableFileSystem) Invoke added in v0.0.7

func (p *MountableFileSystem) Invoke(ctx context.Context, endpoint string, args ...interface{}) (interface{}, error)

Invoke also relies on the prefixed endpoint

func (*MountableFileSystem) MkBucket added in v0.0.7

func (p *MountableFileSystem) MkBucket(ctx context.Context, path string, options interface{}) error

func (*MountableFileSystem) Mount added in v0.0.4

func (p *MountableFileSystem) Mount(mountPoint Path, provider FileSystem)

Mount includes the given provider into the leaf of the path. Important: you cannot mount one provider into another.

func (*MountableFileSystem) Mounted added in v0.0.7

func (p *MountableFileSystem) Mounted(path string) FileSystem

Mounted returns the mounted filesystem or nil if the path cannot be resolved to a mountpoint.

func (*MountableFileSystem) Open added in v0.0.4

func (p *MountableFileSystem) Open(ctx context.Context, path string, flag int, options interface{}) (Blob, error)

func (*MountableFileSystem) ReadAttrs added in v0.0.4

func (p *MountableFileSystem) ReadAttrs(ctx context.Context, path string, args interface{}) (Entry, error)

func (*MountableFileSystem) ReadBucket added in v0.0.7

func (p *MountableFileSystem) ReadBucket(ctx context.Context, path string, options interface{}) (ResultSet, error)

func (*MountableFileSystem) ReadForks added in v0.0.7

func (p *MountableFileSystem) ReadForks(ctx context.Context, path string) ([]string, error)
func (p *MountableFileSystem) RefLink(ctx context.Context, oldPath string, newPath string) error

RefLink is like RefLink

func (*MountableFileSystem) RemoveListener added in v0.0.7

func (p *MountableFileSystem) RemoveListener(ctx context.Context, handle int) error

func (*MountableFileSystem) Rename added in v0.0.4

func (p *MountableFileSystem) Rename(ctx context.Context, oldPath string, newPath string) error

func (*MountableFileSystem) Resolve added in v0.0.4

func (p *MountableFileSystem) Resolve(path string) (mountPoint string, providerPath string, provider FileSystem, err error)

Resolve searches the virtual structure and returns a provider and the according data or nil and empty paths

func (*MountableFileSystem) Rollback added in v0.0.7

func (p *MountableFileSystem) Rollback(ctx context.Context) error

func (*MountableFileSystem) String added in v0.0.7

func (p *MountableFileSystem) String() string
func (p *MountableFileSystem) SymLink(ctx context.Context, oldPath string, newPath string) error

func (*MountableFileSystem) WriteAttrs added in v0.0.4

func (p *MountableFileSystem) WriteAttrs(ctx context.Context, path string, src interface{}) (Entry, error)

type Path

type Path string

A Path must be unique in it's context and has the role of a composite key. It's segments are always separated using a slash, even if they denote paths from windows.

Example

Valid example paths

  • /my/path/may/denote/a/file/or/folder
  • c:/my/windows/folder
  • mydomain.com/myresource
  • mydomain.com:8080/myresource?size=720p#anchor
  • c:/my/ntfs/file:alternate-data-stream

Invalid example paths

Design decisions

There are the following opinionated decisions:

  • In the context of a filesystem, this is equal to the full qualified name of a file entry.

  • It is a string, because defacto all modern APIs are UTF-8 and web based. However there are also a lot of Unix or Linux types which have different local encodings or just have an undefined byte sequence. Providers with such requirements must support the path API through some kind of conversion and normalization, but they should also provide an exact API using byte slices then. One could also argue, that a string is a bad choice for Go, because of these corner case, potential invalid utf-8 sequences and suboptimal string allocations. But using just byte-slices by default would make a lot of things even worse:

  • You cannot simply compare byte slices in Go. You need to compare and acknowledge about a new standard.

  • It can be expected that the developer using this library will certainly need a string representation which will practically always cause additional allocations.

  • Because a path is naturally always a string, you certainly want to use all the provided and standard string handling infrastructures instead of reinventing your own.

  • There are studies which claim that the average filename is between 11 and 15 characters long. Because we want to optimize use cases like keeping 1 million file names in memory, using a fixed size 256 byte array would result in a 17x overhead of memory usage: e.g. 17GiB instead of 1GiB of main memory. To save even more memory and lower GC pressure, we do not use a slice of strings but just a pure string providing helper methods.

func ConcatPaths

func ConcatPaths(paths ...Path) Path

ConcatPaths merges all paths together

func (Path) Add added in v0.0.4

func (p Path) Add(path Path) Path

Add returns this path concated with the given path. Sadly we have no overloading operator.

func (Path) Child

func (p Path) Child(name string) Path

Child returns a new Path with name appended as a child

func (Path) EndsWith

func (p Path) EndsWith(suffix Path) bool

EndsWith tests whether the path ends with prefix.

func (Path) Name

func (p Path) Name() string

Id returns the last element in this path or the empty string if this path is empty.

func (Path) NameAt

func (p Path) NameAt(idx int) string

NameAt returns the name at the given index.

func (Path) NameCount

func (p Path) NameCount() int

NameCount returns how many names are included in this path.

func (Path) Names

func (p Path) Names() []string

Names splits the path by / and returns all segments as a simple string array.

func (Path) Normalize

func (p Path) Normalize() Path

Normalize removes any . and .. and returns a Path without those elements.

func (Path) Parent

func (p Path) Parent() Path

Parent returns the parent path of this path.

func (Path) Resolve added in v0.0.7

func (p Path) Resolve(baseDir Path) Path

Resolve takes the base dir and normalizes this path and returns it. E.g.

  1. "/my/path".Resolve("/some/thing") => /my/path
  2. "./my/path".Resolve("/some/thing") => /some/thing/my/path
  3. "my/path".Resolve("/some/thing") => /some/thing/my/path
  4. "my/path".Resolve("/some/thing") => /some/thing/my/path
  5. "my/path/../../".Resolve("/some/thing") => /some/thing

func (Path) StartsWith

func (p Path) StartsWith(prefix Path) bool

StartsWith tests whether the path begins with prefix.

func (Path) String

func (p Path) String() string

String normalizes the slashes in Path

func (Path) TrimPrefix

func (p Path) TrimPrefix(prefix Path) Path

TrimPrefix returns a path without the prefix

type PathEntry

type PathEntry struct {
	Path  string
	Entry Entry
}

A PathEntry simply provides a Path and the related information

func ReadBucketRecur added in v0.0.7

func ReadBucketRecur(path string) ([]*PathEntry, error)

ReadBucketRecur fully reads the given directory recursively and returns Entries with full qualified paths.

func (*PathEntry) Equals added in v0.0.3

func (e *PathEntry) Equals(other interface{}) bool

Equals checks for equality with another PathEntry

type ResourceListener added in v0.0.7

type ResourceListener interface {
	// OnStatusChanged is called e.g. when a file has been added, removed or changed. event contains
	// implementation specific information. There are many examples of detailed changed events, which
	// we don't like to specify, like onRead, before read, after read, renaming, meta data, quota, lock status,
	// target link, ownership, truncated, ...
	//
	// It is also not defined, when an event is fired and received. However there may be implementation which
	// provide a "before" semantic and may evaluate any returned error, so that a ResourceListener can be used
	// as an interceptor or in a kind of aspect oriented programming.
	OnEvent(path string, event interface{}) error
}

A ResourceListener is used to get notified about changes

type ResultSet added in v0.0.7

type ResultSet interface {
	// ReadAttrs returns the entry for the index >= 0 and < Len() of an already queried response, which
	// is why there is no context here. However args is still in the contract to allows implementation
	// specific allocation free data transformation.
	//
	// See also FileSystem#ReadAttrs()
	ReadAttrs(idx int, args interface{}) Entry

	// Len returns the amount of Entries in the entire result set, which are available without any further I/O
	Len() int

	// Total is the estimated amount of all Entries when all pages have been requested.
	// Is -1 if unknown and you definitely have to loop over to count.
	// However in the meantime, Size may deliver a more correct estimation.
	Total() int64

	// Pages is the estimated amount of all available pages, including the current one. Is -1 if unknown.
	Pages() int64

	// Next loads the next page of results or EOF if no more results are available. The ResultSet is
	// undefined, if an error has been returned. Otherwise you have to evaluate #Len() again and loop over
	// the set again to grab the next results.
	Next(ctx context.Context) error

	// Data returns the actual model behind this list. Implementations which wrap REST APIs typically return
	// a map[string]interface{} or []interface{}. Can return nil. It is called Data to be compatible with os.FileInfo.Data
	Sys() interface{}
}

An EntryList is a collection of loaded bucket Entries, whose Entries are typically the names of other Buckets or Entries. We do not include the ReadForks method, because the determination of available forks may be expensive and usually requires additional I/O. Logically it belongs to the FileSystem#Open() call and is therefore not related to the ResultSet.

type Router added in v0.0.8

type Router struct {
	// contains filtered or unexported fields
}

A Router has a set of patterns which can be registered to be matched in the order of configuration.

func (*Router) Dispatch added in v0.0.8

func (r *Router) Dispatch(ctx context.Context, path Path, args ...interface{}) (interface{}, error)

Dispatch tries to find the correct matcher for the given path. The first matching callback is invoked or if nothing matches, nothing is called at all (and false is returned). Returns io.EOF if no matcher can be applied.

func (*Router) DispatchBlob added in v0.0.12

func (r *Router) DispatchBlob(ctx context.Context, path Path) (Blob, error)

DispatchBlob is required to workaround missing generics

func (*Router) DispatchEntry added in v0.0.12

func (r *Router) DispatchEntry(ctx context.Context, path Path, args ...interface{}) (Entry, error)

DispatchEntry is required to workaround missing generics

func (*Router) DispatchResultSet added in v0.0.12

func (r *Router) DispatchResultSet(ctx context.Context, path Path) (ResultSet, error)

DispatchResultSet is required to workaround missing generics

func (*Router) Match added in v0.0.12

func (r *Router) Match(pattern string, callback func(ctx RoutingContext) (interface{}, error))

Match registers an arbitrary function with a pattern with injection-like semantics.

Supported patterns are:

  • * : matches everything
  • /a/concrete/path : matches the exact path
  • /{name} : matches anything like /a or /b
  • /fix/{var}/fix : matches anything like /fix/a/fix or /fix/b/fix
  • /fix/fix2/* : matches anything like /fix/fix2 or /fix/fix2/a/b/

func (*Router) MatchBlob added in v0.0.12

func (r *Router) MatchBlob(pattern string, f func(ctx RoutingContext) (Blob, error))

MatchBlob is required to workaround missing generics

func (*Router) MatchEntry added in v0.0.12

func (r *Router) MatchEntry(pattern string, f func(ctx RoutingContext) (Entry, error))

MatchEntry is required to workaround missing generics

func (*Router) MatchResultSet added in v0.0.12

func (r *Router) MatchResultSet(pattern string, f func(ctx RoutingContext) (ResultSet, error))

MatchResultSet is required to workaround missing generics

type RoutingContext added in v0.0.8

type RoutingContext interface {
	// ValueOf returns the string value of a named parameter or the empty string if undefined
	ValueOf(name string) string

	// Path returns the actual path
	Path() Path

	// Args may contains additional arguments passed by invocation/dispatching
	Args() []interface{}

	// Context returns the golang execution context
	Context() context.Context
}

The RoutingContext is used to represent a dispatching context or state.

type StrList added in v0.0.7

type StrList struct {
	List
}

A StrList is an ArrayList of strings

func (*StrList) Add added in v0.0.7

func (l *StrList) Add(v string)

type UnavailableDetails added in v0.0.7

type UnavailableDetails interface {
	// A specific message, useful for the user
	UserMessage() string
	// RetryAfter returns the duration to wait before
	RetryAfter() time.Duration
}

UnavailableDetails indicates a temporary downtime and communicates a retry time.

type WalkClosure

type WalkClosure func(path string, info Entry, err error) error

A WalkClosure is invoked for each entry in Walk, as long as no error is returned and Entries are available.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL