qfs

package module
v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 4, 2021 License: MIT Imports: 16 Imported by: 36

Documentation

Index

Constants

View Source
const MemFilestoreType = "mem"

MemFilestoreType uniquely identifies the mem filestore

Variables

View Source
var (
	// ErrNotDirectory is the result of attempting to perform "directory-like" operations on a file
	ErrNotDirectory = errors.New("This file is not a directory")
	// ErrNotFile is the result of attempting to perform "file like" operations on a directory
	ErrNotFile = errors.New("This is a directory")
)
View Source
var (

	// ErrNotFound is the canonical error for not finding a value
	ErrNotFound = errors.New("path not found")
	// ErrReadOnly is a sentinel value for Filesystems that aren't writable
	ErrReadOnly = errors.New("readonly filesystem")
)
View Source
var ErrNotAddingFS = errors.New("this filesystem doesn't support batched adding")

ErrNotAddingFS is the canonical error to return when the AddingFS extension interface is required but not available

Functions

func AbsPath

func AbsPath(path *string) (err error)

AbsPath adjusts the provided string to a path lib functions can work with because paths for Qri can come from the local filesystem, an http url, or the distributed web, Absolutizing is a little tricky

If lib in put params call for a path, running input through AbsPath before calling a lib function should help reduce errors. calling AbsPath on empty string has no effect

func PathKind

func PathKind(path string) string

PathKind estimates what type of resolver string path is referring to

func Walk

func Walk(root File, visit func(f File) error) (err error)

Walk traverses a file tree from the bottom-up calling visit on each file and directory within the tree

func WriteWithHooks added in v0.6.0

func WriteWithHooks(ctx context.Context, fs Filesystem, root File) (string, error)

WriteWithHooks writes a file or directory to a given filestore using merkelization hooks failed writes are rolled back with delete requests for all added files

Types

type AddedFile added in v0.6.0

type AddedFile struct {
	Path  string
	Name  string
	Bytes int64
	Hash  string
	Size  string
}

AddedFile reports on the results of adding a file to the store

type Adder added in v0.6.0

type Adder interface {
	// AddFile adds a file or directory of files to the store
	// this function will return immediately, consumers should read
	// from the Added() channel to see the results of file addition.
	AddFile(context.Context, File) error
	// Added gives a channel to read added files from.
	Added() chan AddedFile
	// In IPFS land close calls adder.Finalize() and adder.PinRoot()
	// (files will only be pinned if the pin flag was set on NewAdder)
	Finalize() (string, error)
}

Adder is the interface for adding files to a Filestore. The addition process is parallelized. Implementers must make all required AddFile calls, then call Close to finalize the addition process. Progress can be monitored through the Added() channel, which emits once per written file

type AddingFS added in v0.6.0

type AddingFS interface {
	// NewAdder allocates an Adder instance for adding files to the filestore
	// Adder gives a higher degree of control over the file adding process at the
	// cost of being harder to work with.
	// "pin" is a flag for recursively pinning this object
	// "wrap" sets weather the top level should be wrapped in a directory
	NewAdder(ctx context.Context, pin, wrap bool) (Adder, error)
}

AddingFS is an interface for filesystems that support batched adding

type CAFS added in v0.6.0

type CAFS interface {
	IsContentAddressedFilesystem()
}

CAFS stands for "content-addressed filesystem". Filesystem that implement this interface declare that all paths to persisted content are reference-by -hash. TODO (b5) - write up a spec test suite for CAFS conformance

type Config added in v0.4.2

type Config struct {
	Type   string                 `json:"type"`
	Config map[string]interface{} `json:"config,omitempty"`
}

Config binds a filesystem type to a configuration map

type Constructor added in v0.4.2

type Constructor func(ctx context.Context, cfg map[string]interface{}) (Filesystem, error)

Constructor is a function that creates a filesystem from a config map the passed in context should last for the duration of the existence of the store. Any resources allocated by the store should be scoped to this context

type Destroyer

type Destroyer interface {
	Destroy() error
}

Destroyer is an optional interface to tear down a filesystem, removing all persisted resources

type File

type File interface {
	// Files implement ReadCloser, but can only be read from or closed if
	// they are not directories
	io.ReadCloser

	// FileName returns a filename associated with this file
	// TODO (b5): consider renaming this to Base
	FileName() string

	// FullPath returns the full path used when adding this file
	FullPath() string

	// IsDirectory returns true if the File is a directory (and therefore
	// supports calling `NextFile`) and false if the File is a normal file
	// (and therefor supports calling `Read` and `Close`)
	IsDirectory() bool

	// NextFile returns the next child file available (if the File is a
	// directory). It will return (nil, io.EOF) if no more files are
	// available. If the file is a regular file (not a directory), NextFile
	// will return a non-nil error.
	NextFile() (File, error)

	// ModTime returns file modification time
	ModTime() time.Time

	// MediaType returns a is a string that indicates the nature and format of a
	// document, file, or assortment of bytes. Media types are described in IETF
	// RFC 6838: https://tools.ietf.org/html/rfc6838
	// MediaTypes expand on Multipurpose Internet Mail Extensions or MIME types,
	// and can include "non-official" media type responses
	MediaType() string
}

File is an interface that provides functionality for handling files/directories as values that can be supplied to commands. For directories, child files are accessed serially by calling `NextFile()`.

func FileString

func FileString(f File) (File, string)

FileString is a utility function that consumes a file, returning a sctring of file byte contents. This is for debugging purposes only, and should never be used for-realsies, as it pulls the *entire* file into a byte slice

func NewWriteHookFile added in v0.6.0

func NewWriteHookFile(file File, cb WriteHook, requiredPaths ...string) File

NewWriteHookFile wraps a File with a hook & set of sibling / child dependencies

type Filesystem

type Filesystem interface {
	// Type returns a string identifier that distinguishes a filesystem from
	// all other implementations, example identifiers include: "local", "ipfs",
	// and "http"
	// types are used as path prefixes when multiplexing filesystems
	Type() string
	// Has returns whether the `path` is mapped to a value.
	// In some contexts, it may be much cheaper only to check for existence of
	// a value, rather than retrieving the value itself. (e.g. HTTP HEAD).
	// The default implementation is found in `GetBackedHas`.
	Has(ctx context.Context, path string) (exists bool, err error)
	// Get fetching files and directories from path strings.
	// in practice path strings can be things like:
	// * a local filesystem
	// * URLS (a "URL path resolver") or
	// * content-addressed file systems like IPFS or Git
	// Datasets & dataset components use a filesource to resolve string references
	Get(ctx context.Context, path string) (File, error)
	// Put places a file or directory on the filesystem, returning the root path.
	// The returned path may or may not honor the path of the given file
	Put(ctx context.Context, file File) (path string, err error)
	// Delete removes a file or directory from the filesystem
	Delete(ctx context.Context, path string) (err error)
}

Filesystem abstracts & unifies filesystem-like behaviour

func NewMemFilesystem added in v0.4.2

func NewMemFilesystem(_ context.Context, cfg map[string]interface{}) (Filesystem, error)

NewMemFilesystem allocates an instace of a mapstore that can be used as a PathResolver satisfies the FSConstructor interface

type MemFS

type MemFS struct {
	Pinned  bool
	Network []*MemFS

	Files map[string]filer
	// contains filtered or unexported fields
}

MemFS implements Filestore in-memory as a map

An example pulled from tests will create a tree of "cafs" with directories & cafs, with paths properly set: NewMemdir("/a",

NewMemfileBytes("a.txt", []byte("foo")),
NewMemfileBytes("b.txt", []byte("bar")),
NewMemdir("/c",
	NewMemfileBytes("d.txt", []byte("baz")),
	NewMemdir("/e",
		NewMemfileBytes("f.txt", []byte("bat")),
	),
),

) File is an interface that provides functionality for handling cafs/directories as values that can be supplied to commands.

func NewMemFS

func NewMemFS() *MemFS

NewMemFS allocates an instance of a mapstore

func (*MemFS) AddConnection added in v0.6.0

func (m *MemFS) AddConnection(other *MemFS)

AddConnection sets up pointers from this MapStore to that, and vice versa.

func (*MemFS) Delete added in v0.6.0

func (m *MemFS) Delete(ctx context.Context, key string) error

Delete removes the file from the store with the key

func (*MemFS) Get

func (m *MemFS) Get(ctx context.Context, key string) (File, error)

Get returns a File from the store

func (*MemFS) Has added in v0.6.0

func (m *MemFS) Has(ctx context.Context, key string) (exists bool, err error)

Has returns whether the store has a File with the key

func (*MemFS) IsContentAddressedFilesystem added in v0.6.0

func (m *MemFS) IsContentAddressedFilesystem()

func (*MemFS) NewAdder added in v0.6.0

func (m *MemFS) NewAdder(ctx context.Context, pin, wrap bool) (Adder, error)

NewAdder returns an Adder for the store

func (*MemFS) ObjectCount added in v0.6.0

func (m *MemFS) ObjectCount() (objects int)

ObjectCount returns the number of content-addressed objects in the store

func (*MemFS) Print added in v0.6.0

func (m *MemFS) Print() (string, error)

Print converts the store to a string

func (*MemFS) Put added in v0.6.0

func (m *MemFS) Put(ctx context.Context, file File) (key string, err error)

Put adds a file to the store

func (*MemFS) PutFileAtKey added in v0.6.0

func (m *MemFS) PutFileAtKey(ctx context.Context, key string, file File) error

PutFileAtKey puts the file at the given key Deprecated - this method breaks CAFS interface assertions. Don't use it.

func (*MemFS) Type added in v0.6.0

func (m *MemFS) Type() string

Type distinguishes this filesystem from others by a unique string prefix

type Memdir

type Memdir struct {
	// contains filtered or unexported fields
}

Memdir is an in-memory directory Currently it only supports either Memfile & Memdir as links

func NewMemdir

func NewMemdir(path string, links ...File) *Memdir

NewMemdir creates a new Memdir, supplying zero or more links

func (*Memdir) AddChildren

func (m *Memdir) AddChildren(fs ...File)

AddChildren allows any sort of file to be added, but only implementations that implement the PathSetter interface will have properly configured paths.

func (*Memdir) ChildDir

func (m *Memdir) ChildDir(dirname string) *Memdir

ChildDir returns a child directory at dirname

func (Memdir) Close

func (Memdir) Close() error

Close does nothing, exists so MemDir implements the File interface

func (Memdir) FileName

func (m Memdir) FileName() string

FileName returns the base of File's internal path

func (Memdir) FullPath

func (m Memdir) FullPath() string

FullPath returns the entire path string

func (Memdir) IsDirectory

func (Memdir) IsDirectory() bool

IsDirectory returns true to indicate MemDir is a Directory

func (*Memdir) MakeDirP

func (m *Memdir) MakeDirP(f File) *Memdir

MakeDirP ensures all directories specified by the given file exist, returning the deepest directory specified

func (*Memdir) MediaType added in v0.4.2

func (m *Memdir) MediaType() string

MediaType is a directory mime-type stand-in

func (*Memdir) ModTime added in v0.4.2

func (m *Memdir) ModTime() time.Time

ModTime returns the last-modified time for this directory TODO (b5) - should modifying children affect this timestamp?

func (*Memdir) NextFile

func (m *Memdir) NextFile() (File, error)

NextFile iterates through each File in the directory on successive calls to File Returning io.EOF when no files remain

func (Memdir) Read

func (Memdir) Read([]byte) (int, error)

Read does nothing, exists so MemDir implements the File interface

func (*Memdir) SetPath

func (m *Memdir) SetPath(path string)

SetPath implements the PathSetter interface

type Memfile

type Memfile struct {
	// contains filtered or unexported fields
}

Memfile is an in-memory file

func NewMemfileBytes

func NewMemfileBytes(path string, data []byte) *Memfile

NewMemfileBytes creates a file from a byte slice

func NewMemfileReader

func NewMemfileReader(path string, r io.Reader) *Memfile

NewMemfileReader creates a file from an io.Reader

func (Memfile) Close

func (m Memfile) Close() error

Close closes the file, if the backing reader implements the io.Closer interface it will call close on the backing Reader

func (Memfile) FileName

func (m Memfile) FileName() string

FileName returns the base of File's internal path

func (Memfile) FullPath

func (m Memfile) FullPath() string

FullPath returns the entire path string

func (Memfile) IsDirectory

func (Memfile) IsDirectory() bool

IsDirectory always returns false 'cause Memfile is a file

func (Memfile) MediaType added in v0.4.2

func (m Memfile) MediaType() string

MediaType for a memfile returns a mime type based on file extension

func (Memfile) ModTime added in v0.4.2

func (m Memfile) ModTime() time.Time

ModTime returns the last-modified time for this file

func (Memfile) NextFile

func (Memfile) NextFile() (File, error)

NextFile does nothing 'cuse Memfile isn't a directory

func (Memfile) Read

func (m Memfile) Read(p []byte) (int, error)

Read implements the io.Reader interface

func (*Memfile) SetPath

func (m *Memfile) SetPath(path string)

SetPath implements the PathSetter interface

type PathResolver

type PathResolver interface {
	Get(ctx context.Context, path string) (File, error)
}

PathResolver is the "get" portion of a Filesystem

type PathSetter

type PathSetter interface {
	SetPath(path string)
}

PathSetter adds the capacity to modify a path property

type PinningFS added in v0.6.0

type PinningFS interface {
	Pin(ctx context.Context, key string, recursive bool) error
	Unpin(ctx context.Context, key string, recursive bool) error
}

PinningFS interface for content stores that support the concept of pinnings

type ReleasingFilesystem added in v0.4.2

type ReleasingFilesystem interface {
	Filesystem
	Done() <-chan struct{}
	DoneErr() error
}

ReleasingFilesystem provides a channel to signal cleanup is finished. It sends after a filesystem has closed & about to release all it's resources

type WriteHook added in v0.6.0

type WriteHook func(ctx context.Context, f File, added map[string]string) (io.Reader, error)

WriteHook is a function that's called when a given path has been written to the content addressed filesystem

type WriteHookFile added in v0.6.0

type WriteHookFile interface {
	File
	RequiredPaths() []string
	HasRequiredPaths(paths map[string]string) bool
	CallAndAdd(ctx context.Context, adder Adder, added map[string]string) error
}

WriteHookFile is a file that can hook into the merkelization process, affecting contents as contents are being rendered immutable

Directories

Path Synopsis
cafs is a "content-addressed-file-systen", which is a generalized interface for working with content-addressed filestores.
cafs is a "content-addressed-file-systen", which is a generalized interface for working with content-addressed filestores.
coreunix
Package coreunix is a fork of github.com/ipfs/go-ipfs/core/coreunix gx paths used: "gx/ipfs/QmUJYo4etAQqFfSS2rarFAE97eNGB8ej64YkRT2SmsYD4r/go-ipfs/pin" ipld "gx/ipfs/QmR7TcHkR9nxkUorfi8XMTAMLUK7GiP64TWWBzY3aacc1o/go-ipld-format" dag "gx/ipfs/QmSei8kFMfqdJq7Q68d2LMnHbTWKKg2daA29ezUYFAUNgc/go-merkledag" posinfo "gx/ipfs/QmQyUyYcpKG1u53V7N25qRTGw5XwaAxTMKXbduqHotQztg/go-ipfs-posinfo" core "gx/ipfs/QmUJYo4etAQqFfSS2rarFAE97eNGB8ej64YkRT2SmsYD4r/go-ipfs/core" unixfs "gx/ipfs/QmfB3oNXGGq9S4B2a9YeCajoATms3Zw2VvDm8fK7VeLSV8/go-unixfs" balanced "gx/ipfs/QmfB3oNXGGq9S4B2a9YeCajoATms3Zw2VvDm8fK7VeLSV8/go-unixfs/importer/balanced" ihelper "gx/ipfs/QmfB3oNXGGq9S4B2a9YeCajoATms3Zw2VvDm8fK7VeLSV8/go-unixfs/importer/helpers" trickle "gx/ipfs/QmfB3oNXGGq9S4B2a9YeCajoATms3Zw2VvDm8fK7VeLSV8/go-unixfs/importer/trickle" coreiface "gx/ipfs/QmUJYo4etAQqFfSS2rarFAE97eNGB8ej64YkRT2SmsYD4r/go-ipfs/core/coreapi/interface" posinfo "gx/ipfs/QmQyUyYcpKG1u53V7N25qRTGw5XwaAxTMKXbduqHotQztg/go-ipfs-posinfo" cid "gx/ipfs/QmPSQnBKM9g7BaUcZCvswUJVscQ1ipjmwxN5PXCjkp9EQ7/go-cid" mfs "gx/ipfs/QmUwXQs8aZ472DmXZ8uJNf7HJNKoMJQVa7RaCz7ujZ3ua9/go-mfs" chunker "gx/ipfs/QmTUTG9Jg9ZRA1EzTPGTDvnwfcfKhDMnqANnP9fe4rSjMR/go-ipfs-chunker" logging "gx/ipfs/QmZChCsSt8DctjceaL56Eibc29CVQq4dGKRXC5JRZ6Ppae/go-log" bstore "gx/ipfs/QmcDDgAXDbpDUpadCJKLr49KYR4HuL7T8Z1dZTHt6ixsoR/go-ipfs-blockstore" files "gx/ipfs/QmZMWMvWMVKCbHetJ4RgndbuEF1io2UpUxwQwtNjtYPzSC/go-ipfs-files"
Package coreunix is a fork of github.com/ipfs/go-ipfs/core/coreunix gx paths used: "gx/ipfs/QmUJYo4etAQqFfSS2rarFAE97eNGB8ej64YkRT2SmsYD4r/go-ipfs/pin" ipld "gx/ipfs/QmR7TcHkR9nxkUorfi8XMTAMLUK7GiP64TWWBzY3aacc1o/go-ipld-format" dag "gx/ipfs/QmSei8kFMfqdJq7Q68d2LMnHbTWKKg2daA29ezUYFAUNgc/go-merkledag" posinfo "gx/ipfs/QmQyUyYcpKG1u53V7N25qRTGw5XwaAxTMKXbduqHotQztg/go-ipfs-posinfo" core "gx/ipfs/QmUJYo4etAQqFfSS2rarFAE97eNGB8ej64YkRT2SmsYD4r/go-ipfs/core" unixfs "gx/ipfs/QmfB3oNXGGq9S4B2a9YeCajoATms3Zw2VvDm8fK7VeLSV8/go-unixfs" balanced "gx/ipfs/QmfB3oNXGGq9S4B2a9YeCajoATms3Zw2VvDm8fK7VeLSV8/go-unixfs/importer/balanced" ihelper "gx/ipfs/QmfB3oNXGGq9S4B2a9YeCajoATms3Zw2VvDm8fK7VeLSV8/go-unixfs/importer/helpers" trickle "gx/ipfs/QmfB3oNXGGq9S4B2a9YeCajoATms3Zw2VvDm8fK7VeLSV8/go-unixfs/importer/trickle" coreiface "gx/ipfs/QmUJYo4etAQqFfSS2rarFAE97eNGB8ej64YkRT2SmsYD4r/go-ipfs/core/coreapi/interface" posinfo "gx/ipfs/QmQyUyYcpKG1u53V7N25qRTGw5XwaAxTMKXbduqHotQztg/go-ipfs-posinfo" cid "gx/ipfs/QmPSQnBKM9g7BaUcZCvswUJVscQ1ipjmwxN5PXCjkp9EQ7/go-cid" mfs "gx/ipfs/QmUwXQs8aZ472DmXZ8uJNf7HJNKoMJQVa7RaCz7ujZ3ua9/go-mfs" chunker "gx/ipfs/QmTUTG9Jg9ZRA1EzTPGTDvnwfcfKhDMnqANnP9fe4rSjMR/go-ipfs-chunker" logging "gx/ipfs/QmZChCsSt8DctjceaL56Eibc29CVQq4dGKRXC5JRZ6Ppae/go-log" bstore "gx/ipfs/QmcDDgAXDbpDUpadCJKLr49KYR4HuL7T8Z1dZTHt6ixsoR/go-ipfs-blockstore" files "gx/ipfs/QmZMWMvWMVKCbHetJ4RgndbuEF1io2UpUxwQwtNjtYPzSC/go-ipfs-files"

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL