cafs

package
v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 4, 2021 License: MIT, MIT Imports: 10 Imported by: 0

README

cafs

Qri GoDoc License Codecov CI

cafs stands for "content-addressed-file-system", which is a generalized interface for working with filestores that determine names content based on the content itself. Examples of a content-addressed file systems include git, IPFS, the DAT project, and like, blockchains or whatever. The long-term goal of cafs is to decouple & interoperate common filestore operations between different content-addressed filestores. This package doesn't aim to implement everything a given filestore can do, but instead focus on basic file & directory i/o. cafs is very early days, starting with a proof of concept based on IPFS and an in-memory implementation. Over time we'll work to add additional stores, which will undoubtably affect the overall interface definition.

Getting Involved

We would love involvement from more people! If you notice any errors or would like to submit changes, please see our Contributing Guidelines.

Documentation

Overview

cafs is a "content-addressed-file-systen", which is a generalized interface for working with content-addressed filestores. real-on-the-real, this is a wrapper for IPFS. It looks a lot like the ipfs datastore interface, except the datastore itself determines keys.

Index

Constants

View Source
const MapFilestoreType = "map"

MapFilestoreType uniquely identifies the map filestore

Variables

View Source
var (
	// ErrNotFound is the canonical error for not finding a value
	ErrNotFound = errors.New("cafs: path not found")
)
View Source
var (
	// SourceAny specifies that content can come from anywhere
	SourceAny = source("any")
)

Functions

func NewMapFilesystem added in v0.4.2

func NewMapFilesystem(_ context.Context, _ map[string]interface{}) (qfs.Filesystem, error)

NewMapFilesystem satisfies the qfs.FSConstructor interface

Types

type AddedFile

type AddedFile struct {
	Path  string
	Name  string
	Bytes int64
	Hash  string
	Size  string
}

AddedFile reports on the results of adding a file to the store TODO - add filepath to this struct

type Adder

type Adder interface {
	// AddFile adds a file or directory of files to the store
	// this function will return immideately, consumers should read
	// from the Added() channel to see the results of file addition.
	AddFile(context.Context, qfs.File) error
	// Added gives a channel to read added files from.
	Added() chan AddedFile
	// In IPFS land close calls adder.Finalize() and adder.PinRoot()
	// (files will only be pinned if the pin flag was set on NewAdder)
	// Close will close the underlying
	Close() error
}

Adder is the interface for adding files to a Filestore. The addition process is parallelized. Implementers must make all required AddFile calls, then call Close to finalize the addition process. Progress can be monitored through the Added() channel

type Fetcher

type Fetcher interface {
	// Fetch gets a file from a source
	Fetch(ctx context.Context, source Source, key string) (qfs.File, error)
}

Fetcher is the interface for getting files from a remote source filestores can opt into the fetcher interface

type Filestore

type Filestore interface {
	// Put places a file or a directory in the store.
	// The most notable difference from a standard file store is the store itself
	// determines the resulting path. paths returned by put must be prefixed with
	// the type:
	// eg. /ipfs/QmZ3KfGaSrb3cnTriJbddCzG7hwQi2j6km7Xe7hVpnsW5S
	Put(ctx context.Context, file qfs.File) (path string, err error)

	// Get retrieves the object `value` named by `key`.
	// Get will return ErrNotFound if the key is not mapped to a value.
	Get(ctx context.Context, path string) (file qfs.File, err error)

	// Has returns whether the `key` is mapped to a `value`.
	// In some contexts, it may be much cheaper only to check for existence of
	// a value, rather than retrieving the value itself. (e.g. HTTP HEAD).
	// The default implementation is found in `GetBackedHas`.
	Has(ctx context.Context, path string) (exists bool, err error)

	// Delete removes the value for given `key`.
	Delete(ctx context.Context, path string) error

	// NewAdder allocates an Adder instance for adding files to the filestore
	// Adder gives a higher degree of control over the file adding process at the
	// cost of being harder to work with.
	// "pin" is a flag for recursively pinning this object
	// "wrap" sets weather the top level should be wrapped in a directory
	NewAdder(pin, wrap bool) (Adder, error)

	// Type is a top-level identifier to distinguish between filestores,
	// for exmple: the "ipfs" in /ipfs/QmZ3KfGaSrb3cnTriJbddCzG7hwQi2j6km7Xe7hVpnsW5S
	// a Filestore implementation should always return the same
	Type() string
}

Filestore is an interface for working with a content-addressed file system. This interface is under active development, expect it to change lots. It's currently form-fitting around IPFS (ipfs.io), with far-off plans to generalize toward compatibility with git (git-scm.com), then maybe other stuff, who knows.

type MapStore

type MapStore struct {
	Pinned  bool
	Network []*MapStore
	Files   map[string]filer
}

MapStore implements Filestore in-memory as a map

An example pulled from tests will create a tree of "cafs" with directories & cafs, with paths properly set: NewMemdir("/a",

NewMemfileBytes("a.txt", []byte("foo")),
NewMemfileBytes("b.txt", []byte("bar")),
NewMemdir("/c",
	NewMemfileBytes("d.txt", []byte("baz")),
	NewMemdir("/e",
		NewMemfileBytes("f.txt", []byte("bat")),
	),
),

) File is an interface that provides functionality for handling cafs/directories as values that can be supplied to commands.

This is pretty close to things that already exist in ipfs and might not be necessary in most situations, but provides a sensible degree of modularity for our purposes: * memdir: github.com/ipfs/go-ipfs/commands/SerialFile * memfs: github.com/ipfs/go-ipfs/commands/ReaderFile

Network simulates IPFS-like behavior, where nodes can connect to each other to retrieve data from other machines

func NewMapstore

func NewMapstore() *MapStore

NewMapstore allocates an instance of a mapstore

func (*MapStore) AddConnection

func (m *MapStore) AddConnection(other *MapStore)

AddConnection sets up pointers from this MapStore to that, and vice versa.

func (MapStore) Delete

func (m MapStore) Delete(ctx context.Context, key string) error

Delete removes the file from the store with the key

func (*MapStore) Fetch

func (m *MapStore) Fetch(ctx context.Context, source Source, key string) (qfs.File, error)

Fetch returns a File from the store

func (*MapStore) Get

func (m *MapStore) Get(ctx context.Context, key string) (qfs.File, error)

Get returns a File from the store

func (MapStore) Has

func (m MapStore) Has(ctx context.Context, key string) (exists bool, err error)

Has returns whether the store has a File with the key

func (MapStore) NewAdder

func (m MapStore) NewAdder(pin, wrap bool) (Adder, error)

NewAdder returns an Adder for the store

func (MapStore) ObjectCount added in v0.6.0

func (m MapStore) ObjectCount() (objects int)

ObjectCount returns the number of content-addressed objects in the store

func (*MapStore) Pin

func (m *MapStore) Pin(ctx context.Context, key string, recursive bool) error

Pin pins a File with the given key

func (MapStore) Print

func (m MapStore) Print() (string, error)

Print converts the store to a string

func (*MapStore) Put

func (m *MapStore) Put(ctx context.Context, file qfs.File) (key string, err error)

Put adds a file to the store and pins

func (*MapStore) PutFileAtKey added in v0.4.2

func (m *MapStore) PutFileAtKey(ctx context.Context, key string, file qfs.File) error

PutFileAtKey puts the file at the given key

func (MapStore) Type added in v0.4.2

func (m MapStore) Type() string

Type distinguishes this filesystem from others by a unique string prefix

func (*MapStore) Unpin

func (m *MapStore) Unpin(ctx context.Context, key string, recursive bool) error

Unpin unpins a File with the given key

type Pinner

type Pinner interface {
	Pin(ctx context.Context, key string, recursive bool) error
	Unpin(ctx context.Context, key string, recursive bool) error
}

Pinner interface for content stores that support the concept of pinning (originated by IPFS). Necessarily asynchronous, with no stateful guarantees, currently not testable.

type Source

type Source interface {
	// address should return the base resource identifier in either content
	// or location based addressing schemes
	Address() string
}

Source identifies where a file should come from. examples of different sources could be an HTTP url or P2P node Identifier

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL