mfs

package module
v0.2.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 25, 2023 License: MIT Imports: 18 Imported by: 81

README

go-mfs

go-mfs implements an in-memory model of a mutable IPFS filesystem.

standard-readme compliant GoDoc Build Status

❗ This repo is no longer maintained.

👉 We highly recommend switching to the maintained version at https://github.com/ipfs/boxo/tree/main/mfs. 🏎️ Good news! There is tooling and documentation to expedite a switch in your repo.

⚠️ If you continue using this repo, please note that security fixes will not be provided (unless someone steps in to maintain it).

📚 Learn more, including how to take the maintainership mantle or ask questions, here.

Table of Contents

Install

go-mfs works like a regular Go module:

> go get github.com/ipfs/go-mfs

It uses Gx to manage dependencies. You can use make all to build it with the gx dependencies.

Usage

import "github.com/ipfs/go-mfs"

Check the GoDoc documentation

Documentation

Documentation around the MFS and the Files API in general around IPFS is a work in progress the following links may be of use:

Repository Structure

This repository contains many files, all belonging to the root mfs package.

  • file.go: MFS File.
  • dir.go: MFS Directory.
  • fd.go: FileDescriptor used to operate on Files.
  • ops.go: Functions that do not belong to either File nor Directory (although they mostly operate on them) that contain common operations to the MFS, e.g., find, move, add a file, make a directory.
  • root.go: MFS Root (a Directory with republishing support).
  • repub.go: Republisher.
  • mfs_test.go: General tests (needs a revision).
  • repub_test.go: Republisher-specific tests (contains only the TestRepublisher function).

License

MIT © Protocol Labs, Inc.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrClosed = errors.New("file closed")

Deprecated: use github.com/ipfs/boxo/mfs.ErrClosed

View Source 
var ErrDirExists = errors.New("directory already has entry by that name")

Deprecated: use github.com/ipfs/boxo/mfs.ErrDirExists

View Source 
var ErrInvalidChild = errors.New("invalid child node")

Deprecated: use github.com/ipfs/boxo/mfs.ErrInvalidChild

View Source 
var ErrIsDirectory = errors.New("error: is a directory")

TODO: Remove if not used.

Deprecated: use github.com/ipfs/boxo/mfs.ErrIsDirectory

View Source 
var ErrNotExist = errors.New("no such rootfs")

TODO: Remove if not used.

Deprecated: use github.com/ipfs/boxo/mfs.ErrNotExist

View Source 
var ErrNotYetImplemented = errors.New("not yet implemented")

Deprecated: use github.com/ipfs/boxo/mfs.ErrNotYetImplemented

Functions

func FlushPath deprecated 

func FlushPath(ctx context.Context, rt *Root, pth string) (ipld.Node, error)

TODO: Document this function and link its functionality with the republisher.

Deprecated: use github.com/ipfs/boxo/mfs.FlushPath

func IsDir deprecated 

func IsDir(fsn FSNode) bool

IsDir checks whether the FSNode is dir type

Deprecated: use github.com/ipfs/boxo/mfs.IsDir

func IsFile deprecated 

func IsFile(fsn FSNode) bool

IsFile checks whether the FSNode is file type

Deprecated: use github.com/ipfs/boxo/mfs.IsFile

func Mkdir deprecated 

func Mkdir(r *Root, pth string, opts MkdirOpts) error

Mkdir creates a directory at 'path' under the directory 'd', creating intermediary directories as needed if 'mkparents' is set to true

Deprecated: use github.com/ipfs/boxo/mfs.Mkdir

func Mv deprecated 

func Mv(r *Root, src, dst string) error

Mv moves the file or directory at 'src' to 'dst' TODO: Document what the strings 'src' and 'dst' represent.

Deprecated: use github.com/ipfs/boxo/mfs.Mv

func PutNode deprecated 

func PutNode(r *Root, path string, nd ipld.Node) error

PutNode inserts 'nd' at 'path' in the given mfs TODO: Rename or clearly document that this is not about nodes but actually MFS files/directories (that in the underlying representation can be considered as just nodes). TODO: Document why are we handling IPLD nodes in the first place when we are actually referring to files/directories (that is, it can't be any node, it has to have a specific format). TODO: Can this function add directories or just files? What would be the difference between adding a directory with this method and creating it with `Mkdir`.

Deprecated: use github.com/ipfs/boxo/mfs.PutNode

Types

type Directory deprecated 

type Directory struct {
	// contains filtered or unexported fields
}

TODO: There's too much functionality associated with this structure, let's organize it (and if possible extract part of it elsewhere) and document the main features of `Directory` here.

Deprecated: use github.com/ipfs/boxo/mfs.Directory

func NewDirectory deprecated 

func NewDirectory(ctx context.Context, name string, node ipld.Node, parent parent, dserv ipld.DAGService) (*Directory, error)

NewDirectory constructs a new MFS directory.

You probably don't want to call this directly. Instead, construct a new root using NewRoot.

Deprecated: use github.com/ipfs/boxo/mfs.NewDirectory

func (*Directory) AddChild 

func (d *Directory) AddChild(name string, nd ipld.Node) error

AddChild adds the node 'nd' under this directory giving it the name 'name'

func (*Directory) Child 

func (d *Directory) Child(name string) (FSNode, error)

Child returns the child of this directory by the given name

func (*Directory) Flush 

func (d *Directory) Flush() error

func (*Directory) ForEachEntry 

func (d *Directory) ForEachEntry(ctx context.Context, f func(NodeListing) error) error

func (*Directory) GetCidBuilder 

func (d *Directory) GetCidBuilder() cid.Builder

GetCidBuilder gets the CID builder of the root node

func (*Directory) GetNode 

func (d *Directory) GetNode() (ipld.Node, error)

func (*Directory) List 

func (d *Directory) List(ctx context.Context) ([]NodeListing, error)

func (*Directory) ListNames 

func (d *Directory) ListNames(ctx context.Context) ([]string, error)

func (*Directory) Mkdir 

func (d *Directory) Mkdir(name string) (*Directory, error)

func (*Directory) Path 

func (d *Directory) Path() string

func (*Directory) SetCidBuilder 

func (d *Directory) SetCidBuilder(b cid.Builder)

SetCidBuilder sets the CID builder

func (*Directory) Type 

func (d *Directory) Type() NodeType

func (*Directory) Uncache 

func (d *Directory) Uncache(name string)
func (d *Directory) Unlink(name string) error

type FSNode deprecated 

type FSNode interface {
	GetNode() (ipld.Node, error)

	Flush() error
	Type() NodeType
}

FSNode abstracts the `Directory` and `File` structures, it represents any child node in the MFS (i.e., all the nodes besides the `Root`). It is the counterpart of the `parent` interface which represents any parent node in the MFS (`Root` and `Directory`). (Not to be confused with the `unixfs.FSNode`.)

Deprecated: use github.com/ipfs/boxo/mfs.FSNode

func DirLookup deprecated 

func DirLookup(d *Directory, pth string) (FSNode, error)

DirLookup will look up a file or directory at the given path under the directory 'd'

Deprecated: use github.com/ipfs/boxo/mfs.DirLookup

func Lookup deprecated 

func Lookup(r *Root, path string) (FSNode, error)

Lookup extracts the root directory and performs a lookup under it. TODO: Now that the root is always a directory, can this function be collapsed with `DirLookup`? Or at least be made a method of `Root`?

Deprecated: use github.com/ipfs/boxo/mfs.Lookup

type File deprecated 

type File struct {
	RawLeaves bool
	// contains filtered or unexported fields
}

File represents a file in the MFS, its logic its mainly targeted to coordinating (potentially many) `FileDescriptor`s pointing to it.

Deprecated: use github.com/ipfs/boxo/mfs.File

func NewFile deprecated 

func NewFile(name string, node ipld.Node, parent parent, dserv ipld.DAGService) (*File, error)

NewFile returns a NewFile object with the given parameters. If the Cid version is non-zero RawLeaves will be enabled.

Deprecated: use github.com/ipfs/boxo/mfs.NewFile

func (*File) Flush 

func (fi *File) Flush() error

TODO: Tight coupling with the `FileDescriptor`, at the very least this should be an independent function that takes a `File` argument and automates the open/flush/close operations. TODO: Why do we need to flush a file that isn't opened? (the `OpenWriteOnly` seems to implicitly be targeting a closed file, a file we forgot to flush? can we close a file without flushing?)

func (*File) GetNode 

func (fi *File) GetNode() (ipld.Node, error)

GetNode returns the dag node associated with this file TODO: Use this method and do not access the `nodeLock` directly anywhere else.

func (*File) Open 

func (fi *File) Open(flags Flags) (_ FileDescriptor, _retErr error)

func (*File) Size 

func (fi *File) Size() (int64, error)

Size returns the size of this file TODO: Should we be providing this API? TODO: There's already a `FileDescriptor.Size()` that through the `DagModifier`'s `fileSize` function is doing pretty much the same thing as here, we should at least call that function and wrap the `ErrNotUnixfs` with an MFS text.

func (*File) Sync 

func (fi *File) Sync() error

func (*File) Type 

func (fi *File) Type() NodeType

Type returns the type FSNode this is

type FileDescriptor deprecated 

type FileDescriptor interface {
	io.Reader
	CtxReadFull(context.Context, []byte) (int, error)

	io.Writer
	io.WriterAt

	io.Closer
	io.Seeker

	Truncate(int64) error
	Size() (int64, error)
	Flush() error
}

One `File` can have many `FileDescriptor`s associated to it (only one if it's RW, many if they are RO, see `File.desclock`). A `FileDescriptor` contains the "view" of the file (through an instance of a `DagModifier`), that's why it (and not the `File`) has the responsibility to `Flush` (which crystallizes that view in the `File`'s `Node`).

Deprecated: use github.com/ipfs/boxo/mfs.FileDescriptor

type Flags deprecated 

type Flags struct {
	Read  bool
	Write bool
	Sync  bool
}

Deprecated: use github.com/ipfs/boxo/mfs.Flags

type MkdirOpts deprecated 

type MkdirOpts struct {
	Mkparents  bool
	Flush      bool
	CidBuilder cid.Builder
}

MkdirOpts is used by Mkdir

Deprecated: use github.com/ipfs/boxo/mfs.MkdirOpts

type NodeListing deprecated 

type NodeListing struct {
	Name string
	Type int
	Size int64
	Hash string
}

Deprecated: use github.com/ipfs/boxo/mfs.NodeListing

type NodeType deprecated 

type NodeType int

Deprecated: use github.com/ipfs/boxo/mfs.NodeType 

const (
	// Deprecated: use github.com/ipfs/boxo/mfs.TFile
	TFile NodeType = iota
	// Deprecated: use github.com/ipfs/boxo/mfs.TDir
	TDir
)

type PubFunc deprecated 

type PubFunc func(context.Context, cid.Cid) error

PubFunc is the user-defined function that determines exactly what logic entails "publishing" a `Cid` value.

Deprecated: use github.com/ipfs/boxo/mfs.PubFunc

type Republisher deprecated 

type Republisher struct {
	TimeoutLong  time.Duration
	TimeoutShort time.Duration
	RetryTimeout time.Duration
	// contains filtered or unexported fields
}

Republisher manages when to publish a given entry.

Deprecated: use github.com/ipfs/boxo/mfs.Republisher

func NewRepublisher deprecated 

func NewRepublisher(ctx context.Context, pf PubFunc, tshort, tlong time.Duration) *Republisher

NewRepublisher creates a new Republisher object to republish the given root using the given short and long time intervals.

Deprecated: use github.com/ipfs/boxo/mfs.NewRepublisher

func (*Republisher) Close 

func (rp *Republisher) Close() error

func (*Republisher) Run 

func (rp *Republisher) Run(lastPublished cid.Cid)

Run contains the core logic of the `Republisher`. It calls the user-defined `pubfunc` function whenever the `Cid` value is updated to a *new* value. The complexity comes from the fact that `pubfunc` may be slow so we need to batch updates.

Algorithm:

  1. When we receive the first update after publishing, we set a `longer` timer.
  2. When we receive any update, we reset the `quick` timer.
  3. If either the `quick` timeout or the `longer` timeout elapses, we call `publish` with the latest updated value.

The `longer` timer ensures that we delay publishing by at most `TimeoutLong`. The `quick` timer allows us to publish sooner if it looks like there are no more updates coming down the pipe.

Note: If a publish fails, we retry repeatedly every TimeoutRetry.

func (*Republisher) Update 

func (rp *Republisher) Update(c cid.Cid)

Update the current value. The value will be published after a delay but each consecutive call to Update may extend this delay up to TimeoutLong.

func (*Republisher) WaitPub 

func (rp *Republisher) WaitPub(ctx context.Context) error

WaitPub waits for the current value to be published (or returns early if it already has).

type Root deprecated 

type Root struct {
	// contains filtered or unexported fields
}

Root represents the root of a filesystem tree.

Deprecated: use github.com/ipfs/boxo/mfs.Root

func NewRoot deprecated 

func NewRoot(parent context.Context, ds ipld.DAGService, node *dag.ProtoNode, pf PubFunc) (*Root, error)

NewRoot creates a new Root and starts up a republisher routine for it.

Deprecated: use github.com/ipfs/boxo/mfs.NewRoot

func (*Root) Close 

func (kr *Root) Close() error

func (*Root) Flush 

func (kr *Root) Flush() error

Flush signals that an update has occurred since the last publish, and updates the Root republisher. TODO: We are definitely abusing the "flush" terminology here.

func (*Root) FlushMemFree 

func (kr *Root) FlushMemFree(ctx context.Context) error

FlushMemFree flushes the root directory and then uncaches all of its links. This has the effect of clearing out potentially stale references and allows them to be garbage collected. CAUTION: Take care not to ever call this while holding a reference to any child directories. Those directories will be bad references and using them may have unintended racy side effects. A better implemented mfs system (one that does smarter internal caching and refcounting) shouldnt need this method. TODO: Review the motivation behind this method once the cache system is refactored.

func (*Root) GetDirectory 

func (kr *Root) GetDirectory() *Directory

GetDirectory returns the root directory.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.