helpers

package
v0.6.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 21, 2021 License: MIT Imports: 16 Imported by: 3

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrMissingFsRef           = errors.New("missing file path or URL, can't create filestore reference")
	ErrUnknownNodeType        = errors.New("unknown node type")
	ErrUnexpectedNodeType     = errors.New("unexpected node type")
	ErrUnexpectedProgramState = errors.New("unexpected program state occurred")
	ErrUnexpectedNilArgument  = errors.New("unexpected nil argument value")
)
View Source 
var BlockSizeLimit = 1048576 // 1 MB

BlockSizeLimit specifies the maximum size an imported block can have.

View Source 
var DefaultLinksPerBlock = roughLinkBlockSize / roughLinkSize

DefaultLinksPerBlock governs how the importer decides how many links there will be per block. This calculation is based on expected distributions of:

  • the expected distribution of block sizes
  • the expected distribution of link sizes
  • desired access speed

For now, we use: 

var roughLinkBlockSize = 1 << 13 // 8KB
var roughLinkSize = 34 + 8 + 5   // sha256 multihash + size + no name
                                 // + protobuf framing
var DefaultLinksPerBlock = (roughLinkBlockSize / roughLinkSize)
                         = ( 8192 / 47 )
                         = (approximately) 174
View Source 
var ErrSizeLimitExceeded = fmt.Errorf("object size limit exceeded")

ErrSizeLimitExceeded signals that a block is larger than BlockSizeLimit.

Functions

This section is empty.

Types

type DagBuilderHelper 

type DagBuilderHelper struct {
	// contains filtered or unexported fields
}

DagBuilderHelper wraps together a bunch of objects needed to efficiently create unixfs dag trees

func (*DagBuilderHelper) Add 

func (db *DagBuilderHelper) Add(node ipld.Node) error

Add inserts the given node in the DAGService.

func (*DagBuilderHelper) AttachMetadataDag added in v0.4.6 

func (db *DagBuilderHelper) AttachMetadataDag(root ipld.Node, fileSize uint64, metaRoot ipld.Node) (ipld.Node, error)

func (*DagBuilderHelper) Done 

func (db *DagBuilderHelper) Done() bool

Done returns whether or not we're done consuming the incoming data.

func (*DagBuilderHelper) FillNodeLayer 

func (db *DagBuilderHelper) FillNodeLayer(node *FSNodeOverDag, fsNodeType pb.Data_DataType) error

FillNodeLayer will add datanodes as children to the give node until it is full in this layer or no more data. NOTE: This function creates raw data nodes so it only works for the `trickle.Layout`.

func (*DagBuilderHelper) GetCidBuilder 

func (db *DagBuilderHelper) GetCidBuilder() cid.Builder

GetCidBuilder returns the internal `cid.CidBuilder` set in the builder.

func (*DagBuilderHelper) GetDagServ 

func (db *DagBuilderHelper) GetDagServ() ipld.DAGService

GetDagServ returns the dagservice object this Helper is using

func (*DagBuilderHelper) GetMetaDb added in v0.4.4 

func (db *DagBuilderHelper) GetMetaDb() *MetaDagBuilderHelper

GetMetaDb returns the metadata DAG build helper this Helper is using

func (*DagBuilderHelper) IsMetaDagBuilt added in v0.4.4 

func (db *DagBuilderHelper) IsMetaDagBuilt() bool

isMetaDagBuilt returns the db.metaDagBuilt bool value.

func (*DagBuilderHelper) IsMultiDagBuilder added in v0.4.0 

func (db *DagBuilderHelper) IsMultiDagBuilder() bool

IsMultiDagBuilder checks if this helper contains multiple dagbuilders.

func (*DagBuilderHelper) IsThereMetaData added in v0.4.4 

func (db *DagBuilderHelper) IsThereMetaData() bool

isThereMetaData returns whether token metadata exists. 

func (db *DagBuilderHelper) Maxlinks() int

Maxlinks returns the configured maximum number for links for nodes built with this helper.

func (*DagBuilderHelper) MultiHelpers added in v0.4.0 

func (db *DagBuilderHelper) MultiHelpers() []*DagBuilderHelper

MultiHelpers returns the sub DagBuilderHelpers under self.

func (*DagBuilderHelper) NewFSNFromDag 

func (db *DagBuilderHelper) NewFSNFromDag(nd *dag.ProtoNode) (*FSNodeOverDag, error)

NewFSNFromDag reconstructs a FSNodeOverDag node from a given dag node

func (*DagBuilderHelper) NewFSNodeOverDag 

func (db *DagBuilderHelper) NewFSNodeOverDag(fsNodeType pb.Data_DataType) *FSNodeOverDag

NewFSNodeOverDag creates a new `dag.ProtoNode` and `ft.FSNode` decoupled from one onther (and will continue in that way until `Commit` is called), with `fsNodeType` specifying the type of the UnixFS layer node (either `File` or `Raw`).

func (*DagBuilderHelper) NewLeafDataNode 

func (db *DagBuilderHelper) NewLeafDataNode(fsNodeType pb.Data_DataType) (node ipld.Node, dataSize uint64, err error)

NewLeafDataNode builds the `node` with the data obtained from the Splitter with the given constraints (BlockSizeLimit, RawLeaves) specified when creating the DagBuilderHelper. It returns `ipld.Node` with the `dataSize` (that will be used to keep track of the DAG file size). The size of the data is computed here because after that it will be hidden by `NewLeafNode` inside a generic `ipld.Node` representation.

func (*DagBuilderHelper) NewLeafNode 

func (db *DagBuilderHelper) NewLeafNode(data []byte, fsNodeType pb.Data_DataType) (ipld.Node, error)

NewLeafNode creates a leaf node filled with data. If rawLeaves is defined then a raw leaf will be returned. Otherwise, it will create and return `FSNodeOverDag` with `fsNodeType`.

func (*DagBuilderHelper) Next 

func (db *DagBuilderHelper) Next() ([]byte, error)

Next returns the next chunk of data to be inserted into the dag if it returns nil, that signifies that the stream is at an end, and that the current building operation should finish.

func (*DagBuilderHelper) ProcessFileStore 

func (db *DagBuilderHelper) ProcessFileStore(node ipld.Node, dataSize uint64) ipld.Node

ProcessFileStore generates, if Filestore is being used, the `FilestoreNode` representation of the `ipld.Node` that contains the file data. If Filestore is not being used just return the same node to continue with its addition to the DAG.

The `db.offset` is updated at this point (instead of when `NewLeafDataNode` is called, both work in tandem but the offset is more related to this function).

func (*DagBuilderHelper) SetMetaDagBuilt added in v0.4.4 

func (db *DagBuilderHelper) SetMetaDagBuilt(v bool)

SetMetaDagBuilt sets db.metaDagBuilt with the given bool value.

func (*DagBuilderHelper) SetMetaDb added in v0.4.11 

func (db *DagBuilderHelper) SetMetaDb(mdb *MetaDagBuilderHelper)

SetMetaDb sets the given `mdb` to the metadata DAG build helper.

type DagBuilderHelperInterface added in v0.4.4 

type DagBuilderHelperInterface interface {
	Next() ([]byte, error)
	NewFSNodeOverDag(pb.Data_DataType) *FSNodeOverDag
	Maxlinks() int
	Done() bool
	Add(ipld.Node) error
	NewLeafDataNode(pb.Data_DataType) (ipld.Node, uint64, error)
	FillNodeLayer(*FSNodeOverDag, pb.Data_DataType) error
	AttachMetadataDag(ipld.Node, uint64, ipld.Node) (ipld.Node, error)
}

type DagBuilderParams 

type DagBuilderParams struct {
	// Maximum number of links per intermediate node
	Maxlinks int

	// RawLeaves signifies that the importer should use raw ipld nodes as leaves
	// instead of using the unixfs TRaw type
	RawLeaves bool

	// CID Builder to use if set
	CidBuilder cid.Builder

	// DAGService to write blocks to (required)
	Dagserv ipld.DAGService

	// NoCopy signals to the chunker that it should track fileinfo for
	// filestore adds
	NoCopy bool

	// Token metadata to be added to the first UnixFs node.
	TokenMetadata []byte

	// Chunk size of the splitter
	ChunkSize uint64

	// MetadataProcessed indicates whether token metadata is processed or not.
	MetadataProcessed bool

	// TrickleFormat indicates the client requested trickle tree format
	TrickleFormat bool
	// contains filtered or unexported fields
}

DagBuilderParams wraps configuration options to create a DagBuilderHelper from a chunker.Splitter.

func (*DagBuilderParams) New 

func (dbp *DagBuilderParams) New(spl chunker.Splitter) (*DagBuilderHelper, error)

New generates a new DagBuilderHelper from the given params and a given chunker.Splitter as data source. If chunker.Splitter is a chunker.MultiSplitter, then DagBuilderHelper will contain underlying DagBuilderHelpers.

type FSNodeOverDag 

type FSNodeOverDag struct {
	// contains filtered or unexported fields
}

FSNodeOverDag encapsulates an `unixfs.FSNode` that will be stored in a `dag.ProtoNode`. Instead of just having a single `ipld.Node` that would need to be constantly (un)packed to access and modify its internal `FSNode` in the process of creating a UnixFS DAG, this structure stores an `FSNode` cache to manipulate it (add child nodes) directly , and only when the node has reached its final (immutable) state (signaled by calling `Commit()`) is it committed to a single (indivisible) `ipld.Node`.

It is used mainly for internal (non-leaf) nodes, and for some representations of data leaf nodes (that don't use raw nodes or Filestore).

It aims to replace the `UnixfsNode` structure which encapsulated too many possible node state combinations.

TODO: Revisit the name.

func NewFSNFromDag 

func NewFSNFromDag(nd *dag.ProtoNode) (*FSNodeOverDag, error)

NewFSNFromDag reconstructs a FSNodeOverDag node from a given dag node

func (*FSNodeOverDag) AddChild 

func (n *FSNodeOverDag) AddChild(child ipld.Node, fileSize uint64, db DagBuilderHelperInterface) error

AddChild adds a `child` `ipld.Node` to both node layers. The `dag.ProtoNode` creates a link to the child node while the `ft.FSNode` stores its file size (that is, not the size of the node but the size of the file data that it is storing at the UnixFS layer). The child is also stored in the `DAGService`.

func (*FSNodeOverDag) AddChildDag added in v0.4.4 

func (n *FSNodeOverDag) AddChildDag(child ipld.Node, fileSize uint64, db *DagBuilderHelper) error

AddChildDag adds the DAG topped by the given "child" whose DAG descendents are already added into blockservice.

func (*FSNodeOverDag) Commit 

func (n *FSNodeOverDag) Commit() (ipld.Node, error)

Commit unifies (resolves) the cache nodes into a single `ipld.Node` that represents them: the `ft.FSNode` is encoded inside the `dag.ProtoNode`.

TODO: Make it read-only after committing, allow to commit only once.

func (*FSNodeOverDag) FileSize 

func (n *FSNodeOverDag) FileSize() uint64

FileSize returns the `Filesize` attribute from the underlying representation of the `ft.FSNode`.

func (*FSNodeOverDag) GetChild 

func (n *FSNodeOverDag) GetChild(ctx context.Context, i int, ds ipld.DAGService) (*FSNodeOverDag, error)

GetChild gets the ith child of this node from the given DAGService.

func (*FSNodeOverDag) GetDagNode 

func (n *FSNodeOverDag) GetDagNode() (ipld.Node, error)

GetDagNode fills out the proper formatting for the FSNodeOverDag node inside of a DAG node and returns the dag node. TODO: Check if we have committed (passed the UnixFS information to the DAG layer) before returning this.

func (*FSNodeOverDag) GetFileNodeType added in v0.4.8 

func (n *FSNodeOverDag) GetFileNodeType() pb.Data_DataType

GetFileNodeType returns the data type of the `ft.FSNode`.

func (*FSNodeOverDag) NumChildren 

func (n *FSNodeOverDag) NumChildren() int

NumChildren returns the number of children of the `ft.FSNode`.

func (*FSNodeOverDag) RemoveChild 

func (n *FSNodeOverDag) RemoveChild(index int, dbh *DagBuilderHelper)

RemoveChild deletes the child node at the given index.

func (*FSNodeOverDag) SetFileData 

func (n *FSNodeOverDag) SetFileData(fileData []byte)

SetFileData stores the `fileData` in the `ft.FSNode`. It should be used only when `FSNodeOverDag` represents a leaf node (internal nodes don't carry data, just file sizes).

func (*FSNodeOverDag) ValidTrickleInnerNodeType added in v0.4.8 

func (n *FSNodeOverDag) ValidTrickleInnerNodeType() (pb.Data_DataType, error)

type MetaDagBuilderHelper added in v0.4.4 

type MetaDagBuilderHelper struct {
	// contains filtered or unexported fields
}

func NewMetaDagBuilderHelper added in v0.4.9 

func NewMetaDagBuilderHelper(idb DagBuilderHelper, spl chunker.Splitter, mroot ipld.Node) *MetaDagBuilderHelper

func (*MetaDagBuilderHelper) Add added in v0.4.4 

func (mdb *MetaDagBuilderHelper) Add(node ipld.Node) error

func (*MetaDagBuilderHelper) AttachMetadataDag added in v0.4.6 

func (mdb *MetaDagBuilderHelper) AttachMetadataDag(root ipld.Node, fileSize uint64, mRoot ipld.Node) (ipld.Node, error)

func (*MetaDagBuilderHelper) Done added in v0.4.4 

func (mdb *MetaDagBuilderHelper) Done() bool

func (*MetaDagBuilderHelper) FillNodeLayer added in v0.4.4 

func (mdb *MetaDagBuilderHelper) FillNodeLayer(node *FSNodeOverDag, fsNodeType pb.Data_DataType) error

func (*MetaDagBuilderHelper) GetDb added in v0.4.4 

func (mdb *MetaDagBuilderHelper) GetDb() DagBuilderHelper

GetDb returns the DagbuilderHelper

func (*MetaDagBuilderHelper) GetMetaDagRoot added in v0.4.4 

func (mdb *MetaDagBuilderHelper) GetMetaDagRoot() ipld.Node

GetMetaDagRoot returns the root of the token metadata DAG 

func (mdb *MetaDagBuilderHelper) Maxlinks() int

func (*MetaDagBuilderHelper) NewFSNodeOverDag added in v0.4.4 

func (mdb *MetaDagBuilderHelper) NewFSNodeOverDag(dataType pb.Data_DataType) *FSNodeOverDag

func (*MetaDagBuilderHelper) NewLeafDataNode added in v0.4.4 

func (mdb *MetaDagBuilderHelper) NewLeafDataNode(fsNodeType pb.Data_DataType) (node ipld.Node, dataSize uint64, err error)

NewMetaLeafDataNode builds a metadata `node` with the meta data obtained from the Metadata Splitter. It returns `ipld.Node` with the `dataSize`.

func (*MetaDagBuilderHelper) NewLeafNode added in v0.4.4 

func (mdb *MetaDagBuilderHelper) NewLeafNode(data []byte, fsNodeType pb.Data_DataType) (ipld.Node, error)

func (*MetaDagBuilderHelper) Next added in v0.4.4 

func (mdb *MetaDagBuilderHelper) Next() ([]byte, error)

func (*MetaDagBuilderHelper) SetDb added in v0.4.4 

func (mdb *MetaDagBuilderHelper) SetDb(db *DagBuilderHelper)

SetDb sets metadata DAG root

func (*MetaDagBuilderHelper) SetMetaDagRoot added in v0.4.4 

func (mdb *MetaDagBuilderHelper) SetMetaDagRoot(root ipld.Node)

SetMetaDagRoot sets metadata DAG root

func (*MetaDagBuilderHelper) SetSpl added in v0.4.4 

func (mdb *MetaDagBuilderHelper) SetSpl()

type SuperMeta added in v0.4.8 

type SuperMeta struct {
	ChunkSize     uint64
	MaxLinks      uint64
	TrickleFormat bool
}

SuperMeta contains the common metadata fields for a BTFS object to be used for metadata readers and modifiers

func GetOrDefaultSuperMeta added in v0.5.1 

func GetOrDefaultSuperMeta(b []byte) (*SuperMeta, error)

func GetSuperMeta added in v0.4.9 

func GetSuperMeta(chunkSize uint64, maxLinks uint64, trickleFormat bool) *SuperMeta

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.