helpers

package
v1.0.12 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 22, 2018 License: MIT Imports: 12 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var BlockSizeLimit = 1048576 // 1 MB

BlockSizeLimit specifies the maximum size an imported block can have.

View Source
var DefaultLinksPerBlock = roughLinkBlockSize / roughLinkSize

DefaultLinksPerBlock governs how the importer decides how many links there will be per block. This calculation is based on expected distributions of:

  • the expected distribution of block sizes
  • the expected distribution of link sizes
  • desired access speed

For now, we use:

var roughLinkBlockSize = 1 << 13 // 8KB
var roughLinkSize = 288          // sha256 + framing + name
var DefaultLinksPerBlock = (roughLinkBlockSize / roughLinkSize)

See calc_test.go

View Source
var ErrSizeLimitExceeded = fmt.Errorf("object size limit exceeded")

ErrSizeLimitExceeded signals that a block is larger than BlockSizeLimit.

Functions

This section is empty.

Types

type DagBuilderHelper

type DagBuilderHelper struct {
	// contains filtered or unexported fields
}

DagBuilderHelper wraps together a bunch of objects needed to efficiently create unixfs dag trees

func (*DagBuilderHelper) Add

func (db *DagBuilderHelper) Add(node *UnixfsNode) (ipld.Node, error)

Add sends a node to the DAGService, and returns it.

func (*DagBuilderHelper) AddNodeAndClose

func (db *DagBuilderHelper) AddNodeAndClose(node ipld.Node) (ipld.Node, error)

AddNodeAndClose adds the last `ipld.Node` from the DAG and closes the builder. It returns the same `node` passed as argument.

func (*DagBuilderHelper) Close

func (db *DagBuilderHelper) Close() error

Close has the DAGService perform a batch Commit operation. It should be called at the end of the building process to make sure all data is persisted.

func (*DagBuilderHelper) Done

func (db *DagBuilderHelper) Done() bool

Done returns whether or not we're done consuming the incoming data.

func (*DagBuilderHelper) FillNodeLayer

func (db *DagBuilderHelper) FillNodeLayer(node *UnixfsNode) error

FillNodeLayer will add datanodes as children to the give node until at most db.indirSize nodes are added.

func (*DagBuilderHelper) GetCidBuilder added in v1.0.8

func (db *DagBuilderHelper) GetCidBuilder() cid.Builder

GetCidBuilder returns the internal `cid.CidBuilder` set in the builder.

func (*DagBuilderHelper) GetDagServ

func (db *DagBuilderHelper) GetDagServ() ipld.DAGService

GetDagServ returns the dagservice object this Helper is using

func (*DagBuilderHelper) GetNextDataNode

func (db *DagBuilderHelper) GetNextDataNode() (*UnixfsNode, error)

GetNextDataNode builds a UnixFsNode with the data obtained from the Splitter, given the constraints (BlockSizeLimit, RawLeaves) specified when creating the DagBuilderHelper.

func (db *DagBuilderHelper) Maxlinks() int

Maxlinks returns the configured maximum number for links for nodes built with this helper.

func (*DagBuilderHelper) NewFSNodeOverDag

func (db *DagBuilderHelper) NewFSNodeOverDag(fsNodeType pb.Data_DataType) *FSNodeOverDag

NewFSNodeOverDag creates a new `dag.ProtoNode` and `ft.FSNode` decoupled from one onther (and will continue in that way until `Commit` is called), with `fsNodeType` specifying the type of the UnixFS layer node (either `File` or `Raw`).

func (*DagBuilderHelper) NewLeaf

func (db *DagBuilderHelper) NewLeaf(data []byte) (*UnixfsNode, error)

NewLeaf creates a leaf node filled with data. If rawLeaves is defined than a raw leaf will be returned. Otherwise, if data is nil the type field will be TRaw (for backwards compatibility), if data is defined (but possibly empty) the type field will be TRaw.

func (*DagBuilderHelper) NewLeafDataNode

func (db *DagBuilderHelper) NewLeafDataNode() (node ipld.Node, dataSize uint64, err error)

NewLeafDataNode is a variation of `GetNextDataNode` that returns an `ipld.Node` instead. It builds the `node` with the data obtained from the Splitter and returns it with the `dataSize` (that will be used to keep track of the DAG file size). The size of the data is computed here because after that it will be hidden by `NewLeafNode` inside a generic `ipld.Node` representation.

func (*DagBuilderHelper) NewLeafNode

func (db *DagBuilderHelper) NewLeafNode(data []byte) (ipld.Node, error)

NewLeafNode is a variation from `NewLeaf` (see its description) that returns an `ipld.Node` instead.

func (*DagBuilderHelper) NewUnixfsNode

func (db *DagBuilderHelper) NewUnixfsNode() *UnixfsNode

NewUnixfsNode creates a new Unixfs node to represent a file.

func (*DagBuilderHelper) Next

func (db *DagBuilderHelper) Next() ([]byte, error)

Next returns the next chunk of data to be inserted into the dag if it returns nil, that signifies that the stream is at an end, and that the current building operation should finish.

func (*DagBuilderHelper) ProcessFileStore

func (db *DagBuilderHelper) ProcessFileStore(node ipld.Node, dataSize uint64) ipld.Node

ProcessFileStore generates, if Filestore is being used, the `FilestoreNode` representation of the `ipld.Node` that contains the file data. If Filestore is not being used just return the same node to continue with its addition to the DAG.

The `db.offset` is updated at this point (instead of when `NewLeafDataNode` is called, both work in tandem but the offset is more related to this function).

type DagBuilderParams

type DagBuilderParams struct {
	// Maximum number of links per intermediate node
	Maxlinks int

	// RawLeaves signifies that the importer should use raw ipld nodes as leaves
	// instead of using the unixfs TRaw type
	RawLeaves bool

	// CID Builder to use if set
	CidBuilder cid.Builder

	// DAGService to write blocks to (required)
	Dagserv ipld.DAGService

	// NoCopy signals to the chunker that it should track fileinfo for
	// filestore adds
	NoCopy bool

	// URL if non-empty (and NoCopy is also true) indicates that the
	// file will not be stored in the datastore but instead retrieved
	// from this location via the urlstore.
	URL string
}

DagBuilderParams wraps configuration options to create a DagBuilderHelper from a chunker.Splitter.

func (*DagBuilderParams) New

New generates a new DagBuilderHelper from the given params and a given chunker.Splitter as data source.

type FSNodeOverDag

type FSNodeOverDag struct {
	// contains filtered or unexported fields
}

FSNodeOverDag encapsulates an `unixfs.FSNode` that will be stored in a `dag.ProtoNode`. Instead of just having a single `ipld.Node` that would need to be constantly (un)packed to access and modify its internal `FSNode` in the process of creating a UnixFS DAG, this structure stores an `FSNode` cache to manipulate it (add child nodes) directly , and only when the node has reached its final (immutable) state (signaled by calling `Commit()`) is it committed to a single (indivisible) `ipld.Node`.

It is used mainly for internal (non-leaf) nodes, and for some representations of data leaf nodes (that don't use raw nodes or Filestore).

It aims to replace the `UnixfsNode` structure which encapsulated too many possible node state combinations.

TODO: Revisit the name.

func (*FSNodeOverDag) AddChild

func (n *FSNodeOverDag) AddChild(child ipld.Node, fileSize uint64, db *DagBuilderHelper) error

AddChild adds a `child` `ipld.Node` to both node layers. The `dag.ProtoNode` creates a link to the child node while the `ft.FSNode` stores its file size (that is, not the size of the node but the size of the file data that it is storing at the UnixFS layer). The child is also stored in the `DAGService`.

func (*FSNodeOverDag) Commit

func (n *FSNodeOverDag) Commit() (ipld.Node, error)

Commit unifies (resolves) the cache nodes into a single `ipld.Node` that represents them: the `ft.FSNode` is encoded inside the `dag.ProtoNode`.

TODO: Evaluate making it read-only after committing.

func (*FSNodeOverDag) FileSize

func (n *FSNodeOverDag) FileSize() uint64

FileSize returns the `Filesize` attribute from the underlying representation of the `ft.FSNode`.

func (*FSNodeOverDag) NumChildren

func (n *FSNodeOverDag) NumChildren() int

NumChildren returns the number of children of the `ft.FSNode`.

func (*FSNodeOverDag) SetFileData

func (n *FSNodeOverDag) SetFileData(fileData []byte)

SetFileData stores the `fileData` in the `ft.FSNode`. It should be used only when `FSNodeOverDag` represents a leaf node (internal nodes don't carry data, just file sizes).

type UnixfsNode

type UnixfsNode struct {
	// contains filtered or unexported fields
}

UnixfsNode is a struct created to aid in the generation of unixfs DAG trees

func NewUnixfsNodeFromDag

func NewUnixfsNodeFromDag(nd *dag.ProtoNode) (*UnixfsNode, error)

NewUnixfsNodeFromDag reconstructs a Unixfs node from a given dag node

func (*UnixfsNode) AddChild

func (n *UnixfsNode) AddChild(child *UnixfsNode, db *DagBuilderHelper) error

AddChild adds the given UnixfsNode as a child of the receiver. The passed in DagBuilderHelper is used to store the child node an pin it locally so it doesnt get lost.

func (*UnixfsNode) FileSize

func (n *UnixfsNode) FileSize() uint64

FileSize returns the total file size of this tree (including children) In the case of raw nodes, it returns the length of the raw data.

func (*UnixfsNode) GetChild

func (n *UnixfsNode) GetChild(ctx context.Context, i int, ds ipld.DAGService) (*UnixfsNode, error)

GetChild gets the ith child of this node from the given DAGService.

func (*UnixfsNode) GetDagNode

func (n *UnixfsNode) GetDagNode() (ipld.Node, error)

GetDagNode fills out the proper formatting for the unixfs node inside of a DAG node and returns the dag node.

func (*UnixfsNode) NumChildren

func (n *UnixfsNode) NumChildren() int

NumChildren returns the number of children referenced by this UnixfsNode.

func (*UnixfsNode) RemoveChild

func (n *UnixfsNode) RemoveChild(index int, dbh *DagBuilderHelper)

RemoveChild deletes the child node at the given index.

func (*UnixfsNode) SetCidBuilder added in v1.0.8

func (n *UnixfsNode) SetCidBuilder(builder cid.Builder)

SetCidBuilder sets the CID Builder

func (*UnixfsNode) SetData

func (n *UnixfsNode) SetData(data []byte)

SetData stores data in this node.

func (*UnixfsNode) SetPosInfo

func (n *UnixfsNode) SetPosInfo(offset uint64, fullPath string, stat os.FileInfo)

SetPosInfo sets information about the offset of the data of this node in a filesystem file.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL