sharding

package
v1.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 6, 2022 License: Apache-2.0, MIT, Apache-2.0, + 1 more Imports: 17 Imported by: 0

Documentation

Overview

Package sharding implements a sharding ClusterDAGService places content in different shards while it's being added, creating a final Cluster DAG and pinning it.

Index

Constants

View Source
const MaxLinks = 5984

MaxLinks is the max number of links that, when serialized fit into a block

Variables

This section is empty.

Functions

func CborDataToNode

func CborDataToNode(raw []byte, format string) (ipld.Node, error)

CborDataToNode parses cbor data into a clusterDAG node while making a few checks

func VerifyShards

func VerifyShards(t *testing.T, rootCid api.Cid, pins MockPinStore, ipfs MockBlockStore, expectedShards int) (map[string]struct{}, error)

VerifyShards checks that a sharded CID has been correctly formed and stored. This is a helper function for testing. It returns a map with all the blocks from all shards.

Types

type DAGService

type DAGService struct {
	adder.BaseDAGService
	// contains filtered or unexported fields
}

DAGService is an implementation of a ClusterDAGService which shards content while adding among several IPFS Cluster peers, creating a Cluster DAG to track and pin that content selectively in the IPFS daemons allocated to it.

func New

func New(ctx context.Context, rpc *rpc.Client, opts api.AddParams, out chan<- api.AddedOutput) *DAGService

New returns a new ClusterDAGService, which uses the given rpc client to perform Allocate, IPFSStream and Pin requests to other cluster components.

func (*DAGService) Add

func (dgs *DAGService) Add(ctx context.Context, node ipld.Node) error

Add puts the given node in its corresponding shard and sends it to the destination peers.

func (*DAGService) AddMany

func (dgs *DAGService) AddMany(ctx context.Context, nodes []ipld.Node) error

AddMany calls Add for every given node.

func (*DAGService) Allocations added in v1.0.0

func (dgs *DAGService) Allocations() []peer.ID

Allocations returns the current allocations for the current shard.

func (*DAGService) Finalize

func (dgs *DAGService) Finalize(ctx context.Context, dataRoot api.Cid) (api.Cid, error)

Finalize finishes sharding, creates the cluster DAG and pins it along with the meta pin for the root node of the content.

type MockBlockStore

type MockBlockStore interface {
	// Gets a block
	BlockGet(context.Context, api.Cid) ([]byte, error)
}

MockBlockStore is used in VerifyShards

type MockPinStore

type MockPinStore interface {
	// Gets a pin
	PinGet(context.Context, api.Cid) (api.Pin, error)
}

MockPinStore is used in VerifyShards

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL