ipfscluster

package module
v1.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 6, 2022 License: Apache-2.0, MIT, Apache-2.0, + 1 more Imports: 56 Imported by: 0

README

IPFS Cluster

Made by Main project Matrix channel GoDoc Go Report Card Build Status codecov

Automated data availability and redundancy on IPFS

logo

IPFS Cluster provides data orchestration across a swarm of IPFS daemons by allocating, replicating and tracking a global pinset distributed among multiple peers.

It provides:

  • A cluster peer application: ipfs-cluster-service, to be run along with go-ipfs.
  • A client CLI application: ipfs-cluster-ctl, which allows easily interacting with the peer's HTTP API.
  • An additional "follower" peer application: ipfs-cluster-follow, focused on simplifying the process of configuring and running follower peers.
Are you using IPFS Cluster?

Please participate in the IPFS Cluster user registry.

Table of Contents

Documentation

Please visit https://cluster.ipfs.io/documentation/ to access user documentation, guides and any other resources, including detailed download and usage instructions.

News & Roadmap

We regularly post project updates to https://cluster.ipfs.io/news/ .

The most up-to-date Roadmap is available at https://cluster.ipfs.io/roadmap/ .

Install

Instructions for different installation methods (including from source) are available at https://cluster.ipfs.io/download .

Usage

Extensive usage information is provided at https://cluster.ipfs.io/documentation/ , including:

Contribute

PRs accepted. As part of the IPFS project, we have some contribution guidelines.

License

This library is dual-licensed under Apache 2.0 and MIT terms.

© 2020. Protocol Labs, Inc.

Documentation

Overview

Package ipfscluster implements a wrapper for the IPFS deamon which allows to orchestrate pinning operations among several IPFS nodes.

IPFS Cluster peers form a separate libp2p swarm. A Cluster peer uses multiple Cluster Components which perform different tasks like managing the underlying IPFS daemons, or providing APIs for external control.

Index

Constants

View Source 
const (
	DefaultEnableRelayHop        = true
	DefaultStateSyncInterval     = 5 * time.Minute
	DefaultPinRecoverInterval    = 12 * time.Minute
	DefaultMonitorPingInterval   = 15 * time.Second
	DefaultPeerWatchInterval     = 5 * time.Second
	DefaultReplicationFactor     = -1
	DefaultLeaveOnShutdown       = false
	DefaultPinOnlyOnTrustedPeers = false
	DefaultDisableRepinning      = true
	DefaultPeerstoreFile         = "peerstore"
	DefaultConnMgrHighWater      = 400
	DefaultConnMgrLowWater       = 100
	DefaultConnMgrGracePeriod    = 2 * time.Minute
	DefaultDialPeerTimeout       = 3 * time.Second
	DefaultFollowerMode          = false
	DefaultMDNSInterval          = 10 * time.Second
)

Configuration defaults

Variables

View Source 
var DefaultListenAddrs = []string{
	"/ip4/0.0.0.0/tcp/9096",
	"/ip4/0.0.0.0/udp/9096/quic",
}

DefaultListenAddrs contains TCP and QUIC listen addresses.

View Source 
var DefaultRPCPolicy = map[string]RPCEndpointType{

	"Cluster.Alerts":               RPCClosed,
	"Cluster.BlockAllocate":        RPCClosed,
	"Cluster.ConnectGraph":         RPCClosed,
	"Cluster.ID":                   RPCOpen,
	"Cluster.IDStream":             RPCOpen,
	"Cluster.IPFSID":               RPCClosed,
	"Cluster.Join":                 RPCClosed,
	"Cluster.PeerAdd":              RPCOpen,
	"Cluster.PeerRemove":           RPCTrusted,
	"Cluster.Peers":                RPCTrusted,
	"Cluster.PeersWithFilter":      RPCClosed,
	"Cluster.Pin":                  RPCClosed,
	"Cluster.PinGet":               RPCClosed,
	"Cluster.PinPath":              RPCClosed,
	"Cluster.Pins":                 RPCClosed,
	"Cluster.Recover":              RPCClosed,
	"Cluster.RecoverAll":           RPCClosed,
	"Cluster.RecoverAllLocal":      RPCTrusted,
	"Cluster.RecoverLocal":         RPCTrusted,
	"Cluster.RepoGC":               RPCClosed,
	"Cluster.RepoGCLocal":          RPCTrusted,
	"Cluster.SendInformerMetrics":  RPCClosed,
	"Cluster.SendInformersMetrics": RPCClosed,
	"Cluster.Status":               RPCClosed,
	"Cluster.StatusAll":            RPCClosed,
	"Cluster.StatusAllLocal":       RPCClosed,
	"Cluster.StatusLocal":          RPCClosed,
	"Cluster.Unpin":                RPCClosed,
	"Cluster.UnpinPath":            RPCClosed,
	"Cluster.Version":              RPCOpen,

	"PinTracker.PinQueueSize": RPCClosed,
	"PinTracker.Recover":      RPCTrusted,
	"PinTracker.RecoverAll":   RPCClosed,
	"PinTracker.Status":       RPCTrusted,
	"PinTracker.StatusAll":    RPCTrusted,
	"PinTracker.Track":        RPCClosed,
	"PinTracker.Untrack":      RPCClosed,

	"IPFSConnector.BlockGet":    RPCClosed,
	"IPFSConnector.BlockStream": RPCTrusted,
	"IPFSConnector.ConfigKey":   RPCClosed,
	"IPFSConnector.Pin":         RPCClosed,
	"IPFSConnector.PinLs":       RPCClosed,
	"IPFSConnector.PinLsCid":    RPCClosed,
	"IPFSConnector.RepoStat":    RPCTrusted,
	"IPFSConnector.Resolve":     RPCClosed,
	"IPFSConnector.SwarmPeers":  RPCTrusted,
	"IPFSConnector.Unpin":       RPCClosed,

	"Consensus.AddPeer":  RPCTrusted,
	"Consensus.LogPin":   RPCTrusted,
	"Consensus.LogUnpin": RPCTrusted,
	"Consensus.Peers":    RPCClosed,
	"Consensus.RmPeer":   RPCTrusted,

	"PeerMonitor.LatestMetrics": RPCClosed,
	"PeerMonitor.MetricNames":   RPCClosed,
}

DefaultRPCPolicy associates all rpc endpoints offered by cluster peers to an endpoint type. See rpcutil/policygen.go as a quick way to generate this without missing any endpoint.

View Source 
var LoggingFacilities = map[string]string{
	"cluster":      "INFO",
	"restapi":      "INFO",
	"restapilog":   "INFO",
	"pinsvcapi":    "INFO",
	"pinsvcapilog": "INFO",
	"ipfsproxy":    "INFO",
	"ipfsproxylog": "INFO",
	"ipfshttp":     "INFO",
	"monitor":      "INFO",
	"dsstate":      "INFO",
	"raft":         "INFO",
	"crdt":         "INFO",
	"pintracker":   "INFO",
	"diskinfo":     "INFO",
	"tags":         "INFO",
	"apitypes":     "INFO",
	"config":       "INFO",
	"shardingdags": "INFO",
	"singledags":   "INFO",
	"adder":        "INFO",
	"optracker":    "INFO",
	"pstoremgr":    "INFO",
	"allocator":    "INFO",
}

LoggingFacilities provides a list of logging identifiers used by cluster and their default logging level.

View Source 
var LoggingFacilitiesExtra = map[string]string{
	"p2p-gorpc":   "ERROR",
	"swarm2":      "ERROR",
	"libp2p-raft": "FATAL",
	"raftlib":     "ERROR",
	"badger":      "INFO",
}

LoggingFacilitiesExtra provides logging identifiers used in ipfs-cluster dependencies, which may be useful to display. Along with their default value.

View Source 
var ReadyTimeout = 30 * time.Second

ReadyTimeout specifies the time before giving up during startup (waiting for consensus to be ready) It may need adjustment according to timeouts in the consensus layer.

Functions

func DecodeClusterSecret added in v0.1.0 

func DecodeClusterSecret(hexSecret string) ([]byte, error)

DecodeClusterSecret parses a hex-encoded string, checks that it is exactly 32 bytes long and returns its value as a byte-slice.x

func EncodeProtectorKey added in v0.3.5 

func EncodeProtectorKey(secretBytes []byte) string

EncodeProtectorKey converts a byte slice to its hex string representation.

func NewClusterHost added in v0.3.5 

func NewClusterHost(
	ctx context.Context,
	ident *config.Identity,
	cfg *Config,
	ds ds.Datastore,
) (host.Host, *pubsub.PubSub, *dual.DHT, error)

NewClusterHost creates a fully-featured libp2p Host with the options from the provided cluster configuration. Using that host, it creates pubsub and a DHT instances (persisting to the given datastore), for shared use by all cluster components. The returned host uses the DHT for routing. Relay and NATService are additionally setup for this host.

func PeersFromMultiaddrs added in v0.3.2 

func PeersFromMultiaddrs(addrs []ma.Multiaddr) []peer.ID

PeersFromMultiaddrs returns all the different peers in the given addresses. each peer only will appear once in the result, even if several multiaddresses for it are provided.

func RPCServiceID added in v0.11.0 

func RPCServiceID(rpcSvc interface{}) string

RPCServiceID returns the Service ID for the given RPCAPI object.

func SetFacilityLogLevel added in v0.0.3 

func SetFacilityLogLevel(f, l string)

SetFacilityLogLevel sets the log level for a given module

Types

type API 

type API interface {
	Component
}

API is a component which offers an API for Cluster. This is a base component.

type Cluster 

type Cluster struct {
	// contains filtered or unexported fields
}

Cluster is the main IPFS cluster component. It provides the go-API for it and orchestrates the components that make up the system.

func NewCluster 

func NewCluster(
	ctx context.Context,
	host host.Host,
	dht *dual.DHT,
	cfg *Config,
	datastore ds.Datastore,
	consensus Consensus,
	apis []API,
	ipfs IPFSConnector,
	tracker PinTracker,
	monitor PeerMonitor,
	allocator PinAllocator,
	informers []Informer,
	tracer Tracer,
) (*Cluster, error)

NewCluster builds a new IPFS Cluster peer. It initializes a LibP2P host, creates and RPC Server and client and sets up all components.

The new cluster peer may still be performing initialization tasks when this call returns (consensus may still be bootstrapping). Use Cluster.Ready() if you need to wait until the peer is fully up.

func (*Cluster) AddFile added in v0.5.0 

func (c *Cluster) AddFile(ctx context.Context, reader *multipart.Reader, params api.AddParams) (api.Cid, error)

AddFile adds a file to the ipfs daemons of the cluster. The ipfs importer pipeline is used to DAGify the file. Depending on input parameters this DAG can be added locally to the calling cluster peer's ipfs repo, or sharded across the entire cluster.

func (*Cluster) Alerts added in v0.13.1 

func (c *Cluster) Alerts() []api.Alert

Alerts returns the last alerts recorded by this cluster peer with the most recent first.

func (*Cluster) ConnectGraph added in v0.3.3 

func (c *Cluster) ConnectGraph() (api.ConnectGraph, error)

ConnectGraph returns a description of which cluster peers and ipfs daemons are connected to each other.

func (*Cluster) Done added in v0.0.3 

func (c *Cluster) Done() <-chan struct{}

Done provides a way to learn if the Peer has been shutdown (for example, because it has been removed from the Cluster)

func (*Cluster) ID 

func (c *Cluster) ID(ctx context.Context) api.ID

ID returns information about the Cluster peer

func (*Cluster) Join added in v0.0.3 

func (c *Cluster) Join(ctx context.Context, addr ma.Multiaddr) error

Join adds this peer to an existing cluster by bootstrapping to a given multiaddress. It works by calling PeerAdd on the destination cluster and making sure that the new peer is ready to discover and contact the rest.

func (*Cluster) PeerAdd added in v0.0.3 

func (c *Cluster) PeerAdd(ctx context.Context, pid peer.ID) (*api.ID, error)

PeerAdd adds a new peer to this Cluster.

For it to work well, the new peer should be discoverable (part of our peerstore or connected to one of the existing peers) and reachable. Since PeerAdd allows to add peers which are not running, or reachable, it is recommended to call Join() from the new peer instead.

The new peer ID will be passed to the consensus component to be added to the peerset.

func (*Cluster) PeerRemove added in v0.0.3 

func (c *Cluster) PeerRemove(ctx context.Context, pid peer.ID) error

PeerRemove removes a peer from this Cluster.

The peer will be removed from the consensus peerset. This may first trigger repinnings for all content if not disabled.

func (*Cluster) Peers added in v0.0.3 

func (c *Cluster) Peers(ctx context.Context, out chan<- api.ID)

Peers returns the IDs of the members of this Cluster on the out channel. This method blocks until it has finished.

func (*Cluster) Pin 

func (c *Cluster) Pin(ctx context.Context, h api.Cid, opts api.PinOptions) (api.Pin, error)

Pin makes the cluster Pin a Cid. This implies adding the Cid to the IPFS Cluster peers shared-state. Depending on the cluster pinning strategy, the PinTracker may then request the IPFS daemon to pin the Cid.

Pin returns the Pin as stored in the global state (with the given allocations and an error if the operation could not be persisted. Pin does not reflect the success or failure of underlying IPFS daemon pinning operations which happen in async fashion.

If the options UserAllocations are non-empty then these peers are pinned with priority over other peers in the cluster. If the max repl factor is less than the size of the specified peerset then peers are chosen from this set in allocation order. If the minimum repl factor is greater than the size of this set then the remaining peers are allocated in order from the rest of the cluster. Priority allocations are best effort. If any priority peers are unavailable then Pin will simply allocate from the rest of the cluster.

If the Update option is set, the pin options (including allocations) will be copied from an existing one. This is equivalent to running PinUpdate.

func (*Cluster) PinGet added in v0.1.0 

func (c *Cluster) PinGet(ctx context.Context, h api.Cid) (api.Pin, error)

PinGet returns information for a single Cid managed by Cluster. The information is obtained from the current global state. The returned api.Pin provides information about the allocations assigned for the requested Cid, but does not indicate if the item is successfully pinned. For that, use Status(). PinGet returns an error if the given Cid is not part of the global state.

func (*Cluster) PinPath added in v0.10.0 

func (c *Cluster) PinPath(ctx context.Context, path string, opts api.PinOptions) (api.Pin, error)

PinPath pins an CID resolved from its IPFS Path. It returns the resolved Pin object.

func (*Cluster) PinUpdate added in v0.11.0 

func (c *Cluster) PinUpdate(ctx context.Context, from api.Cid, to api.Cid, opts api.PinOptions) (api.Pin, error)

PinUpdate pins a new CID based on an existing cluster Pin. The allocations and most pin options (replication factors) are copied from the existing Pin. The options object can be used to set the Name for the new pin and might support additional options in the future.

The from pin is NOT unpinned upon completion. The new pin might take advantage of efficient pin/update operation on IPFS-side (if the IPFSConnector supports it - the default one does). This may offer significant speed when pinning items which are similar to previously pinned content.

func (*Cluster) Pins 

func (c *Cluster) Pins(ctx context.Context, out chan<- api.Pin) error

Pins sends pins on the given out channel as it iterates the full pinset (current global state). This is the source of truth as to which pins are managed and their allocation, but does not indicate if the item is successfully pinned. For that, use the Status*() methods.

The operation can be aborted by cancelling the context. This methods blocks until the operation has completed.

func (*Cluster) Ready added in v0.0.3 

func (c *Cluster) Ready() <-chan struct{}

Ready returns a channel which signals when this peer is fully initialized (including consensus).

func (*Cluster) Recover added in v0.0.3 

func (c *Cluster) Recover(ctx context.Context, h api.Cid) (api.GlobalPinInfo, error)

Recover triggers a recover operation for a given Cid in all cluster peers.

Recover operations ask IPFS to pin or unpin items in error state. Recover is faster than calling Pin on the same CID as it avoids committing an identical pin to the consensus layer.

func (*Cluster) RecoverAll added in v0.12.0 

func (c *Cluster) RecoverAll(ctx context.Context, out chan<- api.GlobalPinInfo) error

RecoverAll triggers a RecoverAllLocal operation on all peers and returns GlobalPinInfo objets for all recovered items. This method blocks until finished. Operation can be aborted by cancelling the context.

func (*Cluster) RecoverAllLocal added in v0.3.1 

func (c *Cluster) RecoverAllLocal(ctx context.Context, out chan<- api.PinInfo) error

RecoverAllLocal triggers a RecoverLocal operation for all Cids tracked by this peer.

Recover operations ask IPFS to pin or unpin items in error state. Recover is faster than calling Pin on the same CID as it avoids committing an identical pin to the consensus layer.

It returns the list of pins that were re-queued for pinning on the out channel. It blocks until done.

RecoverAllLocal is called automatically every PinRecoverInterval.

func (*Cluster) RecoverLocal added in v0.0.3 

func (c *Cluster) RecoverLocal(ctx context.Context, h api.Cid) (api.PinInfo, error)

RecoverLocal triggers a recover operation for a given Cid in this peer only. It returns the updated PinInfo, after recovery.

Recover operations ask IPFS to pin or unpin items in error state. Recover is faster than calling Pin on the same CID as it avoids committing an identical pin to the consensus layer.

func (*Cluster) RepoGC added in v0.12.0 

func (c *Cluster) RepoGC(ctx context.Context) (api.GlobalRepoGC, error)

RepoGC performs garbage collection sweep on all peers' IPFS repo.

func (*Cluster) RepoGCLocal added in v0.12.0 

func (c *Cluster) RepoGCLocal(ctx context.Context) (api.RepoGC, error)

RepoGCLocal performs garbage collection only on the local IPFS deamon.

func (*Cluster) Shutdown 

func (c *Cluster) Shutdown(ctx context.Context) error

Shutdown performs all the necessary operations to shutdown the IPFS Cluster peer: * Save peerstore with the current peers * Remove itself from consensus when LeaveOnShutdown is set * It Shutdowns all the components * Collects all goroutines

Shutdown does not close the libp2p host, the DHT, the datastore or generally anything that Cluster did not create.

func (*Cluster) StateSync 

func (c *Cluster) StateSync(ctx context.Context) error

StateSync performs maintenance tasks on the global state that require looping through all the items. It is triggered automatically on StateSyncInterval. Currently it:

  • Sends unpin for expired items for which this peer is "closest" (skipped for follower peers)

func (*Cluster) Status 

func (c *Cluster) Status(ctx context.Context, h api.Cid) (api.GlobalPinInfo, error)

Status returns the GlobalPinInfo for a given Cid as fetched from all current peers. If an error happens, the GlobalPinInfo should contain as much information as could be fetched from the other peers.

func (*Cluster) StatusAll added in v0.0.3 

func (c *Cluster) StatusAll(ctx context.Context, filter api.TrackerStatus, out chan<- api.GlobalPinInfo) error

StatusAll returns the GlobalPinInfo for all tracked Cids in all peers on the out channel. This is done by broacasting a StatusAll to all peers. If an error happens, it is returned. This method blocks until it finishes. The operation can be aborted by cancelling the context.

func (*Cluster) StatusAllLocal added in v0.3.1 

func (c *Cluster) StatusAllLocal(ctx context.Context, filter api.TrackerStatus, out chan<- api.PinInfo) error

StatusAllLocal returns the PinInfo for all the tracked Cids in this peer on the out channel. It blocks until finished.

func (*Cluster) StatusLocal added in v0.3.1 

func (c *Cluster) StatusLocal(ctx context.Context, h api.Cid) api.PinInfo

StatusLocal returns this peer's PinInfo for a given Cid.

func (*Cluster) Unpin 

func (c *Cluster) Unpin(ctx context.Context, h api.Cid) (api.Pin, error)

Unpin removes a previously pinned Cid from Cluster. It returns the global state Pin object as it was stored before removal, or an error if it was not possible to update the global state.

Unpin does not reflect the success or failure of underlying IPFS daemon unpinning operations, which happen in async fashion.

func (*Cluster) UnpinPath added in v0.10.0 

func (c *Cluster) UnpinPath(ctx context.Context, path string) (api.Pin, error)

UnpinPath unpins a CID resolved from its IPFS Path. If returns the previously pinned Pin object.

func (*Cluster) Version 

func (c *Cluster) Version() string

Version returns the current IPFS Cluster version.

type ClusterRPCAPI added in v0.11.0 

type ClusterRPCAPI struct {
	// contains filtered or unexported fields
}

ClusterRPCAPI is a go-libp2p-gorpc service which provides the internal peer API for the main cluster component.

func (*ClusterRPCAPI) Alerts added in v0.13.1 

func (rpcapi *ClusterRPCAPI) Alerts(ctx context.Context, in struct{}, out *[]api.Alert) error

Alerts runs Cluster.Alerts().

func (*ClusterRPCAPI) BlockAllocate added in v0.11.0 

func (rpcapi *ClusterRPCAPI) BlockAllocate(ctx context.Context, in api.Pin, out *[]peer.ID) error

BlockAllocate returns allocations for blocks. This is used in the adders. It's different from pin allocations when ReplicationFactor < 0.

func (*ClusterRPCAPI) ConnectGraph added in v0.11.0 

func (rpcapi *ClusterRPCAPI) ConnectGraph(ctx context.Context, in struct{}, out *api.ConnectGraph) error

ConnectGraph runs Cluster.GetConnectGraph().

func (*ClusterRPCAPI) ID added in v0.11.0 

func (rpcapi *ClusterRPCAPI) ID(ctx context.Context, in struct{}, out *api.ID) error

ID runs Cluster.ID()

func (*ClusterRPCAPI) IDStream added in v1.0.0 

func (rpcapi *ClusterRPCAPI) IDStream(ctx context.Context, in <-chan struct{}, out chan<- api.ID) error

IDStream runs Cluster.ID() but in streaming form.

func (*ClusterRPCAPI) IPFSID added in v0.14.5 

func (rpcapi *ClusterRPCAPI) IPFSID(ctx context.Context, in peer.ID, out *api.IPFSID) error

IPFSID returns the current cached IPFS ID for a peer.

func (*ClusterRPCAPI) Join added in v0.11.0 

func (rpcapi *ClusterRPCAPI) Join(ctx context.Context, in api.Multiaddr, out *struct{}) error

Join runs Cluster.Join().

func (*ClusterRPCAPI) PeerAdd added in v0.11.0 

func (rpcapi *ClusterRPCAPI) PeerAdd(ctx context.Context, in peer.ID, out *api.ID) error

PeerAdd runs Cluster.PeerAdd().

func (*ClusterRPCAPI) PeerRemove added in v0.11.0 

func (rpcapi *ClusterRPCAPI) PeerRemove(ctx context.Context, in peer.ID, out *struct{}) error

PeerRemove runs Cluster.PeerRm().

func (*ClusterRPCAPI) Peers added in v0.11.0 

func (rpcapi *ClusterRPCAPI) Peers(ctx context.Context, in <-chan struct{}, out chan<- api.ID) error

Peers runs Cluster.Peers().

func (*ClusterRPCAPI) PeersWithFilter added in v1.0.0 

func (rpcapi *ClusterRPCAPI) PeersWithFilter(ctx context.Context, in <-chan []peer.ID, out chan<- api.ID) error

PeersWithFilter runs Cluster.peersWithFilter().

func (*ClusterRPCAPI) Pin added in v0.11.0 

func (rpcapi *ClusterRPCAPI) Pin(ctx context.Context, in api.Pin, out *api.Pin) error

Pin runs Cluster.pin().

func (*ClusterRPCAPI) PinGet added in v0.11.0 

func (rpcapi *ClusterRPCAPI) PinGet(ctx context.Context, in api.Cid, out *api.Pin) error

PinGet runs Cluster.PinGet().

func (*ClusterRPCAPI) PinPath added in v0.11.0 

func (rpcapi *ClusterRPCAPI) PinPath(ctx context.Context, in api.PinPath, out *api.Pin) error

PinPath resolves path into a cid and runs Cluster.Pin().

func (*ClusterRPCAPI) Pins added in v0.11.0 

func (rpcapi *ClusterRPCAPI) Pins(ctx context.Context, in <-chan struct{}, out chan<- api.Pin) error

Pins runs Cluster.Pins().

func (*ClusterRPCAPI) Recover added in v0.11.0 

func (rpcapi *ClusterRPCAPI) Recover(ctx context.Context, in api.Cid, out *api.GlobalPinInfo) error

Recover runs Cluster.Recover().

func (*ClusterRPCAPI) RecoverAll added in v0.12.0 

func (rpcapi *ClusterRPCAPI) RecoverAll(ctx context.Context, in <-chan struct{}, out chan<- api.GlobalPinInfo) error

RecoverAll runs Cluster.RecoverAll().

func (*ClusterRPCAPI) RecoverAllLocal added in v0.11.0 

func (rpcapi *ClusterRPCAPI) RecoverAllLocal(ctx context.Context, in <-chan struct{}, out chan<- api.PinInfo) error

RecoverAllLocal runs Cluster.RecoverAllLocal().

func (*ClusterRPCAPI) RecoverLocal added in v0.11.0 

func (rpcapi *ClusterRPCAPI) RecoverLocal(ctx context.Context, in api.Cid, out *api.PinInfo) error

RecoverLocal runs Cluster.RecoverLocal().

func (*ClusterRPCAPI) RepoGC added in v0.12.0 

func (rpcapi *ClusterRPCAPI) RepoGC(ctx context.Context, in struct{}, out *api.GlobalRepoGC) error

RepoGC performs garbage collection sweep on all peers' repos.

func (*ClusterRPCAPI) RepoGCLocal added in v0.12.0 

func (rpcapi *ClusterRPCAPI) RepoGCLocal(ctx context.Context, in struct{}, out *api.RepoGC) error

RepoGCLocal performs garbage collection sweep only on the local peer's IPFS daemon.

func (*ClusterRPCAPI) SendInformerMetrics added in v0.14.2 

func (rpcapi *ClusterRPCAPI) SendInformerMetrics(ctx context.Context, in struct{}, out *struct{}) error

SendInformerMetrics runs Cluster.sendInformerMetric().

func (*ClusterRPCAPI) SendInformersMetrics added in v0.12.0 

func (rpcapi *ClusterRPCAPI) SendInformersMetrics(ctx context.Context, in struct{}, out *struct{}) error

SendInformersMetrics runs Cluster.sendInformerMetric() on all informers.

func (*ClusterRPCAPI) Status added in v0.11.0 

func (rpcapi *ClusterRPCAPI) Status(ctx context.Context, in api.Cid, out *api.GlobalPinInfo) error

Status runs Cluster.Status().

func (*ClusterRPCAPI) StatusAll added in v0.11.0 

func (rpcapi *ClusterRPCAPI) StatusAll(ctx context.Context, in <-chan api.TrackerStatus, out chan<- api.GlobalPinInfo) error

StatusAll runs Cluster.StatusAll().

func (*ClusterRPCAPI) StatusAllLocal added in v0.11.0 

func (rpcapi *ClusterRPCAPI) StatusAllLocal(ctx context.Context, in <-chan api.TrackerStatus, out chan<- api.PinInfo) error

StatusAllLocal runs Cluster.StatusAllLocal().

func (*ClusterRPCAPI) StatusLocal added in v0.11.0 

func (rpcapi *ClusterRPCAPI) StatusLocal(ctx context.Context, in api.Cid, out *api.PinInfo) error

StatusLocal runs Cluster.StatusLocal().

func (*ClusterRPCAPI) Unpin added in v0.11.0 

func (rpcapi *ClusterRPCAPI) Unpin(ctx context.Context, in api.Pin, out *api.Pin) error

Unpin runs Cluster.Unpin().

func (*ClusterRPCAPI) UnpinPath added in v0.11.0 

func (rpcapi *ClusterRPCAPI) UnpinPath(ctx context.Context, in api.PinPath, out *api.Pin) error

UnpinPath resolves path into a cid and runs Cluster.Unpin().

func (*ClusterRPCAPI) Version added in v0.11.0 

func (rpcapi *ClusterRPCAPI) Version(ctx context.Context, in struct{}, out *api.Version) error

Version runs Cluster.Version().

type Component 

type Component interface {
	SetClient(*rpc.Client)
	Shutdown(context.Context) error
}

Component represents a piece of ipfscluster. Cluster components usually run their own goroutines (a http server for example). They communicate with the main Cluster component and other components (both local and remote), using an instance of rpc.Client.

type Config 

type Config struct {
	config.Saver

	// User-defined peername for use as human-readable identifier.
	Peername string

	// Cluster secret for private network. Peers will be in the same cluster if and
	// only if they have the same ClusterSecret. The cluster secret must be exactly
	// 64 characters and contain only hexadecimal characters (`[0-9a-f]`).
	Secret pnet.PSK

	// RPCPolicy defines access control to RPC endpoints.
	RPCPolicy map[string]RPCEndpointType

	// Leave Cluster on shutdown. Politely informs other peers
	// of the departure and removes itself from the consensus
	// peer set. The Cluster size will be reduced by one.
	LeaveOnShutdown bool

	// Listen parameters for the Cluster libp2p Host. Used by
	// the RPC and Consensus components.
	ListenAddr []ma.Multiaddr

	// Enables HOP relay for the node. If this is enabled, the node will act as
	// an intermediate (Hop Relay) node in relay circuits for connected peers.
	EnableRelayHop bool

	// ConnMgr holds configuration values for the connection manager for
	// the libp2p host.
	// FIXME: This only applies to ipfs-cluster-service.
	ConnMgr ConnMgrConfig

	// Sets the default dial timeout for libp2p connections to other
	// peers.
	DialPeerTimeout time.Duration

	// Time between syncs of the consensus state to the
	// tracker state. Normally states are synced anyway, but this helps
	// when new nodes are joining the cluster. Reduce for faster
	// consistency, increase with larger states.
	StateSyncInterval time.Duration

	// Time between automatic runs of the "recover" operation
	// which will retry to pin/unpin items in error state.
	PinRecoverInterval time.Duration

	// ReplicationFactorMax indicates the target number of nodes
	// that should pin content. For exampe, a replication_factor of
	// 3 will have cluster allocate each pinned hash to 3 peers if
	// possible.
	// See also ReplicationFactorMin. A ReplicationFactorMax of -1
	// will allocate to every available node.
	ReplicationFactorMax int

	// ReplicationFactorMin indicates the minimum number of healthy
	// nodes pinning content. If the number of nodes available to pin
	// is less than this threshold, an error will be returned.
	// In the case of peer health issues, content pinned will be
	// re-allocated if the threshold is crossed.
	// For exampe, a ReplicationFactorMin of 2 will allocate at least
	// two peer to hold content, and return an error if this is not
	// possible.
	ReplicationFactorMin int

	// MonitorPingInterval is the frequency with which a cluster peer
	// sends a "ping" metric. The metric has a TTL set to the double of
	// this value. This metric sends information about this peer to other
	// peers.
	MonitorPingInterval time.Duration

	// PeerWatchInterval is the frequency that we use to watch for changes
	// in the consensus peerset and save new peers to the configuration
	// file. This also affects how soon we realize that we have
	// been removed from a cluster.
	PeerWatchInterval time.Duration

	// MDNSInterval controls the time between mDNS broadcasts to the
	// network announcing the peer addresses. Set to 0 to disable
	// mDNS.
	MDNSInterval time.Duration

	// PinOnlyOnTrustedPeers limits allocations to trusted peers only.
	PinOnlyOnTrustedPeers bool

	// If true, DisableRepinning, ensures that no repinning happens
	// when a node goes down.
	// This is useful when doing certain types of maintenance, or simply
	// when not wanting to rely on the monitoring system which needs a revamp.
	DisableRepinning bool

	// FollowerMode disables broadcast requests from this peer
	// (sync, recover, status) and disallows pinset management
	// operations (Pin/Unpin).
	FollowerMode bool

	// Peerstore file specifies the file on which we persist the
	// libp2p host peerstore addresses. This file is regularly saved.
	PeerstoreFile string

	// PeerAddresses stores additional addresses for peers that may or may
	// not be in the peerstore file. These are considered high priority
	// when bootstrapping the initial cluster connections.
	PeerAddresses []ma.Multiaddr

	// Tracing flag used to skip tracing specific paths when not enabled.
	Tracing bool
}

Config is the configuration object containing customizable variables to initialize the main ipfs-cluster component. It implements the config.ComponentConfig interface.

func (*Config) ApplyEnvVars added in v0.10.0 

func (cfg *Config) ApplyEnvVars() error

ApplyEnvVars fills in any Config fields found as environment variables.

func (*Config) ConfigKey added in v0.2.0 

func (cfg *Config) ConfigKey() string

ConfigKey returns a human-readable string to identify a cluster Config.

func (*Config) Default added in v0.2.0 

func (cfg *Config) Default() error

Default fills in all the Config fields with default working values. This means, it will generate a Secret.

func (*Config) GetPeerstorePath added in v0.4.0 

func (cfg *Config) GetPeerstorePath() string

GetPeerstorePath returns the full path of the PeerstoreFile, obtained by concatenating that value with BaseDir of the configuration, if set. An empty string is returned when BaseDir is not set.

func (*Config) LoadJSON added in v0.2.0 

func (cfg *Config) LoadJSON(raw []byte) error

LoadJSON receives a raw json-formatted configuration and sets the Config fields from it. Note that it should be JSON as generated by ToJSON().

func (*Config) ToDisplayJSON added in v0.13.0 

func (cfg *Config) ToDisplayJSON() ([]byte, error)

ToDisplayJSON returns JSON config as a string.

func (*Config) ToJSON added in v0.2.0 

func (cfg *Config) ToJSON() (raw []byte, err error)

ToJSON generates a human-friendly version of Config.

func (*Config) Validate added in v0.2.0 

func (cfg *Config) Validate() error

Validate will check that the values of this config seem to be working ones.

type ConnMgrConfig added in v0.11.0 

type ConnMgrConfig struct {
	HighWater   int
	LowWater    int
	GracePeriod time.Duration
}

ConnMgrConfig configures the libp2p host connection manager.

type Consensus 

type Consensus interface {
	Component
	// Returns a channel to signal that the consensus layer is ready
	// allowing the main component to wait for it during start.
	Ready(context.Context) <-chan struct{}
	// Logs a pin operation.
	LogPin(context.Context, api.Pin) error
	// Logs an unpin operation.
	LogUnpin(context.Context, api.Pin) error
	AddPeer(context.Context, peer.ID) error
	RmPeer(context.Context, peer.ID) error
	State(context.Context) (state.ReadOnly, error)
	// Provide a node which is responsible to perform
	// specific tasks which must only run in 1 cluster peer.
	Leader(context.Context) (peer.ID, error)
	// Only returns when the consensus state has all log
	// updates applied to it.
	WaitForSync(context.Context) error
	// Clean removes all consensus data.
	Clean(context.Context) error
	// Peers returns the peerset participating in the Consensus.
	Peers(context.Context) ([]peer.ID, error)
	// IsTrustedPeer returns true if the given peer is "trusted".
	// This will grant access to more rpc endpoints and a
	// non-trusted one. This should be fast as it will be
	// called repeatedly for every remote RPC request.
	IsTrustedPeer(context.Context, peer.ID) bool
	// Trust marks a peer as "trusted".
	Trust(context.Context, peer.ID) error
	// Distrust removes a peer from the "trusted" set.
	Distrust(context.Context, peer.ID) error
}

Consensus is a component which keeps a shared state in IPFS Cluster and triggers actions on updates to that state. Currently, Consensus needs to be able to elect/provide a Cluster Leader and the implementation is very tight to the Cluster main component.

type ConsensusRPCAPI added in v0.11.0 

type ConsensusRPCAPI struct {
	// contains filtered or unexported fields
}

ConsensusRPCAPI is a go-libp2p-gorpc service which provides the internal peer API for the Consensus component.

func (*ConsensusRPCAPI) AddPeer added in v0.11.0 

func (rpcapi *ConsensusRPCAPI) AddPeer(ctx context.Context, in peer.ID, out *struct{}) error

AddPeer runs Consensus.AddPeer().

func (*ConsensusRPCAPI) LogPin added in v0.11.0 

func (rpcapi *ConsensusRPCAPI) LogPin(ctx context.Context, in api.Pin, out *struct{}) error

LogPin runs Consensus.LogPin().

func (*ConsensusRPCAPI) LogUnpin added in v0.11.0 

func (rpcapi *ConsensusRPCAPI) LogUnpin(ctx context.Context, in api.Pin, out *struct{}) error

LogUnpin runs Consensus.LogUnpin().

func (*ConsensusRPCAPI) Peers added in v0.11.0 

func (rpcapi *ConsensusRPCAPI) Peers(ctx context.Context, in struct{}, out *[]peer.ID) error

Peers runs Consensus.Peers().

func (*ConsensusRPCAPI) RmPeer added in v0.11.0 

func (rpcapi *ConsensusRPCAPI) RmPeer(ctx context.Context, in peer.ID, out *struct{}) error

RmPeer runs Consensus.RmPeer().

type IPFSConnector 

type IPFSConnector interface {
	Component
	ID(context.Context) (api.IPFSID, error)
	Pin(context.Context, api.Pin) error
	Unpin(context.Context, api.Cid) error
	PinLsCid(context.Context, api.Pin) (api.IPFSPinStatus, error)
	// PinLs returns pins in the pinset of the given types (recursive, direct...)
	PinLs(ctx context.Context, typeFilters []string, out chan<- api.IPFSPinInfo) error
	// ConnectSwarms make sure this peer's IPFS daemon is connected to
	// other peers IPFS daemons.
	ConnectSwarms(context.Context) error
	// SwarmPeers returns the IPFS daemon's swarm peers.
	SwarmPeers(context.Context) ([]peer.ID, error)
	// ConfigKey returns the value for a configuration key.
	// Subobjects are reached with keypaths as "Parent/Child/GrandChild...".
	ConfigKey(keypath string) (interface{}, error)
	// RepoStat returns the current repository size and max limit as
	// provided by "repo stat".
	RepoStat(context.Context) (api.IPFSRepoStat, error)
	// RepoGC performs garbage collection sweep on the IPFS repo.
	RepoGC(context.Context) (api.RepoGC, error)
	// Resolve returns a cid given a path.
	Resolve(context.Context, string) (api.Cid, error)
	// BlockStream adds a stream of blocks to IPFS.
	BlockStream(context.Context, <-chan api.NodeWithMeta) error
	// BlockGet retrieves the raw data of an IPFS block.
	BlockGet(context.Context, api.Cid) ([]byte, error)
}

IPFSConnector is a component which allows cluster to interact with an IPFS daemon. This is a base component.

type IPFSConnectorRPCAPI added in v0.11.0 

type IPFSConnectorRPCAPI struct {
	// contains filtered or unexported fields
}

IPFSConnectorRPCAPI is a go-libp2p-gorpc service which provides the internal peer API for the IPFSConnector component.

func (*IPFSConnectorRPCAPI) BlockGet added in v0.11.0 

func (rpcapi *IPFSConnectorRPCAPI) BlockGet(ctx context.Context, in api.Cid, out *[]byte) error

BlockGet runs IPFSConnector.BlockGet().

func (*IPFSConnectorRPCAPI) BlockStream added in v1.0.0 

func (rpcapi *IPFSConnectorRPCAPI) BlockStream(ctx context.Context, in <-chan api.NodeWithMeta, out chan<- struct{}) error

BlockStream runs IPFSConnector.BlockStream().

func (*IPFSConnectorRPCAPI) ConfigKey added in v0.11.0 

func (rpcapi *IPFSConnectorRPCAPI) ConfigKey(ctx context.Context, in string, out *interface{}) error

ConfigKey runs IPFSConnector.ConfigKey().

func (*IPFSConnectorRPCAPI) Pin added in v0.11.0 

func (rpcapi *IPFSConnectorRPCAPI) Pin(ctx context.Context, in api.Pin, out *struct{}) error

Pin runs IPFSConnector.Pin().

func (*IPFSConnectorRPCAPI) PinLs added in v0.11.0 

func (rpcapi *IPFSConnectorRPCAPI) PinLs(ctx context.Context, in <-chan []string, out chan<- api.IPFSPinInfo) error

PinLs runs IPFSConnector.PinLs().

func (*IPFSConnectorRPCAPI) PinLsCid added in v0.11.0 

func (rpcapi *IPFSConnectorRPCAPI) PinLsCid(ctx context.Context, in api.Pin, out *api.IPFSPinStatus) error

PinLsCid runs IPFSConnector.PinLsCid().

func (*IPFSConnectorRPCAPI) RepoStat added in v0.11.0 

func (rpcapi *IPFSConnectorRPCAPI) RepoStat(ctx context.Context, in struct{}, out *api.IPFSRepoStat) error

RepoStat runs IPFSConnector.RepoStat().

func (*IPFSConnectorRPCAPI) Resolve added in v0.11.0 

func (rpcapi *IPFSConnectorRPCAPI) Resolve(ctx context.Context, in string, out *api.Cid) error

Resolve runs IPFSConnector.Resolve().

func (*IPFSConnectorRPCAPI) SwarmPeers added in v0.11.0 

func (rpcapi *IPFSConnectorRPCAPI) SwarmPeers(ctx context.Context, in struct{}, out *[]peer.ID) error

SwarmPeers runs IPFSConnector.SwarmPeers().

func (*IPFSConnectorRPCAPI) Unpin added in v0.11.0 

func (rpcapi *IPFSConnectorRPCAPI) Unpin(ctx context.Context, in api.Pin, out *struct{}) error

Unpin runs IPFSConnector.Unpin().

type Informer added in v0.0.3 

type Informer interface {
	Component
	Name() string
	// GetMetrics returns the metrics obtained by this Informer.  It must
	// always return at least one metric.
	GetMetrics(context.Context) []api.Metric
}

Informer provides Metric information from a peer. The metrics produced by informers are then passed to a PinAllocator which will use them to determine where to pin content. The metric is agnostic to the rest of Cluster.

type PeerMonitor added in v0.0.3 

type PeerMonitor interface {
	Component
	// LogMetric stores a metric. It can be used to manually inject
	// a metric to a monitor.
	LogMetric(context.Context, api.Metric) error
	// PublishMetric sends a metric to the rest of the peers.
	// How to send it, and to who, is to be decided by the implementation.
	PublishMetric(context.Context, api.Metric) error
	// LatestMetrics returns a map with the latest valid metrics of matching
	// name for the current cluster peers. The result should only contain
	// one metric per peer at most.
	LatestMetrics(ctx context.Context, name string) []api.Metric
	// Returns the latest metric received from a peer. It may be expired.
	LatestForPeer(ctx context.Context, name string, pid peer.ID) api.Metric
	// MetricNames returns a list of metric names.
	MetricNames(ctx context.Context) []string
	// Alerts delivers alerts generated when this peer monitor detects
	// a problem (i.e. metrics not arriving as expected). Alerts can be used
	// to trigger self-healing measures or re-pinnings of content.
	Alerts() <-chan api.Alert
}

PeerMonitor is a component in charge of publishing a peer's metrics and reading metrics from other peers in the cluster. The PinAllocator will use the metrics provided by the monitor as candidates for Pin allocations.

The PeerMonitor component also provides an Alert channel which is signaled when a metric is no longer received and the monitor identifies it as a problem.

type PeerMonitorRPCAPI added in v0.11.0 

type PeerMonitorRPCAPI struct {
	// contains filtered or unexported fields
}

PeerMonitorRPCAPI is a go-libp2p-gorpc service which provides the internal peer API for the PeerMonitor component.

func (*PeerMonitorRPCAPI) LatestMetrics added in v0.11.0 

func (rpcapi *PeerMonitorRPCAPI) LatestMetrics(ctx context.Context, in string, out *[]api.Metric) error

LatestMetrics runs PeerMonitor.LatestMetrics().

func (*PeerMonitorRPCAPI) MetricNames added in v0.12.0 

func (rpcapi *PeerMonitorRPCAPI) MetricNames(ctx context.Context, in struct{}, out *[]string) error

MetricNames runs PeerMonitor.MetricNames().

type Peered 

type Peered interface {
	AddPeer(ctx context.Context, p peer.ID)
	RmPeer(ctx context.Context, p peer.ID)
}

Peered represents a component which needs to be aware of the peers in the Cluster and of any changes to the peer set.

type PinAllocator added in v0.0.3 

type PinAllocator interface {
	Component
	// Allocate returns the list of peers that should be assigned to
	// Pin content in order of preference (from the most preferred to the
	// least). The "current" map contains valid metrics for peers
	// which are currently pinning the content. The candidates map
	// contains the metrics for all peers which are eligible for pinning
	// the content.
	Allocate(ctx context.Context, c api.Cid, current, candidates, priority api.MetricsSet) ([]peer.ID, error)
	// Metrics returns the list of metrics that the allocator needs.
	Metrics() []string
}

PinAllocator decides where to pin certain content. In order to make such decision, it receives the pin arguments, the peers which are currently allocated to the content and metrics available for all peers which could allocate the content.

type PinTracker 

type PinTracker interface {
	Component
	// Track tells the tracker that a Cid is now under its supervision
	// The tracker may decide to perform an IPFS pin.
	Track(context.Context, api.Pin) error
	// Untrack tells the tracker that a Cid is to be forgotten. The tracker
	// may perform an IPFS unpin operation.
	Untrack(context.Context, api.Cid) error
	// StatusAll returns the list of pins with their local status. Takes a
	// filter to specify which statuses to report.
	StatusAll(context.Context, api.TrackerStatus, chan<- api.PinInfo) error
	// Status returns the local status of a given Cid.
	Status(context.Context, api.Cid) api.PinInfo
	// RecoverAll calls Recover() for all pins tracked.
	RecoverAll(context.Context, chan<- api.PinInfo) error
	// Recover retriggers a Pin/Unpin operation in a Cids with error status.
	Recover(context.Context, api.Cid) (api.PinInfo, error)
	// PinQueueSize returns the current size of the pinning queue.
	PinQueueSize(context.Context) (int64, error)
}

PinTracker represents a component which tracks the status of the pins in this cluster and ensures they are in sync with the IPFS daemon. This component should be thread safe.

type PinTrackerRPCAPI added in v0.11.0 

type PinTrackerRPCAPI struct {
	// contains filtered or unexported fields
}

PinTrackerRPCAPI is a go-libp2p-gorpc service which provides the internal peer API for the PinTracker component.

func (*PinTrackerRPCAPI) PinQueueSize added in v1.0.1 

func (rpcapi *PinTrackerRPCAPI) PinQueueSize(ctx context.Context, in struct{}, out *int64) error

PinQueueSize runs PinTracker.PinQueueSize().

func (*PinTrackerRPCAPI) Recover added in v0.11.0 

func (rpcapi *PinTrackerRPCAPI) Recover(ctx context.Context, in api.Cid, out *api.PinInfo) error

Recover runs PinTracker.Recover().

func (*PinTrackerRPCAPI) RecoverAll added in v0.11.0 

func (rpcapi *PinTrackerRPCAPI) RecoverAll(ctx context.Context, in <-chan struct{}, out chan<- api.PinInfo) error

RecoverAll runs PinTracker.RecoverAll().f

func (*PinTrackerRPCAPI) Status added in v0.11.0 

func (rpcapi *PinTrackerRPCAPI) Status(ctx context.Context, in api.Cid, out *api.PinInfo) error

Status runs PinTracker.Status().

func (*PinTrackerRPCAPI) StatusAll added in v0.11.0 

func (rpcapi *PinTrackerRPCAPI) StatusAll(ctx context.Context, in <-chan api.TrackerStatus, out chan<- api.PinInfo) error

StatusAll runs PinTracker.StatusAll().

func (*PinTrackerRPCAPI) Track added in v0.11.0 

func (rpcapi *PinTrackerRPCAPI) Track(ctx context.Context, in api.Pin, out *struct{}) error

Track runs PinTracker.Track().

func (*PinTrackerRPCAPI) Untrack added in v0.11.0 

func (rpcapi *PinTrackerRPCAPI) Untrack(ctx context.Context, in api.Pin, out *struct{}) error

Untrack runs PinTracker.Untrack().

type RPCEndpointType added in v0.11.0 

type RPCEndpointType int

RPCEndpointType controls how access is granted to an RPC endpoint 

const (
	// RPCClosed endpoints can only be called by the local cluster peer
	// on itself.
	RPCClosed RPCEndpointType = iota
	// RPCTrusted endpoints can be called by "trusted" peers.
	// It depends which peers are considered trusted. For example,
	// in "raft" mode, Cluster will all peers as "trusted". In "crdt" mode,
	// trusted peers are those specified in the configuration.
	RPCTrusted
	// RPCOpen endpoints can be called by any peer in the Cluster swarm.
	RPCOpen
)

RPC endpoint types w.r.t. trust level

type Tracer added in v0.9.0 

type Tracer interface {
	Component
}

Tracer implements Component as a way to shutdown and flush and remaining traces.

Source Files

View all Source files

Directories

Path Synopsis
Package adder implements functionality to add content to IPFS daemons managed by the Cluster.
 Package adder implements functionality to add content to IPFS daemons managed by the Cluster.
adderutils
Package adderutils provides some utilities for adding content to cluster.
 Package adderutils provides some utilities for adding content to cluster.
ipfsadd
Package ipfsadd is a simplified copy of go-ipfs/core/coreunix/add.go
 Package ipfsadd is a simplified copy of go-ipfs/core/coreunix/add.go
sharding
Package sharding implements a sharding ClusterDAGService places content in different shards while it's being added, creating a final Cluster DAG and pinning it.
 Package sharding implements a sharding ClusterDAGService places content in different shards while it's being added, creating a final Cluster DAG and pinning it.
single
Package single implements a ClusterDAGService that chunks and adds content to cluster without sharding, before pinning it.
 Package single implements a ClusterDAGService that chunks and adds content to cluster without sharding, before pinning it.
allocator
balanced
Package balanced implements an allocator that can sort allocations based on multiple metrics, where metrics may be an arbitrary way to partition a set of peers.
 Package balanced implements an allocator that can sort allocations based on multiple metrics, where metrics may be an arbitrary way to partition a set of peers.
api
Package api holds declarations for types used in ipfs-cluster APIs to make them re-usable across differen tools.
 Package api holds declarations for types used in ipfs-cluster APIs to make them re-usable across differen tools.
common
Package common implements all the things that an IPFS Cluster API component must do, except the actual routes that it handles.
 Package common implements all the things that an IPFS Cluster API component must do, except the actual routes that it handles.
common/test
Package test provides utility methods to test APIs based on the common API.
 Package test provides utility methods to test APIs based on the common API.
ipfsproxy
Package ipfsproxy implements the Cluster API interface by providing an IPFS HTTP interface as exposed by the go-ipfs daemon.
 Package ipfsproxy implements the Cluster API interface by providing an IPFS HTTP interface as exposed by the go-ipfs daemon.
pb
Package pb provides protobuf definitions for serialized types in Cluster.
 Package pb provides protobuf definitions for serialized types in Cluster.
pinsvcapi
Package pinsvcapi implements an IPFS Cluster API component which provides an IPFS Pinning Services API to the cluster.
 Package pinsvcapi implements an IPFS Cluster API component which provides an IPFS Pinning Services API to the cluster.
pinsvcapi/pinsvc
Package pinsvc contains type definitions for the Pinning Services API
 Package pinsvc contains type definitions for the Pinning Services API
rest
Package rest implements an IPFS Cluster API component.
 Package rest implements an IPFS Cluster API component.
rest/client
Package client provides a Go Client for the IPFS Cluster API provided by the "api/rest" component.
 Package client provides a Go Client for the IPFS Cluster API provided by the "api/rest" component.
cmd
ipfs-cluster-ctl
The ipfs-cluster-ctl application.
 The ipfs-cluster-ctl application.
ipfs-cluster-follow
The ipfs-cluster-follow application.
 The ipfs-cluster-follow application.
ipfs-cluster-service
The ipfs-cluster-service application.
 The ipfs-cluster-service application.
Package cmdutils contains utilities to facilitate building of command line applications launching cluster peers.
 Package cmdutils contains utilities to facilitate building of command line applications launching cluster peers.
Package config provides interfaces and utilities for different Cluster components to register, read, write and validate configuration sections stored in a central configuration file.
 Package config provides interfaces and utilities for different Cluster components to register, read, write and validate configuration sections stored in a central configuration file.
consensus
crdt
Package crdt implements the IPFS Cluster consensus interface using CRDT-datastore to replicate the cluster global state to every peer.
 Package crdt implements the IPFS Cluster consensus interface using CRDT-datastore to replicate the cluster global state to every peer.
raft
Package raft implements a Consensus component for IPFS Cluster which uses Raft (go-libp2p-raft).
 Package raft implements a Consensus component for IPFS Cluster which uses Raft (go-libp2p-raft).
datastore
badger
Package badger provides a configurable BadgerDB go-datastore for use with IPFS Cluster.
 Package badger provides a configurable BadgerDB go-datastore for use with IPFS Cluster.
inmem
Package inmem provides a in-memory thread-safe datastore for use with Cluster.
 Package inmem provides a in-memory thread-safe datastore for use with Cluster.
leveldb
Package leveldb provides a configurable LevelDB go-datastore for use with IPFS Cluster.
 Package leveldb provides a configurable LevelDB go-datastore for use with IPFS Cluster.
informer
disk
Package disk implements an ipfs-cluster informer which can provide different disk-related metrics from the IPFS daemon as an api.Metric.
 Package disk implements an ipfs-cluster informer which can provide different disk-related metrics from the IPFS daemon as an api.Metric.
numpin
Package numpin implements an ipfs-cluster informer which determines how many items this peer is pinning and returns it as api.Metric
 Package numpin implements an ipfs-cluster informer which determines how many items this peer is pinning and returns it as api.Metric
pinqueue
Package pinqueue implements an ipfs-cluster informer which issues the current size of the pinning queue.
 Package pinqueue implements an ipfs-cluster informer which issues the current size of the pinning queue.
tags
Package tags implements an ipfs-cluster informer publishes user-defined tags as metrics.
 Package tags implements an ipfs-cluster informer publishes user-defined tags as metrics.
ipfsconn
ipfshttp
Package ipfshttp implements an IPFS Cluster IPFSConnector component.
 Package ipfshttp implements an IPFS Cluster IPFSConnector component.
monitor
metrics
Package metrics provides common functionality for working with metrics, particularly useful for monitoring components.
 Package metrics provides common functionality for working with metrics, particularly useful for monitoring components.
pubsubmon
Package pubsubmon implements a PeerMonitor component for IPFS Cluster that uses PubSub to send and receive metrics.
 Package pubsubmon implements a PeerMonitor component for IPFS Cluster that uses PubSub to send and receive metrics.
Package observations sets up metric and trace exporting for IPFS cluster.
 Package observations sets up metric and trace exporting for IPFS cluster.
pintracker
optracker
Package optracker implements functionality to track the status of pin and operations as needed by implementations of the pintracker component.
 Package optracker implements functionality to track the status of pin and operations as needed by implementations of the pintracker component.
stateless
Package stateless implements a PinTracker component for IPFS Cluster, which aims to reduce the memory footprint when handling really large cluster states.
 Package stateless implements a PinTracker component for IPFS Cluster, which aims to reduce the memory footprint when handling really large cluster states.
Package pstoremgr provides a Manager that simplifies handling addition, listing and removal of cluster peer multiaddresses from the libp2p Host.
 Package pstoremgr provides a Manager that simplifies handling addition, listing and removal of cluster peer multiaddresses from the libp2p Host.
Package rpcutil provides utility methods to perform go-libp2p-gorpc calls, particularly gorpc.MultiCall().
 Package rpcutil provides utility methods to perform go-libp2p-gorpc calls, particularly gorpc.MultiCall().
policygen
Package state holds the interface that any state implementation for IPFS Cluster must satisfy.
 Package state holds the interface that any state implementation for IPFS Cluster must satisfy.
dsstate
Package dsstate implements the IPFS Cluster state interface using an underlying go-datastore.
 Package dsstate implements the IPFS Cluster state interface using an underlying go-datastore.
Package test offers testing utilities for all the IPFS Cluster codebase, like IPFS daemon and RPC mocks and pre-defined testing CIDs.
 Package test offers testing utilities for all the IPFS Cluster codebase, like IPFS daemon and RPC mocks and pre-defined testing CIDs.
Package version stores version information for IPFS Cluster.
 Package version stores version information for IPFS Cluster.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.