storageimpl

package
v0.0.0-...-75360cb Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 11, 2023 License: Apache-2.0, MIT Imports: 52 Imported by: 0

Documentation

Overview

Package storageimpl provides the primary implementation of storage market top level interfaces

This package provides a production implementation of `StorageClient` and `StorageProvider`.

Index

Constants

View Source
const DefaultMaxTraversalLinks = 2 << 29

DefaultMaxTraversalLinks is the maximum number of links to traverse during CommP calculation before returning an error

View Source
const DefaultPollingInterval = 30 * time.Second

DefaultPollingInterval is the frequency with which we query the provider for a status update

Variables

View Source
var ClientFSMParameterSpec = fsm.Parameters{
	Environment:     &clientDealEnvironment{},
	StateType:       storagemarket.ClientDeal{},
	StateKeyField:   "State",
	Events:          clientstates.ClientEvents,
	StateEntryFuncs: clientstates.ClientStateEntryFuncs,
	FinalityStates:  clientstates.ClientFinalityStates,
}

ClientFSMParameterSpec is a valid set of parameters for a client deal FSM - used in doc generation

View Source
var ProviderFSMParameterSpec = fsm.Parameters{
	Environment:     &providerDealEnvironment{},
	StateType:       storagemarket.MinerDeal{},
	StateKeyField:   "State",
	Events:          providerstates.ProviderEvents,
	StateEntryFuncs: providerstates.ProviderStateEntryFuncs,
	FinalityStates:  providerstates.ProviderFinalityStates,
}

ProviderFSMParameterSpec is a valid set of parameters for a provider FSM - used in doc generation

Functions

func NewProvider

func NewProvider(net network.StorageMarketNetwork,
	ds datastore.Batching,
	fs filestore.FileStore,
	dagStore stores.DAGStoreWrapper,
	indexer provider.Interface,
	pieceStore piecestore.PieceStore,
	dataTransfer datatransfer.Manager,
	spn storagemarket.StorageProviderNode,
	minerAddress address.Address,
	storedAsk StoredAsk,
	meshCreator MeshCreator,
	options ...StorageProviderOption,
) (storagemarket.StorageProvider, error)

NewProvider returns a new storage provider

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is the production implementation of the StorageClient interface

func NewClient

func NewClient(
	net network.StorageMarketNetwork,
	dataTransfer datatransfer.Manager,
	discovery *discoveryimpl.Local,
	ds datastore.Batching,
	scn storagemarket.StorageClientNode,
	bstores storagemarket.BlockstoreAccessor,
	options ...StorageClientOption,
) (*Client, error)

NewClient creates a new storage client

func (*Client) AddPaymentEscrow

func (c *Client) AddPaymentEscrow(ctx context.Context, addr address.Address, amount abi.TokenAmount) error

AddPaymentEscrow adds funds for storage deals

func (*Client) Configure

func (c *Client) Configure(options ...StorageClientOption)

Configure applies the given list of StorageClientOptions after a StorageClient is initialized

func (*Client) GetAsk

GetAsk queries a provider for its current storage ask

The client creates a new `StorageAskStream` for the chosen peer ID, and calls WriteAskRequest on it, which constructs a message and writes it to the Ask stream. When it receives a response, it verifies the signature and returns the validated StorageAsk if successful

func (*Client) GetLocalDeal

func (c *Client) GetLocalDeal(ctx context.Context, cid cid.Cid) (storagemarket.ClientDeal, error)

GetLocalDeal lists deals that are in progress or rejected

func (*Client) GetPaymentEscrow

func (c *Client) GetPaymentEscrow(ctx context.Context, addr address.Address) (storagemarket.Balance, error)

GetPaymentEscrow returns the current funds available for deal payment

func (*Client) GetProviderDealState

func (c *Client) GetProviderDealState(ctx context.Context, proposalCid cid.Cid) (*storagemarket.ProviderDealState, error)

GetProviderDealState queries a provider for the current state of a client's deal

func (*Client) ListLocalDeals

func (c *Client) ListLocalDeals(ctx context.Context) ([]storagemarket.ClientDeal, error)

ListLocalDeals lists deals initiated by this storage client

func (*Client) ListProviders

func (c *Client) ListProviders(ctx context.Context) (<-chan storagemarket.StorageProviderInfo, error)

ListProviders queries chain state and returns active storage providers

func (*Client) OnReady

func (c *Client) OnReady(ready shared.ReadyFunc)

OnReady registers a listener for when the client has finished starting up

func (*Client) PollingInterval

func (c *Client) PollingInterval() time.Duration

PollingInterval is a getter for the polling interval option

func (*Client) ProposeStorageDeal

ProposeStorageDeal initiates the retrieval deal flow, which involves multiple requests and responses.

This function is called after using ListProviders and QueryAs are used to identify an appropriate provider to store data. The parameters passed to ProposeStorageDeal should matched those returned by the miner from QueryAsk to ensure the greatest likelihood the provider will accept the deal.

When called, the client takes the following actions:

1. Calculates the PieceCID for this deal from the given PayloadCID. (by writing the payload to a CAR file then calculating a merkle root for the resulting data)

2. Constructs a `DealProposal` (spec-actors type) with deal terms

3. Signs the `DealProposal` to make a ClientDealProposal

4. Gets the CID for the ClientDealProposal

5. Construct a ClientDeal to track the state of this deal.

6. Tells its statemachine to begin tracking the deal state by the CID of the ClientDealProposal

7. Triggers a `ClientEventOpen` event on its statemachine.

8. Records the Provider as a possible peer for retrieving this data in the future

From then on, the statemachine controls the deal flow in the client. Other components may listen for events in this flow by calling `SubscribeToEvents` on the Client. The Client also provides access to the node and network and other functionality through its implementation of the Client FSM's ClientDealEnvironment.

Documentation of the client state machine can be found at https://godoc.org/github.com/filecoin-project/go-fil-markets/storagemarket/impl/clientstates

func (*Client) Start

func (c *Client) Start(ctx context.Context) error

Start initializes deal processing on a StorageClient, runs migrations and restarts in progress deals

func (*Client) Stop

func (c *Client) Stop() error

Stop ends deal processing on a StorageClient

func (*Client) SubscribeToEvents

func (c *Client) SubscribeToEvents(subscriber storagemarket.ClientSubscriber) shared.Unsubscribe

SubscribeToEvents allows another component to listen for events on the StorageClient in order to track deals as they progress through the deal flow

type DealDeciderFunc

type DealDeciderFunc func(context.Context, storagemarket.MinerDeal) (bool, string, error)

DealDeciderFunc is a function which evaluates an incoming deal to decide if it its accepted It returns: - boolean = true if deal accepted, false if rejected - string = reason deal was not excepted, if rejected - error = if an error occurred trying to decide

type MeshCreator

type MeshCreator interface {
	Connect(context.Context) error
}

type MetadataFunc

type MetadataFunc func(storagemarket.MinerDeal) metadata.Metadata

type Provider

type Provider struct {
	// contains filtered or unexported fields
}

Provider is the production implementation of the StorageProvider interface

func (*Provider) AddStorageCollateral

func (p *Provider) AddStorageCollateral(ctx context.Context, amount abi.TokenAmount) error

AddStorageCollateral adds storage collateral

func (*Provider) AnnounceAllDealsToIndexer

func (p *Provider) AnnounceAllDealsToIndexer(ctx context.Context) error

func (*Provider) AnnounceDealToIndexer

func (p *Provider) AnnounceDealToIndexer(ctx context.Context, proposalCid cid.Cid) error

AnnounceDealToIndexer informs indexer nodes that a new deal was received, so they can download its index

func (*Provider) AwaitReady

func (p *Provider) AwaitReady() error

func (*Provider) Configure

func (p *Provider) Configure(options ...StorageProviderOption)

Configure applies the given list of StorageProviderOptions after a StorageProvider is initialized

func (*Provider) GetAsk

GetAsk returns the storage miner's ask, or nil if one does not exist.

func (*Provider) GetLocalDeal

func (p *Provider) GetLocalDeal(propCid cid.Cid) (storagemarket.MinerDeal, error)

func (*Provider) GetStorageCollateral

func (p *Provider) GetStorageCollateral(ctx context.Context) (storagemarket.Balance, error)

GetStorageCollateral returns the current collateral balance

func (*Provider) HandleAskStream

func (p *Provider) HandleAskStream(s network.StorageAskStream)

HandleAskStream is called by the network implementation whenever a new message is received on the ask protocol

A Provider handling a `AskRequest` does the following:

1. Reads the current signed storage ask from storage

2. Wraps the signed ask in an AskResponse and writes it on the StorageAskStream

The connection is kept open only as long as the request-response exchange.

func (*Provider) HandleDealStatusStream

func (p *Provider) HandleDealStatusStream(s network.DealStatusStream)

HandleDealStatusStream is called by the network implementation whenever a new message is received on the deal status protocol

A Provider handling a `DealStatuRequest` does the following:

1. Lots the deal state from the Provider FSM

2. Verifies the signature on the DealStatusRequest matches the Client for this deal

3. Constructs a ProviderDealState from the deal state

4. Signs the ProviderDealState with its private key

5. Writes a DealStatusResponse with the ProviderDealState and signature onto the DealStatusStream

The connection is kept open only as long as the request-response exchange.

func (*Provider) HandleDealStream

func (p *Provider) HandleDealStream(s network.StorageDealStream)

HandleDealStream is called by the network implementation whenever a new message is received on the deal protocol

It initiates the provider side of the deal flow.

When a provider receives a DealProposal of the deal protocol, it takes the following steps:

1. Calculates the CID for the received ClientDealProposal.

2. Constructs a MinerDeal to track the state of this deal.

3. Tells its statemachine to begin tracking this deal state by CID of the received ClientDealProposal

4. Tracks the received deal stream by the CID of the ClientDealProposal

4. Triggers a `ProviderEventOpen` event on its statemachine.

From then on, the statemachine controls the deal flow in the client. Other components may listen for events in this flow by calling `SubscribeToEvents` on the Provider. The Provider handles loading the next block to send to the client.

Documentation of the client state machine can be found at https://godoc.org/github.com/filecoin-project/go-fil-markets/storagemarket/impl/providerstates

func (*Provider) ImportDataForDeal

func (p *Provider) ImportDataForDeal(ctx context.Context, propCid cid.Cid, data io.Reader) error

ImportDataForDeal manually imports data for an offline storage deal It will verify that the data in the passed io.Reader matches the expected piece cid for the given deal or it will error

func (*Provider) ListLocalDeals

func (p *Provider) ListLocalDeals() ([]storagemarket.MinerDeal, error)

ListLocalDeals lists deals processed by this storage provider

func (*Provider) ListLocalDealsPage

func (p *Provider) ListLocalDealsPage(startPropCid *cid.Cid, offset int, limit int) ([]storagemarket.MinerDeal, error)

func (*Provider) LocalDealCount

func (p *Provider) LocalDealCount() (int, error)

func (*Provider) OnReady

func (p *Provider) OnReady(ready shared.ReadyFunc)

OnReady registers a listener for when the provider has finished starting up

func (*Provider) RetryDealPublishing

func (p *Provider) RetryDealPublishing(propcid cid.Cid) error

func (*Provider) SetAsk

func (p *Provider) SetAsk(price abi.TokenAmount, verifiedPrice abi.TokenAmount, duration abi.ChainEpoch, options ...storagemarket.StorageAskOption) error

SetAsk configures the storage miner's ask with the provided price, duration, and options. Any previously-existing ask is replaced.

func (*Provider) Start

func (p *Provider) Start(ctx context.Context) error

Start initializes deal processing on a StorageProvider and restarts in progress deals. It also registers the provider with a StorageMarketNetwork so it can receive incoming messages on the storage market's libp2p protocols

func (*Provider) Stop

func (p *Provider) Stop() error

Stop terminates processing of deals on a StorageProvider

func (*Provider) SubscribeToEvents

func (p *Provider) SubscribeToEvents(subscriber storagemarket.ProviderSubscriber) shared.Unsubscribe

SubscribeToEvents allows another component to listen for events on the StorageProvider in order to track deals as they progress through the deal flow

type StorageClientOption

type StorageClientOption func(c *Client)

StorageClientOption allows custom configuration of a storage client

func DealPollingInterval

func DealPollingInterval(t time.Duration) StorageClientOption

DealPollingInterval sets the interval at which this client will query the Provider for deal state while waiting for deal acceptance

func MaxTraversalLinks(m uint64) StorageClientOption

MaxTraversalLinks sets the maximum number of links in a DAG to traverse when calculating CommP, sets a budget that limits the depth and density of a DAG that can be traversed

type StorageProviderOption

type StorageProviderOption func(p *Provider)

StorageProviderOption allows custom configuration of a storage provider

func AwaitTransferRestartTimeout

func AwaitTransferRestartTimeout(waitTime time.Duration) StorageProviderOption

AwaitTransferRestartTimeout sets the maximum amount of time a provider will wait for a client to restart a data transfer when the node starts up before failing the deal

func CustomDealDecisionLogic

func CustomDealDecisionLogic(decider DealDeciderFunc) StorageProviderOption

CustomDealDecisionLogic allows a provider to call custom decision logic when validating incoming deal proposals

func CustomMetadataGenerator

func CustomMetadataGenerator(metadataFunc MetadataFunc) StorageProviderOption

type StoredAsk

type StoredAsk interface {
	GetAsk() *storagemarket.SignedStorageAsk
	SetAsk(price abi.TokenAmount, verifiedPrice abi.TokenAmount, duration abi.ChainEpoch, options ...storagemarket.StorageAskOption) error
}

StoredAsk is an interface which provides access to a StorageAsk

Directories

Path Synopsis
Package blockrecorder provides utilits to record locations of CIDs to a temporary metadata file, since writing a CAR happens BEFORE we actually hand off for sealing.
Package blockrecorder provides utilits to record locations of CIDs to a temporary metadata file, since writing a CAR happens BEFORE we actually hand off for sealing.
Package clientstates contains state machine logic relating to the `StorageMarket`.
Package clientstates contains state machine logic relating to the `StorageMarket`.
Package clientutils provides utility functions for the storage client & client FSM
Package clientutils provides utility functions for the storage client & client FSM
Package connmanager tracks open connections maping storage proposal CID -> StorageDealStream
Package connmanager tracks open connections maping storage proposal CID -> StorageDealStream
Package dtutils provides event listeners for the client and provider to listen for events on the data transfer module and dispatch FSM events based on them
Package dtutils provides event listeners for the client and provider to listen for events on the data transfer module and dispatch FSM events based on them
Package providerstates contains state machine logic relating to the `StorageProvider`.
Package providerstates contains state machine logic relating to the `StorageProvider`.
Package providerutils provides utility functions for the storage provider & provider FSM
Package providerutils provides utility functions for the storage provider & provider FSM
Package requestvalidation implements a request validator for the data transfer module to validate data transfer requests for storage deals
Package requestvalidation implements a request validator for the data transfer module to validate data transfer requests for storage deals

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL