stores

package
v1.28.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 14, 2023 License: Apache-2.0, MIT Imports: 31 Imported by: 24

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrNotFound = xerrors.New("not found")

Functions

func AllowDuplicatePuts added in v1.13.4 

func AllowDuplicatePuts(allow bool) carv2.Option

AllowDuplicatePuts is a write option which makes a CAR blockstore not deduplicate blocks in Put and PutMany. The default is to deduplicate, which matches the current semantics of go-ipfs-blockstore v1.

Note that this option only affects the blockstore, and is ignored by the root go-car/v2 package.

func DestroyShardSync added in v1.23.2 

func DestroyShardSync(ctx context.Context, ds DAGStoreWrapper, pieceCid cid.Cid) error

DestroyShardSync calls the DAGStore DestroyShard method and waits synchronously in a dedicated channel until the shard has been destroyed completely.

func FilestoreOf 

func FilestoreOf(bs bstore.Blockstore) (bstore.Blockstore, error)

FilestoreOf returns a FileManager/Filestore backed entirely by a blockstore without requiring a datastore. It achieves this by coercing the blockstore into a datastore. The resulting blockstore is suitable for usage with DagBuilderHelper with DagBuilderParams#NoCopy=true.

func HeaderSize added in v1.13.4 

func HeaderSize(h *CarHeader) (uint64, error)

func IsNotFound 

func IsNotFound(err error) bool

func LdRead added in v1.13.4 

func LdRead(r io.Reader, zeroLenAsEOF bool) ([]byte, error)

func LdSize added in v1.13.4 

func LdSize(d ...[]byte) uint64

func LdWrite added in v1.13.4 

func LdWrite(w io.Writer, d ...[]byte) error

func ReadNode added in v1.13.4 

func ReadNode(r io.Reader, zeroLenAsEOF bool) (cid.Cid, []byte, error)

func RegisterShardSync 

func RegisterShardSync(ctx context.Context, ds DAGStoreWrapper, pieceCid cid.Cid, carPath string, eagerInit bool) error

RegisterShardSync calls the DAGStore RegisterShard method and waits synchronously in a dedicated channel until the registration has completed fully.

func ToByteReader added in v1.13.4 

func ToByteReader(r io.Reader) io.ByteReader

func ToReaderAt added in v1.13.4 

func ToReaderAt(rs io.ReadSeeker) io.ReaderAt

func UseWholeCIDs added in v1.13.4 

func UseWholeCIDs(enable bool) carv2.Option

UseWholeCIDs is a read option which makes a CAR blockstore identify blocks by whole CIDs, and not just their multihashes. The default is to use multihashes, which matches the current semantics of go-ipfs-blockstore v1.

Enabling this option affects a number of methods, including read-only ones:

• Get, Has, and HasSize will only return a block only if the entire CID is present in the CAR file.

• AllKeysChan will return the original whole CIDs, instead of with their multicodec set to "raw" to just provide multihashes.

• If AllowDuplicatePuts isn't set, Put and PutMany will deduplicate by the whole CID, allowing different CIDs with equal multihashes.

Note that this option only affects the blockstore, and is ignored by the root go-car/v2 package.

func WithAsyncErrorHandler added in v1.13.4 

func WithAsyncErrorHandler(ctx context.Context, errHandler func(error)) context.Context

WithAsyncErrorHandler returns a context with async error handling set to the given errHandler. Any errors that occur during asynchronous operations of AllKeysChan will be passed to the given handler.

func WriteCar added in v1.13.4 

func WriteCar(ctx context.Context, ds format.NodeGetter, roots []cid.Cid, w io.Writer) error

func WriteHeader added in v1.13.4 

func WriteHeader(h *CarHeader, w io.Writer) error

Types

type ByteReadSeeker added in v1.13.4 

type ByteReadSeeker interface {
	io.ReadSeeker
	io.ByteReader
}

func ToByteReadSeeker added in v1.13.4 

func ToByteReadSeeker(r io.Reader) ByteReadSeeker

type BytesReader added in v1.13.4 

type BytesReader interface {
	io.Reader
	io.ByteReader
}

type CarHeader added in v1.13.4 

type CarHeader struct {
	Roots   []cid.Cid
	Version uint64
}

func LoadCar added in v1.13.4 

func LoadCar(s Store, r io.Reader) (*CarHeader, error)

func ReadHeader added in v1.13.4 

func ReadHeader(r io.Reader) (*CarHeader, error)

func (CarHeader) Matches added in v1.13.4 

func (h CarHeader) Matches(other CarHeader) bool

Matches checks whether two headers match. Two headers are considered matching if:

  1. They have the same version number, and
  2. They contain the same root CIDs in any order.

Note, this function explicitly ignores the order of roots. If order of roots matter use reflect.DeepEqual instead.

type CarReader added in v1.13.4 

type CarReader struct {
	Header *CarHeader
	// contains filtered or unexported fields
}

func NewCarReader added in v1.13.4 

func NewCarReader(r io.Reader) (*CarReader, error)

func NewCarReaderWithZeroLengthSectionAsEOF added in v1.13.4 

func NewCarReaderWithZeroLengthSectionAsEOF(r io.Reader) (*CarReader, error)

func (*CarReader) Next added in v1.13.4 

func (cr *CarReader) Next() (blocks.Block, error)

type ClosableBlockstore 

type ClosableBlockstore interface {
	bstore.Blockstore
	io.Closer
}

func ReadOnlyFilestore 

func ReadOnlyFilestore(path string) (ClosableBlockstore, error)

ReadOnlyFilestore opens the CAR in the specified path as as a read-only blockstore, and fronts it with a Filestore whose positional mappings are stored inside the CAR itself. It must be closed after done.

func ReadWriteFilestore 

func ReadWriteFilestore(path string, roots ...cid.Cid) (ClosableBlockstore, error)

ReadWriteFilestore opens the CAR in the specified path as as a read-write blockstore, and fronts it with a Filestore whose positional mappings are stored inside the CAR itself. It must be closed after done. Closing will finalize the CAR blockstore.

type DAGStoreWrapper 

type DAGStoreWrapper interface {
	// RegisterShard loads a CAR file into the DAG store and builds an
	// index for it, sending the result on the supplied channel on completion
	RegisterShard(ctx context.Context, pieceCid cid.Cid, carPath string, eagerInit bool, resch chan dagstore.ShardResult) error

	// LoadShard fetches the data for a shard and provides a blockstore
	// interface to it.
	//
	// The blockstore must be closed to release the shard.
	LoadShard(ctx context.Context, pieceCid cid.Cid) (ClosableBlockstore, error)

	// MigrateDeals migrates the supplied storage deals into the DAG store.
	MigrateDeals(ctx context.Context, deals []storagemarket.MinerDeal) (bool, error)

	// GetPiecesContainingBlock returns the CID of all pieces that contain
	// the block with the given CID
	GetPiecesContainingBlock(blockCID cid.Cid) ([]cid.Cid, error)

	GetIterableIndexForPiece(pieceCid cid.Cid) (carindex.IterableIndex, error)

	// DestroyShard initiates the registration of a new shard.
	//
	// This method returns an error synchronously if preliminary validation fails.
	// Otherwise, it queues the shard for destruction. The caller should monitor
	// supplied channel for a result.
	DestroyShard(ctx context.Context, pieceCid cid.Cid, resch chan dagstore.ShardResult) error

	// Close closes the dag store wrapper.
	Close() error
}

DAGStoreWrapper hides the details of the DAG store implementation from the other parts of go-fil-markets.

type OffsetReadSeeker added in v1.13.4 

type OffsetReadSeeker struct {
	// contains filtered or unexported fields
}

OffsetReadSeeker implements Read, and ReadAt on a section of an underlying io.ReaderAt. The main difference between io.SectionReader and OffsetReadSeeker is that NewOffsetReadSeeker does not require the user to know the number of readable bytes.

It also partially implements Seek, where the implementation panics if io.SeekEnd is passed. This is because, OffsetReadSeeker does not know the end of the file therefore cannot seek relative to it.

func NewOffsetReadSeeker added in v1.13.4 

func NewOffsetReadSeeker(r io.ReaderAt, off int64) *OffsetReadSeeker

NewOffsetReadSeeker returns an OffsetReadSeeker that reads from r starting offset offset off and stops with io.EOF when r reaches its end. The Seek function will panic if whence io.SeekEnd is passed.

func (*OffsetReadSeeker) Offset added in v1.13.4 

func (o *OffsetReadSeeker) Offset() int64

func (*OffsetReadSeeker) Position added in v1.13.4 

func (o *OffsetReadSeeker) Position() int64

Position returns the current position of this reader relative to the initial offset.

func (*OffsetReadSeeker) Read added in v1.13.4 

func (o *OffsetReadSeeker) Read(p []byte) (n int, err error)

func (*OffsetReadSeeker) ReadAt added in v1.13.4 

func (o *OffsetReadSeeker) ReadAt(p []byte, off int64) (n int, err error)

func (*OffsetReadSeeker) ReadByte added in v1.13.4 

func (o *OffsetReadSeeker) ReadByte() (byte, error)

func (*OffsetReadSeeker) Seek added in v1.13.4 

func (o *OffsetReadSeeker) Seek(offset int64, whence int) (int64, error)

type OffsetWriteSeeker added in v1.13.4 

type OffsetWriteSeeker struct {
	// contains filtered or unexported fields
}

func NewOffsetWriter added in v1.13.4 

func NewOffsetWriter(w io.WriterAt, off int64) *OffsetWriteSeeker

func (*OffsetWriteSeeker) Position added in v1.13.4 

func (ow *OffsetWriteSeeker) Position() int64

Position returns the current position of this writer relative to the initial offset, i.e. the number of bytes written.

func (*OffsetWriteSeeker) Seek added in v1.13.4 

func (ow *OffsetWriteSeeker) Seek(offset int64, whence int) (int64, error)

func (*OffsetWriteSeeker) Write added in v1.13.4 

func (ow *OffsetWriteSeeker) Write(b []byte) (n int, err error)

type ReadOnly added in v1.13.4 

type ReadOnly struct {
	// contains filtered or unexported fields
}

ReadOnly provides a read-only CAR Block Store.

func NewReadOnly added in v1.13.4 

func NewReadOnly(backing io.ReaderAt, idx index.Index, opts ...carv2.Option) (*ReadOnly, error)

NewReadOnly creates a new ReadOnly blockstore from the backing with a optional index as idx. This function accepts both CARv1 and CARv2 backing. The blockstore is instantiated with the given index if it is not nil.

Otherwise: * For a CARv1 backing an index is generated. * For a CARv2 backing an index is only generated if Header.HasIndex returns false.

There is no need to call ReadOnly.Close on instances returned by this function.

func OpenReadOnly added in v1.13.4 

func OpenReadOnly(path string, opts ...carv2.Option) (*ReadOnly, error)

OpenReadOnly opens a read-only blockstore from a CAR file (either v1 or v2), generating an index if it does not exist. Note, the generated index if the index does not exist is ephemeral and only stored in memory. See car.GenerateIndex and Index.Attach for persisting index onto a CAR file.

func (*ReadOnly) AllKeysChan added in v1.13.4 

func (b *ReadOnly) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error)

AllKeysChan returns the list of keys in the CAR data payload. If the ctx is constructed using WithAsyncErrorHandler any errors that occur during asynchronous retrieval of CIDs will be passed to the error handler function set in context. Otherwise, errors will terminate the asynchronous operation silently.

See WithAsyncErrorHandler

func (*ReadOnly) Close added in v1.13.4 

func (b *ReadOnly) Close() error

Close closes the underlying reader if it was opened by OpenReadOnly. After this call, the blockstore can no longer be used.

Note that this call may block if any blockstore operations are currently in progress, including an AllKeysChan that hasn't been fully consumed or cancelled.

func (*ReadOnly) DeleteBlock added in v1.13.4 

func (b *ReadOnly) DeleteBlock(_ context.Context, _ cid.Cid) error

DeleteBlock is unsupported and always errors.

func (*ReadOnly) Get added in v1.13.4 

func (b *ReadOnly) Get(ctx context.Context, key cid.Cid) (blocks.Block, error)

Get gets a block corresponding to the given key. This API will always return true if the given key has multihash.IDENTITY code.

func (*ReadOnly) GetSize added in v1.13.4 

func (b *ReadOnly) GetSize(ctx context.Context, key cid.Cid) (int, error)

GetSize gets the size of an item corresponding to the given key.

func (*ReadOnly) Has added in v1.13.4 

func (b *ReadOnly) Has(ctx context.Context, key cid.Cid) (bool, error)

Has indicates if the store contains a block that corresponds to the given key. This function always returns true for any given key with multihash.IDENTITY code.

func (*ReadOnly) HashOnRead added in v1.13.4 

func (b *ReadOnly) HashOnRead(bool)

HashOnRead is currently unimplemented; hashing on reads never happens.

func (*ReadOnly) Put added in v1.13.4 

func (b *ReadOnly) Put(context.Context, blocks.Block) error

Put is not supported and always returns an error.

func (*ReadOnly) PutMany added in v1.13.4 

func (b *ReadOnly) PutMany(context.Context, []blocks.Block) error

PutMany is not supported and always returns an error.

func (*ReadOnly) Roots added in v1.13.4 

func (b *ReadOnly) Roots() ([]cid.Cid, error)

Roots returns the root CIDs of the backing CAR.

type ReadOnlyBlockstores 

type ReadOnlyBlockstores struct {
	// contains filtered or unexported fields
}

ReadOnlyBlockstores tracks open read blockstores.

func NewReadOnlyBlockstores 

func NewReadOnlyBlockstores() *ReadOnlyBlockstores

func (*ReadOnlyBlockstores) Get 

func (r *ReadOnlyBlockstores) Get(key string) (bstore.Blockstore, error)

func (*ReadOnlyBlockstores) Track 

func (r *ReadOnlyBlockstores) Track(key string, bs bstore.Blockstore) (bool, error)

func (*ReadOnlyBlockstores) Untrack 

func (r *ReadOnlyBlockstores) Untrack(key string) error

type ReadStore added in v1.13.4 

type ReadStore interface {
	Get(cid.Cid) (blocks.Block, error)
}

type ReadWrite added in v1.13.4 

type ReadWrite struct {
	// contains filtered or unexported fields
}

ReadWrite implements a blockstore that stores blocks in CARv2 format. Blocks put into the blockstore can be read back once they are successfully written. This implementation is preferable for a write-heavy workload. The blocks are written immediately on Put and PutAll calls, while the index is stored in memory and updated incrementally.

The Finalize function must be called once the putting blocks are finished. Upon calling Finalize header is finalized and index is written out. Once finalized, all read and write calls to this blockstore will result in errors.

func OpenReadWrite added in v1.13.4 

func OpenReadWrite(path string, roots []cid.Cid, opts ...carv2.Option) (*ReadWrite, error)

OpenReadWrite creates a new ReadWrite at the given path with a provided set of root CIDs and options.

ReadWrite.Finalize must be called once putting and reading blocks are no longer needed. Upon calling ReadWrite.Finalize the CARv2 header and index are written out onto the file and the backing file is closed. Once finalized, all read and write calls to this blockstore will result in errors. Note, ReadWrite.Finalize must be called on an open instance regardless of whether any blocks were put or not.

If a file at given path does not exist, the instantiation will write car.Pragma and data payload header (i.e. the inner CARv1 header) onto the file before returning.

When the given path already exists, the blockstore will attempt to resume from it. On resumption the existing data sections in file are re-indexed, allowing the caller to continue putting any remaining blocks without having to re-ingest blocks for which previous ReadWrite.Put returned successfully.

Resumption only works on files that were created by a previous instance of a ReadWrite blockstore. This means a file created as a result of a successful call to OpenReadWrite can be resumed from as long as write operations such as ReadWrite.Put, ReadWrite.PutMany returned successfully. On resumption the roots argument and WithDataPadding option must match the previous instantiation of ReadWrite blockstore that created the file. More explicitly, the file resuming from must:

  1. start with a complete CARv2 car.Pragma.
  2. contain a complete CARv1 data header with root CIDs matching the CIDs passed to the constructor, starting at offset optionally padded by WithDataPadding, followed by zero or more complete data sections. If any corrupt data sections are present the resumption will fail. Note, if set previously, the blockstore must use the same WithDataPadding option as before, since this option is used to locate the CARv1 data payload.

Note, resumption should be used with WithCidDeduplication, so that blocks that are successfully written into the file are not re-written. Unless, the user explicitly wants duplicate blocks.

Resuming from finalized files is allowed. However, resumption will regenerate the index regardless by scanning every existing block in file.

func (*ReadWrite) AllKeysChan added in v1.13.4 

func (b *ReadWrite) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error)

func (*ReadWrite) DeleteBlock added in v1.13.4 

func (b *ReadWrite) DeleteBlock(_ context.Context, _ cid.Cid) error

func (*ReadWrite) Discard added in v1.13.4 

func (b *ReadWrite) Discard()

Discard closes this blockstore without finalizing its header and index. After this call, the blockstore can no longer be used.

Note that this call may block if any blockstore operations are currently in progress, including an AllKeysChan that hasn't been fully consumed or cancelled.

func (*ReadWrite) Finalize added in v1.13.4 

func (b *ReadWrite) Finalize() error

Finalize finalizes this blockstore by writing the CARv2 header, along with flattened index for more efficient subsequent read. After this call, the blockstore can no longer be used.

func (*ReadWrite) Get added in v1.13.4 

func (b *ReadWrite) Get(ctx context.Context, key cid.Cid) (blocks.Block, error)

func (*ReadWrite) GetSize added in v1.13.4 

func (b *ReadWrite) GetSize(ctx context.Context, key cid.Cid) (int, error)

func (*ReadWrite) Has added in v1.13.4 

func (b *ReadWrite) Has(ctx context.Context, key cid.Cid) (bool, error)

func (*ReadWrite) HashOnRead added in v1.13.4 

func (b *ReadWrite) HashOnRead(enable bool)

func (*ReadWrite) Put added in v1.13.4 

func (b *ReadWrite) Put(ctx context.Context, blk blocks.Block) error

Put puts a given block to the underlying datastore

func (*ReadWrite) PutMany added in v1.13.4 

func (b *ReadWrite) PutMany(ctx context.Context, blks []blocks.Block) error

PutMany puts a slice of blocks at the same time using batching capabilities of the underlying datastore whenever possible.

func (*ReadWrite) Roots added in v1.13.4 

func (b *ReadWrite) Roots() ([]cid.Cid, error)

type ReadWriteBlockstores 

type ReadWriteBlockstores struct {
	// contains filtered or unexported fields
}

ReadWriteBlockstores tracks open ReadWrite CAR blockstores.

func NewReadWriteBlockstores 

func NewReadWriteBlockstores() *ReadWriteBlockstores

func (*ReadWriteBlockstores) Get 

func (r *ReadWriteBlockstores) Get(key string) (*blockstore.ReadWrite, error)

func (*ReadWriteBlockstores) GetOrOpen 

func (r *ReadWriteBlockstores) GetOrOpen(key string, path string, rootCid cid.Cid) (*blockstore.ReadWrite, error)

func (*ReadWriteBlockstores) Untrack 

func (r *ReadWriteBlockstores) Untrack(key string) error

type Store added in v1.13.4 

type Store interface {
	Put(blocks.Block) error
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.