Documentation
¶
Index ¶
- Variables
- func AllowDuplicatePuts(allow bool) carv2.Option
- func DestroyShardSync(ctx context.Context, ds DAGStoreWrapper, pieceCid cid.Cid) error
- func FilestoreOf(bs bstore.Blockstore) (bstore.Blockstore, error)
- func HeaderSize(h *CarHeader) (uint64, error)
- func IsNotFound(err error) bool
- func LdRead(r io.Reader, zeroLenAsEOF bool) ([]byte, error)
- func LdSize(d ...[]byte) uint64
- func LdWrite(w io.Writer, d ...[]byte) error
- func ReadNode(r io.Reader, zeroLenAsEOF bool) (cid.Cid, []byte, error)
- func RegisterShardSync(ctx context.Context, ds DAGStoreWrapper, pieceCid cid.Cid, carPath string, ...) error
- func ToByteReader(r io.Reader) io.ByteReader
- func ToReaderAt(rs io.ReadSeeker) io.ReaderAt
- func UseWholeCIDs(enable bool) carv2.Option
- func WithAsyncErrorHandler(ctx context.Context, errHandler func(error)) context.Context
- func WriteCar(ctx context.Context, ds format.NodeGetter, roots []cid.Cid, w io.Writer) error
- func WriteHeader(h *CarHeader, w io.Writer) error
- type ByteReadSeeker
- type BytesReader
- type CarHeader
- type CarReader
- type ClosableBlockstore
- type DAGStoreWrapper
- type OffsetReadSeeker
- func (o *OffsetReadSeeker) Offset() int64
- func (o *OffsetReadSeeker) Position() int64
- func (o *OffsetReadSeeker) Read(p []byte) (n int, err error)
- func (o *OffsetReadSeeker) ReadAt(p []byte, off int64) (n int, err error)
- func (o *OffsetReadSeeker) ReadByte() (byte, error)
- func (o *OffsetReadSeeker) Seek(offset int64, whence int) (int64, error)
- type OffsetWriteSeeker
- type ReadOnly
- func (b *ReadOnly) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error)
- func (b *ReadOnly) Close() error
- func (b *ReadOnly) DeleteBlock(_ context.Context, _ cid.Cid) error
- func (b *ReadOnly) Get(ctx context.Context, key cid.Cid) (blocks.Block, error)
- func (b *ReadOnly) GetSize(ctx context.Context, key cid.Cid) (int, error)
- func (b *ReadOnly) Has(ctx context.Context, key cid.Cid) (bool, error)
- func (b *ReadOnly) HashOnRead(bool)
- func (b *ReadOnly) Put(context.Context, blocks.Block) error
- func (b *ReadOnly) PutMany(context.Context, []blocks.Block) error
- func (b *ReadOnly) Roots() ([]cid.Cid, error)
- type ReadOnlyBlockstores
- type ReadStore
- type ReadWrite
- func (b *ReadWrite) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error)
- func (b *ReadWrite) DeleteBlock(_ context.Context, _ cid.Cid) error
- func (b *ReadWrite) Discard()
- func (b *ReadWrite) Finalize() error
- func (b *ReadWrite) Get(ctx context.Context, key cid.Cid) (blocks.Block, error)
- func (b *ReadWrite) GetSize(ctx context.Context, key cid.Cid) (int, error)
- func (b *ReadWrite) Has(ctx context.Context, key cid.Cid) (bool, error)
- func (b *ReadWrite) HashOnRead(enable bool)
- func (b *ReadWrite) Put(ctx context.Context, blk blocks.Block) error
- func (b *ReadWrite) PutMany(ctx context.Context, blks []blocks.Block) error
- func (b *ReadWrite) Roots() ([]cid.Cid, error)
- type ReadWriteBlockstores
- type Store
Constants ¶
This section is empty.
Variables ¶
var ErrNotFound = xerrors.New("not found")
Functions ¶
func AllowDuplicatePuts ¶
AllowDuplicatePuts is a write option which makes a CAR blockstore not deduplicate blocks in Put and PutMany. The default is to deduplicate, which matches the current semantics of go-ipfs-blockstore v1.
Note that this option only affects the blockstore, and is ignored by the root go-car/v2 package.
func DestroyShardSync ¶
func DestroyShardSync(ctx context.Context, ds DAGStoreWrapper, pieceCid cid.Cid) error
DestroyShardSync calls the DAGStore DestroyShard method and waits synchronously in a dedicated channel until the shard has been destroyed completely.
func FilestoreOf ¶
func FilestoreOf(bs bstore.Blockstore) (bstore.Blockstore, error)
FilestoreOf returns a FileManager/Filestore backed entirely by a blockstore without requiring a datastore. It achieves this by coercing the blockstore into a datastore. The resulting blockstore is suitable for usage with DagBuilderHelper with DagBuilderParams#NoCopy=true.
func HeaderSize ¶
func IsNotFound ¶
func RegisterShardSync ¶
func RegisterShardSync(ctx context.Context, ds DAGStoreWrapper, pieceCid cid.Cid, carPath string, eagerInit bool) error
RegisterShardSync calls the DAGStore RegisterShard method and waits synchronously in a dedicated channel until the registration has completed fully.
func ToByteReader ¶
func ToByteReader(r io.Reader) io.ByteReader
func ToReaderAt ¶
func ToReaderAt(rs io.ReadSeeker) io.ReaderAt
func UseWholeCIDs ¶
UseWholeCIDs is a read option which makes a CAR blockstore identify blocks by whole CIDs, and not just their multihashes. The default is to use multihashes, which matches the current semantics of go-ipfs-blockstore v1.
Enabling this option affects a number of methods, including read-only ones:
• Get, Has, and HasSize will only return a block only if the entire CID is present in the CAR file.
• AllKeysChan will return the original whole CIDs, instead of with their multicodec set to "raw" to just provide multihashes.
• If AllowDuplicatePuts isn't set, Put and PutMany will deduplicate by the whole CID, allowing different CIDs with equal multihashes.
Note that this option only affects the blockstore, and is ignored by the root go-car/v2 package.
func WithAsyncErrorHandler ¶
WithAsyncErrorHandler returns a context with async error handling set to the given errHandler. Any errors that occur during asynchronous operations of AllKeysChan will be passed to the given handler.
Types ¶
type ByteReadSeeker ¶
type ByteReadSeeker interface {
io.ReadSeeker
io.ByteReader
}
func ToByteReadSeeker ¶
func ToByteReadSeeker(r io.Reader) ByteReadSeeker
type BytesReader ¶
type BytesReader interface {
io.Reader
io.ByteReader
}
type CarHeader ¶
type CarHeader struct {
Roots []cid.Cid
Version uint64
}
func (CarHeader) Matches ¶
Matches checks whether two headers match. Two headers are considered matching if:
- They have the same version number, and
- They contain the same root CIDs in any order.
Note, this function explicitly ignores the order of roots. If order of roots matter use reflect.DeepEqual instead.
type CarReader ¶
type CarReader struct {
Header *CarHeader
// contains filtered or unexported fields
}
type ClosableBlockstore ¶
type ClosableBlockstore interface {
bstore.Blockstore
io.Closer
}
func ReadOnlyFilestore ¶
func ReadOnlyFilestore(path string) (ClosableBlockstore, error)
ReadOnlyFilestore opens the CAR in the specified path as as a read-only blockstore, and fronts it with a Filestore whose positional mappings are stored inside the CAR itself. It must be closed after done.
func ReadWriteFilestore ¶
func ReadWriteFilestore(path string, roots ...cid.Cid) (ClosableBlockstore, error)
ReadWriteFilestore opens the CAR in the specified path as as a read-write blockstore, and fronts it with a Filestore whose positional mappings are stored inside the CAR itself. It must be closed after done. Closing will finalize the CAR blockstore.
type DAGStoreWrapper ¶
type DAGStoreWrapper interface {
// RegisterShard loads a CAR file into the DAG store and builds an
// index for it, sending the result on the supplied channel on completion
RegisterShard(ctx context.Context, pieceCid cid.Cid, carPath string, eagerInit bool, resch chan dagstore.ShardResult) error
// LoadShard fetches the data for a shard and provides a blockstore
// interface to it.
//
// The blockstore must be closed to release the shard.
LoadShard(ctx context.Context, pieceCid cid.Cid) (ClosableBlockstore, error)
// MigrateDeals migrates the supplied storage deals into the DAG store.
MigrateDeals(ctx context.Context, deals []storagemarket.MinerDeal) (bool, error)
// GetPiecesContainingBlock returns the CID of all pieces that contain
// the block with the given CID
GetPiecesContainingBlock(blockCID cid.Cid) ([]cid.Cid, error)
GetIterableIndexForPiece(pieceCid cid.Cid) (carindex.IterableIndex, error)
// DestroyShard initiates the registration of a new shard.
//
// This method returns an error synchronously if preliminary validation fails.
// Otherwise, it queues the shard for destruction. The caller should monitor
// supplied channel for a result.
DestroyShard(ctx context.Context, pieceCid cid.Cid, resch chan dagstore.ShardResult) error
// Close closes the dag store wrapper.
Close() error
}
DAGStoreWrapper hides the details of the DAG store implementation from the other parts of go-fil-markets.
type OffsetReadSeeker ¶
type OffsetReadSeeker struct {
// contains filtered or unexported fields
}
OffsetReadSeeker implements Read, and ReadAt on a section of an underlying io.ReaderAt. The main difference between io.SectionReader and OffsetReadSeeker is that NewOffsetReadSeeker does not require the user to know the number of readable bytes.
It also partially implements Seek, where the implementation panics if io.SeekEnd is passed. This is because, OffsetReadSeeker does not know the end of the file therefore cannot seek relative to it.
func NewOffsetReadSeeker ¶
func NewOffsetReadSeeker(r io.ReaderAt, off int64) *OffsetReadSeeker
NewOffsetReadSeeker returns an OffsetReadSeeker that reads from r starting offset offset off and stops with io.EOF when r reaches its end. The Seek function will panic if whence io.SeekEnd is passed.
func (*OffsetReadSeeker) Offset ¶
func (o *OffsetReadSeeker) Offset() int64
func (*OffsetReadSeeker) Position ¶
func (o *OffsetReadSeeker) Position() int64
Position returns the current position of this reader relative to the initial offset.
func (*OffsetReadSeeker) ReadAt ¶
func (o *OffsetReadSeeker) ReadAt(p []byte, off int64) (n int, err error)
func (*OffsetReadSeeker) ReadByte ¶
func (o *OffsetReadSeeker) ReadByte() (byte, error)
type OffsetWriteSeeker ¶
type OffsetWriteSeeker struct {
// contains filtered or unexported fields
}
func NewOffsetWriter ¶
func NewOffsetWriter(w io.WriterAt, off int64) *OffsetWriteSeeker
func (*OffsetWriteSeeker) Position ¶
func (ow *OffsetWriteSeeker) Position() int64
Position returns the current position of this writer relative to the initial offset, i.e. the number of bytes written.
type ReadOnly ¶
type ReadOnly struct {
// contains filtered or unexported fields
}
ReadOnly provides a read-only CAR Block Store.
func NewReadOnly ¶
NewReadOnly creates a new ReadOnly blockstore from the backing with a optional index as idx. This function accepts both CARv1 and CARv2 backing. The blockstore is instantiated with the given index if it is not nil.
Otherwise: * For a CARv1 backing an index is generated. * For a CARv2 backing an index is only generated if Header.HasIndex returns false.
There is no need to call ReadOnly.Close on instances returned by this function.
func OpenReadOnly ¶
OpenReadOnly opens a read-only blockstore from a CAR file (either v1 or v2), generating an index if it does not exist. Note, the generated index if the index does not exist is ephemeral and only stored in memory. See car.GenerateIndex and Index.Attach for persisting index onto a CAR file.
func (*ReadOnly) AllKeysChan ¶
AllKeysChan returns the list of keys in the CAR data payload. If the ctx is constructed using WithAsyncErrorHandler any errors that occur during asynchronous retrieval of CIDs will be passed to the error handler function set in context. Otherwise, errors will terminate the asynchronous operation silently.
See WithAsyncErrorHandler
func (*ReadOnly) Close ¶
Close closes the underlying reader if it was opened by OpenReadOnly. After this call, the blockstore can no longer be used.
Note that this call may block if any blockstore operations are currently in progress, including an AllKeysChan that hasn't been fully consumed or cancelled.
func (*ReadOnly) DeleteBlock ¶
DeleteBlock is unsupported and always errors.
func (*ReadOnly) Get ¶
Get gets a block corresponding to the given key. This API will always return true if the given key has multihash.IDENTITY code.
func (*ReadOnly) Has ¶
Has indicates if the store contains a block that corresponds to the given key. This function always returns true for any given key with multihash.IDENTITY code.
func (*ReadOnly) HashOnRead ¶
HashOnRead is currently unimplemented; hashing on reads never happens.
type ReadOnlyBlockstores ¶
type ReadOnlyBlockstores struct {
// contains filtered or unexported fields
}
ReadOnlyBlockstores tracks open read blockstores.
func NewReadOnlyBlockstores ¶
func NewReadOnlyBlockstores() *ReadOnlyBlockstores
func (*ReadOnlyBlockstores) Get ¶
func (r *ReadOnlyBlockstores) Get(key string) (bstore.Blockstore, error)
func (*ReadOnlyBlockstores) Track ¶
func (r *ReadOnlyBlockstores) Track(key string, bs bstore.Blockstore) (bool, error)
func (*ReadOnlyBlockstores) Untrack ¶
func (r *ReadOnlyBlockstores) Untrack(key string) error
type ReadWrite ¶
type ReadWrite struct {
// contains filtered or unexported fields
}
ReadWrite implements a blockstore that stores blocks in CARv2 format. Blocks put into the blockstore can be read back once they are successfully written. This implementation is preferable for a write-heavy workload. The blocks are written immediately on Put and PutAll calls, while the index is stored in memory and updated incrementally.
The Finalize function must be called once the putting blocks are finished. Upon calling Finalize header is finalized and index is written out. Once finalized, all read and write calls to this blockstore will result in errors.
func OpenReadWrite ¶
OpenReadWrite creates a new ReadWrite at the given path with a provided set of root CIDs and options.
ReadWrite.Finalize must be called once putting and reading blocks are no longer needed. Upon calling ReadWrite.Finalize the CARv2 header and index are written out onto the file and the backing file is closed. Once finalized, all read and write calls to this blockstore will result in errors. Note, ReadWrite.Finalize must be called on an open instance regardless of whether any blocks were put or not.
If a file at given path does not exist, the instantiation will write car.Pragma and data payload header (i.e. the inner CARv1 header) onto the file before returning.
When the given path already exists, the blockstore will attempt to resume from it. On resumption the existing data sections in file are re-indexed, allowing the caller to continue putting any remaining blocks without having to re-ingest blocks for which previous ReadWrite.Put returned successfully.
Resumption only works on files that were created by a previous instance of a ReadWrite blockstore. This means a file created as a result of a successful call to OpenReadWrite can be resumed from as long as write operations such as ReadWrite.Put, ReadWrite.PutMany returned successfully. On resumption the roots argument and WithDataPadding option must match the previous instantiation of ReadWrite blockstore that created the file. More explicitly, the file resuming from must:
- start with a complete CARv2 car.Pragma.
- contain a complete CARv1 data header with root CIDs matching the CIDs passed to the constructor, starting at offset optionally padded by WithDataPadding, followed by zero or more complete data sections. If any corrupt data sections are present the resumption will fail. Note, if set previously, the blockstore must use the same WithDataPadding option as before, since this option is used to locate the CARv1 data payload.
Note, resumption should be used with WithCidDeduplication, so that blocks that are successfully written into the file are not re-written. Unless, the user explicitly wants duplicate blocks.
Resuming from finalized files is allowed. However, resumption will regenerate the index regardless by scanning every existing block in file.
func (*ReadWrite) AllKeysChan ¶
func (*ReadWrite) Discard ¶
func (b *ReadWrite) Discard()
Discard closes this blockstore without finalizing its header and index. After this call, the blockstore can no longer be used.
Note that this call may block if any blockstore operations are currently in progress, including an AllKeysChan that hasn't been fully consumed or cancelled.
func (*ReadWrite) Finalize ¶
Finalize finalizes this blockstore by writing the CARv2 header, along with flattened index for more efficient subsequent read. After this call, the blockstore can no longer be used.
func (*ReadWrite) HashOnRead ¶
type ReadWriteBlockstores ¶
type ReadWriteBlockstores struct {
// contains filtered or unexported fields
}
ReadWriteBlockstores tracks open ReadWrite CAR blockstores.
func NewReadWriteBlockstores ¶
func NewReadWriteBlockstores() *ReadWriteBlockstores
func (*ReadWriteBlockstores) Get ¶
func (r *ReadWriteBlockstores) Get(key string) (*blockstore.ReadWrite, error)
func (*ReadWriteBlockstores) GetOrOpen ¶
func (r *ReadWriteBlockstores) GetOrOpen(key string, path string, rootCid cid.Cid) (*blockstore.ReadWrite, error)
func (*ReadWriteBlockstores) Untrack ¶
func (r *ReadWriteBlockstores) Untrack(key string) error
Source Files