localstore

package
v1.4.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 1, 2023 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Overview

Package localstore provides disk storage layer for file Chunk persistence. It uses boson/shed abstractions.

The main type is DB which manages the storage by providing methods to access and add Chunks and to manage their status.

Modes are abstractions that do specific changes to Chunks. There are three mode types:

  • ModeGet, for Chunk access
  • ModePut, for adding Chunks to the database
  • ModeSet, for changing Chunk statuses

Every mode type has a corresponding type (Getter, Putter and Setter) that provides adequate method to perform the opperation and that type should be injected into localstore consumers instead the whole DB. This provides more clear insight which operations consumer is performing on the database.

Getters, Putters and Setters accept different get, put and set modes to perform different actions. For example, ModeGet has two different variables ModeGetRequest and ModeGetSync and two different Getters can be constructed with them that are used when the chunk is requested or when the chunk is synced as this two events are differently changing the database.

Subscription methods are implemented for a specific purpose of continuous iterations over Chunks that should be provided to Push and Pull syncing.

DB implements an internal garbage collector that removes only synced Chunks from the database based on their most recent access time.

Internally, DB stores Chunk data and any required information, such as store and access timestamps in different shed indexes that can be iterated on by garbage collector or subscriptions.

Index

Constants

View Source 
const DbSchemaCode = "code"

DbSchemaCode is the first node schema identifier

View Source 
const DbSchemaNone = ""

There was a time when we had no schema at all.

Variables

View Source 
var (
	// ErrTooShort signals too short input.
	ErrTooShort = errors.New("serialised input too short")
	// ErrInvalidInput signals invalid input to serialise.
	ErrInvalidInput = errors.New("input invalid")
	// ErrInvalidVersionHash signals unknown version of hash.
	ErrInvalidVersionHash = errors.New("invalid version hash")
)
View Source 
var DbSchemaCurrent = DbSchemaCode

The DB schema we want to use. The actual/current DB schema might differ until migrations are run.

View Source 
var (
	// ErrInvalidMode is retuned when an unknown Mode
	// is provided to the function.
	ErrInvalidMode = errors.New("invalid mode")
)

Functions

func IsValueType 

func IsValueType(nodeType uint8) bool

func UnmarshalBinary 

func UnmarshalBinary(data []byte) ([]*node, error)

UnmarshalBinary deserialises a node

Types

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB is the local store implementation and holds database related objects.

func New 

func New(path string, baseKey []byte, stateStore storage.StateStorer, o *Options, logger logging.Logger) (db *DB, err error)

New returns a new DB. All fields and indexes are initialized and possible conflicts with schema from existing database is checked. One goroutine for writing batches is created.

func (*DB) CancelFinder added in v1.1.2 

func (db *DB) CancelFinder(rootCid boson.Address)

func (*DB) ChunkCounter added in v1.2.1 

func (db *DB) ChunkCounter(reference boson.Address) error

func (*DB) Close 

func (db *DB) Close() (err error)

Close closes the underlying database.

func (*DB) DebugIndices 

func (db *DB) DebugIndices() (indexInfo map[string]int, err error)

DebugIndices returns the index sizes for all indexes in localstore the returned map keys are the index name, values are the number of elements in the index

func (*DB) DeleteAllChunk 

func (db *DB) DeleteAllChunk(chunkType chunkstore.ChunkType, reference boson.Address) error

func (*DB) DeleteChunk 

func (db *DB) DeleteChunk(chunkType chunkstore.ChunkType, reference, overlay boson.Address) error

func (*DB) DeleteFile 

func (db *DB) DeleteFile(reference boson.Address) error

func (*DB) DeleteMirror added in v1.2.1 

func (db *DB) DeleteMirror(reference boson.Address) error

func (*DB) Export 

func (db *DB) Export(w io.Writer) (count int64, err error)

Export writes a tar structured data to the writer of all chunks in the retrieval data index. It returns the number of chunks exported.

func (*DB) Get 

func (db *DB) Get(ctx context.Context, mode storage.ModeGet, addr boson.Address, index int64) (ch boson.Chunk, err error)

Get returns a chunk from the database. If the chunk is not found storage.ErrNotFound will be returned. All required indexes will be updated required by the Getter Mode. Get is required to implement chunk.Store interface.

func (*DB) GetAllChunk 

func (db *DB) GetAllChunk(ChunkType chunkstore.ChunkType) (map[string][]chunkstore.Consumer, error)

func (*DB) GetChunk 

func (db *DB) GetChunk(chunkType chunkstore.ChunkType, reference boson.Address) ([]chunkstore.Consumer, error)

func (*DB) GetChunkByOverlay added in v1.1.1 

func (db *DB) GetChunkByOverlay(chunkType chunkstore.ChunkType, reference, overlay boson.Address) (chunkstore.Consumer, error)

func (*DB) GetFile 

func (db *DB) GetFile(reference boson.Address) (filestore.FileView, bool)

func (*DB) GetListFile 

func (db *DB) GetListFile(page filestore.Page, filter []filestore.Filter, sort filestore.Sort) ([]filestore.FileView, int)

func (*DB) GetMirror added in v1.2.1 

func (db *DB) GetMirror(reference boson.Address) (*filestore.FileMirror, error)

func (*DB) GetMirrors added in v1.2.1 

func (db *DB) GetMirrors(reference boson.Address) (fms []*filestore.FileMirror, err error)

func (*DB) GetMulti 

func (db *DB) GetMulti(ctx context.Context, mode storage.ModeGet, addrs ...boson.Address) (chunks []boson.Chunk, err error)

GetMulti returns chunks from the database. If one of the chunks is not found storage.ErrNotFound will be returned. All required indexes will be updated required by the Getter Mode. GetMulti is required to implement chunk.Store interface.

func (*DB) Has 

func (db *DB) Has(ctx context.Context, mode storage.ModeHas, addr boson.Address) (bool, error)

Has returns true if the chunk is stored in database.

func (*DB) HasChunk 

func (db *DB) HasChunk(chunkType chunkstore.ChunkType, reference, overlay boson.Address) (bool, error)

func (*DB) HasChunkBit added in v1.2.1 

func (db *DB) HasChunkBit(chunkType chunkstore.ChunkType, reference boson.Address, bit int) (bool, error)

func (*DB) HasFile 

func (db *DB) HasFile(reference boson.Address) bool

func (*DB) HasMulti 

func (db *DB) HasMulti(ctx context.Context, hasMode storage.ModeHas, addrs ...boson.Address) ([]bool, error)

HasMulti returns a slice of booleans which represent if the provided chunks are stored in database.

func (*DB) Import 

func (db *DB) Import(ctx context.Context, r io.Reader) (count int64, err error)

Import reads a tar structured data from the reader and stores chunks in the database. It returns the number of chunks imported.

func (*DB) Init 

func (db *DB) Init() error

func (*DB) IsFinder added in v1.1.2 

func (db *DB) IsFinder(rootCid boson.Address) bool

func (*DB) Metrics 

func (db *DB) Metrics() []prometheus.Collector

func (*DB) Put 

func (db *DB) Put(ctx context.Context, mode storage.ModePut, chs ...boson.Chunk) (exist []bool, err error)

Put stores Chunks to database and depending on the Putter mode, it updates required indexes. Put is required to implement storage.Store interface.

func (*DB) PutChunk 

func (db *DB) PutChunk(chunkType chunkstore.ChunkType, reference boson.Address, providers []chunkstore.Provider) error

func (*DB) PutFile 

func (db *DB) PutFile(file filestore.FileView) error

func (*DB) PutMirrorFile added in v1.2.1 

func (db *DB) PutMirrorFile(next, reference boson.Address, ope filestore.Operation) error

func (*DB) Set 

func (db *DB) Set(ctx context.Context, mode storage.ModeSet, addrs ...boson.Address) (err error)

Set updates database indexes for chunks represented by provided addresses. Set is required to implement chunk.Store interface.

func (*DB) StartFinder added in v1.1.2 

func (db *DB) StartFinder(rootCid boson.Address)

func (*DB) UpdateFile 

func (db *DB) UpdateFile(file filestore.FileView) error

type Options 

type Options struct {
	// Driver support: leveldb/wiredtiger
	Driver string

	// Capacity is a limit that triggers garbage collection when
	// number of items in gcIndex equals or exceeds it.
	Capacity uint64

	// MetricsPrefix defines a prefix for metrics names.
	MetricsPrefix string

	FullNode bool
}

Options struct holds optional parameters for configuring DB.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.