blobsfile: Index | Files

package blobsfile

import ""

Package blobsfile implement the BlobsFile backend for storing blobs.

It stores multiple blobs (optionally compressed with Snappy) inside "BlobsFile"/fat file/packed file (256MB by default). Blobs are indexed by a kv file (that can be rebuild from the blobsfile).

New blobs are appended to the current file, and when the file exceed the limit, a new fie is created.


Package Files

blobsfile.go index.go


const (
    // Version is the current BlobsFile binary format version
    Version = 1


var (
    // ErrBlobNotFound reports that the blob could not be found
    ErrBlobNotFound = errors.New("blob not found")

    // ErrBlobsfileCorrupted reports that one of the BlobsFile is corrupted and could not be repaired
    ErrBlobsfileCorrupted = errors.New("blobsfile is corrupted")

func ScanBlobsFile Uses

func ScanBlobsFile(path string) ([]string, error)

scanBlobsFile scan a single BlobsFile (#n), and execute `iterFunc` for each indexed blob. `iterFunc` is optional, and without it, this func will check the consistency of each blob, and return a `corruptedError` if a blob is corrupted.

type Blob Uses

type Blob struct {
    Hash string
    Size int
    N    int

Blob represents a blob hash and size when enumerating the DB.

type BlobsFiles Uses

type BlobsFiles struct {
    // contains filtered or unexported fields

BlobsFiles represent the DB

func New Uses

func New(opts *Opts) (*BlobsFiles, error)

New intializes a new BlobsFileBackend.

func (*BlobsFiles) CheckBlobsFiles Uses

func (backend *BlobsFiles) CheckBlobsFiles() error

CheckBlobsFiles will check the consistency of all the BlobsFile

func (*BlobsFiles) Close Uses

func (backend *BlobsFiles) Close() error

Close closes all the indexes and data files.

func (*BlobsFiles) Enumerate Uses

func (backend *BlobsFiles) Enumerate(blobs chan<- *Blob, start, end string, limit int) error

Enumerate outputs all the blobs into the given chan (ordered lexicographically).

func (*BlobsFiles) EnumeratePrefix Uses

func (backend *BlobsFiles) EnumeratePrefix(blobs chan<- *Blob, prefix string, limit int) error

Enumerate outputs all the blobs into the given chan (ordered lexicographically).

func (*BlobsFiles) Exists Uses

func (backend *BlobsFiles) Exists(hash string) (bool, error)

Exists return true if the blobs is already stored.

func (*BlobsFiles) Get Uses

func (backend *BlobsFiles) Get(hash string) ([]byte, error)

Get returns the blob for the given hash.

func (*BlobsFiles) Put Uses

func (backend *BlobsFiles) Put(hash string, data []byte) (err error)

Put save a new blob, hash must be the blake2b hash hex-encoded of the data.

If the blob is already stored, then Put will be a no-op. So it's not necessary to make call Exists before saving a new blob.

func (*BlobsFiles) RebuildIndex Uses

func (backend *BlobsFiles) RebuildIndex() error

RebuildIndex removes the index files and re-build it by re-scanning all the BlobsFiles.

func (*BlobsFiles) SealedPacks Uses

func (backend *BlobsFiles) SealedPacks() []string

func (*BlobsFiles) SetBlobsFilesSealedFunc Uses

func (backend *BlobsFiles) SetBlobsFilesSealedFunc(f func(string))

func (*BlobsFiles) Size Uses

func (backend *BlobsFiles) Size(hash string) (int, error)

Size returns the blob size for the given hash.

func (*BlobsFiles) Stats Uses

func (backend *BlobsFiles) Stats() (*Stats, error)

Stats returns some stats about the DB.

func (*BlobsFiles) String Uses

func (backend *BlobsFiles) String() string

String implements the Stringer interface.

type CompressionAlgorithm Uses

type CompressionAlgorithm byte
const (
    Snappy CompressionAlgorithm = 1 << iota

Compression algorithms flag

type ErrInterventionNeeded Uses

type ErrInterventionNeeded struct {
    // contains filtered or unexported fields

ErrInterventionNeeded is an error indicating an manual action must be performed before being able to use BobsFile

func (*ErrInterventionNeeded) Error Uses

func (ein *ErrInterventionNeeded) Error() string

type Opts Uses

type Opts struct {
    // Compression algorithm
    Compression CompressionAlgorithm

    // The max size of a BlobsFile, will be 256MB by default if not set
    BlobsFileSize int64

    // Where the data and indexes will be stored
    Directory string

    // Allow to catch some events
    LogFunc func(msg string)

    // When trying to self-heal in case of recovery, some step need to be performed by the user
    AskConfirmationFunc func(msg string) bool

    BlobsFilesSealedFunc func(path string)

Opts represents the DB options

type Stats Uses

type Stats struct {
    // The total number of blobs stored
    BlobsCount int

    // The size of all the blobs stored
    BlobsSize int64

    // The number of BlobsFile
    BlobsFilesCount int

    // The size of all the BlobsFile
    BlobsFilesSize int64

Stats represents some stats about the DB state

Package blobsfile imports 17 packages (graph) and is imported by 7 packages. Updated 2020-05-29. Refresh now. Tools for package owners.