blobsfile: a4.io/blobsfile Index | Files

package blobsfile

import "a4.io/blobsfile"

Package blobsfile implement the BlobsFile backend for storing blobs.

It stores multiple blobs (optionally compressed with Snappy) inside "BlobsFile"/fat file/packed file (256MB by default). Blobs are indexed by a kv file (that can be rebuild from the blobsfile).

New blobs are appended to the current file, and when the file exceed the limit, a new fie is created.

Index

Package Files

blobsfile.go index.go

Constants

const (
    // Version is the current BlobsFile binary format version
    Version = 1
)

Variables

var (
    // ErrBlobNotFound reports that the blob could not be found
    ErrBlobNotFound = errors.New("blob not found")

    // ErrBlobsfileCorrupted reports that one of the BlobsFile is corrupted and could not be repaired
    ErrBlobsfileCorrupted = errors.New("blobsfile is corrupted")
)

func ScanBlobsFile Uses

func ScanBlobsFile(path string) ([]string, error)

scanBlobsFile scan a single BlobsFile (#n), and execute `iterFunc` for each indexed blob. `iterFunc` is optional, and without it, this func will check the consistency of each blob, and return a `corruptedError` if a blob is corrupted.

type Blob Uses

type Blob struct {
    Hash string
    Size int
    N    int
}

Blob represents a blob hash and size when enumerating the DB.

type BlobsFiles Uses

type BlobsFiles struct {
    sync.Mutex
    // contains filtered or unexported fields
}

BlobsFiles represent the DB

func New Uses

func New(opts *Opts) (*BlobsFiles, error)

New intializes a new BlobsFileBackend.

func (*BlobsFiles) CheckBlobsFiles Uses

func (backend *BlobsFiles) CheckBlobsFiles() error

CheckBlobsFiles will check the consistency of all the BlobsFile

func (*BlobsFiles) Close Uses

func (backend *BlobsFiles) Close() error

Close closes all the indexes and data files.

func (*BlobsFiles) Enumerate Uses

func (backend *BlobsFiles) Enumerate(blobs chan<- *Blob, start, end string, limit int) error

Enumerate outputs all the blobs into the given chan (ordered lexicographically).

func (*BlobsFiles) EnumeratePrefix Uses

func (backend *BlobsFiles) EnumeratePrefix(blobs chan<- *Blob, prefix string, limit int) error

Enumerate outputs all the blobs into the given chan (ordered lexicographically).

func (*BlobsFiles) Exists Uses

func (backend *BlobsFiles) Exists(hash string) (bool, error)

Exists return true if the blobs is already stored.

func (*BlobsFiles) Get Uses

func (backend *BlobsFiles) Get(hash string) ([]byte, error)

Get returns the blob for the given hash.

func (*BlobsFiles) Put Uses

func (backend *BlobsFiles) Put(hash string, data []byte) (err error)

Put save a new blob, hash must be the blake2b hash hex-encoded of the data.

If the blob is already stored, then Put will be a no-op. So it's not necessary to make call Exists before saving a new blob.

func (*BlobsFiles) RebuildIndex Uses

func (backend *BlobsFiles) RebuildIndex() error

RebuildIndex removes the index files and re-build it by re-scanning all the BlobsFiles.

func (*BlobsFiles) SealedPacks Uses

func (backend *BlobsFiles) SealedPacks() []string

func (*BlobsFiles) SetBlobsFilesSealedFunc Uses

func (backend *BlobsFiles) SetBlobsFilesSealedFunc(f func(string))

func (*BlobsFiles) Size Uses

func (backend *BlobsFiles) Size(hash string) (int, error)

Size returns the blob size for the given hash.

func (*BlobsFiles) Stats Uses

func (backend *BlobsFiles) Stats() (*Stats, error)

Stats returns some stats about the DB.

func (*BlobsFiles) String Uses

func (backend *BlobsFiles) String() string

String implements the Stringer interface.

type CompressionAlgorithm Uses

type CompressionAlgorithm byte
const (
    Snappy CompressionAlgorithm = 1 << iota
)

Compression algorithms flag

type ErrInterventionNeeded Uses

type ErrInterventionNeeded struct {
    // contains filtered or unexported fields
}

ErrInterventionNeeded is an error indicating an manual action must be performed before being able to use BobsFile

func (*ErrInterventionNeeded) Error Uses

func (ein *ErrInterventionNeeded) Error() string

type Opts Uses

type Opts struct {
    // Compression algorithm
    Compression CompressionAlgorithm

    // The max size of a BlobsFile, will be 256MB by default if not set
    BlobsFileSize int64

    // Where the data and indexes will be stored
    Directory string

    // Allow to catch some events
    LogFunc func(msg string)

    // When trying to self-heal in case of recovery, some step need to be performed by the user
    AskConfirmationFunc func(msg string) bool

    BlobsFilesSealedFunc func(path string)
}

Opts represents the DB options

type Stats Uses

type Stats struct {
    // The total number of blobs stored
    BlobsCount int

    // The size of all the blobs stored
    BlobsSize int64

    // The number of BlobsFile
    BlobsFilesCount int

    // The size of all the BlobsFile
    BlobsFilesSize int64
}

Stats represents some stats about the DB state

Package blobsfile imports 17 packages (graph) and is imported by 7 packages. Updated 2020-05-29. Refresh now. Tools for package owners.