Documentation ¶
Overview ¶
package cellar implements a low level, append only data storage engine.
cellar was originally forked from abdullin/cellar. This version focusses on a more standard API, less built in features and more configurability. It is used in carapace/core/pkg/append-db (versioned object DB) as the storage engine.
It is completely embedded, internally buffering writes until chunks are filled up, then automatically flushing them to file. See the example for a full usage of cellar.
Index ¶
- Variables
- func WithNoFileLock(db *DB) error
- func WithReadOnly(db *DB) error
- type AES
- type BoltMetaDB
- func (b *BoltMetaDB) AddChunk(pos int64, dto *pb.ChunkDto) error
- func (b *BoltMetaDB) CellarMeta() (dto *pb.MetaDto, err error)
- func (b *BoltMetaDB) GetBuffer() (buf *pb.BufferDto, err error)
- func (b *BoltMetaDB) GetCheckpoint(name string) (pos int64, err error)
- func (b *BoltMetaDB) Init() error
- func (b *BoltMetaDB) ListChunks() (dto []*pb.ChunkDto, err error)
- func (b *BoltMetaDB) PutBuffer(dto *pb.BufferDto) (err error)
- func (b *BoltMetaDB) PutCheckpoint(name string, pos int64) error
- func (b *BoltMetaDB) SetCellarMeta(dto *pb.MetaDto) (err error)
- type Buffer
- type ChainCompressor
- type ChainDecompressor
- type Cipher
- type CompressionWriter
- type Compressor
- type DB
- func (db *DB) Append(data []byte) (pos int64, err error)
- func (db *DB) Buffer() int64
- func (db *DB) Checkpoint() (pos int64, err error)
- func (db *DB) Close() (err error)
- func (db *DB) Flush() (err error)
- func (db *DB) Folder() string
- func (db *DB) GetUserCheckpoint(name string) (pos int64, err error)
- func (db *DB) PutUserCheckpoint(name string, pos int64) (err error)
- func (db *DB) Reader() *Reader
- func (db *DB) VolatilePos() int64
- type Decompressor
- type FileLock
- type MetaDB
- type MockLock
- type Option
- type ReadFlag
- type ReadOp
- type Reader
- type ReaderInfo
- type Rec
- type Writer
- func (w *Writer) Append(data []byte) (pos int64, err error)
- func (w *Writer) Checkpoint() (int64, error)
- func (w *Writer) Close() error
- func (w *Writer) Flush() error
- func (w *Writer) GetUserCheckpoint(name string) (int64, error)
- func (w *Writer) PutUserCheckpoint(name string, pos int64) error
- func (w *Writer) VolatilePos() int64
Constants ¶
This section is empty.
Variables ¶
var ( CheckPointBucketKey = []byte("a") ChunkTableKey = []byte("b") BufferBucketKey = []byte("c") BufferKey = []byte("d") CellarBucketKey = []byte("e") CellarKey = []byte("f") )
var (
ErrBucketNotExists = errors.New("boltdb: bucket not present")
)
var (
ErrIsFile = errors.New("provided folder is actually a path")
)
Functions ¶
func WithNoFileLock ¶
WithNoFileLock is only recommending in unit tests, as it allows for concurrent writers (which is a big nono if you want data integrity)
func WithReadOnly ¶
Types ¶
type AES ¶
type AES struct {
// contains filtered or unexported fields
}
type BoltMetaDB ¶
func (*BoltMetaDB) CellarMeta ¶
func (b *BoltMetaDB) CellarMeta() (dto *pb.MetaDto, err error)
func (*BoltMetaDB) GetCheckpoint ¶
func (b *BoltMetaDB) GetCheckpoint(name string) (pos int64, err error)
func (*BoltMetaDB) ListChunks ¶
func (b *BoltMetaDB) ListChunks() (dto []*pb.ChunkDto, err error)
func (*BoltMetaDB) PutCheckpoint ¶
func (b *BoltMetaDB) PutCheckpoint(name string, pos int64) error
func (*BoltMetaDB) SetCellarMeta ¶
func (b *BoltMetaDB) SetCellarMeta(dto *pb.MetaDto) (err error)
type ChainCompressor ¶
type ChainCompressor struct {
CompressionLevel int
}
func (ChainCompressor) Compress ¶
func (c ChainCompressor) Compress(w io.Writer) (CompressionWriter, error)
type Cipher ¶
type Cipher interface { Decrypt(src io.Reader) (io.Reader, error) Encrypt(w io.Writer) (*cipher.StreamWriter, error) }
Cipher defines the interface needed to support encryption of the DB
type CompressionWriter ¶
CompressionWriter is based on the functions needed from the lz4 compressor
type Compressor ¶
type Compressor interface {
Compress(io.Writer) (CompressionWriter, error)
}
type DB ¶
type DB struct {
// contains filtered or unexported fields
}
DB is a godlevel/convenience wrapper around Writer and Reader, ensuring only one writer exists per folder, and storing the cipher for faster performance.
func (*DB) Append ¶
Write creates a writer using sync.Once, and then reuses the writer over procedures
func (*DB) Checkpoint ¶
Checkpoint creates an anonymous checkpoint at the current cursor's location.
func (*DB) Close ¶
Close ensures filelocks are cleared and resources closed. Readers derived from this DB instance will remain functional.
func (*DB) GetUserCheckpoint ¶
GetUserCheckpoint returns the position of a named checkpoint
func (*DB) PutUserCheckpoint ¶
PutUserCheckpoint creates a named checkpoint at a given position.
func (*DB) Reader ¶
Reader returns a new db reader. The reader remains active even if the DB is closed
func (*DB) VolatilePos ¶
VolatilePos returns the current cursors location
type MetaDB ¶
type MetaDB interface { GetBuffer() (*pb.BufferDto, error) PutBuffer(*pb.BufferDto) error ListChunks() ([]*pb.ChunkDto, error) AddChunk(int64, *pb.ChunkDto) error CellarMeta() (*pb.MetaDto, error) SetCellarMeta(*pb.MetaDto) error PutCheckpoint(name string, pos int64) error GetCheckpoint(name string) (int64, error) Close() error Init() error }
MetaDB defines an interface for databases storing metadata on the cellar DB. the default implementation is based on either LMDB or Boltdb (K/V stores work best for this purpose)
type Option ¶
func WithCipher ¶
WithCipher allows for customizing the read/write encryption.
func WithLogger ¶
func WithMetaDB ¶
type ReadOp ¶
type ReadOp func(pos *ReaderInfo, data []byte) error
type Reader ¶
type Reader struct { Folder string Flags ReadFlag StartPos int64 EndPos int64 LimitChunks int // contains filtered or unexported fields }
type ReaderInfo ¶
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}