pebble

package
v0.0.0-...-2f989cd Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 3, 2022 License: MIT Imports: 20 Imported by: 0

Documentation

Overview

Package pebble implements the persistent log storage used by Tugboat.

Index

Constants

View Source 
const (
	// MaxKeyLength is the max length of keys allowed.
	MaxKeyLength uint64 = 1024
)

Variables

View Source 
var PebbleLogger pebbleLogger

PebbleLogger is the logger used by pebble.

Functions

func Factory 

func Factory(config LogDBConfig) func(logdb.LogDBCallback, string, string) *ShardedDB

func NewPebbleFS 

func NewPebbleFS(fs vfs.FS) pvfs.FS

NewPebbleFS creates a new pebble/vfs.FS instance.

Types

type IContext 

type IContext interface {
	// Destroy destroys the IContext instance.
	Destroy()
	// Reset resets the IContext instance, all previous returned keys and
	// buffers will be put back to the IContext instance and be ready to
	// be used for the next iteration.
	Reset()
	// GetKey returns a reusable key.
	GetKey() IReusableKey
	// GetValueBuffer returns a byte buffer with at least sz bytes in length.
	GetValueBuffer(sz uint64) []byte
	// GetWriteBatch returns a write batch or transaction instance.
	GetWriteBatch() interface{}
	// SetWriteBatch adds the write batch to the IContext instance.
	SetWriteBatch(wb interface{})
	// GetEntryBatch returns an entry batch instance.
	GetEntryBatch() pb.EntryBatch
	// GetLastEntryBatch returns an entry batch instance.
	GetLastEntryBatch() pb.EntryBatch
}

IContext is the per thread context used in the logdb module. IContext is expected to contain a list of reusable keys and byte slices that are owned per thread so they can be safely reused by the same thread when accessing ILogDB.

type IReusableKey 

type IReusableKey interface {
	// SetEntryKey sets the key to be an entry key for the specified Raft node
	// with the specified entry index.
	SetEntryKey(clusterID uint64, nodeID uint64, index uint64)
	// SetStateKey sets the key to be an persistent state key suitable
	// for the specified Raft cluster node.
	SetStateKey(clusterID uint64, nodeID uint64)
	// SetMaxIndexKey sets the key to be the max possible index key for the
	// specified Raft cluster node.
	SetMaxIndexKey(clusterID uint64, nodeID uint64)
	// Key returns the underlying byte slice of the key.
	Key() []byte
	// Release releases the key instance so it can be reused in the future.
	Release()
}

IReusableKey is the interface for keys that can be reused. A reusable key is usually obtained by calling the GetKey() function of the IContext instance.

type IWriteBatch 

type IWriteBatch interface {
	Destroy()
	Put([]byte, []byte)
	Delete([]byte)
	Clear()
	Count() int
}

IWriteBatch is the interface representing a write batch capable of atomically writing many key-value pairs to the key-value store.

type KV 

type KV struct {
	// contains filtered or unexported fields
}

KV is a pebble based IKVStore type.

func (*KV) BulkRemoveEntries 

func (r *KV) BulkRemoveEntries(fk []byte, lk []byte) (err error)

BulkRemoveEntries ...

func (*KV) Close 

func (r *KV) Close() error

Close closes the RDB object.

func (*KV) CommitWriteBatch 

func (r *KV) CommitWriteBatch(wb *pebbleWriteBatch) error

CommitWriteBatch ...

func (*KV) CompactEntries 

func (r *KV) CompactEntries(fk []byte, lk []byte) error

CompactEntries ...

func (*KV) DeleteValue 

func (r *KV) DeleteValue(key []byte) error

DeleteValue ...

func (*KV) FullCompaction 

func (r *KV) FullCompaction() error

FullCompaction ...

func (*KV) GetValue 

func (r *KV) GetValue(key []byte, op func([]byte) error) (err error)

GetValue ...

func (*KV) GetWriteBatch 

func (r *KV) GetWriteBatch() *pebbleWriteBatch

GetWriteBatch ...

func (*KV) IterateValue 

func (r *KV) IterateValue(fk []byte, lk []byte, inc bool,
	op func(key []byte, data []byte) (bool, error)) (err error)

IterateValue ...

func (*KV) Name 

func (r *KV) Name() string

Name returns the IKVStore type name.

func (*KV) SaveValue 

func (r *KV) SaveValue(key []byte, value []byte) error

SaveValue ...

type Key 

type Key struct {
	// contains filtered or unexported fields
}

Key represents keys that are managed by a sync.Pool to be reused.

func (*Key) Key 

func (k *Key) Key() []byte

Key returns the []byte of the key.

func (*Key) Release 

func (k *Key) Release()

Release puts the key back to the pool.

func (*Key) SetEntryKey 

func (k *Key) SetEntryKey(clusterID uint64, nodeID uint64, index uint64)

SetEntryKey sets the key value to the specified entry key.

func (*Key) SetMaxIndexKey 

func (k *Key) SetMaxIndexKey(clusterID uint64, nodeID uint64)

SetMaxIndexKey sets the key value to the max index record key.

func (*Key) SetMaximumKey 

func (k *Key) SetMaximumKey()

SetMaximumKey sets the key to the maximum possible value.

func (*Key) SetMinimumKey 

func (k *Key) SetMinimumKey()

SetMinimumKey sets the key to the minimum possible value.

func (*Key) SetStateKey 

func (k *Key) SetStateKey(clusterID uint64, nodeID uint64)

SetStateKey sets the key value to the specified State.

type LogDBCallback 

type LogDBCallback func(busy bool)

LogDBCallback is a callback function called by the LogDB.

type LogDBConfig 

type LogDBConfig struct {
	FS                                 vfs.FS
	Shards                             uint64
	KVKeepLogFileNum                   uint64
	KVMaxBackgroundCompactions         uint64
	KVMaxBackgroundFlushes             uint64
	KVLRUCacheSize                     uint64
	KVWriteBufferSize                  uint64
	KVMaxWriteBufferNumber             uint64
	KVLevel0FileNumCompactionTrigger   uint64
	KVLevel0SlowdownWritesTrigger      uint64
	KVLevel0StopWritesTrigger          uint64
	KVMaxBytesForLevelBase             uint64
	KVMaxBytesForLevelMultiplier       uint64
	KVTargetFileSizeBase               uint64
	KVTargetFileSizeMultiplier         uint64
	KVLevelCompactionDynamicLevelBytes uint64
	KVRecycleLogFileNum                uint64
	KVNumOfLevels                      uint64
	KVBlockSize                        uint64
	SaveBufferSize                     uint64
	MaxSaveBufferSize                  uint64
}

LogDBConfig is the configuration object for the LogDB storage engine. This config option is only for advanced users when tuning the balance of I/O performance and memory consumption.

All KV* fields in LogDBConfig had their names derived from RocksDB options, please check RocksDB Tuning Guide wiki for more details.

KVWriteBufferSize and KVMaxWriteBufferNumber are two parameters that directly affect the upper bound of memory size used by the built-in LogDB storage engine.

func GetDefaultLogDBConfig 

func GetDefaultLogDBConfig() LogDBConfig

GetDefaultLogDBConfig returns the default configurations for the LogDB storage engine. The default LogDB configuration use up to 8GBytes memory.

func GetLargeMemLogDBConfig 

func GetLargeMemLogDBConfig() LogDBConfig

GetLargeMemLogDBConfig returns a LogDB config aimed to keep memory size to be large for good I/O performance. It is the default setting used by the system. When using the returned config, LogDB takes up to 8GBytes memory.

func GetMediumMemLogDBConfig 

func GetMediumMemLogDBConfig() LogDBConfig

GetMediumMemLogDBConfig returns a LogDB config aimed to keep memory size at medium level. When using the returned config, LogDB takes up to 4GBytes memory.

func GetSmallMemLogDBConfig 

func GetSmallMemLogDBConfig() LogDBConfig

GetSmallMemLogDBConfig returns a LogDB config aimed to keep memory size at low level. When using the returned config, LogDB takes up to 1GBytes memory.

func GetTinyMemLogDBConfig 

func GetTinyMemLogDBConfig() LogDBConfig

GetTinyMemLogDBConfig returns a LogDB config aimed for minimizing memory size. When using the returned config, LogDB takes up to 256MBytes memory.

func (*LogDBConfig) IsEmpty 

func (cfg *LogDBConfig) IsEmpty() bool

IsEmpty returns a boolean value indicating whether the LogDBConfig instance is empty.

func (*LogDBConfig) MemorySizeMB 

func (cfg *LogDBConfig) MemorySizeMB() uint64

MemorySizeMB returns the estimated upper bound memory size used by the LogDB storage engine. The returned value is in MBytes.

type PebbleFS 

type PebbleFS struct {
	// contains filtered or unexported fields
}

PebbleFS is a wrapper struct that implements the pebble/vfs.FS interface.

func (*PebbleFS) Create 

func (p *PebbleFS) Create(name string) (pvfs.File, error)

Create ...

func (*PebbleFS) GetDiskUsage 

func (p *PebbleFS) GetDiskUsage(path string) (pvfs.DiskUsage, error)

func (*PebbleFS) GetFreeSpace 

func (p *PebbleFS) GetFreeSpace(path string) (uint64, error)

GetFreeSpace ... 

func (p *PebbleFS) Link(oldname, newname string) error

Link ...

func (*PebbleFS) List 

func (p *PebbleFS) List(dir string) ([]string, error)

List ...

func (*PebbleFS) Lock 

func (p *PebbleFS) Lock(name string) (io.Closer, error)

Lock ...

func (*PebbleFS) MkdirAll 

func (p *PebbleFS) MkdirAll(dir string, perm os.FileMode) error

MkdirAll ...

func (*PebbleFS) Open 

func (p *PebbleFS) Open(name string, opts ...pvfs.OpenOption) (pvfs.File, error)

Open ...

func (*PebbleFS) OpenDir 

func (p *PebbleFS) OpenDir(name string) (pvfs.File, error)

OpenDir ...

func (*PebbleFS) PathBase 

func (p *PebbleFS) PathBase(path string) string

PathBase ...

func (*PebbleFS) PathDir 

func (p *PebbleFS) PathDir(path string) string

PathDir ...

func (*PebbleFS) PathJoin 

func (p *PebbleFS) PathJoin(elem ...string) string

PathJoin ...

func (*PebbleFS) Remove 

func (p *PebbleFS) Remove(name string) error

Remove ...

func (*PebbleFS) RemoveAll 

func (p *PebbleFS) RemoveAll(name string) error

RemoveAll ...

func (*PebbleFS) Rename 

func (p *PebbleFS) Rename(oldname, newname string) error

Rename ...

func (*PebbleFS) ReuseForWrite 

func (p *PebbleFS) ReuseForWrite(oldname, newname string) (pvfs.File, error)

ReuseForWrite ...

func (*PebbleFS) Stat 

func (p *PebbleFS) Stat(name string) (os.FileInfo, error)

Stat ...

type ShardedDB 

type ShardedDB struct {
	// contains filtered or unexported fields
}

ShardedDB is a LogDB implementation using sharded rocksdb instances.

func NewLogDB 

func NewLogDB(config LogDBConfig, callback logdb.LogDBCallback, dirs []string, lldirs []string, check bool) (*ShardedDB, error)

NewLogDB creates a Log DB instance based on provided configuration parameters. The underlying KV store used by the Log DB instance is created by the provided factory function.

func OpenShardedDB 

func OpenShardedDB(config LogDBConfig, cb logdb.LogDBCallback, dirs []string, lldirs []string, check bool) (*ShardedDB, error)

OpenShardedDB creates a ShardedDB instance.

func (*ShardedDB) BinaryFormat 

func (s *ShardedDB) BinaryFormat() uint32

BinaryFormat is the binary format supported by the sharded DB.

func (*ShardedDB) Close 

func (s *ShardedDB) Close() (err error)

Close closes the ShardedDB instance.

func (*ShardedDB) CompactEntriesTo 

func (s *ShardedDB) CompactEntriesTo(clusterID uint64,
	nodeID uint64, index uint64) (<-chan struct{}, error)

CompactEntriesTo reclaims underlying storage space used for storing entries up to the specified index.

func (*ShardedDB) GetBootstrapInfo 

func (s *ShardedDB) GetBootstrapInfo(clusterID uint64,
	nodeID uint64) (pb.Bootstrap, error)

GetBootstrapInfo returns the saved bootstrap info for the given node.

func (*ShardedDB) GetLogDBThreadContext 

func (s *ShardedDB) GetLogDBThreadContext() IContext

GetLogDBThreadContext return an IContext instance. This method is expected to be used in benchmarks and tests only.

func (*ShardedDB) GetSnapshot 

func (s *ShardedDB) GetSnapshot(clusterID uint64,
	nodeID uint64) (pb.Snapshot, error)

GetSnapshot returns the most recent snapshot associated with the specified cluster.

func (*ShardedDB) ImportSnapshot 

func (s *ShardedDB) ImportSnapshot(ss pb.Snapshot, nodeID uint64) error

ImportSnapshot imports the snapshot record and other metadata records to the system.

func (*ShardedDB) IterateEntries 

func (s *ShardedDB) IterateEntries(ents []pb.Entry,
	size uint64, clusterID uint64, nodeID uint64, low uint64, high uint64,
	maxSize uint64) ([]pb.Entry, uint64, error)

IterateEntries returns a list of saved entries starting with index low up to index high with a max size of maxSize.

func (*ShardedDB) ListNodeInfo 

func (s *ShardedDB) ListNodeInfo() ([]raftio.NodeInfo, error)

ListNodeInfo lists all available NodeInfo found in the log db.

func (*ShardedDB) Name 

func (s *ShardedDB) Name() string

Name returns the type name of the instance.

func (*ShardedDB) ReadRaftState 

func (s *ShardedDB) ReadRaftState(clusterID uint64,
	nodeID uint64, lastIndex uint64) (raftio.RaftState, error)

ReadRaftState returns the persistent state of the specified raft node.

func (*ShardedDB) RemoveEntriesTo 

func (s *ShardedDB) RemoveEntriesTo(clusterID uint64,
	nodeID uint64, index uint64) error

RemoveEntriesTo removes entries associated with the specified raft node up to the specified index.

func (*ShardedDB) RemoveNodeData 

func (s *ShardedDB) RemoveNodeData(clusterID uint64, nodeID uint64) error

RemoveNodeData deletes all node data that belongs to the specified node.

func (*ShardedDB) SaveBootstrapInfo 

func (s *ShardedDB) SaveBootstrapInfo(clusterID uint64,
	nodeID uint64, bootstrap pb.Bootstrap) error

SaveBootstrapInfo saves the specified bootstrap info for the given node.

func (*ShardedDB) SaveRaftState 

func (s *ShardedDB) SaveRaftState(updates []pb.Update, shardID uint64) error

SaveRaftState saves the raft state and logs found in the raft.Update list to the log db.

func (*ShardedDB) SaveRaftStateCtx 

func (s *ShardedDB) SaveRaftStateCtx(updates []pb.Update, ctx IContext) error

SaveRaftStateCtx saves the raft state and logs found in the raft.Update list to the log db.

func (*ShardedDB) SaveSnapshots 

func (s *ShardedDB) SaveSnapshots(updates []pb.Update) error

SaveSnapshots saves all snapshot metadata found in the raft.Update list.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.