turing

package module
v0.0.0-...-d5de3f3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 28, 2021 License: MIT Imports: 26 Imported by: 0

README

turing

Test GoDoc Release

Turing is a framework for building domain specific databases on top of a replicated key value store. Database commands are implemented as instructions that are managed and executed by a cluster of turing nodes. The framework can be used to build client/server style databases or embedded databases within application/services. The goal is to provide a simple API and toolkit that can be used from standalone in-memory databases up to clusters consisting of many nodes. Under the hood, turing uses the pebble (alpha) for storing the data and dragonboat for reaching consensus among the nodes.

Examples

  • An example implementing a simple counter can be found here.
  • The stdset package implements a set of basic instructions.

License

The MIT License (MIT)

Copyright (c) 2020 Joël Gähwiler

Documentation

Overview

Package turing provides a framework to build domain specific databases.

Index

Constants

View Source
const UnboundedEffect = -1

UnboundedEffect can be used as an effect if the instruction potentially modifies more keys than MaxEffect allows. These instructions may be applied using multiple transactions.

Variables

View Source
var ErrDatabaseClosed = errors.New("turing: database closed")

ErrDatabaseClosed is returned if the database has been closed.

View Source
var ErrMaxEffect = errors.New("turing: max effect")

ErrMaxEffect is returned by a transaction if the effect limit has been reached. The instruction should return with this error to have the current changes persistent and be executed again to persist the remaining changes.

View Source
var ErrReadOnly = errors.New("turing: read only")

ErrReadOnly is returned by a transaction on write operations if the instruction has been flagged as read only.

Functions

func Clone

func Clone(src []byte) []byte

Clone will make a copy of the provided slice.

func PrefixRange

func PrefixRange(prefix []byte) ([]byte, []byte)

PrefixRange will compute the lower and upper bound of a prefix range.

func SetLogger

func SetLogger(sink io.Writer)

SetLogger can used to set a custom writer for all logs.

Types

type Cache

type Cache interface {
	// Set will set the specified key to the new value.
	Set(key, value interface{})

	// Get will lookup the the specified key and return the value and whether
	// it has been found.
	Get(key interface{}) (interface{}, bool)
}

Cache is used by instructions to cache concrete values.

type Config

type Config struct {

	// The id of this member.
	ID uint64

	// All cluster members.
	Members []Member

	// The storage directory. If empty, an in-memory filesystem is used.
	Directory string

	// The used instructions.
	Instructions []Instruction

	// In standalone mode the database is not replicated.
	Standalone bool

	// The maximum effect that can be reported by an instruction. Instructions
	// with a bigger effect must report an unbounded effect. Increasing the
	// value will allow more throughput as more instructions are executed using
	// the same transaction.
	//
	// Default: 10_000.
	MaxEffect int

	// The average round trip time.
	//
	// Default: 10ms.
	RoundTripTime time.Duration

	// The number of concurrent database readers.
	//
	// Default: min(NumCPUs - 3, 2).
	ConcurrentReaders int

	// The number of concurrent raft proposers.
	//
	// Default: NumCPUs.
	ConcurrentProposers int

	// The maximum instruction batch sizes.
	//
	// Default: 200, 200, 200.
	UpdateBatchSize   int
	LookupBatchSize   int
	ProposalBatchSize int

	// The time after a proposal times out.
	//
	// Default: 10s.
	ProposalTimeout time.Duration

	// The time after a linear read times out.
	//
	// Default: 10s.
	LinearReadTimeout time.Duration
}

Config is used to configure a machine.

func (Config) DatabaseDir

func (c Config) DatabaseDir() string

DatabaseDir returns the directory used for the database files.

func (Config) DatabaseFS

func (c Config) DatabaseFS() pfs.FS

DatabaseFS returns the filesystem used for the database files.

func (*Config) Local

func (c *Config) Local() *Member

Local will return the local member.

func (Config) RaftDir

func (c Config) RaftDir() string

RaftDir returns the directory used for the raft files.

func (Config) RaftFS

func (c Config) RaftFS() dfs.FS

RaftFS returns the filesystem used for the raft files.

func (*Config) Validate

func (c *Config) Validate() error

Validate will validate the configuration and ensure defaults.

type Description

type Description struct {
	// The unique name of the instruction. The notation "path/package/Type" is
	// recommended to ease discoverability.
	Name string

	// The builder can be set to provide a custom builder. If not set, the
	// default reflect based builder will be used.
	Builder func() Instruction

	// The recycler can be used in conjunction with the custom builder to
	// recycle built instructions.
	Recycler func(ins Instruction)

	// The operators used by this instruction. Deprecated operators must be
	// retained to ensure they can be used to compact older database levels.
	Operators []*Operator

	// NoResult may be set to true to indicate that the write instruction does
	// not carry a result. This potentially reduces some RPC traffic.
	NoResult bool
	// contains filtered or unexported fields
}

Description is a description of an instruction.

func (Description) Validate

func (d Description) Validate() error

Validate will validate the instruction description.

type Instruction

type Instruction interface {
	// Describe should return a description of the instruction. This method is
	// called often, therefore it should just return a pointer to a statically
	// allocated object and never build the object on request.
	Describe() *Description

	// Effect should return the amount of modifications this instruction will
	// make. A positive number is interpreted as the maximum amount of set,
	// unset merged and deleted keys during the execution. A zero value
	// indicates that the instruction is read only and will not set or delete
	// any keys. A negative number indicates that the effect is unbounded and
	// may modify many keys.
	Effect() int

	// Execute should execute the instruction using the provided memory.
	Execute(mem Memory, cache Cache) error

	// Encode should encode the instruction.
	Encode() ([]byte, Ref, error)

	// Decode should decode the instruction.
	Decode([]byte) error
}

Instruction is the interface that is implemented by instructions that are executed by the machine.

type Iterator

type Iterator interface {
	// SeekGE will seek to the exact key or the next greater key.
	SeekGE(key []byte) bool

	// SeekLT will seek to the exact key or the next smaller key.
	SeekLT(key []byte) bool

	// First will seek to the first key in the range.
	First() bool

	// Last will seek to the last key in the range.
	Last() bool

	// Valid will return whether a valid key/value pair is present.
	Valid() bool

	// Next will move on to the next key.
	Next() bool

	// Prev will go back to the previous key.
	Prev() bool

	// Key will return a buffered key that can be used until released. Leaking
	// the buffer brings a performance penalty. Use Clone and TempKey instead.
	Key() ([]byte, Ref)

	// TempKey will return the temporary key which is only valid until the next
	// iteration or until the iterator is closed.
	TempKey() []byte

	// Value will return a buffered value that can be used until released. Leaking
	// the buffer brings a performance penalty. Use Clone and TempValue instead.
	Value() ([]byte, Ref, error)

	// TempValue will return the temporary value which is only valid until the
	// next iteration or the iterator is closed.
	TempValue() ([]byte, error)

	// Use will yield the temporary key and value to the provided function.
	Use(fn func(key, value []byte) error) error

	// Error will return the error.
	Error() error

	// Close will close the iterator.
	Close() error
}

Iterator is used to iterate over the memory.

type Machine

type Machine struct {
	// contains filtered or unexported fields
}

Machine maintains a raft cluster with members and maintains consensus about the executed instructions on the replicated database.

func Start

func Start(config Config) (*Machine, error)

Start will create a new machine using the specified configuration.

func Test

func Test(ins ...Instruction) *Machine

Test will start and return a machine for testing purposes.

func (*Machine) Execute

func (m *Machine) Execute(ins Instruction, opts ...Options) error

Execute will execute the specified instruction.

func (*Machine) ExecuteAsync

func (m *Machine) ExecuteAsync(ins Instruction, fn func(error), opts ...Options) error

ExecuteAsync will execute the specified instruction asynchronously. The specified function is called once the instruction has been executed.

func (*Machine) Status

func (m *Machine) Status() Status

Status will return the current status.

func (*Machine) Stop

func (m *Machine) Stop()

Stop will stop the machine.

func (*Machine) Subscribe

func (m *Machine) Subscribe(observer Observer)

Subscribe will subscribe the provided observer.

func (*Machine) Unsubscribe

func (m *Machine) Unsubscribe(observer Observer)

Unsubscribe will unsubscribe the provided observer.

type Member

type Member struct {
	ID   uint64
	Host string
	Port int
}

Member specifies a cluster member.

func ParseMember

func ParseMember(str string) (Member, error)

ParseMember will parse the provided string and return a member. The string is expected to have the form "id@[host]:port".

func ParseMembers

func ParseMembers(str string) ([]Member, error)

ParseMembers will parse the provided string and return a list of members. The string is expected to have the form "id@[host]:port,...".

func (Member) Address

func (m Member) Address() string

Address will return the members full address in the form "host:port".

func (Member) String

func (m Member) String() string

String will return a formatted member string "id@host:port".

func (Member) Validate

func (m Member) Validate() error

Validate will validate the member.

type Memory

type Memory interface {
	// Iterate will construct and return a new iterator. The iterator must be
	// closed as soon as it is not used anymore.
	Iterate(prefix []byte) Iterator

	// Get will lookup the specified key. The returned slice must not be modified
	// by the caller. A closer is returned that must be closed once the value is
	// not used anymore. Consider using Use() if the value is only used temporarily.
	Get(key []byte) ([]byte, bool, io.Closer, error)

	// Use will lookup the specified key and yield it to the provided function if
	// it exists.
	Use(key []byte, fn func(value []byte) error) error

	// Set will set the specified key to the new value. This operation will count
	// as one towards the effect of the backing transaction.
	Set(key, value []byte) error

	// Unset will remove the specified key. This operation will count as one towards
	// the effect of the backing transaction.
	Unset(key []byte) error

	// Delete deletes all of the keys in the range [start, end] (inclusive on start,
	// exclusive on end). This operation will count as one towards the effect of the
	// backing transaction.
	Delete(start, end []byte) error

	// Merge merges existing values with the provided value using the specified
	// operator.
	Merge(key, value []byte, operator *Operator) error

	// Effect will return the current effect of the backing transaction.
	Effect() int
}

Memory is used by instructions to read and write to the database.

type Observer

type Observer interface {
	// Init is called when the source instruction stream has been (re)opened.
	// This happens when the machine starts and whenever the node fails and
	// restarts.
	Init()

	// Process is called repeatedly with every instruction processed by the
	// machine. The implementation must ensure that the function returns as fast
	// as possible as it blocks the execution of other instructions. If false is
	// returned, the observer will be unsubscribed.
	Process(ins Instruction) bool
}

Observer is the interface implemented by observers that want to observe the stream of instructions processed by the machine.

type Operator

type Operator struct {
	// The name of the operator.
	Name string

	// The zero value used as the base value if there is none.
	Zero []byte

	// The function called to apply operands to a value.
	Apply func(value []byte, ops [][]byte) ([]byte, Ref, error)

	// An optional function called to combine operands.
	Combine func(ops [][]byte) ([]byte, Ref, error)
	// contains filtered or unexported fields
}

Operator describes a merge operator.

type Options

type Options struct {
	// StaleRead can be set to execute a stale read. While this is much faster
	// the instruction might read stale data in relationship to the current
	// state of the cluster. In other words, settings this value will reduce
	// the default linearizable guarantee to a serializable guarantee.
	StaleRead bool
}

Options define options used during instruction execution.

type Ref

type Ref interface {
	Release()
}

Ref manages the reference to buffer that can be released.

type Role

type Role int

Role specifies the role of a cluster member.

const (

	// RoleLeader is the elected leader of a cluster.
	RoleLeader Role

	// RoleFollower is a electable cluster member.
	RoleFollower

	// RoleObserver is non-electable cluster member.
	RoleObserver
)

func (Role) String

func (r Role) String() string

String returns the name of the role.

type Status

type Status struct {
	// The id of this member.
	ID uint64

	// The role of this member.
	Role Role

	// The cluster leader.
	Leader *Member

	// The cluster members.
	Members []Member
}

Status contains information about the cluster.

func (Status) String

func (s Status) String() string

String returns the status formatted as a string.

Directories

Path Synopsis
examples

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL