dvid: github.com/janelia-flyem/dvid/storage Index | Files | Directories

package storage

import "github.com/janelia-flyem/dvid/storage"

Package storage provides a unified interface to a number of storage engines. Since each storage engine has different capabilities, this package defines a number of interfaces in addition to the core Engine interface, which all storage engines should satisfy.

Keys are specified as a combination of Context and a datatype-specific byte slice, typically called an "type-specific key" (TKey) in DVID docs and code. The Context provides DVID-wide namespacing and as such, must use one of the Context implementations within the storage package. (This is enforced by making Context a Go opaque interface.) The type-specific key formatting is entirely up to the datatype designer, although use of dvid.Index is suggested.

Initially we are concentrating on key-value backends but expect to support graph and perhaps relational databases, either using specialized databases or software layers on top of an ordered key-value store.

Although we assume lexicographically ordering for range queries, there is some variation in how variable size keys are treated. We assume all storage engines, after appropriate DVID drivers, use the following definition of ordering:

A string s precedes a string t in lexicographic order if:

* s is a prefix of t, or
* if c and d are respectively the first character of s and t in which s and t differ,
  then c precedes d in character order.
* if s and t are equivalent for all of s, but t is longer

Note: For the characters that are alphabetical letters, the character order coincides
with the alphabetical order. Digits precede letters, and uppercase letters precede
lowercase ones.

Examples:

composer precedes computer
house precedes household
Household precedes house
H2O precedes HOTEL
mydex precedes mydexterity

Note that the above is different than shortlex order, which would group strings
based on length first.

The above lexicographical ordering is used by default for levedb variants.

Index

Package Files

context.go graphdb.go graphkeyvalue.go groupcache.go kafka.go keyvalue.go log.go monitor.go push.go storage.go storage_local.go utils.go

Constants

const (
    // MarkData is a byte indicating real data stored and should be the last byte of any
    // versioned key.
    MarkData = 0x03

    // MarkTombstone is a byte indicating a tombstone -- a marker for deleted data.
    MarkTombstone = 0x4F
)
const (
    TKeyMinClass = 0x00
    TKeyMaxClass = 0xFF
)
const KafkaMaxMessageSize = 980 * dvid.Kilo

KafkaMaxMessageSize is the max message size in bytes for a Kafka message.

const MonitorBuffer = 10000

Variables

var (
    // Number of bytes read in last second from storage engine.
    StoreKeyBytesReadPerSec int

    // Number of bytes written in last second to storage engine.
    StoreKeyBytesWrittenPerSec int

    // Number of bytes read in last second from storage engine.
    StoreValueBytesReadPerSec int

    // Number of bytes written in last second to storage engine.
    StoreValueBytesWrittenPerSec int

    // Number of bytes read in last second from file system.
    FileBytesReadPerSec int

    // Number of bytes written in last second to file system.
    FileBytesWrittenPerSec int

    // Number of key-value GET calls in last second.
    GetsPerSec int

    // Number of key-value PUT calls in last second.
    PutsPerSec int

    // Channel to notify bytes read from a storage engine.
    StoreKeyBytesRead chan int

    // Channel to notify bytes written to a storage engine.
    StoreKeyBytesWritten chan int

    // Channel to notify bytes read from a storage engine.
    StoreValueBytesRead chan int

    // Channel to notify bytes written to a storage engine.
    StoreValueBytesWritten chan int

    // Channel to notify bytes read from file system.
    FileBytesRead chan int

    // Channel to notify bytes written to file system.
    FileBytesWritten chan int
)
var ErrLogEOF = fmt.Errorf("log EOF")

ErrLogEOF is an error returned by a log Read() on an end of file

var (

    // the kafka topic prefix for mutation logging
    KafkaTopicPrefix string
)

func AllStores Uses

func AllStores() (map[Alias]dvid.Store, error)

func DataFromFile Uses

func DataFromFile(filename string) ([]byte, error)

DataFromFile returns data from a file.

func DataKeyToLocalIDs Uses

func DataKeyToLocalIDs(k Key) (dvid.InstanceID, dvid.VersionID, dvid.ClientID, error)

DataKeyToLocalIDs parses a key and returns instance, version and client ids.

func DefaultKVStore Uses

func DefaultKVStore() (dvid.Store, error)

func DefaultLogStore Uses

func DefaultLogStore() (dvid.Store, error)

func DeleteDataInstance Uses

func DeleteDataInstance(data dvid.Data) error

DeleteDataInstance removes a data instance.

func EnginesAvailable Uses

func EnginesAvailable() string

EnginesAvailable returns a description of the available storage engines.

func GetAssignedLog Uses

func GetAssignedLog(dataname dvid.InstanceName, root dvid.UUID, tags map[string]string, typename dvid.TypeString) (dvid.Store, error)

GetAssignedLog returns the append-only log assigned based on (instance name, root uuid) or type.

func GetAssignedStore Uses

func GetAssignedStore(dataname dvid.InstanceName, root dvid.UUID, tags map[string]string, typename dvid.TypeString) (dvid.Store, error)

GetAssignedStore returns the store assigned based on (instance name, root uuid), tag, or type, in that order. In some cases, this store may include a caching wrapper if the data instance has been configured to use groupcache.

func GetDataSizes Uses

func GetDataSizes(store dvid.Store, instances []dvid.InstanceID) (map[dvid.InstanceID]uint64, error)

GetDataSizes returns a list of storage sizes in bytes for each data instance in the store. A list of InstanceID can be optionally supplied so only those instances are queried. This requires some scanning of the database so could take longer than normal requests, particularly if a list of instances is not given. Note that the underlying store must support both the OrderedKeyValueGetter and SizeViewer interface, else this function returns nil.

func GetStoreByAlias Uses

func GetStoreByAlias(alias Alias) (dvid.Store, error)

GetStoreByAlias returns a store by the alias given to it in the configuration TOML file, e.g., "raid6".

func Initialize Uses

func Initialize(cmdline dvid.Config, backend *Backend) (createdMetadata bool, err error)

Initialize the storage systems. Returns a bool + error where the bool is true if the metadata store is newly created and needs initialization. The map of store configurations should be keyed by either a datatype name, "default", or "metadata".

func KafkaProduceMsg Uses

func KafkaProduceMsg(value []byte, topic string) error

KafkaProduceMsg sends a message to kafka

func KafkaShutdown Uses

func KafkaShutdown()

KafkaShutdown makes sure that the kafka queue is flushed before stopping.

func KafkaTopicSuffix Uses

func KafkaTopicSuffix(dataUUID dvid.UUID) string

KafkaTopicSuffix returns any configured suffix for the given data UUID or the empty string.

func LogActivityToKafka Uses

func LogActivityToKafka(activity map[string]interface{})

LogActivityToKafka publishes activity

func MetadataKeyPrefix Uses

func MetadataKeyPrefix() []byte

func NewStore Uses

func NewStore(c dvid.StoreConfig) (db dvid.Store, created bool, err error)

NewStore checks if a given engine is available and if so, returns a store opened with the configuration.

func RegisterEngine Uses

func RegisterEngine(e Engine)

RegisterEngine registers an Engine for DVID use.

func Repair Uses

func Repair(name, path string) error

Repair repairs a named engine's store at given path.

func Shutdown Uses

func Shutdown()

Shutdown handles any storage-specific shutdown procedures.

func SplitKey Uses

func SplitKey(k Key) (unversioned Key, versioned Key, err error)

SplitKey returns key components depending on whether the passed Key is a metadata or data key. If metadata, it returns the key and a 0 version id. If it is a data key, it returns the unversioned portion of the Key and the version id.

func UpdateDataKey Uses

func UpdateDataKey(k Key, instance dvid.InstanceID, version dvid.VersionID, client dvid.ClientID) error

func VersionFromDataKey Uses

func VersionFromDataKey(key Key) (dvid.VersionID, error)

VersionFromDataKey returns a version ID from a data key.

type Alias Uses

type Alias string

Alias is a nickname for a storage configuration, e.g., "raid6" for basholeveldb on RAID6 drives. It is used in the DVID TOML configuration file like [storage.alias]

type Backend Uses

type Backend struct {
    Metadata    Alias // The store that should be used for metadata storage.
    DefaultLog  Alias // The log that should be used by default.
    DefaultKVDB Alias // The key-value datastore that should be used by default.
    Stores      map[Alias]dvid.StoreConfig
    KVStore     DataMap
    LogStore    DataMap
    Groupcache  GroupcacheConfig
}

Backend provide data instance to store mappings gleaned from DVID configuration. Currently, a data instance can be mapped to one KV and one Log store.

func GetTestableBackend Uses

func GetTestableBackend(kvMap, logMap DataMap) (map[Alias]TestableEngine, *Backend, error)

GetTestableBackend returns a testable engine and backend.

func (Backend) StoreConfig Uses

func (b Backend) StoreConfig(d dvid.DataSpecifier) (config dvid.StoreConfig, found bool)

StoreConfig returns a data specifier's assigned store configuration. The DataSpecifier can be "default" or "metadata" as well as datatype names and data instance specifications.

type Batch Uses

type Batch interface {
    // Delete removes from the batch a put using the given key.
    Delete(TKey)

    // Put adds to the batch a put using the given key-value.
    Put(k TKey, v []byte)

    // Commits a batch of operations and closes the write batch.
    Commit() error
}

Batch groups operations into a transaction. Clear() and Close() were removed due to how other key-value stores implement batches. It's easier to implement cross-database handling of a simple write/delete batch that commits then closes rather than something that clears.

type BlobStore Uses

type BlobStore interface {
    PutBlob([]byte) (ref string, err error)
    GetBlob(ref string) ([]byte, error)
}

BlobStore allows writing and retrieving unversioned data. It differs from KeyValueDB in that it auto-generates and returns a key on writing, typically the content hash.

type BufferableOps Uses

type BufferableOps interface {
    KeyValueSetter

    // Put key-value pairs.  Note that it could be more efficient to use the Batcher
    // interface so you don't have to create and keep a slice of KeyValue.  Some
    // databases like leveldb will copy on batch put anyway.
    PutRange(Context, []TKeyValue) error

    // DeleteRange removes all key-value pairs with keys in the given range.
    // If versioned data in mutable stores, this will create tombstones in the version
    // unlike RawDelete or DeleteAll.
    DeleteRange(ctx Context, kStart, kEnd TKey) error

    // ProcessRange will process all gets when flush is called
    ProcessRange(ctx Context, kStart, kEnd TKey, op *ChunkOp, f ChunkFunc) error
}

RequestBufferSubset implements a subset of the ordered key/value interface. It declares interface common to both ordered key value and RequestBuffer

type Chunk Uses

type Chunk struct {
    *ChunkOp
    *TKeyValue
}

Chunk is the unit passed down channels to chunk handlers. Chunks can be passed from lower-level database access functions to type-specific chunk processing.

type ChunkFunc Uses

type ChunkFunc func(*Chunk) error

ChunkFunc is a function that accepts a Chunk.

type ChunkOp Uses

type ChunkOp struct {
    Op  interface{}
    Wg  *sync.WaitGroup
}

ChunkOp is a type-specific operation with an optional WaitGroup to sync mapping before reduce.

type Context Uses

type Context interface {
    // VersionID returns the local version ID of the DAG node being operated on.
    // If not versioned, the version is the root ID.
    VersionID() dvid.VersionID

    // RepoRoot returns the root uuid.
    RepoRoot() (dvid.UUID, error)

    // ConstructKey takes a type-specific key component, and generates a
    // namespaced key that fits with the DVID-wide key space partitioning.
    ConstructKey(TKey) Key

    // ConstructKeyVersion constructs a key like ConstructKey
    // but using specified version
    ConstructKeyVersion(TKey, dvid.VersionID) Key

    // KeyRange returns the minimum and maximum keys for this context.
    KeyRange() (min, max Key)

    // String prints a description of the Context
    String() string

    // Returns a sync.Mutex specific to this context.
    Mutex() *sync.Mutex

    // Versioned is true if this Context is also a VersionedCtx.
    Versioned() bool

    // SplitKey returns key components useful to store all versiones in a familyColumn if the storage engine supports it
    SplitKey(tk TKey) (Key, []byte, error)
}

Context allows encapsulation of data that defines the partitioning of the DVID key space.

type DataContext Uses

type DataContext struct {
    // contains filtered or unexported fields
}

DataContext supports both unversioned and versioned data persistence.

func NewDataContext Uses

func NewDataContext(data dvid.Data, versionID dvid.VersionID) *DataContext

NewDataContext provides a way for datatypes to create a Context that adheres to DVID key space partitioning. Since Context and VersionedCtx interfaces are opaque, i.e., can only be implemented within package storage, we force compatible implementations to embed DataContext and initialize it via this function.

func (*DataContext) ClientFromKey Uses

func (ctx *DataContext) ClientFromKey(key Key) (dvid.ClientID, error)

ClientFromKey returns a clientID from a full key. Any DataContext is sufficient as receiver.

func (*DataContext) ClientID Uses

func (ctx *DataContext) ClientID() dvid.ClientID

func (*DataContext) ConstructKey Uses

func (ctx *DataContext) ConstructKey(tk TKey) Key

func (*DataContext) ConstructKeyVersion Uses

func (ctx *DataContext) ConstructKeyVersion(tk TKey, version dvid.VersionID) Key

func (*DataContext) Data Uses

func (ctx *DataContext) Data() dvid.Data

func (*DataContext) DataName Uses

func (ctx *DataContext) DataName() dvid.InstanceName

func (*DataContext) GetRequestID Uses

func (ctx *DataContext) GetRequestID() string

GetRequestID returns a string identifier or the empty string if none have been set.

func (*DataContext) InstanceFromKey Uses

func (ctx *DataContext) InstanceFromKey(key Key) (dvid.InstanceID, error)

InstanceFromKey returns an InstanceID from a full key. Any DataContext is sufficient as receiver.

func (*DataContext) InstanceID Uses

func (ctx *DataContext) InstanceID() dvid.InstanceID

func (*DataContext) InstanceVersion Uses

func (ctx *DataContext) InstanceVersion() dvid.InstanceVersion

func (*DataContext) KeyRange Uses

func (ctx *DataContext) KeyRange() (min, max Key)

KeyRange returns the min and max full keys. The DataContext can have any version since min/max keys for a data instance is independent of the current context's version.

func (*DataContext) MaxVersionKey Uses

func (ctx *DataContext) MaxVersionKey(tk TKey) (Key, error)

Returns upper bound key for versions of given byte slice key representation.

func (*DataContext) MinVersionKey Uses

func (ctx *DataContext) MinVersionKey(tk TKey) (Key, error)

Returns lower bound key for versions of given byte slice key representation.

func (*DataContext) Mutex Uses

func (ctx *DataContext) Mutex() *sync.Mutex

func (*DataContext) RepoRoot Uses

func (ctx *DataContext) RepoRoot() (dvid.UUID, error)

func (*DataContext) RequestID Uses

func (ctx *DataContext) RequestID() string

func (*DataContext) SetRequestID Uses

func (ctx *DataContext) SetRequestID(id string)

SetRequestID sets a string identifier.

func (*DataContext) SplitKey Uses

func (ctx *DataContext) SplitKey(tk TKey) (Key, []byte, error)

Split the key in two parts: the first one call unversioned key, and the second one called versioned key

func (*DataContext) String Uses

func (ctx *DataContext) String() string

func (*DataContext) TKeyClassRange Uses

func (ctx *DataContext) TKeyClassRange(c TKeyClass) (min, max Key)

TKeyClassRange returns the min and max full keys for this class of TKeys.

func (*DataContext) TKeyRange Uses

func (ctx *DataContext) TKeyRange() (min, max TKey)

TKeyRange returns min and max type-specific keys. The max key is not guaranteed to be the theoretical maximum TKey but should be so for any TKey of 128 bytes or less. The DataContext can be nil.

func (*DataContext) TombstoneKey Uses

func (ctx *DataContext) TombstoneKey(tk TKey) Key

func (*DataContext) TombstoneKeyVersion Uses

func (ctx *DataContext) TombstoneKeyVersion(tk TKey, version dvid.VersionID) Key

func (*DataContext) UnversionedKey Uses

func (ctx *DataContext) UnversionedKey(tk TKey) (Key, dvid.VersionID, error)

UnversionedKey returns a unversioned Key and the version id as separate components. This can be useful for storage systems like column stores where the row key is the unversioned Key and the column qualifier is the version id.

func (*DataContext) UnversionedKeyPrefix Uses

func (ctx *DataContext) UnversionedKeyPrefix(tk TKey) Key

UnversionedKeyPrefix returns a byte slice that is the common prefix for all versions of this particular TKey.

func (*DataContext) UpdateInstance Uses

func (ctx *DataContext) UpdateInstance(k Key) error

func (*DataContext) ValidKV Uses

func (ctx *DataContext) ValidKV(kv *KeyValue, versions map[dvid.VersionID]struct{}) bool

ValidKV returns true if a key-value pair is in an allowed set of versions. A nil kv always returns true. An uninterpretable key returns false.

func (*DataContext) VersionFromKey Uses

func (ctx *DataContext) VersionFromKey(key Key) (dvid.VersionID, error)

VersionFromKey returns a version ID from a full key. Any DataContext is sufficient as receiver.

func (*DataContext) VersionID Uses

func (ctx *DataContext) VersionID() dvid.VersionID

func (*DataContext) Versioned Uses

func (ctx *DataContext) Versioned() bool

Versioned returns false. This can be overriden by embedding DataContext in structures that will support the VersionedCtx interface.

type DataMap Uses

type DataMap map[dvid.DataSpecifier]Alias

DataMap describes how data instances and types are mapped to available storage systems.

type Engine Uses

type Engine interface {
    fmt.Stringer

    // GetName returns a simple driver identifier like "basholeveldb", "kvautobus" or "bigtable".
    GetName() string

    // IsDistributed returns whether the engine is a distributed DB (engine should manage request throttling)
    IsDistributed() bool

    // GetSemVer returns the semantic versioning info.
    GetSemVer() semver.Version

    // NewStore returns a new storage engine given the passed configuration.
    // Should return true for initMetadata if the store needs initialization of metadata.
    NewStore(dvid.StoreConfig) (db dvid.Store, initMetadata bool, err error)
}

Engine is a storage engine that can create a storage instance, dvid.Store, which could be a database directory in the case of an embedded database Engine implementation. Engine implementations can fulfill a variety of interfaces, checkable by runtime cast checks, e.g., myGetter, ok := myEngine.(OrderedKeyValueGetter) Data types can throw a warning at init time if the backend doesn't support required interfaces, or they can choose to implement multiple ways of handling data. Each Engine implementation should call storage.Register() to register its availability.

func GetEngine Uses

func GetEngine(name string) Engine

GetEngine returns an Engine of the given name.

type Filter Uses

type Filter interface {
    // Check filters type-specific key-value pairs.
    Check(tkv *TKeyValue) (skip bool, err error)
}

Filter can filter key-value pairs based on some criteria.

type FilterSpec Uses

type FilterSpec string

FilterSpec is a string specification of type-specific filters to apply to key-value pairs before sending them to a remote DVID. For example, a FilterSpec could look like:

roi:seven_column,3f8a/tile:xy,xz

The above specifies two filters joined by a forward slash. The first is an "roi" filters that lists a ROI data instance name ("seven_column") and its version as a partial, unique UUID. The second is a "tile" filter that specifies two types of tile plane: xy and xz.

func (FilterSpec) GetFilterSpec Uses

func (f FilterSpec) GetFilterSpec(ftype string) (value string, found bool)

GetFilterSpec parses a FilterSpec and returns the filter spec of given type. If no filter spec of ftype is available, the second argument is false.

type Filterer Uses

type Filterer interface {
    NewFilter(FilterSpec) (Filter, error)
}

Filterer is an interface that can provide a send filter given a spec. Datatypes can fulfill this interface if they want to filter key-values sent to peer DVID servers. An example is the use of ROIs to filter kv pairs based on spatial constraints.

See datatype/imageblk/distributed.go

type GraphDB Uses

type GraphDB interface {
    GraphSetter
    GraphGetter
    Close()
}

GraphDB defines the entire interface that a graph database should support

func GraphStore Uses

func GraphStore() (GraphDB, error)

func NewGraphStore Uses

func NewGraphStore(kvdb OrderedKeyValueDB) (GraphDB, error)

NewGraphStore returns a graph backend that uses the provided keyvalue datastore. If a first-class graph store is used in the future, this function must be changed.

type GraphGetter Uses

type GraphGetter interface {
    // GetVertices retrieves a list of all vertices in the graph
    GetVertices(ctx Context) ([]dvid.GraphVertex, error)
    // GetEdges retrieves a list of all edges in the graph
    GetEdges(ctx Context) ([]dvid.GraphEdge, error)
    // GetVertex retrieves a vertex given a vertex id
    GetVertex(ctx Context, id dvid.VertexID) (dvid.GraphVertex, error)
    // GetVertex retrieves an edges between two vertex IDs
    GetEdge(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID) (dvid.GraphEdge, error)
    // GetVertexProperty retrieves a property as a byte array given a vertex id
    GetVertexProperty(ctx Context, id dvid.VertexID, key string) ([]byte, error)
    // GetEdgeProperty retrieves a property as a byte array given an edge defined by id1 and id2
    GetEdgeProperty(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, key string) ([]byte, error)
}

GraphGetter defines operations that retrieve information from a graph

type GraphKeyValueDB Uses

type GraphKeyValueDB struct {
    OrderedKeyValueDB
    // contains filtered or unexported fields
}

GraphKeyValueDB defines a type that embeds a KeyValueDB using that engine to store all graph objects

func (*GraphKeyValueDB) AddEdge Uses

func (db *GraphKeyValueDB) AddEdge(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, weight float64) error

AddEdge reads both vertices, modifies the vertex edge lists, and then creates an edge (2 read ops, 3 write ops)

func (*GraphKeyValueDB) AddVertex Uses

func (db *GraphKeyValueDB) AddVertex(ctx Context, id dvid.VertexID, weight float64) error

AddVertex requires one key-value put

func (*GraphKeyValueDB) Close Uses

func (db *GraphKeyValueDB) Close()

Close does nothing, explicitly close the key value DB instead

func (*GraphKeyValueDB) CreateGraph Uses

func (db *GraphKeyValueDB) CreateGraph(ctx Context) error

CreateGraph does nothing as the graph keyspace uniquely defines a graph when using a single key-value datastore

func (*GraphKeyValueDB) GetEdge Uses

func (db *GraphKeyValueDB) GetEdge(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID) (dvid.GraphEdge, error)

GetEdge performs 1 read

func (*GraphKeyValueDB) GetEdgeProperty Uses

func (db *GraphKeyValueDB) GetEdgeProperty(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, key string) ([]byte, error)

GetEdgeProperty performs 1 read (property name with vertex ids encoded in key)

func (*GraphKeyValueDB) GetEdges Uses

func (db *GraphKeyValueDB) GetEdges(ctx Context) ([]dvid.GraphEdge, error)

GetEdges uses a range query to get all edges (#reads = #edges)

func (*GraphKeyValueDB) GetVertex Uses

func (db *GraphKeyValueDB) GetVertex(ctx Context, id dvid.VertexID) (dvid.GraphVertex, error)

GetVertex performs 1 read

func (*GraphKeyValueDB) GetVertexProperty Uses

func (db *GraphKeyValueDB) GetVertexProperty(ctx Context, id dvid.VertexID, key string) ([]byte, error)

GetVertexProperty performs 1 read (property name with vertex id encoded in key)

func (*GraphKeyValueDB) GetVertices Uses

func (db *GraphKeyValueDB) GetVertices(ctx Context) ([]dvid.GraphVertex, error)

GetVertices uses a range query to get all vertices (#reads = #vertices)

func (*GraphKeyValueDB) RemoveEdge Uses

func (db *GraphKeyValueDB) RemoveEdge(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID) error

RemoveEdge removes the edge and all of its properties and modifies affected vertices (2 read, 3 + num properties writes)

func (*GraphKeyValueDB) RemoveEdgeProperty Uses

func (db *GraphKeyValueDB) RemoveEdgeProperty(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, key string) error

RemoveEdgeProperty retrieves edge and batch removes property and modifies edge (1 read, 2 writes)

func (*GraphKeyValueDB) RemoveGraph Uses

func (db *GraphKeyValueDB) RemoveGraph(ctx Context) error

RemoveGraph retrieves all keys with the graph prefix and deletes them

func (*GraphKeyValueDB) RemoveVertex Uses

func (db *GraphKeyValueDB) RemoveVertex(ctx Context, id dvid.VertexID) error

RemoveVertex removes the vertex and all of its edges and properties (1 read, 1 + num edges + num properties writes)

func (*GraphKeyValueDB) RemoveVertexProperty Uses

func (db *GraphKeyValueDB) RemoveVertexProperty(ctx Context, id dvid.VertexID, key string) error

RemoveVertexProperty retrieves vertex and batch removes property and modifies vertex (1 read, 2 writes)

func (*GraphKeyValueDB) SetEdgeProperty Uses

func (db *GraphKeyValueDB) SetEdgeProperty(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, key string, value []byte) error

SetEdgeProperty modifies the edge and adds a property vertex key (1 read, 2 writes)

func (*GraphKeyValueDB) SetEdgeWeight Uses

func (db *GraphKeyValueDB) SetEdgeWeight(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, weight float64) error

SetEdgeWeight requires 1 read, 1 write

func (*GraphKeyValueDB) SetVertexProperty Uses

func (db *GraphKeyValueDB) SetVertexProperty(ctx Context, id dvid.VertexID, key string, value []byte) error

SetVertexProperty modifies the vertex and adds a property vertex key (1 read, 2 writes)

func (*GraphKeyValueDB) SetVertexWeight Uses

func (db *GraphKeyValueDB) SetVertexWeight(ctx Context, id dvid.VertexID, weight float64) error

SetVertexWeight requires 1 read, 1 write

func (*GraphKeyValueDB) String Uses

func (db *GraphKeyValueDB) String() string

GetName returns the name of theengien

type GraphSetter Uses

type GraphSetter interface {
    // CreateGraph creates a graph with the given context.
    CreateGraph(ctx Context) error

    // AddVertex inserts an id of a given weight into the graph
    AddVertex(ctx Context, id dvid.VertexID, weight float64) error

    // AddEdge adds an edge between vertex id1 and id2 with the provided weight
    AddEdge(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, weight float64) error

    // SetVertexWeight modifies the weight of vertex id
    SetVertexWeight(ctx Context, id dvid.VertexID, weight float64) error

    // SetEdgeWeight modifies the weight of the edge defined by id1 and id2
    SetEdgeWeight(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, weight float64) error

    // SetVertexProperty adds arbitrary data to a vertex using a string key
    SetVertexProperty(ctx Context, id dvid.VertexID, key string, value []byte) error
    // SetEdgeProperty adds arbitrary data to an edge using a string key
    SetEdgeProperty(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, key string, value []byte) error
    // RemoveVertex removes the vertex and its properties and edges
    RemoveVertex(ctx Context, id dvid.VertexID) error
    // RemoveEdge removes the edge defined by id1 and id2 and its properties
    RemoveEdge(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID) error
    // RemoveGraph removes the entire graph including all vertices, edges, and properties
    RemoveGraph(ctx Context) error
    // RemoveVertexProperty removes the property data for vertex id at the key
    RemoveVertexProperty(ctx Context, id dvid.VertexID, key string) error
    // RemoveEdgeProperty removes the property data for edge at the key
    RemoveEdgeProperty(ctx Context, id1 dvid.VertexID, id2 dvid.VertexID, key string) error
}

GraphSetter defines operations that modify a graph

type GroupcacheConfig Uses

type GroupcacheConfig struct {
    GB        int
    Host      string   // The http address of this DVID server's groupcache port.
    Peers     []string // The http addresses of the peer groupcache group.
    Instances []string // Data instances that use groupcache in form "<name>:<uuid>""
}

GroupcacheConfig handles settings for the groupcache library.

type GroupcacheCtx Uses

type GroupcacheCtx struct {
    Context
    KeyValueDB
}

GroupcacheCtx packages both a storage.Context and a KeyValueDB for GETs.

type GroupcacheStats Uses

type GroupcacheStats struct {
    Gets           int64 // any GET request, including from peers
    CacheHits      int64 // either cache was good
    PeerLoads      int64 // either remote load or remote cache hit (not an error)
    PeerErrors     int64
    Loads          int64 // gets - cacheHits
    LoadsDeduped   int64 // after singleflight
    LocalLoads     int64 // total good local loads
    LocalLoadErrs  int64 // total bad local loads
    ServerRequests int64 // gets that came over network from peers

    // Cache stats for items owned by the host.
    MainCache groupcache.CacheStats

    // Cache stats for replicated items from a peer.
    HotCache groupcache.CacheStats
}

GroupcacheStats returns all the stats of interest for this groupcache service.

func GetGroupcacheStats Uses

func GetGroupcacheStats() (stats GroupcacheStats, err error)

GetGroupcacheStats returns all kinds of stats about the groupcache system.

type KafkaConfig Uses

type KafkaConfig struct {
    TopicActivity string   // if supplied, will be override topic for activity log
    TopicPrefix   string   // if supplied, will be prefixed to any mutation logging
    TopicSuffixes []string // optional topic suffixes per data UUID
    Servers       []string
}

KafkaConfig describes kafka servers and an optional local file directory into which failed messages will be stored.

func (KafkaConfig) Initialize Uses

func (kc KafkaConfig) Initialize(hostID string) error

Initialize sets up default activity topic and launches goroutine for handling async kafka messages.

type Key Uses

type Key []byte

Key is the slice of bytes used to store a value in a storage engine. It internally represents a number of DVID features like a data instance ID, version, and a type-specific key component.

func ConstructBlobKey Uses

func ConstructBlobKey(k []byte) Key

ConstructBlobKey returns a blob Key, partitioned from other key spaces, for a given key.

func DataKeyRange Uses

func DataKeyRange() (minKey, maxKey Key)

DataKeyRange returns the min and max Key across all data keys.

func MaxDataKey Uses

func MaxDataKey() Key

func MergeKey Uses

func MergeKey(unvKey Key, verKey []byte) Key

func MinDataKey Uses

func MinDataKey() Key

func (Key) IsBlobKey Uses

func (k Key) IsBlobKey() bool

IsBlobKey returns true if the given key is in the metadata keyspace (instead of data or blob).

func (Key) IsDataKey Uses

func (k Key) IsDataKey() bool

IsDataKey returns true if the given key is in the data keyspace (instead of metadata or blob).

func (Key) IsMetadataKey Uses

func (k Key) IsMetadataKey() bool

IsMetadataKey returns true if the given key is in the metadata keyspace (instead of data or blob).

func (Key) IsTombstone Uses

func (k Key) IsTombstone() bool

IsTombstone returns true if the given key is a tombstone key.

type KeyChan Uses

type KeyChan chan Key

KeyChan is a channel of full (not type-specific) keys.

type KeyRange Uses

type KeyRange struct {
    Start   Key // Range includes this Key.
    OpenEnd Key // Range extend to but does not include this Key.
}

KeyRange is a range of keys that is closed at the beginning and open at the end.

type KeyValue Uses

type KeyValue struct {
    K   Key
    V   []byte
}

KeyValue stores a full storage key-value pair.

type KeyValueBatcher Uses

type KeyValueBatcher interface {
    NewBatch(ctx Context) Batch
}

KeyValueBatcher allow batching operations into an atomic update or transaction. For example: "Atomic Updates" in http://leveldb.googlecode.com/svn/trunk/doc/index.html

type KeyValueDB Uses

type KeyValueDB interface {
    dvid.Store
    KeyValueGetter
    KeyValueSetter
}

KeyValueDB provides an interface to the simplest storage API: a key-value store.

func DefaultKVDB Uses

func DefaultKVDB() (KeyValueDB, error)

type KeyValueGetter Uses

type KeyValueGetter interface {
    // Get returns a value given a key.
    Get(ctx Context, k TKey) ([]byte, error)

    // Exists returns true if a key has been set.
    Exists(ctx Context, k TKey) (bool, error)
}

type KeyValueIngestable Uses

type KeyValueIngestable interface {
    // KeyValueIngest accepts mutations without any guarantee that the ingested key value
    // will be immediately readable.  This allows immutable stores to accept data that will
    // be later processed into its final readable, immutable form.
    KeyValueIngest(Context, TKey, []byte) error
}

KeyValueIngestable implementations allow ingestion of data without necessarily allowing immediate reads of the ingested data.

type KeyValueRequester Uses

type KeyValueRequester interface {
    NewBuffer(ctx Context) RequestBuffer
}

KeyValueRequester allows operations to be queued so that they can be handled as a batch job. (See RequestBuffer for more information.)

type KeyValueSetter Uses

type KeyValueSetter interface {
    // Put writes a value with given key in a possibly versioned context.
    Put(Context, TKey, []byte) error

    // Delete deletes a key-value pair so that subsequent Get on the key returns nil.
    // For versioned data in mutable stores, Delete() will create a tombstone for the version
    // unlike RawDelete or DeleteAll.
    Delete(Context, TKey) error

    // RawPut is a low-level function that puts a key-value pair using full keys.
    // This can be used in conjunction with RawRangeQuery.  It does not automatically
    // delete any associated tombstone, unlike the Delete() function, so tombstone
    // deletion must be handled via RawDelete().
    RawPut(Key, []byte) error

    // RawDelete is a low-level function.  It deletes a key-value pair using full keys
    // without any context.  This can be used in conjunction with RawRangeQuery.
    RawDelete(Key) error
}

type KeyValueTimestampGetter Uses

type KeyValueTimestampGetter interface {
    // Get returns a value as well as last modification time
    GetWithTimestamp(ctx Context, k TKey) ([]byte, time.Time, error)
}

type LogMessage Uses

type LogMessage struct {
    EntryType uint16
    Data      []byte
}

LogMessage is a single message to a log

type LogReadable Uses

type LogReadable interface {
    ReadLogRequired() bool
    GetReadLog() ReadLog
}

type LogWritable Uses

type LogWritable interface {
    WriteLogRequired() bool
    GetWriteLog() WriteLog
}

type Logable Uses

type Logable interface {
    LogReadable
    LogWritable
}

type MetadataContext Uses

type MetadataContext struct{}

MetadataContext is an implementation of Context for MetadataContext persistence.

func NewMetadataContext Uses

func NewMetadataContext() MetadataContext

func (MetadataContext) ConstructKey Uses

func (ctx MetadataContext) ConstructKey(tk TKey) Key

func (MetadataContext) ConstructKeyVersion Uses

func (ctx MetadataContext) ConstructKeyVersion(tk TKey, version dvid.VersionID) Key

Note: needed to satisfy interface but should not be called

func (MetadataContext) KeyRange Uses

func (ctx MetadataContext) KeyRange() (min, max Key)

func (MetadataContext) Mutex Uses

func (ctx MetadataContext) Mutex() *sync.Mutex

func (MetadataContext) RepoRoot Uses

func (ctx MetadataContext) RepoRoot() (dvid.UUID, error)

func (MetadataContext) RequestID Uses

func (ctx MetadataContext) RequestID() string

func (MetadataContext) SplitKey Uses

func (ctx MetadataContext) SplitKey(tk TKey) (Key, []byte, error)

func (MetadataContext) String Uses

func (ctx MetadataContext) String() string

func (MetadataContext) VersionID Uses

func (ctx MetadataContext) VersionID() dvid.VersionID

func (MetadataContext) Versioned Uses

func (ctx MetadataContext) Versioned() bool

type Op Uses

type Op uint8

Op enumerates the types of single key-value operations that can be performed for storage engines.

const (
    GetOp Op  = iota
    PutOp
    DeleteOp
    CommitOp
)

type OrderedKeyValueDB Uses

type OrderedKeyValueDB interface {
    dvid.Store
    OrderedKeyValueGetter
    OrderedKeyValueSetter
}

OrderedKeyValueDB addes range queries and range puts to a base KeyValueDB.

func DefaultOrderedKVDB Uses

func DefaultOrderedKVDB() (OrderedKeyValueDB, error)

func MetaDataKVStore Uses

func MetaDataKVStore() (OrderedKeyValueDB, error)

type OrderedKeyValueGetter Uses

type OrderedKeyValueGetter interface {
    KeyValueGetter

    // GetRange returns a range of values spanning (kStart, kEnd) keys.
    GetRange(ctx Context, kStart, kEnd TKey) ([]*TKeyValue, error)

    // KeysInRange returns a range of type-specific key components spanning (kStart, kEnd).
    KeysInRange(ctx Context, kStart, kEnd TKey) ([]TKey, error)

    // SendKeysInRange sends a range of keys down a key channel.
    SendKeysInRange(ctx Context, kStart, kEnd TKey, ch KeyChan) error

    // ProcessRange sends a range of type key-value pairs to type-specific chunk handlers,
    // allowing chunk processing to be concurrent with key-value sequential reads.
    // Since the chunks are typically sent during sequential read iteration, the
    // receiving function can be organized as a pool of chunk handling goroutines.
    // See datatype/imageblk.ProcessChunk() for an example.  If the ChunkFunc returns
    // an error, it is expected that the ProcessRange should immediately terminate and
    // propagate the error.
    ProcessRange(ctx Context, kStart, kEnd TKey, op *ChunkOp, f ChunkFunc) error

    // RawRangeQuery sends a range of full keys.  This is to be used for low-level data
    // retrieval like DVID-to-DVID communication and should not be used by data type
    // implementations if possible because each version's key-value pairs are sent
    // without filtering by the current version and its ancestor graph.  A nil is sent
    // down the channel when the range is complete.  The query can be cancelled by sending
    // a value down the cancel channel.
    RawRangeQuery(kStart, kEnd Key, keysOnly bool, out chan *KeyValue, cancel <-chan struct{}) error
}

type OrderedKeyValueSetter Uses

type OrderedKeyValueSetter interface {
    KeyValueSetter

    // Put key-value pairs.  Note that it could be more efficient to use the Batcher
    // interface so you don't have to create and keep a slice of KeyValue.  Some
    // databases like leveldb will copy on batch put anyway.
    PutRange(Context, []TKeyValue) error

    // DeleteRange removes all key-value pairs with keys in the given range.
    // If versioned data in mutable stores, this will create tombstones in the version
    // unlike RawDelete or DeleteAll.
    DeleteRange(ctx Context, kStart, kEnd TKey) error

    // DeleteAll removes all key-value pairs for the context.
    DeleteAll(ctx Context) error
}

type PatchFunc Uses

type PatchFunc func([]byte) ([]byte, error)

PatchFunc is a function that accepts a value and patches that value in place

type ReadLog Uses

type ReadLog interface {
    dvid.Store
    ReadBinary(dataID, version dvid.UUID) ([]byte, error)
    ReadAll(dataID, version dvid.UUID) ([]LogMessage, error)
    StreamAll(dataID, version dvid.UUID, ch chan LogMessage, wg *sync.WaitGroup) error
}

type RepairableEngine Uses

type RepairableEngine interface {
    Engine
    Repair(path string) error
}

RepairableEngine is a storage engine that can be repaired.

type RequestBuffer Uses

type RequestBuffer interface {
    BufferableOps

    // ProcessList will process all gets when flush is called
    ProcessList(ctx Context, tkeys []TKey, op *ChunkOp, f ChunkFunc) error

    // PutCallback writes a value with given key in a possibly versioned context and signals the callback
    PutCallback(Context, TKey, []byte, chan error) error

    // Flush processes all the queued jobs
    Flush() error
}

RequestBuffer allows one to queue up several requests and then process them all as a batch operation (if the driver supports batch operations). There is no guarantee on the order or atomicity of these operations.

type RequestCtx Uses

type RequestCtx interface {
    // GetRequestID returns a string identifier or the empty string if none have been set.
    GetRequestID() string

    // SetRequestID sets a string identifier.
    SetRequestID(id string)
}

RequestCtx is associated with a particular request (typically a web request) and can set and retrieve information about it.

type Requirements Uses

type Requirements struct {
    BulkIniter bool
    BulkWriter bool
    Batcher    bool
    GraphDB    bool
}

Requirements lists required backend interfaces for a type.

type SizeViewer Uses

type SizeViewer interface {
    GetApproximateSizes(ranges []KeyRange) ([]uint64, error)
}

SizeViewer stores are able to return the size in bytes stored for a given range of Key.

type TKey Uses

type TKey []byte

TKey is the type-specific component of a key. Each data instance will insert key components into a class of TKey. Typically, the first byte is unique for a given class of keys within a datatype. For a given class of type-specific keys, they must either be (1) identical length or (2) if varying length, not share a common prefix. The second case can be trivially fulfilled by ending a type-specific key with a unique (not occurring within these type-specific keys) termination byte like 0x00.

func MaxTKey Uses

func MaxTKey(class TKeyClass) TKey

MaxTKey returns the lexicographically largest TKey for this class.

func MinTKey Uses

func MinTKey(class TKeyClass) TKey

MinTKey returns the lexicographically smallest TKey for this class.

func NewTKey Uses

func NewTKey(class TKeyClass, tkey []byte) TKey

func TKeyFromKey Uses

func TKeyFromKey(key Key) (TKey, error)

TKeyFromKey returns a type-specific key from a full key. An error is returned if the key is not one with a type-specific key component.

func (TKey) Class Uses

func (tk TKey) Class() (TKeyClass, error)

Class returns the TKeyClass of a TKey.

func (TKey) ClassBytes Uses

func (tk TKey) ClassBytes(class TKeyClass) ([]byte, error)

ClassBytes returns the bytes for a class of TKey, suitable for decoding by each data instance.

type TKeyClass Uses

type TKeyClass byte

TKeyClass partitions the TKey space into a maximum of 256 classes.

type TKeyValue Uses

type TKeyValue struct {
    K   TKey
    V   []byte
}

TKeyValue stores a type-specific key-value pair.

func (TKeyValue) Deserialize Uses

func (kv TKeyValue) Deserialize(uncompress bool) (TKeyValue, error)

Deserialize returns a type key-value pair where the value has been deserialized.

type TKeyValues Uses

type TKeyValues []TKeyValue

TKeyValues is a slice of type-specific key-value pairs that can be sorted.

func (TKeyValues) Len Uses

func (kv TKeyValues) Len() int

func (TKeyValues) Less Uses

func (kv TKeyValues) Less(i, j int) bool

func (TKeyValues) Swap Uses

func (kv TKeyValues) Swap(i, j int)

type TestableEngine Uses

type TestableEngine interface {
    Engine
    AddTestConfig(*Backend) (Alias, error)
    Delete(dvid.StoreConfig) error
}

TestableEngine is a storage engine that allows creation of temporary stores for testing.

type TransactionDB Uses

type TransactionDB interface {
    // LockKey uses atomic get/put to use the key as a shared lock.
    // If the lock is being used, the lock will be queried with some backoff.
    LockKey(Key) error

    // Release lock on key
    UnlockKey(Key) error

    // Patch patches the value at the given key with function f
    // The patching function should work on uninitialized data.
    Patch(Context, TKey, PatchFunc) error
}

TransactionDB allows multiple database operations to execute as an atomic transaction

type VersionedCtx Uses

type VersionedCtx interface {
    Context

    // UnversionedKeyPrefix returns a byte slice that is the common prefix
    // for all versions of this particular TKey.
    UnversionedKeyPrefix(TKey) Key

    // UnversionedKey returns a unversioned Key and the version id
    // as separate components.  This can be useful for storage systems
    // like column stores where the row key is the unversioned Key and
    // the column qualifier is the version id.
    UnversionedKey(TKey) (Key, dvid.VersionID, error)

    // VersionFromKey returns a version ID from a full key.  Any VersionedContext is sufficient as receiver
    VersionFromKey(Key) (dvid.VersionID, error)

    // TombstoneKey takes a type-specific key component and returns a key that
    // signals a deletion of any ancestor values.  The returned key must have
    // as its last byte storage.MarkTombstone.
    TombstoneKey(TKey) Key

    // TombstoneKeyVersion implement TombstoneKey but for the specified version
    TombstoneKeyVersion(TKey, dvid.VersionID) Key

    // Head checks whether this the open head of the master branch
    Head() bool

    // MasterVersion checks whether current version is on master branch
    MasterVersion(dvid.VersionID) bool

    // NumVersions returns the number of version in the current DAG
    NumVersions() int32
    // Returns lower bound key for versions.
    MinVersionKey(TKey) (Key, error)

    // Returns upper bound key for versions.
    MaxVersionKey(TKey) (Key, error)

    // GetBestKeyVersion returns the key that most closely matches this context's version.
    // If no suitable key or a tombstone is encountered closed to the version, nil is returned.
    GetBestKeyVersion([]Key) (Key, error)

    // VersionedKeyValue returns the key-value pair corresponding to this key's version
    // given a list of key-value pairs across many versions.  If no suitable key-value
    // pair is found, nil is returned.
    VersionedKeyValue([]*KeyValue) (*KeyValue, error)

    // Data returns the data associated with this context
    Data() dvid.Data
}

VersionedCtx extends a Context with the minimal functions necessary to handle versioning in storage engines.

type WriteLog Uses

type WriteLog interface {
    dvid.Store
    Append(dataID, version dvid.UUID, msg LogMessage) error
    CloseLog(dataID, version dvid.UUID) error
    TopicAppend(topic string, msg LogMessage) error
    TopicClose(topic string) error
}

WriteLog is an append-only log of messages specific to a topic or a data instance and UUID.

Directories

PathSynopsis
filelog
swiftPackage swift adds Openstack Swift support to DVID.

Package storage imports 16 packages (graph) and is imported by 55 packages. Updated 2019-07-19. Refresh now. Tools for package owners.