dvid: github.com/janelia-flyem/dvid/datastore Index | Files

package datastore

import "github.com/janelia-flyem/dvid/datastore"

Package datastore provides versioning and persisting supported data types using one of the supported storage engines.


Package Files

copy_local.go datainstance.go datastore.go datatype.go doc.go pubsub.go push_local.go repo.go repo_local.go swift.go test_support.go


const (
    StartDataMsg = "datastore.startData"
    PutKVMsg     = "datastore.putKV"
const DataShutdownTime = 20

DataShutdownTime is the maximum number of seconds a data instance can delay when terminating goroutines during Shutdown.

const InitialMutationID = 1000000000 // billion

InitialMutationID is the initial mutation ID for all new repositories. It was chosen to be a large number to distinguish between mutations with repo-wide persistent IDs and the legacy data instance-specific mutation IDs that would reset on server restart.

const PropertyTKeyClass = 1

PropertyTKeyClass is the TKeyClass reserved for data instance properties, and should be reserved in each data type implementation.

const RepoFormatVersion = 1

RepoFormatVersion is the current repo metadata format version

const (
    ServerLockKey storage.TKeyClass // name of key for locking metadata globally


Key space handling for metadata

const StrideMutationID = 100

StrideMutationID is a way to minimize cost of persisting mutation IDs for a repo. We persist the current mutation ID plus the stride, so that we don't have to update the persisted mutation ID frequently. The downside is that there may be jumps in mutation IDs on restarts, since the persisted ID is loaded on next restart.

const (
    Version = "0.10.0"


var (
    ErrManagerNotInitialized = errors.New("datastore repo manager not initialized")
    ErrBadMergeType          = errors.New("bad merge type")

    ErrInvalidUUID         = errors.New("UUID is not present in datastore")
    ErrInvalidVersion      = errors.New("server-specific version id is invalid")
    ErrInvalidRepoID       = errors.New("server-specific repo id is invalid")
    ErrExistingUUID        = errors.New("UUID already exists in datastore")
    ErrExistingDataName    = errors.New("data name already exists")
    ErrInvalidDataName     = errors.New("invalid data instance name")
    ErrInvalidDataInstance = errors.New("invalid data instance id")
    ErrInvalidDataUUID     = errors.New("invalid data instance UUID")

    ErrInvalidStore = errors.New("invalid data store")

    ErrModifyLockedNode   = errors.New("can't modify locked node")
    ErrBranchUnlockedNode = errors.New("can't branch an unlocked node")
    ErrBranchUnique       = errors.New("branch already exists with given name")
var (
    // Compiled is the set of registered datatypes compiled into DVID and
    // held as a global variable initialized at runtime.
    Compiled map[dvid.URLString]TypeService

func AddToNodeLog Uses

func AddToNodeLog(uuid dvid.UUID, msgs []string) error

func AddToRepoLog Uses

func AddToRepoLog(uuid dvid.UUID, msgs []string) error

func BlockOnUpdating Uses

func BlockOnUpdating(uuid dvid.UUID, name dvid.InstanceName) error

BlockOnUpdating blocks until the given data is not updating from syncs or has events waiting in sync channels. Primarily used during testing.

func CloseReopenTest Uses

func CloseReopenTest(datamap ...DataStorageMap)

CloseReopenTest forces close and then reopening of the datastore, useful for testing persistence. We only allow close/reopen when all tests not avaiting close/reopen are finished.

func CloseTest Uses

func CloseTest()

func Commit Uses

func Commit(uuid dvid.UUID, note string, log []string) error

func CompiledChart Uses

func CompiledChart() string

CompiledChart returns a chart (names/urls) of datatypes compiled into this DVID.

func CompiledTypes Uses

func CompiledTypes() map[dvid.TypeString]TypeService

CompiledTypes returns a map of datatypes compiled into this DVID.

func CompiledURLs Uses

func CompiledURLs() string

CompiledURLs returns a list of datatype urls supported by this DVID.

func CopyInstance Uses

func CopyInstance(uuid dvid.UUID, source, target dvid.InstanceName, c dvid.Config) error

CopyInstance copies a data instance locally, perhaps to a different storage engine if the new instance uses a different backend per a data instance-specific configuration. (See sample config.example.toml file in root dvid source directory.)

func DeleteConflicts Uses

func DeleteConflicts(uuid dvid.UUID, data DataService, oldParents, newParents []dvid.UUID) error

DeleteConflicts removes all conflicted kv pairs for the given data instance using the priority established by parents. As a side effect, newParents are modified by new children of parents.

func DeleteDataByName Uses

func DeleteDataByName(uuid dvid.UUID, name dvid.InstanceName, passcode string) error

DeleteDataByName returns a data service given an instance name and UUID.

func DeleteDataByVersion Uses

func DeleteDataByVersion(v dvid.VersionID, name dvid.InstanceName, passcode string) error

DeleteDataByVersion returns a data service given an instance name and UUID.

func DeleteRepo Uses

func DeleteRepo(uuid dvid.UUID, passcode string) error

DeleteRepo deletes a Repo holding a node with UUID.

func FlattenMetadata Uses

func FlattenMetadata(uuid dvid.UUID, configFName string) error

FlattenMetadata stores the metadata of a reduced set of nodes into the destination store.

func GetAncestry Uses

func GetAncestry(v dvid.VersionID) ([]dvid.VersionID, error)

GetAncestry returns a list of ancestor versions from the given version to the root.

func GetBlobStore Uses

func GetBlobStore(d dvid.Data) (store storage.BlobStore, err error)

GetBlobStore returns a blob store or nil if not available. TODO -- Add configurable option to use a file system.

func GetBranchVersions Uses

func GetBranchVersions(uuid dvid.UUID, name string) ([]dvid.UUID, error)

GetBranchVersions returns a slice of UUIDs for the given branch from the HEAD (leaf) to the root.

func GetBranchVersionsJSON Uses

func GetBranchVersionsJSON(uuid dvid.UUID, name string) (string, error)

func GetChildrenByVersion Uses

func GetChildrenByVersion(v dvid.VersionID) ([]dvid.VersionID, error)

GetChildren returns the child nodes of the given version id.

func GetGraphDB Uses

func GetGraphDB(d dvid.Data) (db storage.GraphDB, err error)

GetGraphDB returns a graph store assigned to this data instance. If the store is nil or not available, an error is returned.

func GetKeyValueBatcher Uses

func GetKeyValueBatcher(d dvid.Data) (db storage.KeyValueBatcher, err error)

GetKeyValueBatcher returns a batch-capable kv data store assigned to this data instance. If the store is nil or not available, an error is returned.

func GetKeyValueDB Uses

func GetKeyValueDB(d dvid.Data) (db storage.KeyValueDB, err error)

GetKeyValueDB returns a kv data store assigned to this data instance. If the store is nil or not available, an error is returned.

func GetNodeLog Uses

func GetNodeLog(uuid dvid.UUID) ([]string, error)

func GetNodeNote Uses

func GetNodeNote(uuid dvid.UUID) (string, error)

func GetOrderedKeyValueDB Uses

func GetOrderedKeyValueDB(d dvid.Data) (db storage.OrderedKeyValueDB, err error)

GetOrderedKeyValueDB returns the ordered kv data store assigned to this data instance. If the store is nil or not available, an error is returned.

func GetParentsByVersion Uses

func GetParentsByVersion(v dvid.VersionID) ([]dvid.VersionID, error)

GetParents returns the parent nodes of the given version id.

func GetRepoAlias Uses

func GetRepoAlias(uuid dvid.UUID) (string, error)

func GetRepoDescription Uses

func GetRepoDescription(uuid dvid.UUID) (string, error)

func GetRepoJSON Uses

func GetRepoJSON(uuid dvid.UUID) (string, error)

func GetRepoLog Uses

func GetRepoLog(uuid dvid.UUID) ([]string, error)

func GetRepoRoot Uses

func GetRepoRoot(uuid dvid.UUID) (dvid.UUID, error)

func GetRepoRootVersion Uses

func GetRepoRootVersion(v dvid.VersionID) (dvid.VersionID, error)

func GetStorageDetails Uses

func GetStorageDetails() (map[string]StorageStats, error)

GetStorageDetails scans all key-value stores and returns detailed stats per instances.

func GetStorageSummary Uses

func GetStorageSummary() (string, error)

GetStorageSummary returns JSON for all the data instances in the stores.

func Initialize Uses

func Initialize(initMetadata bool, iconfig Config) error

Initialize creates a repositories manager that is handled through package functions.

func LimitVersions Uses

func LimitVersions(uui dvid.UUID, configFName string) error

LimitVersions removes versions from the metadata that are not present in a configuration file.

func LockedUUID Uses

func LockedUUID(uuid dvid.UUID) (bool, error)

LockedUUID returns true if a given UUID is locked.

func LockedVersion Uses

func LockedVersion(v dvid.VersionID) (bool, error)

LockedVersion returns true if a given version is locked.

func LogRepoOpToKafka Uses

func LogRepoOpToKafka(uuid dvid.UUID, b []byte) error

LogRepoOpToKafka logs a repo operation to kafka

func MarshalJSON Uses

func MarshalJSON() ([]byte, error)

MarshalJSON returns JSON of object where each repo is a property with root UUID name and value corresponding to repo info.

func MatchingUUID Uses

func MatchingUUID(uuidStr string) (dvid.UUID, dvid.VersionID, error)

MatchingUUID returns a local version ID and the full UUID from a potentially shortened UUID string. Partial matches are accepted as long as they are unique for a datastore. So if a datastore has nodes with UUID strings 3FA22..., 7CD11..., and 836EE..., we can still find a match even if given the minimum 3 letters. (We don't allow UUID strings of less than 3 letters just to prevent mistakes.) If the passed string has a colon, the string after the colon is parsed as a case-sensitive branch name of the repo with the given UUID, and the UUID returned will be the HEAD or uncommitted leaf of that branch. Example "3FA22:master" returns the leaf UUID of branch "master" for the repo containing

the 3FA22 UUID.

func Merge Uses

func Merge(parents []dvid.UUID, note string, mt MergeType) (dvid.UUID, error)

func MigrateBatch Uses

func MigrateBatch(uuid dvid.UUID, configFName string) (err error)

MigrateBatch migrates instances specified in the config file from a source to a destination store, similar to the above "migrate" command but in a batch. Before running this command, you must modify the config TOML file so the destination store is available.

The migrate config file contains JSON with the following format: {

"Versions": ["2881e9","52a13","57e8d"],
"Migrations": [
		"Name": "instance-name",
		"SrcStore": "source store",
		"DstStore": {
			"Path": "/path/to/store",
			"Engine": "basholeveldb"
		"Name": "#datatype",
		"SrcStore": "source store",
		"DstStore": {
			"Path": "/another/store",
			"Engine": "badger"
"Exclusions": ["name1", "name2"]


func MigrateInstance Uses

func MigrateInstance(uuid dvid.UUID, source dvid.InstanceName, srcStore, dstStore dvid.Store, c dvid.Config, done chan bool) error

MigrateInstance migrates a data instance locally from an old storage engine to the current configured storage.

func ModifyDataConfigByName Uses

func ModifyDataConfigByName(uuid dvid.UUID, name dvid.InstanceName, c dvid.Config) error

func NewRepo Uses

func NewRepo(alias, description string, assign *dvid.UUID, passcode string) (dvid.UUID, error)

NewRepo creates a new Repo and returns its UUID, either an assigned UUID if provided or creating a new UUID.

func NewTestRepo Uses

func NewTestRepo() (dvid.UUID, dvid.VersionID)

NewTestRepo returns a new datastore.Repo suitable for testing.

func NewUUID Uses

func NewUUID(assign *dvid.UUID) (dvid.UUID, dvid.VersionID, error)

func NewVersion Uses

func NewVersion(parent dvid.UUID, note string, branchname string, assign *dvid.UUID) (dvid.UUID, error)

NewVersion creates a new version as a child of the given parent. If the assign parameter is not nil, the new node is given the UUID.

func NotifySubscribers Uses

func NotifySubscribers(e SyncEvent, m SyncMessage) error

NotifySubscribers sends a message to any data instances subscribed to the event.

func OpenTest Uses

func OpenTest(datamap ...DataStorageMap)

func PushData Uses

func PushData(d dvid.Data, p *PushSession) error

PushData transfers all key-value pairs pertinent to the given data instance. Each datatype can implement filters that can restrict the transmitted key-value pairs based on the given FilterSpec. Note that because of the generality of this function, a particular datatype implementation could be much more efficient when implementing filtering. For example, the imageblk datatype could scan its key-values using the ROI to generate keys (since imageblk keys will likely be a vast superset of ROI spans), while this generic routine will scan every key-value pair for a data instance and query the ROI to see if this key is ok to send.

func PushRepo Uses

func PushRepo(uuid dvid.UUID, target string, config dvid.Config) error

PushRepo pushes a Repo to a remote DVID server at the target address.

func Register Uses

func Register(t TypeService)

Register registers a datatype for DVID use.

func ReloadMetadata Uses

func ReloadMetadata() error

ReloadMetadata reloads the repositories manager from an existing metadata store. This shuts off all other dvid requests.

func RenameData Uses

func RenameData(uuid dvid.UUID, oldname, newname dvid.InstanceName, passcode string) error

RenameData renames a data service given an old instance name and UUID.

func SaveDataByUUID Uses

func SaveDataByUUID(uuid dvid.UUID, data DataService) error

SaveDataByUUID persists metadata for a data instance with given uuid. TODO -- Make this more efficient by storing data metadata separately from repo.

Currently we save entire repo.

func SaveDataByVersion Uses

func SaveDataByVersion(v dvid.VersionID, data DataService) error

SaveDataByVersion persists metadata for a data instance with given version. TODO -- Make this more efficient by storing data metadata separately from repo.

Currently we save entire repo.

func SetNodeNote Uses

func SetNodeNote(uuid dvid.UUID, note string) error

func SetRepoAlias Uses

func SetRepoAlias(uuid dvid.UUID, alias string) error

func SetRepoDescription Uses

func SetRepoDescription(uuid dvid.UUID, desc string) error

func SetSyncByJSON Uses

func SetSyncByJSON(d dvid.Data, uuid dvid.UUID, replace bool, in io.ReadCloser) error

SetSyncByJSON takes a JSON object of sync names and UUID, and creates the sync graph and sets the data instance's sync. If replace is false (default), the new sync is appended to the current syncs.

func SetSyncData Uses

func SetSyncData(data dvid.Data, syncs dvid.UUIDSet, replace bool) error

SetSyncData modfies the manager sync graphs and data instance's sync list. If replace is false (default), the new sync is appended to the current syncs.

func SetTagsByJSON Uses

func SetTagsByJSON(d dvid.Data, uuid dvid.UUID, replace bool, in io.ReadCloser) error

SetTagsByJSON takes a JSON object of tags and either appends or replaces the current data's tags depending on the replace parameter.

func Shutdown Uses

func Shutdown()

Shutdown sends signal for all goroutines for data processing to be terminated.

func TransferData Uses

func TransferData(uuid dvid.UUID, srcStore, dstStore dvid.Store, configFName string) error

TransferData copies key-value pairs from one repo to store and apply filtering as specified by the JSON configuration in the file specified by configFName. An example of the transfer JSON configuration file format: {

"Versions": [
"Metadata": true,


All ancestors of desired leaf nodes should be specified because key-value pair transfer only occurs if the version in which it was saved is specified on the list. This is useful for editing a preexisting store with new versions.

If Metadata property is true, then if metadata exists in the old store, it is transferred to the new store with only the versions specified appearing in the DAG.

func Types Uses

func Types() (map[dvid.URLString]TypeService, error)

Types returns the types currently within the DVID server.

func UUIDFromVersion Uses

func UUIDFromVersion(v dvid.VersionID) (dvid.UUID, error)

func VersionFromUUID Uses

func VersionFromUUID(uuid dvid.UUID) (dvid.VersionID, error)

type BlobService Uses

type BlobService interface {
    GetBlob(ref string) (data []byte, err error)
    PutBlob(data []byte) (ref string, err error)

BlobService is an interface for storing and retrieving data based on its content.

type CommitSyncer Uses

type CommitSyncer interface {
    // SyncOnCommit is an asynchronous function that should be called when a node is committed.
    SyncOnCommit(dvid.UUID, dvid.VersionID)

CommitSyncer want to be notified when a node is committed.

type Config Uses

type Config struct {
    InstanceGen   string
    InstanceStart dvid.InstanceID
    MutationStart uint64
    ReadOnly      bool

Config specifies new instance and mutation ID generation

type Data Uses

type Data struct {
    // contains filtered or unexported fields

Data is the base struct of repo-specific data instances. It should be embedded in a datatype's DataService implementation and handle datastore-wide key partitioning.

func NewDataService Uses

func NewDataService(t TypeService, rootUUID dvid.UUID, id dvid.InstanceID, name dvid.InstanceName, c dvid.Config) (*Data, error)

NewDataService returns a new Data instance that fulfills the DataService interface. The UUID passed in corresponds to the root UUID of the DAG subgraph that should hold the data. This returned Data struct is usually embedded by datatype-specific data instances. By default, LZ4 and the default checksum is used.

func (*Data) Checksum Uses

func (d *Data) Checksum() dvid.Checksum

func (*Data) Compression Uses

func (d *Data) Compression() dvid.Compression

func (*Data) DAGRootUUID Uses

func (d *Data) DAGRootUUID() (dvid.UUID, error)

func (*Data) DataName Uses

func (d *Data) DataName() dvid.InstanceName

func (*Data) DataUUID Uses

func (d *Data) DataUUID() dvid.UUID

func (*Data) DescribeTKeyClass Uses

func (d *Data) DescribeTKeyClass(tkc storage.TKeyClass) string

DescribeTKeyClass returns a string explanation of what a particular TKeyClass is used for. This will be overriden in data types, but if not, this provides a fallback for all data types.

func (*Data) Equals Uses

func (d *Data) Equals(d2 *Data) bool

Equals returns true if the two data instances are identical, not just referring to the same data instance (e.g., different names but same data UUID, etc).

func (*Data) GetBlob Uses

func (d *Data) GetBlob(ref string) (data []byte, err error)

GetBlob retrieves data given a reference that was received during PutBlob.

func (*Data) GetReadLog Uses

func (d *Data) GetReadLog() storage.ReadLog

func (*Data) GetType Uses

func (d *Data) GetType() TypeService

func (*Data) GetWriteLog Uses

func (d *Data) GetWriteLog() storage.WriteLog

func (*Data) GobDecode Uses

func (d *Data) GobDecode(b []byte) error

func (*Data) GobEncode Uses

func (d *Data) GobEncode() ([]byte, error)

func (*Data) InstanceID Uses

func (d *Data) InstanceID() dvid.InstanceID

func (*Data) IsDeleted Uses

func (d *Data) IsDeleted() bool

IsDeleted returns true if data has been deleted or is deleting.

func (*Data) IsMutationRequest Uses

func (d *Data) IsMutationRequest(action, endpoint string) bool

IsMutationRequest is the default definition of mutation requests. All POST, PUT, and DELETE actions return true, while others return false.

func (*Data) KVStore Uses

func (d *Data) KVStore() (dvid.Store, error)

func (*Data) MarshalJSON Uses

func (d *Data) MarshalJSON() ([]byte, error)

func (*Data) MarshalJSONTags Uses

func (d *Data) MarshalJSONTags() ([]byte, error)

func (*Data) ModifyConfig Uses

func (d *Data) ModifyConfig(config dvid.Config) error

func (*Data) MutAdd Uses

func (d *Data) MutAdd(mutID uint64) (newOp bool)

MutAdd adds a wait to this operation. Returns true if this is a new operation for this Data

func (*Data) MutDelete Uses

func (d *Data) MutDelete(mutID uint64)

MutDelete removes a wait for this operation. Should only be done after all MutWait() have completed.

func (*Data) MutDone Uses

func (d *Data) MutDone(mutID uint64)

MutDone marks the end of operations corresponding to MutAdd.

func (*Data) MutWait Uses

func (d *Data) MutWait(mutID uint64)

MutWait blocks until all operations with the given ID are completed.

func (*Data) NewMutationID Uses

func (d *Data) NewMutationID() uint64

func (*Data) PersistMetadata Uses

func (d *Data) PersistMetadata() error

func (*Data) ProduceKafkaMsg Uses

func (d *Data) ProduceKafkaMsg(b []byte) error

func (*Data) PushData Uses

func (d *Data) PushData(p *PushSession) error

PushData is the base implementation of pushing data instance key-value pairs to a remote DVID without any datatype-specific filtering of data.

func (*Data) PutBlob Uses

func (d *Data) PutBlob(b []byte) (ref string, err error)

PutBlob stores data and returns a reference to that data.

func (*Data) RootUUID Uses

func (d *Data) RootUUID() dvid.UUID

func (*Data) RootVersionID Uses

func (d *Data) RootVersionID() (dvid.VersionID, error)

func (*Data) SetDataUUID Uses

func (d *Data) SetDataUUID(uuid dvid.UUID)

func (*Data) SetDeleted Uses

func (d *Data) SetDeleted(deleted bool)

SetDeleted is used to indicate whether data has been deleted or is deleting.

func (*Data) SetInstanceID Uses

func (d *Data) SetInstanceID(id dvid.InstanceID)

func (*Data) SetKVStore Uses

func (d *Data) SetKVStore(kvStore dvid.Store)

func (*Data) SetLogStore Uses

func (d *Data) SetLogStore(logStore dvid.Store)

func (*Data) SetName Uses

func (d *Data) SetName(name dvid.InstanceName)

func (*Data) SetRootUUID Uses

func (d *Data) SetRootUUID(uuid dvid.UUID)

func (*Data) SetSync Uses

func (d *Data) SetSync(syncs dvid.UUIDSet)

SetSync sets the list of synced UUIDs for this data instance. It does not modify the sync graph.

func (*Data) SetTags Uses

func (d *Data) SetTags(tags map[string]string)

func (*Data) SyncPending Uses

func (d *Data) SyncPending() bool

SyncPending returns true if there are outstanding sync events in this instance's subscription.

func (*Data) SyncedData Uses

func (d *Data) SyncedData() dvid.UUIDSet

SyncedData returns a set of data UUIDs to which it is synced.

func (*Data) SyncedNames Uses

func (d *Data) SyncedNames() []dvid.InstanceName

SyncedNames returns a set of data instance names to which it is synced. Legacy and will be removed after metadata refactor.

func (*Data) Tags Uses

func (d *Data) Tags() map[string]string

func (*Data) TypeName Uses

func (d *Data) TypeName() dvid.TypeString

func (*Data) TypeURL Uses

func (d *Data) TypeURL() dvid.URLString

func (*Data) TypeVersion Uses

func (d *Data) TypeVersion() string

func (*Data) Versioned Uses

func (d *Data) Versioned() bool

type DataAvail Uses

type DataAvail uint8

DataAvail gives the availability of data within a node or whether parent nodes must be traversed to check for key-value pairs.

const (
    // DataDelta = For any query, we must also traverse this node's ancestors in the DAG
    // up to any DataComplete ancestor.  Used if a node is marked as archived.
    DataDelta DataAvail = iota

    // DataComplete = All key-value pairs are available within this node.

    // DataRoot = Queries are redirected to Root since this is unversioned.

    // DataDeleted = key-value pairs have been explicitly deleted at this node and is no longer available.

func (DataAvail) String Uses

func (avail DataAvail) String() string

type DataInitializer Uses

type DataInitializer interface {
    InitDataHandlers() error

DataInitializer is a data instance that needs to be initialized, e.g., start long-lived goroutines that handle data syncs, etc. Initialization should only constitute supporting data and goroutines and not change the data itself like InstanceMutator, so no saveNeeded flag needs to be returned.

type DataService Uses

type DataService interface {

    Help() string
    GetType() TypeService

    // ModifyConfig modifies a configuration in a type-specific way.
    ModifyConfig(dvid.Config) error

    // IsMutationRequest returns true if the given HTTP method on the endpoint results in mutations.
    // This allows datatypes to define actions on an endpoint as immutable even if they use POST.
    // The endpoint is the keyword immediately following a data instance name in the URL:
    //    /api/node/483f/grayscale/raw/xy/...
    // In the above URL, "raw" is the endpoint.
    IsMutationRequest(action, endpoint string) bool

    // DoRPC handles command line and RPC commands specific to a data type
    DoRPC(Request, *Response) error

    // ServeHTTP handles HTTP requests in the context of a particular version and
    // returns activity information that can be logged for monitoring.
    ServeHTTP(dvid.UUID, *VersionedCtx, http.ResponseWriter, *http.Request) (activity map[string]interface{})

    // DescribeTKeyClass explains in a string what a particular TKeyClass
    // is used for.  For example, one class of TKey for the label data types is the block-indexed
    // voxel values.
    DescribeTKeyClass(storage.TKeyClass) string

    // PushData handles DVID-to-DVID transmission of key-value pairs, optionally
    // delimited by type-specific filter specifications (e.g., "roi:roiname,uuid")
    // and a set of versions.  DataService implementations can automatically embed
    // this via datastore.Data or can add filters by providing their own PushData(),
    // Filterer, and Filter implementations.  (datatype/imageblk/distributed.go)
    PushData(*PushSession) error

DataService is an interface for operations on an instance of a supported datatype.

func GetDataByDataUUID Uses

func GetDataByDataUUID(dataUUID dvid.UUID) (DataService, error)

GetDataByDataUUID returns a data service given a data UUID.

func GetDataByUUIDName Uses

func GetDataByUUIDName(uuid dvid.UUID, name dvid.InstanceName) (DataService, error)

GetDataByUUIDName returns a data service given an instance name and UUID.

func GetDataByVersionName Uses

func GetDataByVersionName(v dvid.VersionID, name dvid.InstanceName) (DataService, error)

GetDataByVersionName returns a data service given an instance name and version.

func NewData Uses

func NewData(uuid dvid.UUID, t TypeService, name dvid.InstanceName, c dvid.Config) (DataService, error)

NewData adds a new, named instance of a datatype to repo. Settings can be passed via the 'config' argument. For example, config["versioned"] with a bool value will specify whether the data is versioned.

type DataStorageMap Uses

type DataStorageMap struct {
    KVStores  storage.DataMap
    LogStores storage.DataMap

DataStorageMap describes mappings from various instance and data type specifications to KV and Log stores.

type DataTxInit Uses

type DataTxInit struct {
    Session    rpc.SessionID
    DataName   dvid.InstanceName
    TypeName   dvid.TypeString
    InstanceID dvid.InstanceID
    Tags       map[string]string

type Initializer Uses

type Initializer interface {

Initializer is a data instance that can be initialized. Typically this sets up any goroutines necessary after server configuration, etc.

type InstanceMutator Uses

type InstanceMutator interface {
    // Loads all mutable properties and applies any necessary migration to
    // transform the internal data from the stored to expected version.
    LoadMutable(root dvid.VersionID, storedVersion, expectedVersion uint64) (saveNeeded bool, err error)

InstanceMutator provides a hook for data instances to load mutable data on startup. It is assumed that the data instances store data whenever its data mutates, e.g., extents for labelblk or max label for labelvol.

type InstanceStats Uses

type InstanceStats map[string]KeyStats // index is description of tkey class

type KVMessage Uses

type KVMessage struct {
    Session   rpc.SessionID
    KV        storage.KeyValue
    Terminate bool // true if this message is the last txn for this data instance and KV is invalid.

KVMessage packages a key-value pair for transmission to a remote DVID as well as control of the receiving FSM.

type KeyStats Uses

type KeyStats struct {
    LeafKV, IntKV       uint64 // # of kv pairs in leaf and interior nodes, respectively
    LeafBytes, IntBytes uint64 // # of bytes in leaf and interior nodes, respectively

type MergeType Uses

type MergeType uint8

MergeType describes the expectation of processing for the merge, e.g., is it expected to be free of conflicts at the key-value level, require automated type-specific key-value merging, or require external guidance.

const (
    // MergeConflictFree assumes that changes in nodes-to-be-merged are disjoint
    // at the key-value level.
    MergeConflictFree MergeType = iota

    // MergeTypeSpecificAuto requires datatype-specific code for merging at each
    // key-value pair.

    // MergeExternalData requires external data to reconcile merging of nodes.

type MutationDumper Uses

type MutationDumper interface {
    DumpMutations(startUUID, endUUID dvid.UUID, filename string) (comment string, err error)

MutationDumper is a dataservice that suppports the flatten-mutations command via a DumpMutations() function.

type MutationMutexer Uses

type MutationMutexer interface {
    MutAdd(mutID uint64) (newOp bool)
    MutDone(mutID uint64)
    MutWait(mutID uint64)
    MutDelete(mutID uint64)

MutationMutexer is an interface for mutexes on particular mutation IDs.

type PropertyCopier Uses

type PropertyCopier interface {
    CopyPropertiesFrom(DataService, storage.FilterSpec) error

PropertyCopier are types that can copy data instance properties from another (typically identically typed) data instance with an optional filter. This is used to create copies of data instances locally or when pushing to a remote DVID.

type PushSession Uses

type PushSession struct {
    Filter   storage.FilterSpec
    Versions map[dvid.VersionID]struct{}
    // contains filtered or unexported fields

PushSession encapsulates parameters necessary for DVID-to-DVID push/pull processing.

func (*PushSession) EndInstancePush Uses

func (p *PushSession) EndInstancePush() error

EndInstancePush terminates a data instance push.

func (*PushSession) SendKV Uses

func (p *PushSession) SendKV(kv *storage.KeyValue) error

SendKV sends a key-value pair. The key-values may be buffered before sending for efficiency of transmission.

func (*PushSession) StartInstancePush Uses

func (p *PushSession) StartInstancePush(d dvid.Data) error

StartInstancePush initiates a data instance push. After some number of Send calls, the EndInstancePush must be called.

type Request Uses

type Request struct {
    Input []byte

Request supports RPC requests to the DVID server.

type Response Uses

type Response struct {
    Output []byte

Response supports RPC responses from DVID.

func (*Response) Write Uses

func (r *Response) Write(w io.Writer) error

Writes a RPC response to a writer.

type Shutdowner Uses

type Shutdowner interface {
    Shutdown(wg *sync.WaitGroup) // Expects wg.Done() to be called in Shutdown function.

Shutdowner is a data instance that has a function to call during shutdown. Typically, this exits goroutines used for background data processing.

type StorageStats Uses

type StorageStats map[string]InstanceStats

func (StorageStats) String Uses

func (stats StorageStats) String() string

type SyncEvent Uses

type SyncEvent struct {
    Data  dvid.UUID
    Event string

SyncEvent identifies an event in which a data instance has modified its data

type SyncMessage Uses

type SyncMessage struct {
    Event   string
    Version dvid.VersionID
    Delta   interface{}

SyncMessage describes changes to a data instance for a given version.

type SyncSub Uses

type SyncSub struct {
    Event  SyncEvent
    Notify dvid.UUID // the data UUID of data instance to notify
    Ch     chan SyncMessage

SyncSub is a subscription request from an instance to be notified via a channel when a given data instance has a given event.

type SyncSubs Uses

type SyncSubs []SyncSub

SyncSubs is a slice of sync subscriptions.

func (SyncSubs) Add Uses

func (subs SyncSubs) Add(added SyncSub) SyncSubs

Add returns a SyncSubs with the added SyncSub, making sure that only one subscription exists for any (Event, Notify) tuple. If a previous (Event, Notify) exists, it is replaced by the passed SyncSub.

type Syncer Uses

type Syncer interface {
    // GetSyncSubs returns the subscriptions that need to be created to keep this data
    // synced and may launch goroutines that will consume inbound channels of changes
    // from associated data.
    GetSyncSubs(dvid.Data) (SyncSubs, error)

    // SyncedNames returns the set of data instance UUIDs to which the data is synced.
    SyncedNames() []dvid.InstanceName

    // SyncedNames returns the set of data instance UUIDs to which the data is synced.
    SyncedData() dvid.UUIDSet

    // SyncPending returns true if there are outstanding sync events in this instance's subscription.
    SyncPending() bool

Syncer types are typically DataService that know how to sync with other data.

type Type Uses

type Type struct {
    // Data type name and may not be unique.
    Name dvid.TypeString

    // The unique package name that fulfills the DVID Data interface
    URL dvid.URLString

    // The version identifier of this datatype code
    Version string

    // A list of interface requirements for the backend datastore
    Requirements *storage.Requirements

Type identifies the datatype underlying a DataService.

func (*Type) Do Uses

func (t *Type) Do(cmd Request, reply *Response) error

Do error message. Do should be overridden

func (*Type) GetStorageRequirements Uses

func (t *Type) GetStorageRequirements() *storage.Requirements

func (*Type) GetTypeName Uses

func (t *Type) GetTypeName() dvid.TypeString

func (*Type) GetTypeURL Uses

func (t *Type) GetTypeURL() dvid.URLString

func (*Type) GetTypeVersion Uses

func (t *Type) GetTypeVersion() string

type TypeMigrator Uses

type TypeMigrator interface {
    MigrateData([]dvid.VersionID) (DataService, error)

TypeMigrator is an interface for a DataService that can migrate itself to another DataService. A deprecated DataService implementation can implement this interface to auto-convert on metadata load.

type TypeService Uses

type TypeService interface {
    GetTypeName() dvid.TypeString

    GetTypeURL() dvid.URLString

    GetTypeVersion() string

    GetStorageRequirements() *storage.Requirements

    // Create an instance of this datatype in the given repo (identified by its root UUID)
    // with local instance ID and name, passing configuration parameters via dvid.Config.
    NewDataService(dvid.UUID, dvid.InstanceID, dvid.InstanceName, dvid.Config) (DataService, error)

    // Help returns a string explaining how to use a datatype's service
    Help() string

    // Do executes type-specific commands from command line.
    Do(Request, *Response) error

TypeService is an interface all datatype implementations must fulfill. New types can be made by embedding Type and will automatically fulfill the Get* functions.

func TypeServiceByName Uses

func TypeServiceByName(name dvid.TypeString) (TypeService, error)

TypeServiceByName returns a TypeService given a type name. Note that the type name is possibly ambiguous, particularly if using type names across different DVID servers.

func TypeServiceByURL Uses

func TypeServiceByURL(url dvid.URLString) (TypeService, error)

TypeServiceByURL returns a TypeService given its URL. This is the preferred method for accessing datatype implementations since they should work across different DVID servers.

type TypeUpgrader Uses

type TypeUpgrader interface {
    UpgradeData() (upgraded bool, err error)

TypeUpgrader is an interface for a DataService that can upgrade itself to another version of data storage. A deprecated DataService implementation can implement this interface to auto-convert on metadata load.

type Updater Uses

type Updater struct {
    // contains filtered or unexported fields

func (*Updater) StartUpdate Uses

func (u *Updater) StartUpdate()

func (*Updater) StopUpdate Uses

func (u *Updater) StopUpdate()

func (*Updater) Updating Uses

func (u *Updater) Updating() bool

Returns true if the data is currently being updated.

type VersionInitializer Uses

type VersionInitializer interface {
    InitVersion(dvid.UUID, dvid.VersionID) error

VersionInitializer provides a hook for data instances to receive branch (new version) events and modify their properties as needed.

type VersionRemapper Uses

type VersionRemapper interface {
    RemapVersions(dvid.VersionMap) error

VersionRemapper provides a hook for data instances to remap properties that depend on server-specific version ids. During DVID-to-DVID transfer of data, these version ids need to be remapped as part of the push/pull.

type VersionedCtx Uses

type VersionedCtx struct {
    User string

VersionedCtx implements storage.VersionedCtx for data instances that have a version DAG.

func NewVersionedCtx Uses

func NewVersionedCtx(data dvid.Data, versionID dvid.VersionID) *VersionedCtx

NewVersionedCtx creates a new versioned context that also has ability to hold a user string.

func NewVersionedCtxMasterLeaf Uses

func NewVersionedCtxMasterLeaf(data dvid.Data) (*VersionedCtx, error)

NewVersionedCtxMasterLeaf creates a new versioned context for the master branch leaf.

func (*VersionedCtx) GetBestKeyVersion Uses

func (vctx *VersionedCtx) GetBestKeyVersion(keys []storage.Key) (storage.Key, error)

GetBestKeyVersion returns the key that most closely matches this context's version. If no suitable key or a tombstone is encountered closed to the version, nil is returned.

func (*VersionedCtx) GetGraphDB Uses

func (vctx *VersionedCtx) GetGraphDB() (storage.GraphDB, error)

GetGraphDB returns a graph store associated with this context or an error if one is not available.

func (*VersionedCtx) GetKeyValueDB Uses

func (vctx *VersionedCtx) GetKeyValueDB() (storage.KeyValueDB, error)

GetKeyValueDB returns a key-value store associated with this context on an error if one is unavailable.

func (*VersionedCtx) GetOrderedKeyValueDB Uses

func (vctx *VersionedCtx) GetOrderedKeyValueDB() (storage.OrderedKeyValueDB, error)

GetOrderedKeyValueDB returns an ordered key-value store associated with this context or an error if one is unavailable.

func (*VersionedCtx) Head Uses

func (vctx *VersionedCtx) Head() bool

Head checks whether this the open head of the master branch

func (*VersionedCtx) MasterVersion Uses

func (vctx *VersionedCtx) MasterVersion(version dvid.VersionID) bool

Head checks whether specified version is on the master branch

func (*VersionedCtx) NumVersions Uses

func (vctx *VersionedCtx) NumVersions() int32

NumVersions returns the number of versions for a given

func (*VersionedCtx) RepoRoot Uses

func (vctx *VersionedCtx) RepoRoot() (dvid.UUID, error)

RepoRoot returns the root uuid.

func (*VersionedCtx) String Uses

func (vctx *VersionedCtx) String() string

func (*VersionedCtx) VersionUUID Uses

func (vctx *VersionedCtx) VersionUUID() dvid.UUID

VersionUUID returns the UUID associated with this versioned context.

func (*VersionedCtx) Versioned Uses

func (vctx *VersionedCtx) Versioned() bool

func (*VersionedCtx) VersionedKeyValue Uses

func (vctx *VersionedCtx) VersionedKeyValue(values []*storage.KeyValue) (*storage.KeyValue, error)

VersionedKeyValue returns the key-value pair corresponding to this key's version given a list of key-value pairs across many versions. If no suitable key-value pair is found or a tombstone is encounterd closest to version, nil is returned.

Package datastore imports 22 packages (graph) and is imported by 57 packages. Updated 2020-07-09. Refresh now. Tools for package owners.