datastore

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 18, 2023 License: BSD-3-Clause Imports: 22 Imported by: 34

Documentation

Overview

Package datastore provides versioning and persisting supported data types using one of the supported storage engines.

Index

Constants

View Source 
const (
	StartDataMsg = "datastore.startData"
	PutKVMsg     = "datastore.putKV"
)
View Source 
const DataShutdownTime = 20

DataShutdownTime is the maximum number of seconds a data instance can delay when terminating goroutines during Shutdown.

View Source 
const InitialMutationID = 1000000000 // billion

InitialMutationID is the initial mutation ID for all new repositories. It was chosen to be a large number to distinguish between mutations with repo-wide persistent IDs and the legacy data instance-specific mutation IDs that would reset on server restart.

View Source 
const PropertyTKeyClass = 1

PropertyTKeyClass is the TKeyClass reserved for data instance properties, and should be reserved in each data type implementation.

View Source 
const RepoFormatVersion = 1

RepoFormatVersion is the current repo metadata format version

View Source 
const (
	ServerLockKey storage.TKeyClass // name of key for locking metadata globally

)

Key space handling for metadata

View Source 
const StrideMutationID = 100

StrideMutationID is a way to minimize cost of persisting mutation IDs for a repo. We persist the current mutation ID plus the stride, so that we don't have to update the persisted mutation ID frequently. The downside is that there may be jumps in mutation IDs on restarts, since the persisted ID is loaded on next restart.

View Source 
const (
	Version = "0.11.0"
)

Variables

View Source 
var (
	ErrManagerNotInitialized = errors.New("datastore repo manager not initialized")
	ErrBadMergeType          = errors.New("bad merge type")

	ErrInvalidUUID         = errors.New("UUID is not present in datastore")
	ErrInvalidVersion      = errors.New("server-specific version id is invalid")
	ErrInvalidRepoID       = errors.New("server-specific repo id is invalid")
	ErrExistingUUID        = errors.New("UUID already exists in datastore")
	ErrExistingDataName    = errors.New("data name already exists")
	ErrInvalidDataName     = errors.New("invalid data instance name")
	ErrInvalidDataInstance = errors.New("invalid data instance id")
	ErrInvalidDataUUID     = errors.New("invalid data instance UUID")

	ErrInvalidStore = errors.New("invalid data store")

	ErrModifyLockedNode   = errors.New("can't modify locked node")
	ErrBranchUnlockedNode = errors.New("can't branch an unlocked node")
	ErrBranchUnique       = errors.New("branch already exists with given name")
)
View Source 
var (
	// Compiled is the set of registered datatypes compiled into DVID and
	// held as a global variable initialized at runtime.
	Compiled map[dvid.URLString]TypeService
)

Functions

func AddToNodeLog 

func AddToNodeLog(uuid dvid.UUID, msgs []string) error

func AddToRepoLog 

func AddToRepoLog(uuid dvid.UUID, msgs []string) error

func BlockOnUpdating 

func BlockOnUpdating(uuid dvid.UUID, name dvid.InstanceName) error

BlockOnUpdating blocks until the given data is not updating from syncs or has events waiting in sync channels. Primarily used during testing.

func CloseReopenTest 

func CloseReopenTest(datamap ...DataStorageMap)

CloseReopenTest forces close and then reopening of the datastore, useful for testing persistence. We only allow close/reopen when all tests not avaiting close/reopen are finished.

func CloseTest 

func CloseTest()

func Commit 

func Commit(uuid dvid.UUID, note string, log []string) error

func CompiledChart 

func CompiledChart() string

CompiledChart returns a chart (names/urls) of datatypes compiled into this DVID.

func CompiledTypes 

func CompiledTypes() map[dvid.TypeString]TypeService

CompiledTypes returns a map of datatypes compiled into this DVID.

func CompiledURLs 

func CompiledURLs() string

CompiledURLs returns a list of datatype urls supported by this DVID.

func CopyInstance 

func CopyInstance(uuid dvid.UUID, source, target dvid.InstanceName, c dvid.Config) error

CopyInstance copies a data instance locally, perhaps to a different storage engine if the new instance uses a different backend per a data instance-specific configuration. (See sample config.example.toml file in root dvid source directory.)

func DeleteConflicts 

func DeleteConflicts(uuid dvid.UUID, data DataService, oldParents, newParents []dvid.UUID) error

DeleteConflicts removes all conflicted kv pairs for the given data instance using the priority established by parents. As a side effect, newParents are modified by new children of parents.

func DeleteDataByName 

func DeleteDataByName(uuid dvid.UUID, name dvid.InstanceName, passcode string) error

DeleteDataByName returns a data service given an instance name and UUID.

func DeleteDataByVersion 

func DeleteDataByVersion(v dvid.VersionID, name dvid.InstanceName, passcode string) error

DeleteDataByVersion returns a data service given an instance name and UUID.

func DeleteRepo 

func DeleteRepo(uuid dvid.UUID, passcode string) error

DeleteRepo deletes a Repo holding a node with UUID.

func FlattenMetadata 

func FlattenMetadata(uuid dvid.UUID, configFName string) error

FlattenMetadata stores the metadata of a reduced set of nodes into the destination store.

func GetAncestry 

func GetAncestry(v dvid.VersionID) ([]dvid.VersionID, error)

GetAncestry returns a list of ancestor versions from the given version to the root.

func GetBlobStore 

func GetBlobStore(d dvid.Data) (store storage.BlobStore, err error)

GetBlobStore returns a blob store or nil if not available. TODO -- Add configurable option to use a file system.

func GetBranchHead added in v1.0.0 

func GetBranchHead(uuid dvid.UUID, name string) (branchUUID dvid.UUID, branchV dvid.VersionID, err error)

func GetBranchVersions 

func GetBranchVersions(uuid dvid.UUID, name string) ([]dvid.UUID, error)

GetBranchVersions returns a slice of UUIDs for the given branch from the HEAD (leaf) to the root.

func GetBranchVersionsJSON 

func GetBranchVersionsJSON(uuid dvid.UUID, name string) (string, error)

func GetChildrenByVersion 

func GetChildrenByVersion(v dvid.VersionID) ([]dvid.VersionID, error)

GetChildren returns the child nodes of the given version id.

func GetKeyValueBatcher 

func GetKeyValueBatcher(d dvid.Data) (db storage.KeyValueBatcher, err error)

GetKeyValueBatcher returns a batch-capable kv data store assigned to this data instance. If the store is nil or not available, an error is returned.

func GetKeyValueDB 

func GetKeyValueDB(d dvid.Data) (db storage.KeyValueDB, err error)

GetKeyValueDB returns a kv data store assigned to this data instance. If the store is nil or not available, an error is returned.

func GetNodeLog 

func GetNodeLog(uuid dvid.UUID) ([]string, error)

func GetNodeNote 

func GetNodeNote(uuid dvid.UUID) (string, error)

func GetOrderedKeyValueDB 

func GetOrderedKeyValueDB(d dvid.Data) (db storage.OrderedKeyValueDB, err error)

GetOrderedKeyValueDB returns the ordered kv data store assigned to this data instance. If the store is nil or not available, an error is returned.

func GetParentsByVersion 

func GetParentsByVersion(v dvid.VersionID) ([]dvid.VersionID, error)

GetParents returns the parent nodes of the given version id.

func GetRepoAlias 

func GetRepoAlias(uuid dvid.UUID) (string, error)

func GetRepoDescription 

func GetRepoDescription(uuid dvid.UUID) (string, error)

func GetRepoJSON 

func GetRepoJSON(uuid dvid.UUID) (string, error)

func GetRepoLog 

func GetRepoLog(uuid dvid.UUID) ([]string, error)

func GetRepoRoot 

func GetRepoRoot(uuid dvid.UUID) (dvid.UUID, error)

func GetRepoRootVersion 

func GetRepoRootVersion(v dvid.VersionID) (dvid.VersionID, error)

func GetStorageDetails 

func GetStorageDetails() (map[string]StorageStats, error)

GetStorageDetails scans all key-value stores and returns detailed stats per instances.

func GetStorageSummary 

func GetStorageSummary() (string, error)

GetStorageSummary returns JSON for all the data instances in the stores.

func HideBranch added in v0.9.13 

func HideBranch(uuid dvid.UUID, branchName string) error

HideBranch makes the branch at given UUID (that node and all its children) not visible from a metadata perspective. The actual data associated with the branch are not deleted from the store.

func Initialize 

func Initialize(initMetadata bool, iconfig Config) error

Initialize creates a repositories manager that is handled through package functions.

func LimitVersions 

func LimitVersions(uui dvid.UUID, configFName string) error

LimitVersions removes versions from the metadata that are not present in a configuration file.

func LockedUUID 

func LockedUUID(uuid dvid.UUID) (bool, error)

LockedUUID returns true if a given UUID is locked.

func LockedVersion 

func LockedVersion(v dvid.VersionID) (bool, error)

LockedVersion returns true if a given version is locked.

func LogRepoOpToKafka 

func LogRepoOpToKafka(uuid dvid.UUID, b []byte) error

LogRepoOpToKafka logs a repo operation to kafka

func MakeMaster added in v0.9.13 

func MakeMaster(newMasterNode dvid.UUID, oldMasterBranchName string) error

MakeMaster makes the branch at given UUID (that node and all its children) the new master branch and renames the old master branch to the given branch name. NOTE: This command will fail if the given UUID is not a node that is directly branched off master.

func MarshalJSON 

func MarshalJSON() ([]byte, error)

MarshalJSON returns JSON of object where each repo is a property with root UUID name and value corresponding to repo info.

func MatchingUUID 

func MatchingUUID(uuidStr string) (dvid.UUID, dvid.VersionID, error)

MatchingUUID returns a local version ID and the full UUID from a potentially shortened UUID string. Partial matches are accepted as long as they are unique for a datastore. So if a datastore has nodes with UUID strings 3FA22... (root), 7CD11..., and 836EE... (leaf), we can still find a match even if given the minimum 3 letters. (We don't allow UUID strings of less than 3 letters just to prevent mistakes.) If the passed string has a colon, the string after the colon is parsed as a case-sensitive branch name of the repo with the given UUID, and the UUID returned will be the HEAD or uncommitted leaf of that branch.

Examples: "3FA22:master" returns the leaf UUID 836ee of branch "master" for the repo containing 

the 3FA22 UUID.

"3FA22:master~1" returns the 1st parent of the leaf UUID of branch "master" for the 

repo containing, which is 7cd11.

func Merge 

func Merge(parents []dvid.UUID, note string, mt MergeType) (dvid.UUID, error)

func MigrateBatch 

func MigrateBatch(uuid dvid.UUID, configFName string) (err error)

MigrateBatch migrates instances specified in the config file from a source to a destination store, similar to the above "migrate" command but in a batch. Before running this command, you must modify the config TOML file so the destination store is available.

The migrate config file contains JSON with the following format: 

{
	"Versions": ["2881e9","52a13","57e8d"],
	"Migrations": [
		{
			"Name": "instance-name",
			"SrcStore": "source store",
			"DstStore": {
				"Path": "/path/to/store",
				"Engine": "basholeveldb"
			}
		},
		{
			"Name": "#datatype",
			"SrcStore": "source store",
			"DstStore": {
				"Path": "/another/store",
				"Engine": "badger"
			}
		},
	],
	"Exclusions": ["name1", "name2"]
}

func MigrateInstance 

func MigrateInstance(uuid dvid.UUID, source dvid.InstanceName, srcStore, dstStore dvid.Store, c dvid.Config, done chan bool) error

MigrateInstance migrates a data instance locally from an old storage engine to the current configured storage.

func ModifyDataConfigByName 

func ModifyDataConfigByName(uuid dvid.UUID, name dvid.InstanceName, c dvid.Config) error

func NewRepo 

func NewRepo(alias, description string, assign *dvid.UUID, passcode string) (dvid.UUID, error)

NewRepo creates a new Repo and returns its UUID, either an assigned UUID if provided or creating a new UUID.

func NewTestRepo 

func NewTestRepo() (dvid.UUID, dvid.VersionID)

NewTestRepo returns a new datastore.Repo suitable for testing.

func NewUUID 

func NewUUID(assign *dvid.UUID) (dvid.UUID, dvid.VersionID, error)

func NewVersion 

func NewVersion(parent dvid.UUID, note string, branchname string, assign *dvid.UUID) (dvid.UUID, error)

NewVersion creates a new version as a child of the given parent. If the assign parameter is not nil, the new node is given the UUID.

func NotifySubscribers 

func NotifySubscribers(e SyncEvent, m SyncMessage) error

NotifySubscribers sends a message to any data instances subscribed to the event.

func OpenTest 

func OpenTest(datamap ...DataStorageMap)

func PushData 

func PushData(d dvid.Data, p *PushSession) error

PushData transfers all key-value pairs pertinent to the given data instance. Each datatype can implement filters that can restrict the transmitted key-value pairs based on the given FilterSpec. Note that because of the generality of this function, a particular datatype implementation could be much more efficient when implementing filtering. For example, the imageblk datatype could scan its key-values using the ROI to generate keys (since imageblk keys will likely be a vast superset of ROI spans), while this generic routine will scan every key-value pair for a data instance and query the ROI to see if this key is ok to send.

func PushRepo 

func PushRepo(uuid dvid.UUID, target string, config dvid.Config) error

PushRepo pushes a Repo to a remote DVID server at the target address.

func Register 

func Register(t TypeService)

Register registers a datatype for DVID use.

func ReloadMetadata 

func ReloadMetadata() error

ReloadMetadata reloads the repositories manager from an existing metadata store. This shuts off all other dvid requests.

func RenameData 

func RenameData(uuid dvid.UUID, oldname, newname dvid.InstanceName, passcode string) error

RenameData renames a data service given an old instance name and UUID.

func SaveDataByUUID 

func SaveDataByUUID(uuid dvid.UUID, data DataService) error

SaveDataByUUID persists metadata for a data instance with given uuid. TODO -- Make this more efficient by storing data metadata separately from repo. 

Currently we save entire repo.

func SaveDataByVersion 

func SaveDataByVersion(v dvid.VersionID, data DataService) error

SaveDataByVersion persists metadata for a data instance with given version. TODO -- Make this more efficient by storing data metadata separately from repo. 

Currently we save entire repo.

func SetNodeNote 

func SetNodeNote(uuid dvid.UUID, note string) error

func SetRepoAlias 

func SetRepoAlias(uuid dvid.UUID, alias string) error

func SetRepoDescription 

func SetRepoDescription(uuid dvid.UUID, desc string) error

func SetSyncByJSON 

func SetSyncByJSON(d dvid.Data, uuid dvid.UUID, replace bool, in io.ReadCloser) error

SetSyncByJSON takes a JSON object of sync names and UUID, and creates the sync graph and sets the data instance's sync. If replace is false (default), the new sync is appended to the current syncs.

func SetSyncData 

func SetSyncData(data dvid.Data, syncs dvid.UUIDSet, replace bool) error

SetSyncData modfies the manager sync graphs and data instance's sync list. If replace is false (default), the new sync is appended to the current syncs.

func SetTagsByJSON 

func SetTagsByJSON(d dvid.Data, uuid dvid.UUID, replace bool, in io.ReadCloser) error

SetTagsByJSON takes a JSON object of tags and either appends or replaces the current data's tags depending on the replace parameter.

func Shutdown 

func Shutdown()

Shutdown sends signal for all goroutines for data processing to be terminated.

func TransferData 

func TransferData(uuid dvid.UUID, srcStore, dstStore dvid.Store, configFName string) error

TransferData copies key-value pairs from one repo to store and apply filtering as specified by the JSON configuration in the file specified by configFName. An example of the transfer JSON configuration file format: 

{
	"Versions": [
		"8a90ec0d257c415cae29f8c46603bcae",
		"a5682904bb824c06aba470c0a0cbffab",
		...
	},
	"Metadata": true,
}

All ancestors of desired leaf nodes should be specified because key-value pair transfer only occurs if the version in which it was saved is specified on the list. This is useful for editing a preexisting store with new versions.

If Metadata property is true, then if metadata exists in the old store, it is transferred to the new store with only the versions specified appearing in the DAG.

func Types 

func Types() (map[dvid.URLString]TypeService, error)

Types returns the types currently within the DVID server.

func UUIDFromVersion 

func UUIDFromVersion(v dvid.VersionID) (dvid.UUID, error)

func VersionFromUUID 

func VersionFromUUID(uuid dvid.UUID) (dvid.VersionID, error)

Types

type BlobService 

type BlobService interface {
	GetBlob(ref string) (data []byte, err error)
	PutBlob(data []byte) (ref string, err error)
}

BlobService is an interface for storing and retrieving data based on its content.

type CommitSyncer 

type CommitSyncer interface {
	// SyncOnCommit is an asynchronous function that should be called when a node is committed.
	SyncOnCommit(dvid.UUID, dvid.VersionID)
}

CommitSyncer want to be notified when a node is committed.

type Config 

type Config struct {
	InstanceGen   string
	InstanceStart dvid.InstanceID
	MutationStart uint64
	ReadOnly      bool
}

Config specifies new instance and mutation ID generation

type Data 

type Data struct {
	// contains filtered or unexported fields
}

Data is the base struct of repo-specific data instances. It should be embedded in a datatype's DataService implementation and handle datastore-wide key partitioning.

func NewDataService 

func NewDataService(t TypeService, rootUUID dvid.UUID, id dvid.InstanceID, name dvid.InstanceName, c dvid.Config) (*Data, error)

NewDataService returns a new Data instance that fulfills the DataService interface. The UUID passed in corresponds to the root UUID of the DAG subgraph that should hold the data. This returned Data struct is usually embedded by datatype-specific data instances. By default, LZ4 and the default checksum is used.

func (*Data) Checksum 

func (d *Data) Checksum() dvid.Checksum

func (*Data) Compression 

func (d *Data) Compression() dvid.Compression

func (*Data) DAGRootUUID 

func (d *Data) DAGRootUUID() (dvid.UUID, error)

func (*Data) DataName 

func (d *Data) DataName() dvid.InstanceName

func (*Data) DataUUID 

func (d *Data) DataUUID() dvid.UUID

func (*Data) DescribeTKeyClass 

func (d *Data) DescribeTKeyClass(tkc storage.TKeyClass) string

DescribeTKeyClass returns a string explanation of what a particular TKeyClass is used for. This will be overriden in data types, but if not, this provides a fallback for all data types.

func (*Data) Equals 

func (d *Data) Equals(d2 *Data) bool

Equals returns true if the two data instances are identical, not just referring to the same data instance (e.g., different names but same data UUID, etc).

func (*Data) GetBlob 

func (d *Data) GetBlob(ref string) (data []byte, err error)

GetBlob retrieves data given a reference that was received during PutBlob.

func (*Data) GetReadLog 

func (d *Data) GetReadLog() storage.ReadLog

func (*Data) GetType 

func (d *Data) GetType() TypeService

func (*Data) GetWriteLog 

func (d *Data) GetWriteLog() storage.WriteLog

func (*Data) GobDecode 

func (d *Data) GobDecode(b []byte) error

func (*Data) GobEncode 

func (d *Data) GobEncode() ([]byte, error)

func (*Data) InstanceID 

func (d *Data) InstanceID() dvid.InstanceID

func (*Data) IsDeleted 

func (d *Data) IsDeleted() bool

IsDeleted returns true if data has been deleted or is deleting.

func (*Data) IsMutationRequest 

func (d *Data) IsMutationRequest(action, endpoint string) bool

IsMutationRequest is the default definition of mutation requests. All POST, PUT, and DELETE actions return true, while others return false.

func (*Data) KVStore 

func (d *Data) KVStore() (dvid.Store, error)

func (*Data) MarshalJSON 

func (d *Data) MarshalJSON() ([]byte, error)

func (*Data) MarshalJSONTags 

func (d *Data) MarshalJSONTags() ([]byte, error)

func (*Data) ModifyConfig 

func (d *Data) ModifyConfig(config dvid.Config) error

func (*Data) MutAdd 

func (d *Data) MutAdd(mutID uint64) (newOp bool)

MutAdd adds a wait to this operation. Returns true if this is a new operation for this Data

func (*Data) MutDelete 

func (d *Data) MutDelete(mutID uint64)

MutDelete removes a wait for this operation. Should only be done after all MutWait() have completed.

func (*Data) MutDone 

func (d *Data) MutDone(mutID uint64)

MutDone marks the end of operations corresponding to MutAdd.

func (*Data) MutWait 

func (d *Data) MutWait(mutID uint64)

MutWait blocks until all operations with the given ID are completed.

func (*Data) NewMutationID 

func (d *Data) NewMutationID() uint64

func (*Data) PersistMetadata 

func (d *Data) PersistMetadata() error

func (*Data) PublishKafkaMsg added in v0.9.16 

func (d *Data) PublishKafkaMsg(b []byte) error

func (*Data) PushData 

func (d *Data) PushData(p *PushSession) error

PushData is the base implementation of pushing data instance key-value pairs to a remote DVID without any datatype-specific filtering of data.

func (*Data) PutBlob 

func (d *Data) PutBlob(b []byte) (ref string, err error)

PutBlob stores data and returns a reference to that data.

func (*Data) RootUUID 

func (d *Data) RootUUID() dvid.UUID

func (*Data) RootVersionID 

func (d *Data) RootVersionID() (dvid.VersionID, error)

func (*Data) SetDataUUID 

func (d *Data) SetDataUUID(uuid dvid.UUID)

func (*Data) SetDeleted 

func (d *Data) SetDeleted(deleted bool)

SetDeleted is used to indicate whether data has been deleted or is deleting.

func (*Data) SetInstanceID 

func (d *Data) SetInstanceID(id dvid.InstanceID)

func (*Data) SetKVStore 

func (d *Data) SetKVStore(kvStore dvid.Store)

func (*Data) SetLogStore 

func (d *Data) SetLogStore(logStore dvid.Store)

func (*Data) SetName 

func (d *Data) SetName(name dvid.InstanceName)

func (*Data) SetRootUUID 

func (d *Data) SetRootUUID(uuid dvid.UUID)

func (*Data) SetSync 

func (d *Data) SetSync(syncs dvid.UUIDSet)

SetSync sets the list of synced UUIDs for this data instance. It does not modify the sync graph.

func (*Data) SetTags 

func (d *Data) SetTags(tags map[string]string)

func (*Data) SyncPending 

func (d *Data) SyncPending() bool

SyncPending returns true if there are outstanding sync events in this instance's subscription.

func (*Data) SyncedData 

func (d *Data) SyncedData() dvid.UUIDSet

SyncedData returns a set of data UUIDs to which it is synced.

func (*Data) SyncedNames 

func (d *Data) SyncedNames() []dvid.InstanceName

SyncedNames returns a set of data instance names to which it is synced. Legacy and will be removed after metadata refactor.

func (*Data) Tags 

func (d *Data) Tags() map[string]string

func (*Data) TypeName 

func (d *Data) TypeName() dvid.TypeString

func (*Data) TypeURL 

func (d *Data) TypeURL() dvid.URLString

func (*Data) TypeVersion 

func (d *Data) TypeVersion() string

func (*Data) Versioned 

func (d *Data) Versioned() bool

type DataAvail 

type DataAvail uint8

DataAvail gives the availability of data within a node or whether parent nodes must be traversed to check for key-value pairs. 

const (
	// DataDelta = For any query, we must also traverse this node's ancestors in the DAG
	// up to any DataComplete ancestor.  Used if a node is marked as archived.
	DataDelta DataAvail = iota

	// DataComplete = All key-value pairs are available within this node.
	DataComplete

	// DataRoot = Queries are redirected to Root since this is unversioned.
	DataRoot

	// DataDeleted = key-value pairs have been explicitly deleted at this node and is no longer available.
	DataDeleted
)

func (DataAvail) String 

func (avail DataAvail) String() string

type DataInitializer 

type DataInitializer interface {
	InitDataHandlers() error
}

DataInitializer is a data instance that needs to be initialized, e.g., start long-lived goroutines that handle data syncs, etc. Initialization should only constitute ephemeral data and goroutines and not require persisting data itself like InstanceMutator, so no saveNeeded flag needs to be returned.

type DataService 

type DataService interface {
	dvid.Data
	json.Marshaler // Supply JSON describing service.
	BlobService

	Help() string
	GetType() TypeService

	// ModifyConfig modifies a configuration in a type-specific way.
	ModifyConfig(dvid.Config) error

	// IsMutationRequest returns true if the given HTTP method on the endpoint results in mutations.
	// This allows datatypes to define actions on an endpoint as immutable even if they use POST.
	// The endpoint is the keyword immediately following a data instance name in the URL:
	//    /api/node/483f/grayscale/raw/xy/...
	// In the above URL, "raw" is the endpoint.
	IsMutationRequest(action, endpoint string) bool

	// DoRPC handles command line and RPC commands specific to a data type
	DoRPC(Request, *Response) error

	// ServeHTTP handles HTTP requests in the context of a particular version and
	// returns activity information that can be logged for monitoring.
	ServeHTTP(dvid.UUID, *VersionedCtx, http.ResponseWriter, *http.Request) (activity map[string]interface{})

	// DescribeTKeyClass explains in a string what a particular TKeyClass
	// is used for.  For example, one class of TKey for the label data types is the block-indexed
	// voxel values.
	DescribeTKeyClass(storage.TKeyClass) string

	// PushData handles DVID-to-DVID transmission of key-value pairs, optionally
	// delimited by type-specific filter specifications (e.g., "roi:roiname,uuid")
	// and a set of versions.  DataService implementations can automatically embed
	// this via datastore.Data or can add filters by providing their own PushData(),
	// Filterer, and Filter implementations.  (datatype/imageblk/distributed.go)
	PushData(*PushSession) error
}

DataService is an interface for operations on an instance of a supported datatype.

func GetDataByDataUUID 

func GetDataByDataUUID(dataUUID dvid.UUID) (DataService, error)

GetDataByDataUUID returns a data service given a data UUID.

func GetDataByUUIDName 

func GetDataByUUIDName(uuid dvid.UUID, name dvid.InstanceName) (DataService, error)

GetDataByUUIDName returns a data service given an instance name and UUID.

func GetDataByVersionName 

func GetDataByVersionName(v dvid.VersionID, name dvid.InstanceName) (DataService, error)

GetDataByVersionName returns a data service given an instance name and version.

func NewData 

func NewData(uuid dvid.UUID, t TypeService, name dvid.InstanceName, c dvid.Config) (DataService, error)

NewData adds a new, named instance of a datatype to repo. Settings can be passed via the 'config' argument. For example, config["versioned"] with a bool value will specify whether the data is versioned.

type DataSpec added in v1.0.0 

type DataSpec struct {
	// contains filtered or unexported fields
}

func (*DataSpec) DataName added in v1.0.0 

func (d *DataSpec) DataName() dvid.InstanceName

func (*DataSpec) DataUUID added in v1.0.0 

func (d *DataSpec) DataUUID() dvid.UUID

func (*DataSpec) RootUUID added in v1.0.0 

func (d *DataSpec) RootUUID() dvid.UUID

func (*DataSpec) Tags added in v1.0.0 

func (d *DataSpec) Tags() map[string]string

func (*DataSpec) TypeName added in v1.0.0 

func (d *DataSpec) TypeName() dvid.TypeString

type DataStorageMap 

type DataStorageMap struct {
	KVStores  storage.DataMap
	LogStores storage.DataMap
}

DataStorageMap describes mappings from various instance and data type specifications to KV and Log stores.

type DataTxInit 

type DataTxInit struct {
	Session    rpc.SessionID
	DataName   dvid.InstanceName
	DataUUID   dvid.UUID
	RootUUID   dvid.UUID
	TypeName   dvid.TypeString
	InstanceID dvid.InstanceID
	Tags       map[string]string
}

type Initializer 

type Initializer interface {
	Initialize()
}

Initializer is a data instance that can be initialized. Typically this sets up any goroutines necessary after server configuration, etc.

type InstanceMutator 

type InstanceMutator interface {
	// Loads all mutable properties and applies any necessary migration to
	// transform the internal data from the stored to expected version.
	LoadMutable(root dvid.VersionID, storedVersion, expectedVersion uint64) (saveNeeded bool, err error)
}

InstanceMutator provides a hook for data instances to load mutable data on startup. It is assumed that the data instances store data whenever its data mutates, e.g., extents for labelblk or max label for labelvol.

type InstanceStats 

type InstanceStats map[string]KeyStats // index is description of tkey class

type KVMessage 

type KVMessage struct {
	Session   rpc.SessionID
	KV        storage.KeyValue
	Terminate bool // true if this message is the last txn for this data instance and KV is invalid.
}

KVMessage packages a key-value pair for transmission to a remote DVID as well as control of the receiving FSM.

type KeyStats 

type KeyStats struct {
	LeafKV, IntKV       uint64 // # of kv pairs in leaf and interior nodes, respectively
	LeafBytes, IntBytes uint64 // # of bytes in leaf and interior nodes, respectively
}

type MergeType 

type MergeType uint8

MergeType describes the expectation of processing for the merge, e.g., is it expected to be free of conflicts at the key-value level, require automated type-specific key-value merging, or require external guidance. 

const (
	// MergeConflictFree assumes that changes in nodes-to-be-merged are disjoint
	// at the key-value level.
	MergeConflictFree MergeType = iota

	// MergeTypeSpecificAuto requires datatype-specific code for merging at each
	// key-value pair.
	MergeTypeSpecificAuto

	// MergeExternalData requires external data to reconcile merging of nodes.
	MergeExternalData
)

type MutationDumper 

type MutationDumper interface {
	DumpMutations(startUUID, endUUID dvid.UUID, filename string) (comment string, err error)
}

MutationDumper is a dataservice that suppports the flatten-mutations command via a DumpMutations() function.

type MutationMutexer 

type MutationMutexer interface {
	MutAdd(mutID uint64) (newOp bool)
	MutDone(mutID uint64)
	MutWait(mutID uint64)
	MutDelete(mutID uint64)
}

MutationMutexer is an interface for mutexes on particular mutation IDs.

type PropertyCopier 

type PropertyCopier interface {
	CopyPropertiesFrom(DataService, storage.FilterSpec) error
}

PropertyCopier are types that can copy data instance properties from another (typically identically typed) data instance with an optional filter. This is used to create copies of data instances locally or when pushing to a remote DVID.

type PushSession 

type PushSession struct {
	Filter   storage.FilterSpec
	Versions map[dvid.VersionID]struct{}
	// contains filtered or unexported fields
}

PushSession encapsulates parameters necessary for DVID-to-DVID push/pull processing.

func (*PushSession) EndInstancePush 

func (p *PushSession) EndInstancePush() error

EndInstancePush terminates a data instance push.

func (*PushSession) SendKV 

func (p *PushSession) SendKV(kv *storage.KeyValue) error

SendKV sends a key-value pair. The key-values may be buffered before sending for efficiency of transmission.

func (*PushSession) StartInstancePush 

func (p *PushSession) StartInstancePush(d dvid.Data) error

StartInstancePush initiates a data instance push. After some number of Send calls, the EndInstancePush must be called.

type Request 

type Request struct {
	dvid.Command
	Input []byte
}

Request supports RPC requests to the DVID server.

type Response 

type Response struct {
	dvid.Response
	Output []byte
}

Response supports RPC responses from DVID.

func (*Response) Write 

func (r *Response) Write(w io.Writer) error

Writes a RPC response to a writer.

type Shutdowner 

type Shutdowner interface {
	Shutdown(wg *sync.WaitGroup) // Expects wg.Done() to be called in Shutdown function.
}

Shutdowner is a data instance that has a function to call during shutdown. Typically, this exits goroutines used for background data processing.

type StorageStats 

type StorageStats map[string]InstanceStats

func (StorageStats) String 

func (stats StorageStats) String() string

type SyncEvent 

type SyncEvent struct {
	Data  dvid.UUID
	Event string
}

SyncEvent identifies an event in which a data instance has modified its data

type SyncMessage 

type SyncMessage struct {
	Event   string
	Version dvid.VersionID
	Delta   interface{}
}

SyncMessage describes changes to a data instance for a given version.

type SyncSub 

type SyncSub struct {
	Event  SyncEvent
	Notify dvid.UUID // the data UUID of data instance to notify
	Ch     chan SyncMessage
}

SyncSub is a subscription request from an instance to be notified via a channel when a given data instance has a given event.

type SyncSubs 

type SyncSubs []SyncSub

SyncSubs is a slice of sync subscriptions.

func (SyncSubs) Add 

func (subs SyncSubs) Add(added SyncSub) SyncSubs

Add returns a SyncSubs with the added SyncSub, making sure that only one subscription exists for any (Event, Notify) tuple. If a previous (Event, Notify) exists, it is replaced by the passed SyncSub.

type Syncer 

type Syncer interface {
	// GetSyncSubs returns the subscriptions that need to be created to keep this data
	// synced and may launch goroutines that will consume inbound channels of changes
	// from associated data.
	GetSyncSubs(dvid.Data) (SyncSubs, error)

	// SyncedNames returns the set of data instance UUIDs to which the data is synced.
	SyncedNames() []dvid.InstanceName

	// SyncedNames returns the set of data instance UUIDs to which the data is synced.
	SyncedData() dvid.UUIDSet

	// SyncPending returns true if there are outstanding sync events in this instance's subscription.
	SyncPending() bool
}

Syncer types are typically DataService that know how to sync with other data.

type Type 

type Type struct {
	// Data type name and may not be unique.
	Name dvid.TypeString

	// The unique package name that fulfills the DVID Data interface
	URL dvid.URLString

	// The version identifier of this datatype code
	Version string

	// A list of interface requirements for the backend datastore
	Requirements *storage.Requirements
}

Type identifies the datatype underlying a DataService.

func (*Type) Do 

func (t *Type) Do(cmd Request, reply *Response) error

Do error message. Do should be overridden

func (*Type) GetStorageRequirements 

func (t *Type) GetStorageRequirements() *storage.Requirements

func (*Type) GetTypeName 

func (t *Type) GetTypeName() dvid.TypeString

func (*Type) GetTypeURL 

func (t *Type) GetTypeURL() dvid.URLString

func (*Type) GetTypeVersion 

func (t *Type) GetTypeVersion() string

type TypeMigrator 

type TypeMigrator interface {
	MigrateData([]dvid.VersionID) (DataService, error)
}

TypeMigrator is an interface for a DataService that can migrate itself to another DataService. A deprecated DataService implementation can implement this interface to auto-convert on metadata load.

type TypeService 

type TypeService interface {
	GetTypeName() dvid.TypeString

	GetTypeURL() dvid.URLString

	GetTypeVersion() string

	GetStorageRequirements() *storage.Requirements

	// Create an instance of this datatype in the given repo (identified by its root UUID)
	// with local instance ID and name, passing configuration parameters via dvid.Config.
	NewDataService(dvid.UUID, dvid.InstanceID, dvid.InstanceName, dvid.Config) (DataService, error)

	// Help returns a string explaining how to use a datatype's service
	Help() string

	// Do executes type-specific commands from command line.
	Do(Request, *Response) error
}

TypeService is an interface all datatype implementations must fulfill. New types can be made by embedding Type and will automatically fulfill the Get* functions.

func TypeServiceByName 

func TypeServiceByName(name dvid.TypeString) (TypeService, error)

TypeServiceByName returns a TypeService given a type name. Note that the type name is possibly ambiguous, particularly if using type names across different DVID servers.

func TypeServiceByURL 

func TypeServiceByURL(url dvid.URLString) (TypeService, error)

TypeServiceByURL returns a TypeService given its URL. This is the preferred method for accessing datatype implementations since they should work across different DVID servers.

type TypeUpgrader 

type TypeUpgrader interface {
	DataService
	UpgradeData() (upgraded bool, err error)
}

TypeUpgrader is an interface for a DataService that can upgrade itself to another version of data storage. A deprecated DataService implementation can implement this interface to auto-convert on metadata load.

type Updater 

type Updater struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

func (*Updater) StartUpdate 

func (u *Updater) StartUpdate()

func (*Updater) StopUpdate 

func (u *Updater) StopUpdate()

func (*Updater) Updating 

func (u *Updater) Updating() bool

Returns true if the data is currently being updated.

type VersionInitializer 

type VersionInitializer interface {
	InitVersion(dvid.UUID, dvid.VersionID) error
}

VersionInitializer provides a hook for data instances to receive branch (new version) events and modify their properties as needed.

type VersionRemapper 

type VersionRemapper interface {
	RemapVersions(dvid.VersionMap) error
}

VersionRemapper provides a hook for data instances to remap properties that depend on server-specific version ids. During DVID-to-DVID transfer of data, these version ids need to be remapped as part of the push/pull.

type VersionedCtx 

type VersionedCtx struct {
	*storage.DataContext
	User string
	App  string
}

VersionedCtx implements storage.VersionedCtx for data instances that have a version DAG.

func NewVersionedCtx 

func NewVersionedCtx(data dvid.Data, versionID dvid.VersionID) *VersionedCtx

NewVersionedCtx creates a new versioned context that also has ability to hold a user string.

func NewVersionedCtxMasterLeaf 

func NewVersionedCtxMasterLeaf(data dvid.Data) (*VersionedCtx, error)

NewVersionedCtxMasterLeaf creates a new versioned context for the master branch leaf.

func (*VersionedCtx) GetBestKeyVersion 

func (vctx *VersionedCtx) GetBestKeyVersion(keys []storage.Key) (storage.Key, error)

GetBestKeyVersion returns the key that most closely matches this context's version. If no suitable key or a tombstone is encountered closed to the version, nil is returned.

func (*VersionedCtx) GetKeyValueDB 

func (vctx *VersionedCtx) GetKeyValueDB() (storage.KeyValueDB, error)

GetKeyValueDB returns a key-value store associated with this context on an error if one is unavailable.

func (*VersionedCtx) GetOrderedKeyValueDB 

func (vctx *VersionedCtx) GetOrderedKeyValueDB() (storage.OrderedKeyValueDB, error)

GetOrderedKeyValueDB returns an ordered key-value store associated with this context or an error if one is unavailable.

func (*VersionedCtx) Head 

func (vctx *VersionedCtx) Head() bool

Head checks whether this the open head of the master branch

func (*VersionedCtx) MasterVersion 

func (vctx *VersionedCtx) MasterVersion(version dvid.VersionID) bool

Head checks whether specified version is on the master branch

func (*VersionedCtx) NumVersions 

func (vctx *VersionedCtx) NumVersions() int32

NumVersions returns the number of versions for a given

func (*VersionedCtx) RepoRoot 

func (vctx *VersionedCtx) RepoRoot() (dvid.UUID, error)

RepoRoot returns the root uuid.

func (*VersionedCtx) String 

func (vctx *VersionedCtx) String() string

func (*VersionedCtx) VersionUUID 

func (vctx *VersionedCtx) VersionUUID() dvid.UUID

VersionUUID returns the UUID associated with this versioned context.

func (*VersionedCtx) Versioned 

func (vctx *VersionedCtx) Versioned() bool

func (*VersionedCtx) VersionedKeyValue 

func (vctx *VersionedCtx) VersionedKeyValue(values []*storage.KeyValue) (*storage.KeyValue, error)

VersionedKeyValue returns the key-value pair corresponding to this key's version given a list of key-value pairs across many versions. If no suitable key-value pair is found or a tombstone is encounterd closest to version, nil is returned.

TODO: This function doesn't rely on the values being pre-sorted by version because any storage backend could be used. If we want to optimize this, we could require backends to return pre-sorted kv pairs. However migrating to DAGStore as sole backend would make this function obsolete.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.