labelmap

package
v0.8.18 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 18, 2018 License: BSD-3-Clause Imports: 32 Imported by: 0

Documentation

Overview

Package labelmap handles both volumes of label data as well as indexing to quickly find and generate sparse volumes of any particular label. It differs from labelmap in using in-memory label maps to greatly decrease changes to underlying label data key-value pairs, instead handling all merges through changes in label maps.

Index

Constants

View Source 
const (
	Version  = "0.1"
	RepoURL  = "github.com/janelia-flyem/dvid/datatype/labelmap"
	TypeName = "labelmap"
)
View Source 
const KVWriteSize = 500

KVWriteSize is the # of key-value pairs we will write as one atomic batch write.

Variables

View Source 
var (
	DefaultBlockSize int32   = 64
	DefaultRes       float32 = imageblk.DefaultRes
	DefaultUnits             = imageblk.DefaultUnits
)

Functions

func ChangeLabelIndex 

func ChangeLabelIndex(d dvid.Data, v dvid.VersionID, label uint64, delta labels.SupervoxelChanges) error

ChangeLabelIndex applies changes to a label's index and then stores the result. Supervoxel size changes for blocks should be passed into the function. The passed SupervoxelDelta can contain more supervoxels than the label index.

func CleaveIndex 

func CleaveIndex(d dvid.Data, v dvid.VersionID, op labels.CleaveOp, info dvid.ModInfo) error

CleaveIndex modifies the label index to remove specified supervoxels and create another label index for this cleaved body.

func DecodeAffinitiesTKey 

func DecodeAffinitiesTKey(tk storage.TKey) (label uint64, err error)

DecodeAffinitiesTKey parses a TKey and returns the corresponding label.

func DecodeBlockTKey 

func DecodeBlockTKey(tk storage.TKey) (scale uint8, idx *dvid.IndexZYX, err error)

DecodeBlockTKey returns a spatial index from a label block key. TODO: Extend this when necessary to allow any form of spatial indexing like CZYX.

func DecodeLabelIndexTKey 

func DecodeLabelIndexTKey(tk storage.TKey) (label uint64, err error)

DecodeLabelIndexTKey parses a TKey and returns the corresponding label.

func DeleteLabelIndex 

func DeleteLabelIndex(d dvid.Data, v dvid.VersionID, label uint64) error

DeleteLabelIndex deletes the index for a given label set.

func EncodeFormat 

func EncodeFormat() dvid.DataValues

func GetBinaryData 

func GetBinaryData(compression string, in io.ReadCloser, estsize int64) ([]byte, error)

GetBinaryData returns label data from a potentially compressed ("lz4", "gzip") reader.

func GetBoundedIndex 

func GetBoundedIndex(d dvid.Data, v dvid.VersionID, label uint64, bounds dvid.Bounds, isSupervoxel bool) (*labels.Index, error)

GetBoundedIndex gets bounded label index data from storage for a given data instance.

func GetLabelIndex 

func GetLabelIndex(d dvid.Data, v dvid.VersionID, label uint64, isSupervoxel bool) (*labels.Index, error)

GetLabelIndex gets label set index data from storage for a given data instance and version. If isSupervoxel is true, the label is interpreted as a supervoxel and the label set index containing the given supervoxel is returned. Concurrency-safe access and supports caching. If a label has been mapped to another, a nil Index is returned.

func GetLabelSize 

func GetLabelSize(d dvid.Data, v dvid.VersionID, label uint64, isSupervoxel bool) (uint64, error)

GetLabelSize returns the # of voxels in the given label. If isSupervoxel = true, the given label is interpreted as a supervoxel id and the size is of a supervoxel. If a label doesn't exist, a zero (not error) is returned.

func GetLabelSizes added in v0.8.18 

func GetLabelSizes(d dvid.Data, v dvid.VersionID, labels []uint64, isSupervoxel bool) ([]uint64, error)

GetLabelSizes returns the # of voxels in the given labels. If isSupervoxel = true, the given labels are interpreted as supervoxel ids and the sizes are of a supervoxel. If a label doesn't exist, a zero (not error) is returned.

func GetMultiLabelIndex 

func GetMultiLabelIndex(d dvid.Data, v dvid.VersionID, lbls labels.Set, bounds dvid.Bounds) (*labels.Index, error)

GetMultiLabelIndex gets index data for all labels is a set with possible bounds.

func GetSupervoxelBlocks 

func GetSupervoxelBlocks(d dvid.Data, v dvid.VersionID, supervoxel uint64) (dvid.IZYXSlice, error)

GetSupervoxelBlocks gets the blocks corresponding to a supervoxel id.

func NewAffinitiesTKey 

func NewAffinitiesTKey(label uint64) storage.TKey

NewAffinitiesTKey returns a TKey corresponding to a label's affinities.

func NewBlockTKey 

func NewBlockTKey(scale uint8, idx dvid.Index) storage.TKey

NewBlockTKey returns a TKey for a label block, which is a slice suitable for lexicographical ordering on zyx coordinates.

func NewBlockTKeyByCoord 

func NewBlockTKeyByCoord(scale uint8, izyx dvid.IZYXString) storage.TKey

NewBlockTKeyByCoord returns a TKey for a block coord in string format.

func NewLabelIndexTKey 

func NewLabelIndexTKey(label uint64) storage.TKey

NewLabelIndexTKey returns a TKey corresponding to a label.

func PutLabelIndex 

func PutLabelIndex(d dvid.Data, v dvid.VersionID, label uint64, idx *labels.Index) error

PutLabelIndex persists a label index data for a given data instance and version. If the given index is nil, the index is deleted. Concurrency-safe and supports caching.

func ZeroBytes 

func ZeroBytes() []byte

ZeroBytes returns a slice of bytes that represents the zero label.

Types

type Data 

type Data struct {
	*imageblk.Data
	datastore.Updater

	// The maximum label id found in each version of this instance.
	// Can be unset if no new label was added at that version, in which case
	// you must traverse DAG to find max label of parent.
	MaxLabel map[dvid.VersionID]uint64

	// The maximum label for this instance in the entire repo.  This allows us to do
	// conflict-free merges without any relabeling.  Note that relabeling (rebasing)
	// is required if we move data between repos, e.g., when pushing remote nodes,
	// since we have no control over which labels were created remotely and there
	// could be conflicts between the local and remote repos.  When mutations only
	// occur within a single repo, however, this atomic label allows us to prevent
	// conflict across all versions within this repo.
	MaxRepoLabel uint64

	// True if sparse volumes (split, merge, sparse volume optimized GET) are supported
	// for this data instance.  (Default true)
	IndexedLabels bool

	// Maximum down-resolution level supported.  Each down-res level is 2x scope of
	// the higher level.
	MaxDownresLevel uint8
	// contains filtered or unexported fields
}

Data of labelmap type is an extended form of imageblk Data

func GetByDataUUID 

func GetByDataUUID(dataUUID dvid.UUID) (*Data, error)

GetByDataUUID returns a pointer to labelmap data given a data UUID. Returns error if not found.

func GetByUUIDName 

func GetByUUIDName(uuid dvid.UUID, name dvid.InstanceName) (*Data, error)

GetByUUIDName returns a pointer to labelmap data given a UUID and data name.

func GetByVersionName 

func GetByVersionName(v dvid.VersionID, name dvid.InstanceName) (*Data, error)

GetByVersionName returns a pointer to labelmap data given a version and data name.

func NewData 

func NewData(uuid dvid.UUID, id dvid.InstanceID, name dvid.InstanceName, c dvid.Config) (*Data, error)

NewData returns a pointer to labelmap data.

func (*Data) AnyScaleUpdating 

func (d *Data) AnyScaleUpdating() bool

func (*Data) BlockSize 

func (d *Data) BlockSize() dvid.Point

func (*Data) CleaveLabel 

func (d *Data) CleaveLabel(v dvid.VersionID, label uint64, info dvid.ModInfo, r io.ReadCloser) (cleaveLabel uint64, err error)

CleaveLabel cleaves a label given supervoxels to be cleaved. Requires JSON in request body using the following format: 

[supervoxel1, supervoxel2, ...]

Each element of the JSON array is a supervoxel to be cleaved from the label and either given a new label or the one optionally supplied via the "cleavelabel" query string. A cleave label can be specified via the "toLabel" parameter, which if 0 will have an automatic label ID selected for the cleaved body.

func (*Data) CopyPropertiesFrom 

func (d *Data) CopyPropertiesFrom(src datastore.DataService, fs storage.FilterSpec) error

CopyPropertiesFrom copies the data instance-specific properties from a given data instance into the receiver's properties. Fulfills the datastore.PropertyCopier interface.

func (*Data) CreateCompositeChunk 

func (d *Data) CreateCompositeChunk(chunk *storage.Chunk) error

CreateCompositeChunk processes each chunk of labels and grayscale data, saving the composited result into an rgba8. Only some multiple of the # of CPU cores can be used for chunk handling before it waits for chunk processing to abate via the buffered server.HandlerToken channel.

func (*Data) DoRPC 

func (d *Data) DoRPC(req datastore.Request, reply *datastore.Response) error

DoRPC acts as a switchboard for RPC commands.

func (*Data) Equals 

func (d *Data) Equals(d2 *Data) bool

func (*Data) Extents 

func (d *Data) Extents() *dvid.Extents

func (*Data) FoundSparseVol 

func (d *Data) FoundSparseVol(ctx *datastore.VersionedCtx, label uint64, bounds dvid.Bounds, isSupervoxel bool) (bool, error)

FoundSparseVol returns true if a sparse volume is found for the given label within the given bounds.

func (*Data) GetImage 

func (d *Data) GetImage(v dvid.VersionID, vox *Labels, supervoxels bool, scale uint8, roiname dvid.InstanceName) (*dvid.Image, error)

GetImage retrieves a 2d image from a version node given a geometry of labels.

func (*Data) GetLabelAtPoint 

func (d *Data) GetLabelAtPoint(v dvid.VersionID, pt dvid.Point) (uint64, error)

GetLabelAtPoint returns the 64-bit unsigned int label for a given point.

func (*Data) GetLabelAtScaledPoint 

func (d *Data) GetLabelAtScaledPoint(v dvid.VersionID, pt dvid.Point, scale uint8, supervoxels bool) (uint64, error)

GetLabelAtScaledPoint returns the 64-bit unsigned int label for a given point.

func (*Data) GetLabelBlock 

func (d *Data) GetLabelBlock(v dvid.VersionID, bcoord dvid.ChunkPoint3d, scale uint8) (*labels.Block, error)

GetLabelBlock returns a compressed label Block of the given block coordinate.

func (*Data) GetLabelBytes 

func (d *Data) GetLabelBytes(v dvid.VersionID, bcoord dvid.ChunkPoint3d) ([]byte, error)

GetLabelBytes returns a block of hi-res (body) labels (scale 0) in packed little-endian uint64 format

func (*Data) GetLabelBytesWithScale 

func (d *Data) GetLabelBytesWithScale(v dvid.VersionID, bcoord dvid.ChunkPoint3d, scale uint8, supervoxels bool) ([]byte, error)

GetLabelBytesWithScale returns a block of labels at given scale in packed little-endian uint64 format.

func (*Data) GetLabels 

func (d *Data) GetLabels(v dvid.VersionID, supervoxels bool, scale uint8, vox *Labels, r *imageblk.ROI) error

GetLabels copies labels from the storage engine to Labels, a requested subvolume or 2d image. If supervoxels is true, the returned labels are not mapped but are the raw supervoxels.

func (*Data) GetMappedLabels 

func (d *Data) GetMappedLabels(v dvid.VersionID, supervoxels []uint64) ([]uint64, error)

GetMappedLabels returns an array of mapped labels, which could be the same as the passed slice, for the given version of the data instance.

func (*Data) GetMaxDownresLevel 

func (d *Data) GetMaxDownresLevel() uint8

GetMaxDownresLevel returns the number of down-res levels, where level 0 = high-resolution and each subsequent level has one-half the resolution.

func (*Data) GetSparseCoarseVol 

func (d *Data) GetSparseCoarseVol(ctx *datastore.VersionedCtx, label uint64, bounds dvid.Bounds, isSupervoxel bool) ([]byte, error)

GetSparseCoarseVol returns an encoded sparse volume given a label. This will return nil slice if the given label was not found. The encoding has the following format where integers are little endian and blocks are returned in sorted ZYX order (small Z first): 

		byte     Set to 0
		uint8    Number of dimensions
		uint8    Dimension of run (typically 0 = X)
		byte     Reserved (to be used later)
		uint32    # Blocks [TODO.  0 for now]
		uint32    # Spans
		Repeating unit of:
    		int32   Block coordinate of run start (dimension 0)
    		int32   Block coordinate of run start (dimension 1)
    		int32   Block coordinate of run start (dimension 2)
    		int32   Length of run

func (*Data) GetSupervoxelAtPoint 

func (d *Data) GetSupervoxelAtPoint(v dvid.VersionID, pt dvid.Point) (uint64, error)

GetSupervoxelAtPoint returns the 64-bit unsigned int supervoxel id for a given point.

func (*Data) GetSupervoxels added in v0.8.18 

func (d *Data) GetSupervoxels(v dvid.VersionID, label uint64) (labels.Set, error)

GetSupervoxels returns the set of supervoxel ids that compose the given label

func (*Data) GetVolume 

func (d *Data) GetVolume(v dvid.VersionID, vox *Labels, supervoxels bool, scale uint8, roiname dvid.InstanceName) ([]byte, error)

GetVolume retrieves a n-d volume from a version node given a geometry of labels.

func (*Data) GobDecode 

func (d *Data) GobDecode(b []byte) error

func (*Data) GobEncode 

func (d *Data) GobEncode() ([]byte, error)

func (*Data) InitDataHandlers 

func (d *Data) InitDataHandlers() error

InitDataHandlers launches goroutines to handle each labelmap instance's syncs.

func (*Data) Initialize 

func (d *Data) Initialize()

Initialize makes sure index caching is initialized if cache size is specified in the server configuration.

func (*Data) LoadImages 

func (d *Data) LoadImages(v dvid.VersionID, offset dvid.Point, filenames []string) error

LoadImages bulk loads images using different techniques if it is a multidimensional file like HDF5 or a sequence of PNG/JPG/TIF images.

func (*Data) LoadMutable 

func (d *Data) LoadMutable(root dvid.VersionID, storedVersion, expectedVersion uint64) (bool, error)

LoadMutable loads mutable properties of label volumes like the maximum labels for each version. Note that we load these max labels from key-value pairs rather than data instance properties persistence, because in the case of a crash, the actually stored repo data structure may be out-of-date compared to the guaranteed up-to-date key-value pairs for max labels.

func (*Data) MarshalJSON 

func (d *Data) MarshalJSON() ([]byte, error)

func (*Data) MarshalJSONExtents 

func (d *Data) MarshalJSONExtents(ctx *datastore.VersionedCtx) ([]byte, error)

func (*Data) MergeLabels 

func (d *Data) MergeLabels(v dvid.VersionID, op labels.MergeOp, info dvid.ModInfo) error

MergeLabels handles merging of any number of labels throughout the various label data structures. It assumes that the merges aren't cascading, e.g., there is no attempt to merge label 3 into 4 and also 4 into 5. The caller should have flattened the merges. TODO: Provide some indication that subset of labels are under evolution, returning 

an "unavailable" status or 203 for non-authoritative response.  This might not be
feasible for clustered DVID front-ends due to coordination issues.

EVENTS

labels.MergeStartEvent occurs at very start of merge and transmits labels.DeltaMergeStart struct.

labels.MergeBlockEvent occurs for every block of a merged label and transmits labels.DeltaMerge struct.

labels.MergeEndEvent occurs at end of merge and transmits labels.DeltaMergeEnd struct.

func (*Data) NewLabel 

func (d *Data) NewLabel(v dvid.VersionID) (uint64, error)

NewLabel returns a new label for the given version.

func (*Data) NewLabels 

func (d *Data) NewLabels(geom dvid.Geometry, img interface{}) (*Labels, error)

NewLabels returns labelmap Labels, a representation of externally usable subvolume or slice data, given some geometry and optional image data. If img is passed in, the function will initialize Voxels with data from the image. Otherwise, it will allocate a zero buffer of appropriate size.

func (*Data) PushData 

func (d *Data) PushData(p *datastore.PushSession) error

PushData pushes labelmap data to a remote DVID.

func (*Data) PutLabels 

func (d *Data) PutLabels(v dvid.VersionID, subvol *dvid.Subvolume, data []byte, roiname dvid.InstanceName, mutate bool) error

PutLabels persists voxels from a subvolume into the storage engine. This involves transforming a supplied volume of uint64 with known geometry into many labels.Block that tiles the subvolume. Messages are sent to subscribers for ingest events.

func (*Data) ReadChunk 

func (d *Data) ReadChunk(chunk *storage.Chunk) error

ReadChunk reads a chunk of data as part of a mapped operation. Only some multiple of the # of CPU cores can be used for chunk handling before it waits for chunk processing to abate via the buffered server.HandlerToken channel.

func (*Data) ReceiveBlocks 

func (d *Data) ReceiveBlocks(ctx *datastore.VersionedCtx, r io.ReadCloser, scale uint8, downscale bool, compression string, indexing bool) error

ReceiveBlocks stores a slice of bytes corresponding to specified blocks

func (*Data) ScaleUpdating 

func (d *Data) ScaleUpdating(scale uint8) bool

func (*Data) SendBlocks 

func (d *Data) SendBlocks(ctx *datastore.VersionedCtx, w http.ResponseWriter, supervoxels bool, scale uint8, subvol *dvid.Subvolume, compression string) error

SendBlocks returns a series of blocks covering the given block-aligned subvolume.

func (*Data) ServeHTTP 

func (d *Data) ServeHTTP(uuid dvid.UUID, ctx *datastore.VersionedCtx, w http.ResponseWriter, r *http.Request)

ServeHTTP handles all incoming HTTP requests for this data.

func (*Data) SetResolution 

func (d *Data) SetResolution(uuid dvid.UUID, jsonBytes []byte) error

SetResolution loads JSON data giving Resolution.

func (*Data) Shutdown 

func (d *Data) Shutdown(wg *sync.WaitGroup)

Shutdown terminates blocks until syncs are done then terminates background goroutines processing data.

func (*Data) SplitLabels 

func (d *Data) SplitLabels(v dvid.VersionID, fromLabel uint64, r io.ReadCloser, info dvid.ModInfo) (toLabel uint64, err error)

SplitLabels splits a portion of a label's voxels into a given split label or, if the given split label is 0, a new label, which is returned. The input is a binary sparse volume and should preferably be the smaller portion of a labeled region. In other words, the caller should chose to submit for relabeling the smaller portion of any split. It is assumed that the given split voxels are within the fromLabel set of voxels and will generate unspecified behavior if this is not the case.

func (*Data) SplitSupervoxel 

func (d *Data) SplitSupervoxel(v dvid.VersionID, svlabel uint64, r io.ReadCloser, info dvid.ModInfo) (splitSupervoxel, remainSupervoxel uint64, err error)

SplitSupervoxel splits a portion of a supervoxel's voxels into two new labels. The input is a binary sparse volume and should be totally contained by the given supervoxel. The first returned label is assigned to the split voxels while the second returned label is assigned to the remainder voxels.

func (*Data) StartScaleUpdate 

func (d *Data) StartScaleUpdate(scale uint8)

func (*Data) StopScaleUpdate 

func (d *Data) StopScaleUpdate(scale uint8)

func (*Data) StoreDownres 

func (d *Data) StoreDownres(v dvid.VersionID, hiresScale uint8, hires downres.BlockMap) (downres.BlockMap, error)

StoreDownres computes a downscale representation of a set of mutated blocks.

func (*Data) WriteSparseCoarseVols 

func (d *Data) WriteSparseCoarseVols(ctx *datastore.VersionedCtx, w io.Writer, begLabel, endLabel uint64, bounds dvid.Bounds) error

WriteSparseCoarseVols returns a stream of sparse volumes with blocks of the given label in encoded RLE format: 

	uint64   label
	<coarse sparse vol as given below>

	uint64   label
	<coarse sparse vol as given below>

	...

The coarse sparse vol has the following format where integers are little endian and the order
of data is exactly as specified below:

	int32    # Spans
	Repeating unit of:
		int32   Block coordinate of run start (dimension 0)
		int32   Block coordinate of run start (dimension 1)
		int32   Block coordinate of run start (dimension 2)
		int32   Length of run

type IngestedBlock 

type IngestedBlock struct {
	MutID  uint64
	BCoord dvid.IZYXString
	Data   *labels.Block
}

IngestedBlock is the unit of delta for a IngestBlockEvent.

type Labels 

type Labels struct {
	*imageblk.Voxels
}

Labels are voxels that have uint64 labels.

func (*Labels) ComputeTransform 

func (v *Labels) ComputeTransform(tkey storage.TKey, blockSize dvid.Point) (blockBeg, dataBeg, dataEnd dvid.Point, err error)

ComputeTransform determines the block coordinate and beginning + ending voxel points for the data corresponding to the given Block.

func (*Labels) Interpolable 

func (l *Labels) Interpolable() bool

func (*Labels) String 

func (l *Labels) String() string

type MutatedBlock 

type MutatedBlock struct {
	MutID  uint64
	BCoord dvid.IZYXString
	Prev   *labels.Block
	Data   *labels.Block
}

MutatedBlock tracks previous and updated block data. It is the unit of delta for a MutateBlockEvent.

type SVMap 

type SVMap struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

SVMap is a version-aware supervoxel map that tries to be memory efficient and allows up to 256 versions per SVMap instance. Splits are also cached by version.

func (*SVMap) MappedLabel 

func (svm *SVMap) MappedLabel(v dvid.VersionID, label uint64) (uint64, bool)

MappedLabel returns the mapped label and a boolean: true if a mapping was found and false if none was found. For faster mapping, large scale transformations, e.g. block-level output, should not use this routine but work directly with mapLabel() doing locking and ancestry lookup outside loops.

func (*SVMap) MappedLabels 

func (svm *SVMap) MappedLabels(v dvid.VersionID, supervoxels []uint64) ([]uint64, error)

MappedLabels returns an array of mapped labels, which could be the same as the passed slice.

func (*SVMap) SupervoxelSplitsJSON added in v0.8.18 

func (svm *SVMap) SupervoxelSplitsJSON(v dvid.VersionID) (string, error)

SupervoxelSplitsJSON returns a JSON string giving all the supervoxel splits from this version to the root.

type SparseVolFormat 

type SparseVolFormat uint8

SparseVolFormat indicates the type of encoding used for sparse volume representation. 

const (
	// FormatLegacyRLE is Legacy RLE encoding with header that gives # spans.
	FormatLegacyRLE SparseVolFormat = iota

	// FormatStreamingRLE specifies Streaming RLE encoding
	FormatStreamingRLE

	// FormatBinaryBlocks specifies a streaming set of binary Blocks
	FormatBinaryBlocks
)

type Type 

type Type struct {
	imageblk.Type
}

Type uses imageblk data type by composition.

func (*Type) Help 

func (dtype *Type) Help() string

func (*Type) NewDataService 

func (dtype *Type) NewDataService(uuid dvid.UUID, id dvid.InstanceID, name dvid.InstanceName, c dvid.Config) (datastore.DataService, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.