labelarray

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 18, 2023 License: BSD-3-Clause Imports: 29 Imported by: 2

Documentation

Overview

Package labelarray handles both volumes of label data as well as indexing to quickly find and generate sparse volumes of any particular label.

Index

Constants

View Source 
const (
	Version  = "0.1"
	RepoURL  = "github.com/janelia-flyem/dvid/datatype/labelarray"
	TypeName = "labelarray"
)
View Source 
const KVWriteSize = 500

KVWriteSize is the # of key-value pairs we will write as one atomic batch write.

Variables

View Source 
var (
	DefaultBlockSize int32   = 64
	DefaultRes       float32 = imageblk.DefaultRes
	DefaultUnits             = imageblk.DefaultUnits
)

Functions

func ChangeLabelIndex 

func ChangeLabelIndex(d dvid.Data, v dvid.VersionID, label uint64, delta blockDiffMap) error

ChangeLabelIndex applies changes to a label's index and then stores the result. Concurrency-safe and supports caching.

func DecodeBlockTKey 

func DecodeBlockTKey(tk storage.TKey) (scale uint8, idx *dvid.IndexZYX, err error)

DecodeBlockTKey returns a spatial index from a label block key. TODO: Extend this when necessary to allow any form of spatial indexing like CZYX.

func DecodeLabelIndexTKey 

func DecodeLabelIndexTKey(tk storage.TKey) (label uint64, err error)

DecodeLabelIndexTKey parses a TKey and returns the corresponding label.

func DeleteLabelIndex 

func DeleteLabelIndex(d dvid.Data, v dvid.VersionID, label uint64) error

DeleteLabelIndex deletes the index for a given label.

func EncodeFormat 

func EncodeFormat() dvid.DataValues

func GetBinaryData 

func GetBinaryData(compression string, in io.ReadCloser, estsize int64) ([]byte, error)

GetBinaryData returns label data from a potentially compressed ("lz4", "gzip") reader.

func NewBlockTKey 

func NewBlockTKey(scale uint8, idx dvid.Index) storage.TKey

NewBlockTKey returns a TKey for a label block, which is a slice suitable for lexicographical ordering on zyx coordinates.

func NewBlockTKeyByCoord 

func NewBlockTKeyByCoord(scale uint8, izyx dvid.IZYXString) storage.TKey

NewBlockTKeyByCoord returns a TKey for a block coord in string format.

func NewLabelIndexTKey 

func NewLabelIndexTKey(label uint64) storage.TKey

NewLabelIndexTKey returns a TKey corresponding to a label. Value will hold block coords that contain the label.

func SetLabelIndex 

func SetLabelIndex(d dvid.Data, v dvid.VersionID, label uint64, m *Meta) error

SetLabelIndex sets label index data from storage for a given data instance and version. If the given Meta is nil, the label index is deleted. Concurrency-safe and supports caching.

func ZeroBytes 

func ZeroBytes() []byte

ZeroBytes returns a slice of bytes that represents the zero label.

Types

type Data 

type Data struct {
	*imageblk.Data
	datastore.Updater

	// The maximum label id found in each version of this instance.
	// Can be unset if no new label was added at that version, in which case
	// you must traverse DAG to find max label of parent.
	MaxLabel map[dvid.VersionID]uint64

	// The maximum label for this instance in the entire repo.  This allows us to do
	// conflict-free merges without any relabeling.  Note that relabeling (rebasing)
	// is required if we move data between repos, e.g., when pushing remote nodes,
	// since we have no control over which labels were created remotely and there
	// could be conflicts between the local and remote repos.  When mutations only
	// occur within a single repo, however, this atomic label allows us to prevent
	// conflict across all versions within this repo.
	MaxRepoLabel uint64

	// True if sparse volumes (split, merge, sparse volume optimized GET) are supported
	// for this data instance.  (Default true)
	IndexedLabels bool

	// True if we keep track of # voxels per label.  (Default true)
	CountLabels bool

	// Maximum down-resolution level supported.  Each down-res level is 2x scope of
	// the higher level.
	MaxDownresLevel uint8
	// contains filtered or unexported fields
}

Data of labelarray type is an extended form of imageblk Data

func GetByDataUUID 

func GetByDataUUID(dataUUID dvid.UUID) (*Data, error)

GetByDataUUID returns a pointer to labelarray data given a data UUID. Returns error if not found.

func GetByUUIDName 

func GetByUUIDName(uuid dvid.UUID, name dvid.InstanceName) (*Data, error)

GetByUUIDName returns a pointer to labelarray data given a UUID and data name.

func GetByVersionName 

func GetByVersionName(v dvid.VersionID, name dvid.InstanceName) (*Data, error)

GetByVersionName returns a pointer to labelarray data given a version and data name.

func NewData 

func NewData(uuid dvid.UUID, id dvid.InstanceID, name dvid.InstanceName, c dvid.Config) (*Data, error)

NewData returns a pointer to labelarray data.

func (*Data) AnyScaleUpdating 

func (d *Data) AnyScaleUpdating() bool

func (*Data) BlockSize 

func (d *Data) BlockSize() dvid.Point

func (*Data) CopyPropertiesFrom 

func (d *Data) CopyPropertiesFrom(src datastore.DataService, fs storage.FilterSpec) error

CopyPropertiesFrom copies the data instance-specific properties from a given data instance into the receiver's properties. Fulfills the datastore.PropertyCopier interface.

func (*Data) CreateComposite 

func (d *Data) CreateComposite(request datastore.Request, reply *datastore.Response) error

CreateComposite creates a new rgba8 image by combining hash of labels + the grayscale

func (*Data) CreateCompositeChunk 

func (d *Data) CreateCompositeChunk(chunk *storage.Chunk) error

CreateCompositeChunk processes each chunk of labels and grayscale data, saving the composited result into an rgba8. Only some multiple of the # of CPU cores can be used for chunk handling before it waits for chunk processing to abate via the buffered server.HandlerToken channel.

func (*Data) DescribeTKeyClass 

func (d *Data) DescribeTKeyClass(tkc storage.TKeyClass) string

DescribeTKeyClass returns a string explanation of what a particular TKeyClass is used for. Implements the datastore.TKeyClassDescriber interface.

func (*Data) DoRPC 

func (d *Data) DoRPC(req datastore.Request, reply *datastore.Response) error

DoRPC acts as a switchboard for RPC commands.

func (*Data) Equals 

func (d *Data) Equals(d2 *Data) bool

func (*Data) Extents 

func (d *Data) Extents() *dvid.Extents

func (*Data) FoundSparseVol 

func (d *Data) FoundSparseVol(ctx *datastore.VersionedCtx, label uint64, bounds dvid.Bounds) (bool, error)

FoundSparseVol returns true if a sparse volume is found for the given label within the given bounds.

func (*Data) GetImage 

func (d *Data) GetImage(v dvid.VersionID, vox *Labels, scale uint8, roiname dvid.InstanceName) (*dvid.Image, error)

GetImage retrieves a 2d image from a version node given a geometry of labels.

func (*Data) GetLabelAtPoint 

func (d *Data) GetLabelAtPoint(v dvid.VersionID, pt dvid.Point) (uint64, error)

GetLabelAtPoint returns the 64-bit unsigned int label for a given point.

func (*Data) GetLabelAtScaledPoint 

func (d *Data) GetLabelAtScaledPoint(v dvid.VersionID, pt dvid.Point, scale uint8) (uint64, error)

GetLabelAtScaledPoint returns the 64-bit unsigned int label for a given point.

func (*Data) GetLabelBlock 

func (d *Data) GetLabelBlock(v dvid.VersionID, bcoord dvid.ChunkPoint3d, scale uint8) (*labels.Block, error)

GetLabelBlock returns a compressed label Block of the given block coordinate.

func (*Data) GetLabelBytes 

func (d *Data) GetLabelBytes(v dvid.VersionID, bcoord dvid.ChunkPoint3d) ([]byte, error)

GetLabelBytes returns a block of hi-res labels (scale 0) in packed little-endian uint64 format

func (*Data) GetLabelBytesAtScaledPoint 

func (d *Data) GetLabelBytesAtScaledPoint(v dvid.VersionID, pt dvid.Point, scale uint8) ([]byte, error)

GetLabelBytesAtScaledPoint returns the 8 byte slice corresponding to a 64-bit label at a point.

func (*Data) GetLabelBytesWithScale 

func (d *Data) GetLabelBytesWithScale(v dvid.VersionID, bcoord dvid.ChunkPoint3d, scale uint8) ([]byte, error)

GetLabelBytesWithScale returns a block of labels in packed little-endian uint64 format at the given scale.

func (*Data) GetLabels 

func (d *Data) GetLabels(v dvid.VersionID, scale uint8, vox *Labels, r *imageblk.ROI) error

GetLabels copies labels from the storage engine to Labels, a requested subvolume or 2d image.

func (*Data) GetMaxDownresLevel 

func (d *Data) GetMaxDownresLevel() uint8

GetMaxDownresLevel returns the number of down-res levels, where level 0 = high-resolution and each subsequent level has one-half the resolution.

func (*Data) GetSparseCoarseVol 

func (d *Data) GetSparseCoarseVol(ctx *datastore.VersionedCtx, label uint64, bounds dvid.Bounds) ([]byte, error)

GetSparseCoarseVol returns an encoded sparse volume given a label. This will return nil slice if the given label was not found. The encoding has the following format where integers are little endian: 

		byte     Set to 0
		uint8    Number of dimensions
		uint8    Dimension of run (typically 0 = X)
		byte     Reserved (to be used later)
		uint32    # Blocks [TODO.  0 for now]
		uint32    # Spans
		Repeating unit of:
    		int32   Block coordinate of run start (dimension 0)
    		int32   Block coordinate of run start (dimension 1)
    		int32   Block coordinate of run start (dimension 2)
    		int32   Length of run

func (*Data) GetVolume 

func (d *Data) GetVolume(v dvid.VersionID, vox *Labels, scale uint8, roiname dvid.InstanceName) ([]byte, error)

GetVolume retrieves a n-d volume from a version node given a geometry of labels.

func (*Data) GobDecode 

func (d *Data) GobDecode(b []byte) error

func (*Data) GobEncode 

func (d *Data) GobEncode() ([]byte, error)

func (*Data) InitDataHandlers 

func (d *Data) InitDataHandlers() error

InitDataHandlers launches goroutines to handle each labelarray instance's syncs.

func (*Data) Initialize 

func (d *Data) Initialize()

Initialize makes sure index caching is initialized if cache size is specified in the server configuration.

func (*Data) LoadImages 

func (d *Data) LoadImages(v dvid.VersionID, offset dvid.Point, filenames []string) error

LoadImages bulk loads images using different techniques if it is a multidimensional file like HDF5 or a sequence of PNG/JPG/TIF images.

func (*Data) LoadMutable 

func (d *Data) LoadMutable(root dvid.VersionID, storedVersion, expectedVersion uint64) (bool, error)

LoadMutable loads mutable properties of label volumes like the maximum labels for each version. Note that we load these max labels from key-value pairs rather than data instance properties persistence, because in the case of a crash, the actually stored repo data structure may be out-of-date compared to the guaranteed up-to-date key-value pairs for max labels.

func (*Data) MarshalJSON 

func (d *Data) MarshalJSON() ([]byte, error)

func (*Data) MarshalJSONExtents 

func (d *Data) MarshalJSONExtents(ctx *datastore.VersionedCtx) ([]byte, error)

func (*Data) MergeLabels 

func (d *Data) MergeLabels(v dvid.VersionID, op labels.MergeOp) error

MergeLabels handles merging of any number of labels throughout the various label data structures. It assumes that the merges aren't cascading, e.g., there is no attempt to merge label 3 into 4 and also 4 into 5. The caller should have flattened the merges. TODO: Provide some indication that subset of labels are under evolution, returning 

an "unavailable" status or 203 for non-authoritative response.  This might not be
feasible for clustered DVID front-ends due to coordination issues.

EVENTS

labels.MergeStartEvent occurs at very start of merge and transmits labels.DeltaMergeStart struct.

labels.MergeBlockEvent occurs for every block of a merged label and transmits labels.DeltaMerge struct.

labels.MergeEndEvent occurs at end of merge and transmits labels.DeltaMergeEnd struct.

func (*Data) NewLabel 

func (d *Data) NewLabel(v dvid.VersionID) (uint64, error)

NewLabel returns a new label for the given version.

func (*Data) NewLabels 

func (d *Data) NewLabels(geom dvid.Geometry, img interface{}) (*Labels, error)

NewLabels returns labelarray Labels, a representation of externally usable subvolume or slice data, given some geometry and optional image data. If img is passed in, the function will initialize Voxels with data from the image. Otherwise, it will allocate a zero buffer of appropriate size.

func (*Data) PushData 

func (d *Data) PushData(p *datastore.PushSession) error

PushData pushes labelarray data to a remote DVID.

func (*Data) PutLabels 

func (d *Data) PutLabels(v dvid.VersionID, subvol *dvid.Subvolume, data []byte, roiname dvid.InstanceName, mutate bool) error

PutLabels persists voxels from a subvolume into the storage engine. This involves transforming a supplied volume of uint64 with known geometry into many labels.Block that tiles the subvolume. Messages are sent to subscribers for ingest or mutate events.

func (*Data) ReadChunk 

func (d *Data) ReadChunk(chunk *storage.Chunk) error

ReadChunk reads a chunk of data as part of a mapped operation. Only some multiple of the # of CPU cores can be used for chunk handling before it waits for chunk processing to abate via the buffered server.HandlerToken channel.

func (*Data) ReceiveBlocks 

func (d *Data) ReceiveBlocks(ctx *datastore.VersionedCtx, r io.ReadCloser, scale uint8, downscale bool, compression string) error

ReceiveBlocks stores a slice of bytes corresponding to specified blocks

func (*Data) ScaleUpdating 

func (d *Data) ScaleUpdating(scale uint8) bool

func (*Data) SendBlocks 

func (d *Data) SendBlocks(ctx *datastore.VersionedCtx, w http.ResponseWriter, scale uint8, subvol *dvid.Subvolume, compression string) error

SendBlocks returns a series of blocks covering the given block-aligned subvolume.

func (*Data) ServeHTTP 

func (d *Data) ServeHTTP(uuid dvid.UUID, ctx *datastore.VersionedCtx, w http.ResponseWriter, r *http.Request) (activity map[string]interface{})

ServeHTTP handles all incoming HTTP requests for this data.

func (*Data) SetResolution 

func (d *Data) SetResolution(uuid dvid.UUID, jsonBytes []byte) error

SetResolution loads JSON data giving Resolution.

func (*Data) Shutdown 

func (d *Data) Shutdown(wg *sync.WaitGroup)

Shutdown terminates blocks until syncs are done then terminates background goroutines processing data.

func (*Data) SplitCoarseLabels 

func (d *Data) SplitCoarseLabels(v dvid.VersionID, fromLabel, splitLabel uint64, r io.ReadCloser) (toLabel uint64, err error)

SplitCoarseLabels splits a portion of a label's voxels into a given split label or, if the given split label is 0, a new label, which is returned. The input is a binary sparse volume defined by block coordinates and should be the smaller portion of a labeled region-to-be-split.

EVENTS

labels.SplitStartEvent occurs at very start of split and transmits labels.DeltaSplitStart struct.

labels.SplitBlockEvent occurs for every block of a split label and transmits labels.DeltaSplit struct.

labels.SplitEndEvent occurs at end of split and transmits labels.DeltaSplitEnd struct.

func (*Data) SplitLabels 

func (d *Data) SplitLabels(v dvid.VersionID, fromLabel, splitLabel uint64, r io.ReadCloser) (toLabel uint64, err error)

SplitLabels splits a portion of a label's voxels into a given split label or, if the given split label is 0, a new label, which is returned. The input is a binary sparse volume and should preferably be the smaller portion of a labeled region. In other words, the caller should chose to submit for relabeling the smaller portion of any split. It is assumed that the given split voxels are within the fromLabel set of voxels and will generate unspecified behavior if this is not the case.

EVENTS

labels.SplitStartEvent occurs at very start of split and transmits labels.DeltaSplitStart struct.

labels.SplitBlockEvent occurs for every block of a split label and transmits labels.DeltaSplit struct.

labels.SplitEndEvent occurs at end of split and transmits labels.DeltaSplitEnd struct.

func (*Data) StartScaleUpdate 

func (d *Data) StartScaleUpdate(scale uint8)

func (*Data) StopScaleUpdate 

func (d *Data) StopScaleUpdate(scale uint8)

func (*Data) StoreDownres 

func (d *Data) StoreDownres(v dvid.VersionID, hiresScale uint8, hires downres.BlockMap) (downres.BlockMap, error)

func (*Data) WriteSparseCoarseVols 

func (d *Data) WriteSparseCoarseVols(ctx *datastore.VersionedCtx, w io.Writer, begLabel, endLabel uint64, bounds dvid.Bounds) error

WriteSparseCoarseVols returns a stream of sparse volumes with blocks of the given label in encoded RLE format: 

	uint64   label
	<coarse sparse vol as given below>

	uint64   label
	<coarse sparse vol as given below>

	...

The coarse sparse vol has the following format where integers are little endian and the order
of data is exactly as specified below:

	int32    # Spans
	Repeating unit of:
		int32   Block coordinate of run start (dimension 0)
		int32   Block coordinate of run start (dimension 1)
		int32   Block coordinate of run start (dimension 2)
		int32   Length of run

type IngestedBlock 

type IngestedBlock struct {
	MutID  uint64
	BCoord dvid.IZYXString
	Data   *labels.Block
}

IngestedBlock is the unit of delta for a IngestBlockEvent.

type Labels 

type Labels struct {
	*imageblk.Voxels
}

Labels are voxels that have uint64 labels.

func (*Labels) ComputeTransform 

func (v *Labels) ComputeTransform(tkey storage.TKey, blockSize dvid.Point) (blockBeg, dataBeg, dataEnd dvid.Point, err error)

ComputeTransform determines the block coordinate and beginning + ending voxel points for the data corresponding to the given Block.

func (*Labels) Interpolable 

func (l *Labels) Interpolable() bool

func (*Labels) String 

func (l *Labels) String() string

type Meta 

type Meta struct {
	Voxels uint64         // Total # of voxels in label.
	Blocks dvid.IZYXSlice // Sorted block coordinates occupied by label.

	LastModTime time.Time
	LastModUser string
}

Meta gives a high-level overview of a label's voxels including a block index. Some properties are only used if the associated data instance has features enabled, e.g., size tracking.

func GetLabelIndex 

func GetLabelIndex(d dvid.Data, v dvid.VersionID, label uint64) (*Meta, error)

GetLabelIndex gets label index data from storage for a given data instance and version. Concurrency-safe access and supports caching.

func GetLabelProcessedIndex 

func GetLabelProcessedIndex(d dvid.Data, v dvid.VersionID, label uint64, scale uint8, bounds dvid.Bounds) (*Meta, error)

GetLabelProcessedIndex gets label index data from storage for a given data instance and version with scaling and/or bounds. Concurrency-safe access and supports caching. If scale > 0, then the number of voxels is set to zero since an accurate count is not available without more processing.

func GetMappedLabelIndex 

func GetMappedLabelIndex(d dvid.Data, v dvid.VersionID, label uint64, scale uint8, bounds dvid.Bounds) (*Meta, labels.Set, error)

GetMappedLabelIndex gets index data for all labels potentially merged to the given label, allowing scaling and bounds. If bounds are used, the returned Meta will not include a voxel count. Concurrency-safe access and supports caching.

func GetMappedLabelSetIndex 

func GetMappedLabelSetIndex(d dvid.Data, v dvid.VersionID, lbls labels.Set, scale uint8, bounds dvid.Bounds) (*Meta, labels.Set, error)

GetMappedLabelSetIndex gets index data for all labels potentially merged to the given labels, allowing scaling and bounds. If bounds are used, the returned Meta will not include a voxel count. Concurrency-safe access and supports caching.

func (Meta) MarshalBinary 

func (m Meta) MarshalBinary() ([]byte, error)

MarshalBinary implements the encoding.BinaryMarshaler interface

func (*Meta) UnmarshalBinary 

func (m *Meta) UnmarshalBinary(b []byte) error

UnmarshalBinary implements the encoding.BinaryUnmarshaler interface.

type MutatedBlock 

type MutatedBlock struct {
	MutID  uint64
	BCoord dvid.IZYXString
	Prev   *labels.Block
	Data   *labels.Block
}

MutatedBlock tracks previous and updated block data. It is the unit of delta for a MutateBlockEvent.

type SparseVolFormat 

type SparseVolFormat uint8

SparseVolFormat indicates the type of encoding used for sparse volume representation. 

const (
	// FormatLegacyRLE is Legacy RLE encoding with header that gives # spans.
	FormatLegacyRLE SparseVolFormat = iota

	// FormatStreamingRLE specifies Streaming RLE encoding
	FormatStreamingRLE

	// FormatBinaryBlocks specifies a streaming set of binary Blocks
	FormatBinaryBlocks
)

type Type 

type Type struct {
	imageblk.Type
}

Type uses imageblk data type by composition.

func (*Type) Help 

func (dtype *Type) Help() string

func (*Type) NewDataService 

func (dtype *Type) NewDataService(uuid dvid.UUID, id dvid.InstanceID, name dvid.InstanceName, c dvid.Config) (datastore.DataService, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.