dvid: github.com/janelia-flyem/dvid/datatype/labelarray Index | Files

package labelarray

import "github.com/janelia-flyem/dvid/datatype/labelarray"

Package labelarray handles both volumes of label data as well as indexing to quickly find and generate sparse volumes of any particular label.

Index

Package Files

composite.go downres.go ingest.go keys.go labelarray.go labelidx.go mutate.go read.go sync.go write.go

Constants

const (
    Version  = "0.1"
    RepoURL  = "github.com/janelia-flyem/dvid/datatype/labelarray"
    TypeName = "labelarray"
)
const KVWriteSize = 500

KVWriteSize is the # of key-value pairs we will write as one atomic batch write.

Variables

var (
    DefaultBlockSize int32   = 64
    DefaultRes       float32 = imageblk.DefaultRes
    DefaultUnits             = imageblk.DefaultUnits
)

func ChangeLabelIndex Uses

func ChangeLabelIndex(d dvid.Data, v dvid.VersionID, label uint64, delta blockDiffMap) error

ChangeLabelIndex applies changes to a label's index and then stores the result. Concurrency-safe and supports caching.

func DecodeBlockTKey Uses

func DecodeBlockTKey(tk storage.TKey) (scale uint8, idx *dvid.IndexZYX, err error)

DecodeBlockTKey returns a spatial index from a label block key. TODO: Extend this when necessary to allow any form of spatial indexing like CZYX.

func DecodeLabelIndexTKey Uses

func DecodeLabelIndexTKey(tk storage.TKey) (label uint64, err error)

DecodeLabelIndexTKey parses a TKey and returns the corresponding label.

func DeleteLabelIndex Uses

func DeleteLabelIndex(d dvid.Data, v dvid.VersionID, label uint64) error

DeleteLabelIndex deletes the index for a given label.

func EncodeFormat Uses

func EncodeFormat() dvid.DataValues

func GetBinaryData Uses

func GetBinaryData(compression string, in io.ReadCloser, estsize int64) ([]byte, error)

GetBinaryData returns label data from a potentially compressed ("lz4", "gzip") reader.

func NewBlockTKey Uses

func NewBlockTKey(scale uint8, idx dvid.Index) storage.TKey

NewBlockTKey returns a TKey for a label block, which is a slice suitable for lexicographical ordering on zyx coordinates.

func NewBlockTKeyByCoord Uses

func NewBlockTKeyByCoord(scale uint8, izyx dvid.IZYXString) storage.TKey

NewBlockTKeyByCoord returns a TKey for a block coord in string format.

func NewLabelIndexTKey Uses

func NewLabelIndexTKey(label uint64) storage.TKey

NewLabelIndexTKey returns a TKey corresponding to a label. Value will hold block coords that contain the label.

func SetLabelIndex Uses

func SetLabelIndex(d dvid.Data, v dvid.VersionID, label uint64, m *Meta) error

SetLabelIndex sets label index data from storage for a given data instance and version. If the given Meta is nil, the label index is deleted. Concurrency-safe and supports caching.

func ZeroBytes Uses

func ZeroBytes() []byte

ZeroBytes returns a slice of bytes that represents the zero label.

type Data Uses

type Data struct {
    *imageblk.Data
    datastore.Updater

    // The maximum label id found in each version of this instance.
    // Can be unset if no new label was added at that version, in which case
    // you must traverse DAG to find max label of parent.
    MaxLabel map[dvid.VersionID]uint64

    // The maximum label for this instance in the entire repo.  This allows us to do
    // conflict-free merges without any relabeling.  Note that relabeling (rebasing)
    // is required if we move data between repos, e.g., when pushing remote nodes,
    // since we have no control over which labels were created remotely and there
    // could be conflicts between the local and remote repos.  When mutations only
    // occur within a single repo, however, this atomic label allows us to prevent
    // conflict across all versions within this repo.
    MaxRepoLabel uint64

    // True if sparse volumes (split, merge, sparse volume optimized GET) are supported
    // for this data instance.  (Default true)
    IndexedLabels bool

    // True if we keep track of # voxels per label.  (Default true)
    CountLabels bool

    // Maximum down-resolution level supported.  Each down-res level is 2x scope of
    // the higher level.
    MaxDownresLevel uint8
    // contains filtered or unexported fields
}

Data of labelarray type is an extended form of imageblk Data

func GetByDataUUID Uses

func GetByDataUUID(dataUUID dvid.UUID) (*Data, error)

GetByDataUUID returns a pointer to labelarray data given a data UUID. Returns error if not found.

func GetByUUIDName Uses

func GetByUUIDName(uuid dvid.UUID, name dvid.InstanceName) (*Data, error)

GetByUUIDName returns a pointer to labelarray data given a UUID and data name.

func GetByVersionName Uses

func GetByVersionName(v dvid.VersionID, name dvid.InstanceName) (*Data, error)

GetByVersionName returns a pointer to labelarray data given a version and data name.

func NewData Uses

func NewData(uuid dvid.UUID, id dvid.InstanceID, name dvid.InstanceName, c dvid.Config) (*Data, error)

NewData returns a pointer to labelarray data.

func (*Data) AnyScaleUpdating Uses

func (d *Data) AnyScaleUpdating() bool

func (*Data) BlockSize Uses

func (d *Data) BlockSize() dvid.Point

func (*Data) CopyPropertiesFrom Uses

func (d *Data) CopyPropertiesFrom(src datastore.DataService, fs storage.FilterSpec) error

CopyPropertiesFrom copies the data instance-specific properties from a given data instance into the receiver's properties. Fulfills the datastore.PropertyCopier interface.

func (*Data) CreateComposite Uses

func (d *Data) CreateComposite(request datastore.Request, reply *datastore.Response) error

CreateComposite creates a new rgba8 image by combining hash of labels + the grayscale

func (*Data) CreateCompositeChunk Uses

func (d *Data) CreateCompositeChunk(chunk *storage.Chunk) error

CreateCompositeChunk processes each chunk of labels and grayscale data, saving the composited result into an rgba8. Only some multiple of the # of CPU cores can be used for chunk handling before it waits for chunk processing to abate via the buffered server.HandlerToken channel.

func (*Data) DescribeTKeyClass Uses

func (d *Data) DescribeTKeyClass(tkc storage.TKeyClass) string

DescribeTKeyClass returns a string explanation of what a particular TKeyClass is used for. Implements the datastore.TKeyClassDescriber interface.

func (*Data) DoRPC Uses

func (d *Data) DoRPC(req datastore.Request, reply *datastore.Response) error

DoRPC acts as a switchboard for RPC commands.

func (*Data) Equals Uses

func (d *Data) Equals(d2 *Data) bool

func (*Data) Extents Uses

func (d *Data) Extents() *dvid.Extents

func (*Data) FoundSparseVol Uses

func (d *Data) FoundSparseVol(ctx *datastore.VersionedCtx, label uint64, bounds dvid.Bounds) (bool, error)

FoundSparseVol returns true if a sparse volume is found for the given label within the given bounds.

func (*Data) GetImage Uses

func (d *Data) GetImage(v dvid.VersionID, vox *Labels, scale uint8, roiname dvid.InstanceName) (*dvid.Image, error)

GetImage retrieves a 2d image from a version node given a geometry of labels.

func (*Data) GetLabelAtPoint Uses

func (d *Data) GetLabelAtPoint(v dvid.VersionID, pt dvid.Point) (uint64, error)

GetLabelAtPoint returns the 64-bit unsigned int label for a given point.

func (*Data) GetLabelAtScaledPoint Uses

func (d *Data) GetLabelAtScaledPoint(v dvid.VersionID, pt dvid.Point, scale uint8) (uint64, error)

GetLabelAtScaledPoint returns the 64-bit unsigned int label for a given point.

func (*Data) GetLabelBlock Uses

func (d *Data) GetLabelBlock(v dvid.VersionID, bcoord dvid.ChunkPoint3d, scale uint8) (*labels.Block, error)

GetLabelBlock returns a compressed label Block of the given block coordinate.

func (*Data) GetLabelBytes Uses

func (d *Data) GetLabelBytes(v dvid.VersionID, bcoord dvid.ChunkPoint3d) ([]byte, error)

GetLabelBytes returns a block of hi-res labels (scale 0) in packed little-endian uint64 format

func (*Data) GetLabelBytesAtScaledPoint Uses

func (d *Data) GetLabelBytesAtScaledPoint(v dvid.VersionID, pt dvid.Point, scale uint8) ([]byte, error)

GetLabelBytesAtScaledPoint returns the 8 byte slice corresponding to a 64-bit label at a point.

func (*Data) GetLabelBytesWithScale Uses

func (d *Data) GetLabelBytesWithScale(v dvid.VersionID, bcoord dvid.ChunkPoint3d, scale uint8) ([]byte, error)

GetLabelBytesWithScale returns a block of labels in packed little-endian uint64 format at the given scale.

func (*Data) GetLabels Uses

func (d *Data) GetLabels(v dvid.VersionID, scale uint8, vox *Labels, r *imageblk.ROI) error

GetLabels copies labels from the storage engine to Labels, a requested subvolume or 2d image.

func (*Data) GetMaxDownresLevel Uses

func (d *Data) GetMaxDownresLevel() uint8

GetMaxDownresLevel returns the number of down-res levels, where level 0 = high-resolution and each subsequent level has one-half the resolution.

func (*Data) GetSparseCoarseVol Uses

func (d *Data) GetSparseCoarseVol(ctx *datastore.VersionedCtx, label uint64, bounds dvid.Bounds) ([]byte, error)

GetSparseCoarseVol returns an encoded sparse volume given a label. This will return nil slice if the given label was not found. The encoding has the following format where integers are little endian:

		byte     Set to 0
		uint8    Number of dimensions
		uint8    Dimension of run (typically 0 = X)
		byte     Reserved (to be used later)
		uint32    # Blocks [TODO.  0 for now]
		uint32    # Spans
		Repeating unit of:
    		int32   Block coordinate of run start (dimension 0)
    		int32   Block coordinate of run start (dimension 1)
    		int32   Block coordinate of run start (dimension 2)
    		int32   Length of run

func (*Data) GetVolume Uses

func (d *Data) GetVolume(v dvid.VersionID, vox *Labels, scale uint8, roiname dvid.InstanceName) ([]byte, error)

GetVolume retrieves a n-d volume from a version node given a geometry of labels.

func (*Data) GobDecode Uses

func (d *Data) GobDecode(b []byte) error

func (*Data) GobEncode Uses

func (d *Data) GobEncode() ([]byte, error)

func (*Data) InitDataHandlers Uses

func (d *Data) InitDataHandlers() error

InitDataHandlers launches goroutines to handle each labelarray instance's syncs.

func (*Data) Initialize Uses

func (d *Data) Initialize()

Initialize makes sure index caching is initialized if cache size is specified in the server configuration.

func (*Data) LoadImages Uses

func (d *Data) LoadImages(v dvid.VersionID, offset dvid.Point, filenames []string) error

LoadImages bulk loads images using different techniques if it is a multidimensional file like HDF5 or a sequence of PNG/JPG/TIF images.

func (*Data) LoadMutable Uses

func (d *Data) LoadMutable(root dvid.VersionID, storedVersion, expectedVersion uint64) (bool, error)

LoadMutable loads mutable properties of label volumes like the maximum labels for each version. Note that we load these max labels from key-value pairs rather than data instance properties persistence, because in the case of a crash, the actually stored repo data structure may be out-of-date compared to the guaranteed up-to-date key-value pairs for max labels.

func (*Data) MarshalJSON Uses

func (d *Data) MarshalJSON() ([]byte, error)

func (*Data) MarshalJSONExtents Uses

func (d *Data) MarshalJSONExtents(ctx *datastore.VersionedCtx) ([]byte, error)

func (*Data) MergeLabels Uses

func (d *Data) MergeLabels(v dvid.VersionID, op labels.MergeOp) error

MergeLabels handles merging of any number of labels throughout the various label data structures. It assumes that the merges aren't cascading, e.g., there is no attempt to merge label 3 into 4 and also 4 into 5. The caller should have flattened the merges. TODO: Provide some indication that subset of labels are under evolution, returning

an "unavailable" status or 203 for non-authoritative response.  This might not be
feasible for clustered DVID front-ends due to coordination issues.

EVENTS

labels.MergeStartEvent occurs at very start of merge and transmits labels.DeltaMergeStart struct.

labels.MergeBlockEvent occurs for every block of a merged label and transmits labels.DeltaMerge struct.

labels.MergeEndEvent occurs at end of merge and transmits labels.DeltaMergeEnd struct.

func (*Data) NewLabel Uses

func (d *Data) NewLabel(v dvid.VersionID) (uint64, error)

NewLabel returns a new label for the given version.

func (*Data) NewLabels Uses

func (d *Data) NewLabels(geom dvid.Geometry, img interface{}) (*Labels, error)

NewLabels returns labelarray Labels, a representation of externally usable subvolume or slice data, given some geometry and optional image data. If img is passed in, the function will initialize Voxels with data from the image. Otherwise, it will allocate a zero buffer of appropriate size.

func (*Data) PushData Uses

func (d *Data) PushData(p *datastore.PushSession) error

PushData pushes labelarray data to a remote DVID.

func (*Data) PutLabels Uses

func (d *Data) PutLabels(v dvid.VersionID, subvol *dvid.Subvolume, data []byte, roiname dvid.InstanceName, mutate bool) error

PutLabels persists voxels from a subvolume into the storage engine. This involves transforming a supplied volume of uint64 with known geometry into many labels.Block that tiles the subvolume. Messages are sent to subscribers for ingest or mutate events.

func (*Data) ReadChunk Uses

func (d *Data) ReadChunk(chunk *storage.Chunk) error

ReadChunk reads a chunk of data as part of a mapped operation. Only some multiple of the # of CPU cores can be used for chunk handling before it waits for chunk processing to abate via the buffered server.HandlerToken channel.

func (*Data) ReceiveBlocks Uses

func (d *Data) ReceiveBlocks(ctx *datastore.VersionedCtx, r io.ReadCloser, scale uint8, downscale bool, compression string) error

ReceiveBlocks stores a slice of bytes corresponding to specified blocks

func (*Data) ScaleUpdating Uses

func (d *Data) ScaleUpdating(scale uint8) bool

func (*Data) SendBlocks Uses

func (d *Data) SendBlocks(ctx *datastore.VersionedCtx, w http.ResponseWriter, scale uint8, subvol *dvid.Subvolume, compression string) error

SendBlocks returns a series of blocks covering the given block-aligned subvolume.

func (*Data) ServeHTTP Uses

func (d *Data) ServeHTTP(uuid dvid.UUID, ctx *datastore.VersionedCtx, w http.ResponseWriter, r *http.Request) (activity map[string]interface{})

ServeHTTP handles all incoming HTTP requests for this data.

func (*Data) SetResolution Uses

func (d *Data) SetResolution(uuid dvid.UUID, jsonBytes []byte) error

SetResolution loads JSON data giving Resolution.

func (*Data) Shutdown Uses

func (d *Data) Shutdown(wg *sync.WaitGroup)

Shutdown terminates blocks until syncs are done then terminates background goroutines processing data.

func (*Data) SplitCoarseLabels Uses

func (d *Data) SplitCoarseLabels(v dvid.VersionID, fromLabel, splitLabel uint64, r io.ReadCloser) (toLabel uint64, err error)

SplitCoarseLabels splits a portion of a label's voxels into a given split label or, if the given split label is 0, a new label, which is returned. The input is a binary sparse volume defined by block coordinates and should be the smaller portion of a labeled region-to-be-split.

EVENTS

labels.SplitStartEvent occurs at very start of split and transmits labels.DeltaSplitStart struct.

labels.SplitBlockEvent occurs for every block of a split label and transmits labels.DeltaSplit struct.

labels.SplitEndEvent occurs at end of split and transmits labels.DeltaSplitEnd struct.

func (*Data) SplitLabels Uses

func (d *Data) SplitLabels(v dvid.VersionID, fromLabel, splitLabel uint64, r io.ReadCloser) (toLabel uint64, err error)

SplitLabels splits a portion of a label's voxels into a given split label or, if the given split label is 0, a new label, which is returned. The input is a binary sparse volume and should preferably be the smaller portion of a labeled region. In other words, the caller should chose to submit for relabeling the smaller portion of any split. It is assumed that the given split voxels are within the fromLabel set of voxels and will generate unspecified behavior if this is not the case.

EVENTS

labels.SplitStartEvent occurs at very start of split and transmits labels.DeltaSplitStart struct.

labels.SplitBlockEvent occurs for every block of a split label and transmits labels.DeltaSplit struct.

labels.SplitEndEvent occurs at end of split and transmits labels.DeltaSplitEnd struct.

func (*Data) StartScaleUpdate Uses

func (d *Data) StartScaleUpdate(scale uint8)

func (*Data) StopScaleUpdate Uses

func (d *Data) StopScaleUpdate(scale uint8)

func (*Data) StoreDownres Uses

func (d *Data) StoreDownres(v dvid.VersionID, hiresScale uint8, hires downres.BlockMap) (downres.BlockMap, error)

func (*Data) WriteSparseCoarseVols Uses

func (d *Data) WriteSparseCoarseVols(ctx *datastore.VersionedCtx, w io.Writer, begLabel, endLabel uint64, bounds dvid.Bounds) error

WriteSparseCoarseVols returns a stream of sparse volumes with blocks of the given label in encoded RLE format:

	uint64   label
	<coarse sparse vol as given below>

	uint64   label
	<coarse sparse vol as given below>

	...

The coarse sparse vol has the following format where integers are little endian and the order
of data is exactly as specified below:

	int32    # Spans
	Repeating unit of:
		int32   Block coordinate of run start (dimension 0)
		int32   Block coordinate of run start (dimension 1)
		int32   Block coordinate of run start (dimension 2)
		int32   Length of run

type IngestedBlock Uses

type IngestedBlock struct {
    MutID  uint64
    BCoord dvid.IZYXString
    Data   *labels.Block
}

IngestedBlock is the unit of delta for a IngestBlockEvent.

type Labels Uses

type Labels struct {
    *imageblk.Voxels
}

Labels are voxels that have uint64 labels.

func (*Labels) ComputeTransform Uses

func (v *Labels) ComputeTransform(tkey storage.TKey, blockSize dvid.Point) (blockBeg, dataBeg, dataEnd dvid.Point, err error)

ComputeTransform determines the block coordinate and beginning + ending voxel points for the data corresponding to the given Block.

func (*Labels) Interpolable Uses

func (l *Labels) Interpolable() bool

func (*Labels) String Uses

func (l *Labels) String() string

type Meta Uses

type Meta struct {
    Voxels uint64         // Total # of voxels in label.
    Blocks dvid.IZYXSlice // Sorted block coordinates occupied by label.

    LastModTime time.Time
    LastModUser string
}

Meta gives a high-level overview of a label's voxels including a block index. Some properties are only used if the associated data instance has features enabled, e.g., size tracking.

func GetLabelIndex Uses

func GetLabelIndex(d dvid.Data, v dvid.VersionID, label uint64) (*Meta, error)

GetLabelIndex gets label index data from storage for a given data instance and version. Concurrency-safe access and supports caching.

func GetLabelProcessedIndex Uses

func GetLabelProcessedIndex(d dvid.Data, v dvid.VersionID, label uint64, scale uint8, bounds dvid.Bounds) (*Meta, error)

GetLabelProcessedIndex gets label index data from storage for a given data instance and version with scaling and/or bounds. Concurrency-safe access and supports caching.

func GetMappedLabelIndex Uses

func GetMappedLabelIndex(d dvid.Data, v dvid.VersionID, label uint64, scale uint8, bounds dvid.Bounds) (*Meta, labels.Set, error)

GetMappedLabelIndex gets index data for all labels potentially merged to the given label, allowing scaling and bounds. If bounds are used, the returned Meta will not include a voxel count. Concurrency-safe access and supports caching.

func GetMappedLabelSetIndex Uses

func GetMappedLabelSetIndex(d dvid.Data, v dvid.VersionID, lbls labels.Set, scale uint8, bounds dvid.Bounds) (*Meta, labels.Set, error)

GetMappedLabelSetIndex gets index data for all labels potentially merged to the given labels, allowing scaling and bounds. If bounds are used, the returned Meta will not include a voxel count. Concurrency-safe access and supports caching.

func (Meta) MarshalBinary Uses

func (m Meta) MarshalBinary() ([]byte, error)

MarshalBinary implements the encoding.BinaryMarshaler interface

func (*Meta) UnmarshalBinary Uses

func (m *Meta) UnmarshalBinary(b []byte) error

UnmarshalBinary implements the encoding.BinaryUnmarshaler interface.

type MutatedBlock Uses

type MutatedBlock struct {
    MutID  uint64
    BCoord dvid.IZYXString
    Prev   *labels.Block
    Data   *labels.Block
}

MutatedBlock tracks previous and updated block data. It is the unit of delta for a MutateBlockEvent.

type SparseVolFormat Uses

type SparseVolFormat uint8

SparseVolFormat indicates the type of encoding used for sparse volume representation.

const (
    // FormatLegacyRLE is Legacy RLE encoding with header that gives # spans.
    FormatLegacyRLE SparseVolFormat = iota

    // FormatStreamingRLE specifies Streaming RLE encoding
    FormatStreamingRLE

    // FormatBinaryBlocks specifies a streaming set of binary Blocks
    FormatBinaryBlocks
)

type Type Uses

type Type struct {
    imageblk.Type
}

Type uses imageblk data type by composition.

func (*Type) Help Uses

func (dtype *Type) Help() string

func (*Type) NewDataService Uses

func (dtype *Type) NewDataService(uuid dvid.UUID, id dvid.InstanceID, name dvid.InstanceName, c dvid.Config) (datastore.DataService, error)

Package labelarray imports 29 packages (graph) and is imported by 2 packages. Updated 2019-06-02. Refresh now. Tools for package owners.