imageblk

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 18, 2023 License: BSD-3-Clause Imports: 22 Imported by: 9

Documentation

Overview 

	Package imageblk implements DVID support for image blocks of various formats (uint8, uint16, rgba8).
    For label data, use labelblk.

Index

Constants

View Source 
const (
	Version = "0.2"
	RepoURL = "github.com/janelia-flyem/dvid/datatype/imageblk"
)
View Source 
const (
	IngestBlockEvent = "BLOCK_INGEST"
	MutateBlockEvent = "BLOCK_MUTATE"
	DeleteBlockEvent = "BLOCK_DELETE"
)

Events for imageblk

View Source 
const KVWriteSize = 500

KVWriteSize is the # of key-value pairs we will write as one atomic batch write.

Variables

View Source 
var (
	// DefaultBlockSize specifies the default size for each block of this data type.
	DefaultBlockSize int32 = 32

	DefaultRes float32 = 8

	DefaultUnits = "nanometers"
)

Functions

func DecodeTKey 

func DecodeTKey(tk storage.TKey) (*dvid.IndexZYX, error)

DecodeTKey returns a spatial index from a image block key. TODO: Extend this when necessary to allow any form of spatial indexing like CZYX.

func MetaTKey 

func MetaTKey() storage.TKey

MetaTKey provides a TKey for metadata (extents)

func NewTKey 

func NewTKey(idx dvid.Index) storage.TKey

NewTKey returns a type-specific key component for an image block. TKey = s

func NewTKeyByCoord 

func NewTKeyByCoord(izyx dvid.IZYXString) storage.TKey

NewTKeyByCoord returns a TKey for a block coord in string format.

Types

type ArbSlice 

type ArbSlice struct {
	// contains filtered or unexported fields
}

ArbSlice is a 2d rectangle that can be positioned arbitrarily in 3D.

func (ArbSlice) String 

func (s ArbSlice) String() string

type Block 

type Block struct {
	Index *dvid.IndexZYX
	Data  []byte
	MutID uint64
}

Block encodes a 3d block coordinate and block data. It is the unit of delta for a IngestBlockEvent.

type Data 

type Data struct {
	*datastore.Data
	Properties
	sync.Mutex // to protect extent updates
}

Data embeds the datastore's Data and extends it with voxel-specific properties.

func (*Data) AllocateBlock 

func (d *Data) AllocateBlock() []byte

func (*Data) BackgroundBlock 

func (d *Data) BackgroundBlock() []byte

BackgroundBlock returns a block buffer that has been preinitialized to the background value.

func (*Data) BlankImage 

func (d *Data) BlankImage(dstW, dstH int32) (*dvid.Image, error)

BlankImage initializes a blank image of appropriate size and depth for the current data values. Returns an error if the geometry is not 2d.

func (*Data) BlockSize 

func (d *Data) BlockSize() dvid.Point

func (*Data) CopyPropertiesFrom 

func (d *Data) CopyPropertiesFrom(src datastore.DataService, fs storage.FilterSpec) error

CopyPropertiesFrom copies the data instance-specific properties from a given data instance into the receiver's properties. Fulfills the datastore.PropertyCopier interface.

func (*Data) DescribeTKeyClass 

func (d *Data) DescribeTKeyClass(tkc storage.TKeyClass) string

DescribeTKeyClass returns a string explanation of what a particular TKeyClass is used for. Implements the datastore.TKeyClassDescriber interface.

func (*Data) DoRPC 

func (d *Data) DoRPC(req datastore.Request, reply *datastore.Response) error

DoRPC acts as a switchboard for RPC commands.

func (*Data) Equals 

func (d *Data) Equals(d2 *Data) bool

func (*Data) ForegroundROI 

func (d *Data) ForegroundROI(req datastore.Request, reply *datastore.Response) error

ForegroundROI creates a new ROI by determining all non-background blocks.

func (*Data) GetArbitraryImage 

func (d *Data) GetArbitraryImage(ctx storage.Context, tlStr, trStr, blStr, resStr string) (*dvid.Image, error)

func (*Data) GetBlock 

func (d *Data) GetBlock(v dvid.VersionID, k storage.TKey) ([]byte, error)

GetBlock returns a block with data from this instance's preferred storage.

func (*Data) GetBlocks 

func (d *Data) GetBlocks(v dvid.VersionID, start dvid.ChunkPoint3d, span int32) ([]byte, error)

GetBlocks returns a slice of bytes corresponding to all the blocks along a span in X

func (*Data) GetExtents 

func (d *Data) GetExtents(ctx *datastore.VersionedCtx) (dvidextents dvid.Extents, err error)

GetExtents retrieves current extent (and updates extents cache) TODO -- refactor return since MinIndex / MaxIndex not used so should use extents3d.

func (*Data) GetImage 

func (d *Data) GetImage(v dvid.VersionID, vox *Voxels, roiname dvid.InstanceName) (*dvid.Image, error)

GetImage retrieves a 2d image from a version node given a geometry of voxels.

func (*Data) GetVolume 

func (d *Data) GetVolume(v dvid.VersionID, vox *Voxels, roiname dvid.InstanceName) ([]byte, error)

GetVolume retrieves a n-d volume from a version node given a geometry of voxels.

func (*Data) GetVoxels 

func (d *Data) GetVoxels(v dvid.VersionID, vox *Voxels, roiname dvid.InstanceName) error

GetVoxels copies voxels from the storage engine to Voxels, a requested subvolume or 2d image.

func (*Data) GobDecode 

func (d *Data) GobDecode(b []byte) error

func (*Data) GobEncode 

func (d *Data) GobEncode() ([]byte, error)

func (*Data) Help 

func (d *Data) Help() string

func (*Data) IngestVoxels 

func (d *Data) IngestVoxels(v dvid.VersionID, mutID uint64, vox *Voxels, roiname dvid.InstanceName) error

IngestVoxels ingests voxels from a subvolume into the storage engine. The subvolume must be aligned to blocks of the data instance, which simplifies the routine since we are simply replacing a value instead of modifying values (GET + PUT).

func (*Data) LoadImages 

func (d *Data) LoadImages(v dvid.VersionID, offset dvid.Point, filenames []string) error

LoadImages bulk loads images using different techniques if it is a multidimensional file like HDF5 or a sequence of PNG/JPG/TIF images.

func (*Data) LoadOldBlocks 

func (d *Data) LoadOldBlocks(v dvid.VersionID, vox *Voxels, blocks storage.TKeyValues) error

LoadOldBlocks loads blocks with old data if they exist.

func (*Data) MarshalJSON 

func (d *Data) MarshalJSON() ([]byte, error)

MarshalJSON returns a JSON representation of the data assuming that any extents are for the master branch leaf.

func (*Data) MarshalJSONExtents 

func (d *Data) MarshalJSONExtents(ctx *datastore.VersionedCtx) ([]byte, error)

func (*Data) ModifyConfig 

func (d *Data) ModifyConfig(config dvid.Config) error

func (*Data) MutateVoxels 

func (d *Data) MutateVoxels(v dvid.VersionID, mutID uint64, vox *Voxels, roiname dvid.InstanceName) error

MutateVoxels mutates voxels from a subvolume into the storage engine. This differs from the IngestVoxels function in firing off a MutateBlockEvent instead of an IngestBlockEvent, which tells subscribers that a previous value has changed instead of a completely new key/value being inserted. There will be some decreased performance due to cleanup of prior denormalizations compared to IngestVoxels.

func (*Data) NdDataMetadata 

func (d *Data) NdDataMetadata(ctx *datastore.VersionedCtx) (string, error)

NdDataSchema returns the metadata in JSON for this Data

func (*Data) NewArbSlice 

func (d *Data) NewArbSlice(topLeft, topRight, bottomLeft dvid.Vector3d, res float64) (*ArbSlice, error)

NewArbSlice returns an image with arbitrary 3D orientation. The 3d points are in real world space definited by resolution, e.g., nanometer space.

func (*Data) NewArbSliceFromStrings 

func (d *Data) NewArbSliceFromStrings(tlStr, trStr, blStr, resStr, sep string) (*ArbSlice, error)

NewArbSliceFromStrings returns an image with arbitrary 3D orientation given string parameters. The 3d points are in real world space definited by resolution, e.g., nanometer space.

func (*Data) NewFilter 

func (d *Data) NewFilter(fs storage.FilterSpec) (storage.Filter, error)

NewFilter returns a Filter for use with a push of key-value pairs.

func (*Data) NewVoxels 

func (d *Data) NewVoxels(geom dvid.Geometry, img interface{}) (*Voxels, error)

NewVoxels returns Voxels with given geometry and optional image data. If img is passed in, the function will initialize the Voxels with data from the image. Otherwise, it will allocate a zero buffer of appropriate size.

func (*Data) PostExtents 

func (d *Data) PostExtents(ctx *datastore.VersionedCtx, start dvid.Point, end dvid.Point) error

PostExtents updates extents with the new points (always growing)

func (*Data) PropertiesWithExtents 

func (d *Data) PropertiesWithExtents(ctx *datastore.VersionedCtx) (props Properties, err error)

func (*Data) PushData 

func (d *Data) PushData(p *datastore.PushSession) error

PushData does an imageblk-specific push using optional ROI filters.

func (*Data) PutBlocks 

func (d *Data) PutBlocks(v dvid.VersionID, mutID uint64, start dvid.ChunkPoint3d, span int, data io.ReadCloser, mutate bool) error

PutBlocks stores blocks of data in a span along X

func (*Data) PutChunk 

func (d *Data) PutChunk(chunk *storage.Chunk, hasbuffer bool, finishedRequests chan error) error

PutChunk puts a chunk of data as part of a mapped operation. Only some multiple of the # of CPU cores can be used for chunk handling before it waits for chunk processing to abate via the buffered server.HandlerToken channel.

func (*Data) PutLocal 

func (d *Data) PutLocal(request datastore.Request, reply *datastore.Response) error

PutLocal adds image data to a version node, altering underlying blocks if the image intersects the block.

The image filename glob MUST BE absolute file paths that are visible to the server. This function is meant for mass ingestion of large data files, and it is inappropriate to read gigabytes of data just to send it over the network to a local DVID.

func (*Data) PutVoxels 

func (d *Data) PutVoxels(v dvid.VersionID, mutID uint64, vox *Voxels, roiname dvid.InstanceName, mutate bool) error

PutVoxels persists voxels from a subvolume into the storage engine. The subvolume must be aligned to blocks of the data instance, which simplifies the routine if the PUT is a mutation (signals MutateBlockEvent) instead of ingestion.

func (*Data) ReadChunk 

func (d *Data) ReadChunk(chunk *storage.Chunk) error

ReadChunk reads a chunk of data as part of a mapped operation. Only some multiple of the # of CPU cores can be used for chunk handling before it waits for chunk processing to abate via the buffered server.HandlerToken channel.

func (*Data) Resolution 

func (d *Data) Resolution() dvid.Resolution

func (*Data) SendBlocks 

func (d *Data) SendBlocks(ctx *datastore.VersionedCtx, w http.ResponseWriter, subvol *dvid.Subvolume, compression string) error

SendBlocks returns a slice of bytes corresponding to all the blocks along a span in X

func (*Data) SendBlocksSpecific 

func (d *Data) SendBlocksSpecific(ctx *datastore.VersionedCtx, w http.ResponseWriter, compression string, blockstring string, isprefetch bool) (numBlocks int, err error)

SendBlocksSpecific writes data to the blocks specified -- best for non-ordered backend

func (*Data) SendSerializedBlock 

func (d *Data) SendSerializedBlock(w http.ResponseWriter, x, y, z int32, v []byte, compression string) error

SendSerializedBlock writes a serialized data block to the writer with given compression.

func (*Data) SendUnserializedBlock 

func (d *Data) SendUnserializedBlock(w http.ResponseWriter, x, y, z int32, v []byte, compression string) error

SendUnserializedBlock writes a raw data block to the writer with given compression.

func (*Data) ServeHTTP 

func (d *Data) ServeHTTP(uuid dvid.UUID, ctx *datastore.VersionedCtx, w http.ResponseWriter, r *http.Request) (activity map[string]interface{})

ServeHTTP handles all incoming HTTP requests for this data.

func (*Data) SetExtents 

func (d *Data) SetExtents(ctx *datastore.VersionedCtx, uuid dvid.UUID, jsonBytes []byte) error

SetExtents saves JSON data giving MinPoint and MaxPoint (put operation)

func (*Data) SetResolution 

func (d *Data) SetResolution(uuid dvid.UUID, jsonBytes []byte) error

SetResolution loads JSON data giving Resolution.

func (*Data) String 

func (d *Data) String() string

type ExtentsJSON 

type ExtentsJSON struct {
	MinPoint dvid.Point
	MaxPoint dvid.Point
}

ExtentsJSON is extents encoding for imageblk

type Filter 

type Filter struct {
	*Data
	// contains filtered or unexported fields
}

func (Filter) Check 

func (f Filter) Check(tkv *storage.TKeyValue) (skip bool, err error)

type KeyFunc 

type KeyFunc func(dvid.Point3d) []byte

type MutatedBlock 

type MutatedBlock struct {
	Index *dvid.IndexZYX
	Prev  []byte
	Data  []byte
	MutID uint64
}

MutatedBlock encodes a 3d block coordinate and previous and updated block data. It is the unit of delta for a MutateBlockEvent.

type PopulateFunc 

type PopulateFunc func([]byte) ([]byte, error)

type Properties 

type Properties struct {
	// Values describes the data type/label for each value within a voxel.
	Values dvid.DataValues

	// Interpolable is true if voxels can be interpolated when resizing.
	Interpolable bool

	// Block size for this repo
	BlockSize dvid.Point

	dvid.Resolution

	// leave field in metadata but no longer updated!!
	dvid.Extents

	// Background value for data
	Background uint8

	// GridStore designates store to be used for immutable data access
	GridStore storage.Alias

	// ScaleLevel designates resolution from 0 (high-res) to an int N with 2^N down-res
	ScaleLevel int
}

Properties are additional properties for image block data instances beyond those in standard datastore.Data. These will be persisted to metadata storage.

type ROI 

type ROI struct {
	Iter *roi.Iterator
	// contains filtered or unexported fields
}

ROI encapsulates a request-specific ROI check with a given scaling for voxels outside the ROI.

func GetROI 

func GetROI(v dvid.VersionID, roiname dvid.InstanceName, bnd dvid.Bounder) (*ROI, error)

GetROI returns an imageblk.ROI that can iterate over the provided Voxels unless roiname is empty, which prompts a nil ROI returned.

type Type 

type Type struct {
	datastore.Type
	// contains filtered or unexported fields
}

Type embeds the datastore's Type to create a unique type with voxel functions. Refinements of general voxel types can be implemented by embedding this type, choosing appropriate # of values and bytes/value, overriding functions as needed, and calling datastore.Register(). Note that these fields are invariant for all instances of this type. Fields that can change depending on the type of data (e.g., resolution) should be in the Data type.

func NewType 

func NewType(values dvid.DataValues, interpolable bool) Type

NewType returns a pointer to a new imageblk Type with default values set.

func (*Type) Help 

func (dtype *Type) Help() string

func (*Type) NewData 

func (dtype *Type) NewData(uuid dvid.UUID, id dvid.InstanceID, name dvid.InstanceName, c dvid.Config) (*Data, error)

NewData returns a pointer to a new Voxels with default values.

func (*Type) NewDataService 

func (dtype *Type) NewDataService(uuid dvid.UUID, id dvid.InstanceID, name dvid.InstanceName, c dvid.Config) (datastore.DataService, error)

NewDataService returns a pointer to a new Voxels with default values.

type ValueCache 

type ValueCache struct {
	sync.Mutex
	// contains filtered or unexported fields
}

ValueCache is a concurrency-friendly cache

func NewValueCache 

func NewValueCache(size int) *ValueCache

func (*ValueCache) Clear 

func (vc *ValueCache) Clear()

Clear clears the cache.

func (*ValueCache) Get 

func (vc *ValueCache) Get(key []byte, pf PopulateFunc) ([]byte, bool, error)

Get returns the cached value of a key. On a miss, it uses the passed PopulateFunc to retrieve the key and stores it in the cache. If nil is passed for the PopulateFunc, the function just returns a "false" with no value.

type Voxels 

type Voxels struct {
	dvid.Geometry
	// contains filtered or unexported fields
}

Voxels represents subvolumes or slices and implements the ExtData interface.

func NewVoxels 

func NewVoxels(geom dvid.Geometry, values dvid.DataValues, data []byte, stride int32) *Voxels

func (*Voxels) BytesPerVoxel 

func (v *Voxels) BytesPerVoxel() int32

func (*Voxels) ComputeTransform 

func (v *Voxels) ComputeTransform(block *storage.TKeyValue, blockSize dvid.Point) (blockBeg, dataBeg, dataEnd dvid.Point, err error)

ComputeTransform determines the block coordinate and beginning + ending voxel points for the data corresponding to the given Block.

func (*Voxels) Data 

func (v *Voxels) Data() []byte

func (*Voxels) DownRes 

func (v *Voxels) DownRes(magnification dvid.Point) error

DownRes downsamples 2d Voxels data by averaging where the down-magnification are integers. If the source image size in Voxels is not an integral multiple of the reduction factor, the edge voxels on the right and bottom side are truncated. This function modifies the Voxels data. An error is returned if a non-2d Voxels receiver is used.

func (*Voxels) GetImage2d 

func (v *Voxels) GetImage2d() (*dvid.Image, error)

GetImage2d returns a 2d image suitable for use external to DVID. TODO -- Create more comprehensive handling of endianness and encoding of multibytes/voxel data into appropriate images.

func (*Voxels) Interpolable 

func (v *Voxels) Interpolable() bool

func (*Voxels) NewChunkIndex 

func (v *Voxels) NewChunkIndex() dvid.ChunkIndexer

func (*Voxels) NewIndexIterator 

func (v *Voxels) NewIndexIterator(chunkSize dvid.Point) (dvid.IndexIterator, error)

NewIndexIterator returns an iterator that can move across the voxel geometry, generating indices or index spans.

func (*Voxels) ReadBlock 

func (v *Voxels) ReadBlock(block *storage.TKeyValue, blockSize dvid.Point, attenuation uint8) error

ReadBlock reads the possibly intersecting block data into the receiver Voxels.

func (*Voxels) SetData 

func (v *Voxels) SetData(data []byte)

func (*Voxels) SetGeometry 

func (v *Voxels) SetGeometry(geom dvid.Geometry)

func (*Voxels) SetStride 

func (v *Voxels) SetStride(stride int32)

func (*Voxels) SetValues 

func (v *Voxels) SetValues(values dvid.DataValues)

func (*Voxels) Stride 

func (v *Voxels) Stride() int32

func (*Voxels) String 

func (v *Voxels) String() string

func (*Voxels) Values 

func (v *Voxels) Values() dvid.DataValues

func (*Voxels) WriteBlock 

func (v *Voxels) WriteBlock(block *storage.TKeyValue, blockSize dvid.Point) error

WriteBlock writes a subvolume or 2d image into a possibly intersecting block.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.