dvid: github.com/janelia-flyem/dvid/datatype/imageblk Index | Files

package imageblk

import "github.com/janelia-flyem/dvid/datatype/imageblk"

	Package imageblk implements DVID support for image blocks of various formats (uint8, uint16, rgba8).
    For label data, use labelblk.

Index

Package Files

arb_image.go float32.go imageblk.go ingest.go keys.go push.go read.go rgba8.go sync.go uint16.go uint32.go uint64.go uint8.go write.go

Constants

const (
    Version = "0.2"
    RepoURL = "github.com/janelia-flyem/dvid/datatype/imageblk"
)
const (
    IngestBlockEvent = "BLOCK_INGEST"
    MutateBlockEvent = "BLOCK_MUTATE"
    DeleteBlockEvent = "BLOCK_DELETE"
)

Events for imageblk

const KVWriteSize = 500

KVWriteSize is the # of key-value pairs we will write as one atomic batch write.

Variables

var (
    // DefaultBlockSize specifies the default size for each block of this data type.
    DefaultBlockSize int32 = 32

    DefaultRes float32 = 8

    DefaultUnits = "nanometers"
)
var ExtentsUnchanged = errors.New("extents does not change")

ExtentsUnchanged is an error for patch failure

func DecodeTKey Uses

func DecodeTKey(tk storage.TKey) (*dvid.IndexZYX, error)

DecodeTKey returns a spatial index from a image block key. TODO: Extend this when necessary to allow any form of spatial indexing like CZYX.

func MetaTKey Uses

func MetaTKey() storage.TKey

MetaTKey provides a TKey for metadata (extents)

func NewTKey Uses

func NewTKey(idx dvid.Index) storage.TKey

NewTKey returns a type-specific key component for an image block. TKey = s

func NewTKeyByCoord Uses

func NewTKeyByCoord(izyx dvid.IZYXString) storage.TKey

NewTKeyByCoord returns a TKey for a block coord in string format.

type ArbSlice Uses

type ArbSlice struct {
    // contains filtered or unexported fields
}

ArbSlice is a 2d rectangle that can be positioned arbitrarily in 3D.

func (ArbSlice) String Uses

func (s ArbSlice) String() string

type Block Uses

type Block struct {
    Index *dvid.IndexZYX
    Data  []byte
    MutID uint64
}

Block encodes a 3d block coordinate and block data. It is the unit of delta for a IngestBlockEvent.

type Data Uses

type Data struct {
    *datastore.Data
    Properties
    sync.Mutex // to protect extent updates
}

Data embeds the datastore's Data and extends it with voxel-specific properties.

func (*Data) AllocateBlock Uses

func (d *Data) AllocateBlock() []byte

func (*Data) BackgroundBlock Uses

func (d *Data) BackgroundBlock() []byte

BackgroundBlock returns a block buffer that has been preinitialized to the background value.

func (*Data) BlankImage Uses

func (d *Data) BlankImage(dstW, dstH int32) (*dvid.Image, error)

BlankImage initializes a blank image of appropriate size and depth for the current data values. Returns an error if the geometry is not 2d.

func (*Data) BlockSize Uses

func (d *Data) BlockSize() dvid.Point

func (*Data) CopyPropertiesFrom Uses

func (d *Data) CopyPropertiesFrom(src datastore.DataService, fs storage.FilterSpec) error

CopyPropertiesFrom copies the data instance-specific properties from a given data instance into the receiver's properties. Fulfills the datastore.PropertyCopier interface.

func (*Data) DescribeTKeyClass Uses

func (d *Data) DescribeTKeyClass(tkc storage.TKeyClass) string

DescribeTKeyClass returns a string explanation of what a particular TKeyClass is used for. Implements the datastore.TKeyClassDescriber interface.

func (*Data) DoRPC Uses

func (d *Data) DoRPC(req datastore.Request, reply *datastore.Response) error

DoRPC acts as a switchboard for RPC commands.

func (*Data) Equals Uses

func (d *Data) Equals(d2 *Data) bool

func (*Data) ForegroundROI Uses

func (d *Data) ForegroundROI(req datastore.Request, reply *datastore.Response) error

ForegroundROI creates a new ROI by determining all non-background blocks.

func (*Data) GetArbitraryImage Uses

func (d *Data) GetArbitraryImage(ctx storage.Context, tlStr, trStr, blStr, resStr string) (*dvid.Image, error)

func (*Data) GetBlock Uses

func (d *Data) GetBlock(v dvid.VersionID, k storage.TKey) ([]byte, error)

GetBlock returns a block with data from this instance's preferred storage.

func (*Data) GetBlocks Uses

func (d *Data) GetBlocks(v dvid.VersionID, start dvid.ChunkPoint3d, span int32) ([]byte, error)

GetBlocks returns a slice of bytes corresponding to all the blocks along a span in X

func (*Data) GetExtents Uses

func (d *Data) GetExtents(ctx *datastore.VersionedCtx) (dvidextents dvid.Extents, err error)

GetExtents retrieves current extent (and updates extents cache) TODO -- refactor return since MinIndex / MaxIndex not used so should use extents3d.

func (*Data) GetImage Uses

func (d *Data) GetImage(v dvid.VersionID, vox *Voxels, roiname dvid.InstanceName) (*dvid.Image, error)

GetImage retrieves a 2d image from a version node given a geometry of voxels.

func (*Data) GetVolume Uses

func (d *Data) GetVolume(v dvid.VersionID, vox *Voxels, roiname dvid.InstanceName) ([]byte, error)

GetVolume retrieves a n-d volume from a version node given a geometry of voxels.

func (*Data) GetVoxels Uses

func (d *Data) GetVoxels(v dvid.VersionID, vox *Voxels, roiname dvid.InstanceName) error

GetVoxels copies voxels from the storage engine to Voxels, a requested subvolume or 2d image.

func (*Data) GobDecode Uses

func (d *Data) GobDecode(b []byte) error

func (*Data) GobEncode Uses

func (d *Data) GobEncode() ([]byte, error)

func (*Data) Help Uses

func (d *Data) Help() string

func (*Data) IngestVoxels Uses

func (d *Data) IngestVoxels(v dvid.VersionID, mutID uint64, vox *Voxels, roiname dvid.InstanceName) error

IngestVoxels ingests voxels from a subvolume into the storage engine. The subvolume must be aligned to blocks of the data instance, which simplifies the routine since we are simply replacing a value instead of modifying values (GET + PUT).

func (*Data) LoadImages Uses

func (d *Data) LoadImages(v dvid.VersionID, offset dvid.Point, filenames []string) error

LoadImages bulk loads images using different techniques if it is a multidimensional file like HDF5 or a sequence of PNG/JPG/TIF images.

func (*Data) LoadOldBlocks Uses

func (d *Data) LoadOldBlocks(v dvid.VersionID, vox *Voxels, blocks storage.TKeyValues) error

LoadOldBlocks loads blocks with old data if they exist.

func (*Data) MarshalJSON Uses

func (d *Data) MarshalJSON() ([]byte, error)

func (*Data) MarshalJSONExtents Uses

func (d *Data) MarshalJSONExtents(ctx *datastore.VersionedCtx) ([]byte, error)

func (*Data) ModifyConfig Uses

func (d *Data) ModifyConfig(config dvid.Config) error

func (*Data) MutateVoxels Uses

func (d *Data) MutateVoxels(v dvid.VersionID, mutID uint64, vox *Voxels, roiname dvid.InstanceName) error

MutateVoxels mutates voxels from a subvolume into the storage engine. This differs from the IngestVoxels function in firing off a MutateBlockEvent instead of an IngestBlockEvent, which tells subscribers that a previous value has changed instead of a completely new key/value being inserted. There will be some decreased performance due to cleanup of prior denormalizations compared to IngestVoxels.

func (*Data) NdDataMetadata Uses

func (d *Data) NdDataMetadata(ctx *datastore.VersionedCtx) (string, error)

NdDataSchema returns the metadata in JSON for this Data

func (*Data) NewArbSlice Uses

func (d *Data) NewArbSlice(topLeft, topRight, bottomLeft dvid.Vector3d, res float64) (*ArbSlice, error)

NewArbSlice returns an image with arbitrary 3D orientation. The 3d points are in real world space definited by resolution, e.g., nanometer space.

func (*Data) NewArbSliceFromStrings Uses

func (d *Data) NewArbSliceFromStrings(tlStr, trStr, blStr, resStr, sep string) (*ArbSlice, error)

NewArbSliceFromStrings returns an image with arbitrary 3D orientation given string parameters. The 3d points are in real world space definited by resolution, e.g., nanometer space.

func (*Data) NewFilter Uses

func (d *Data) NewFilter(fs storage.FilterSpec) (storage.Filter, error)

NewFilter returns a Filter for use with a push of key-value pairs.

func (*Data) NewVoxels Uses

func (d *Data) NewVoxels(geom dvid.Geometry, img interface{}) (*Voxels, error)

NewVoxels returns Voxels with given geometry and optional image data. If img is passed in, the function will initialize the Voxels with data from the image. Otherwise, it will allocate a zero buffer of appropriate size.

func (*Data) PostExtents Uses

func (d *Data) PostExtents(ctx *datastore.VersionedCtx, start dvid.Point, end dvid.Point) error

PostExtents updates extents with the new points (always growing)

func (*Data) PropertiesWithExtents Uses

func (d *Data) PropertiesWithExtents(ctx *datastore.VersionedCtx) (props Properties, err error)

func (*Data) PushData Uses

func (d *Data) PushData(p *datastore.PushSession) error

PushData does an imageblk-specific push using optional ROI filters.

func (*Data) PutBlocks Uses

func (d *Data) PutBlocks(v dvid.VersionID, mutID uint64, start dvid.ChunkPoint3d, span int, data io.ReadCloser, mutate bool) error

PutBlocks stores blocks of data in a span along X

func (*Data) PutChunk Uses

func (d *Data) PutChunk(chunk *storage.Chunk, hasbuffer bool, patchgeo *patchGeo, finishedRequests chan error) error

PutChunk puts a chunk of data as part of a mapped operation. Only some multiple of the # of CPU cores can be used for chunk handling before it waits for chunk processing to abate via the buffered server.HandlerToken channel.

func (*Data) PutLocal Uses

func (d *Data) PutLocal(request datastore.Request, reply *datastore.Response) error

PutLocal adds image data to a version node, altering underlying blocks if the image intersects the block.

The image filename glob MUST BE absolute file paths that are visible to the server. This function is meant for mass ingestion of large data files, and it is inappropriate to read gigabytes of data just to send it over the network to a local DVID.

func (*Data) PutVoxels Uses

func (d *Data) PutVoxels(v dvid.VersionID, mutID uint64, vox *Voxels, roiname dvid.InstanceName, mutate bool) error

PutVoxels persists voxels from a subvolume into the storage engine. The subvolume must be aligned to blocks of the data instance, which simplifies the routine if the PUT is a mutation (signals MutateBlockEvent) instead of ingestion.

func (*Data) ReadChunk Uses

func (d *Data) ReadChunk(chunk *storage.Chunk) error

ReadChunk reads a chunk of data as part of a mapped operation. Only some multiple of the # of CPU cores can be used for chunk handling before it waits for chunk processing to abate via the buffered server.HandlerToken channel.

func (*Data) Resolution Uses

func (d *Data) Resolution() dvid.Resolution

func (*Data) SendBlockSimple Uses

func (d *Data) SendBlockSimple(w http.ResponseWriter, x, y, z int32, v []byte, compression string) error

func (*Data) SendBlocks Uses

func (d *Data) SendBlocks(ctx *datastore.VersionedCtx, w http.ResponseWriter, subvol *dvid.Subvolume, compression string) error

GetBlocks returns a slice of bytes corresponding to all the blocks along a span in X

func (*Data) SendBlocksSpecific Uses

func (d *Data) SendBlocksSpecific(ctx *datastore.VersionedCtx, w http.ResponseWriter, compression string, blockstring string, isprefetch bool) (numBlocks int, err error)

SendBlocksSpecific writes data to the blocks specified -- best for non-ordered backend

func (*Data) ServeHTTP Uses

func (d *Data) ServeHTTP(uuid dvid.UUID, ctx *datastore.VersionedCtx, w http.ResponseWriter, r *http.Request) (activity map[string]interface{})

ServeHTTP handles all incoming HTTP requests for this data.

func (*Data) SetExtents Uses

func (d *Data) SetExtents(ctx *datastore.VersionedCtx, uuid dvid.UUID, jsonBytes []byte) error

SetExtents saves JSON data giving MinPoint and MaxPoint (put operation)

func (*Data) SetResolution Uses

func (d *Data) SetResolution(uuid dvid.UUID, jsonBytes []byte) error

SetResolution loads JSON data giving Resolution.

func (*Data) String Uses

func (d *Data) String() string

type ExtentsJSON Uses

type ExtentsJSON struct {
    MinPoint dvid.Point
    MaxPoint dvid.Point
}

ExtentsJSON is extents encoding for imageblk

type Filter Uses

type Filter struct {
    *Data
    // contains filtered or unexported fields
}

func (Filter) Check Uses

func (f Filter) Check(tkv *storage.TKeyValue) (skip bool, err error)

type KeyFunc Uses

type KeyFunc func(dvid.Point3d) []byte

type MutatedBlock Uses

type MutatedBlock struct {
    Index *dvid.IndexZYX
    Prev  []byte
    Data  []byte
    MutID uint64
}

MutatedBlock encodes a 3d block coordinate and previous and updated block data. It is the unit of delta for a MutateBlockEvent.

type PopulateFunc Uses

type PopulateFunc func([]byte) ([]byte, error)

type Properties Uses

type Properties struct {
    // Values describes the data type/label for each value within a voxel.
    Values dvid.DataValues

    // Interpolable is true if voxels can be interpolated when resizing.
    Interpolable bool

    // Block size for this repo
    BlockSize dvid.Point

    dvid.Resolution

    // leave field in metadata but no longer updated!!
    dvid.Extents

    // Background value for data
    Background uint8
}

Properties are additional properties for image block data instances beyond those in standard datastore.Data. These will be persisted to metadata storage.

type ROI Uses

type ROI struct {
    Iter *roi.Iterator
    // contains filtered or unexported fields
}

ROI encapsulates a request-specific ROI check with a given scaling for voxels outside the ROI.

func GetROI Uses

func GetROI(v dvid.VersionID, roiname dvid.InstanceName, bnd dvid.Bounder) (*ROI, error)

GetROI returns an imageblk.ROI that can iterate over the provided Voxels unless roiname is empty, which prompts a nil ROI returned.

type Type Uses

type Type struct {
    datastore.Type
    // contains filtered or unexported fields
}

Type embeds the datastore's Type to create a unique type with voxel functions. Refinements of general voxel types can be implemented by embedding this type, choosing appropriate # of values and bytes/value, overriding functions as needed, and calling datastore.Register(). Note that these fields are invariant for all instances of this type. Fields that can change depending on the type of data (e.g., resolution) should be in the Data type.

func NewType Uses

func NewType(values dvid.DataValues, interpolable bool) Type

NewType returns a pointer to a new imageblk Type with default values set.

func (*Type) Help Uses

func (dtype *Type) Help() string

func (*Type) NewData Uses

func (dtype *Type) NewData(uuid dvid.UUID, id dvid.InstanceID, name dvid.InstanceName, c dvid.Config) (*Data, error)

NewData returns a pointer to a new Voxels with default values.

func (*Type) NewDataService Uses

func (dtype *Type) NewDataService(uuid dvid.UUID, id dvid.InstanceID, name dvid.InstanceName, c dvid.Config) (datastore.DataService, error)

NewDataService returns a pointer to a new Voxels with default values.

type ValueCache Uses

type ValueCache struct {
    sync.Mutex
    // contains filtered or unexported fields
}

ValueCache is a concurrency-friendly cache

func NewValueCache Uses

func NewValueCache(size int) *ValueCache

func (*ValueCache) Clear Uses

func (vc *ValueCache) Clear()

Clear clears the cache.

func (*ValueCache) Get Uses

func (vc *ValueCache) Get(key []byte, pf PopulateFunc) ([]byte, bool, error)

Get returns the cached value of a key. On a miss, it uses the passed PopulateFunc to retrieve the key and stores it in the cache. If nil is passed for the PopulateFunc, the function just returns a "false" with no value.

type Voxels Uses

type Voxels struct {
    dvid.Geometry
    // contains filtered or unexported fields
}

Voxels represents subvolumes or slices and implements the ExtData interface.

func NewVoxels Uses

func NewVoxels(geom dvid.Geometry, values dvid.DataValues, data []byte, stride int32) *Voxels

func (*Voxels) BytesPerVoxel Uses

func (v *Voxels) BytesPerVoxel() int32

func (*Voxels) ComputeTransform Uses

func (v *Voxels) ComputeTransform(block *storage.TKeyValue, blockSize dvid.Point) (blockBeg, dataBeg, dataEnd dvid.Point, err error)

ComputeTransform determines the block coordinate and beginning + ending voxel points for the data corresponding to the given Block.

func (*Voxels) Data Uses

func (v *Voxels) Data() []byte

func (*Voxels) DownRes Uses

func (v *Voxels) DownRes(magnification dvid.Point) error

DownRes downsamples 2d Voxels data by averaging where the down-magnification are integers. If the source image size in Voxels is not an integral multiple of the reduction factor, the edge voxels on the right and bottom side are truncated. This function modifies the Voxels data. An error is returned if a non-2d Voxels receiver is used.

func (*Voxels) GetImage2d Uses

func (v *Voxels) GetImage2d() (*dvid.Image, error)

GetImage2d returns a 2d image suitable for use external to DVID. TODO -- Create more comprehensive handling of endianness and encoding of multibytes/voxel data into appropriate images.

func (*Voxels) Interpolable Uses

func (v *Voxels) Interpolable() bool

func (*Voxels) NewChunkIndex Uses

func (v *Voxels) NewChunkIndex() dvid.ChunkIndexer

func (*Voxels) NewIndexIterator Uses

func (v *Voxels) NewIndexIterator(chunkSize dvid.Point) (dvid.IndexIterator, error)

NewIndexIterator returns an iterator that can move across the voxel geometry, generating indices or index spans.

func (*Voxels) ReadBlock Uses

func (v *Voxels) ReadBlock(block *storage.TKeyValue, blockSize dvid.Point, attenuation uint8) error

ReadBlock reads the possibly intersecting block data into the receiver Voxels.

func (*Voxels) SetData Uses

func (v *Voxels) SetData(data []byte)

func (*Voxels) SetGeometry Uses

func (v *Voxels) SetGeometry(geom dvid.Geometry)

func (*Voxels) SetStride Uses

func (v *Voxels) SetStride(stride int32)

func (*Voxels) SetValues Uses

func (v *Voxels) SetValues(values dvid.DataValues)

func (*Voxels) Stride Uses

func (v *Voxels) Stride() int32

func (*Voxels) String Uses

func (v *Voxels) String() string

func (*Voxels) Values Uses

func (v *Voxels) Values() dvid.DataValues

func (*Voxels) WriteBlock Uses

func (v *Voxels) WriteBlock(block *storage.TKeyValue, blockSize dvid.Point) error

WriteBlock writes a subvolume or 2d image into a possibly intersecting block.

Package imageblk imports 22 packages (graph) and is imported by 15 packages. Updated 2019-08-13. Refresh now. Tools for package owners.