cockroach: Index | Files

package coldata

import ""


Package Files

batch.go bytes.go nulls.go vec.go


const FlatBytesOverhead = unsafe.Sizeof(Bytes{})

FlatBytesOverhead is the overhead of Bytes in bytes.

func BatchSize Uses

func BatchSize() uint16

BatchSize is the maximum number of tuples that fit in a column batch. TODO(jordan): tune

func BytesFromArrowSerializationFormat Uses

func BytesFromArrowSerializationFormat(b *Bytes, data []byte, offsets []int32)

BytesFromArrowSerializationFormat takes an Arrow byte slice and accompanying offsets and populates b.

type Batch Uses

type Batch interface {
    // Length returns the number of values in the columns in the batch.
    Length() uint16
    // SetLength sets the number of values in the columns in the batch.
    // Width returns the number of columns in the batch.
    Width() int
    // ColVec returns the ith Vec in this batch.
    ColVec(i int) Vec
    // ColVecs returns all of the underlying Vecs in this batch.
    ColVecs() []Vec
    // Selection, if not nil, returns the selection vector on this batch: a
    // densely-packed list of the indices in each column that have not been
    // filtered out by a previous step.
    Selection() []uint16
    // SetSelection sets whether this batch is using its selection vector or not.
    // AppendCol appends a Vec with the specified type to this batch.
    // Reset modifies the caller in-place to have the given length and columns
    // with the given coltypes. If it's possible, Reset will reuse the existing
    // columns and allocations, invalidating existing references to the Batch or
    // its Vecs. However, Reset does _not_ zero out the column data.
    Reset(types []coltypes.T, length int)
    // ResetInternalBatch resets a batch and its underlying Vecs for reuse. It's
    // important for callers to call ResetInternalBatch if they own internal
    // batches that they reuse as not doing this could result in correctness
    // or memory blowup issues.

Batch is the type that columnar operators receive and produce. It represents a set of column vectors (partial data columns) as well as metadata about a batch, like the selection vector (which rows in the column batch are selected).

func NewMemBatch Uses

func NewMemBatch(types []coltypes.T) Batch

NewMemBatch allocates a new in-memory Batch. TODO(jordan): pool these allocations.

func NewMemBatchWithSize Uses

func NewMemBatchWithSize(types []coltypes.T, size int) Batch

NewMemBatchWithSize allocates a new in-memory Batch with the given column size. Use for operators that have a precisely-sized output batch.

type Bytes Uses

type Bytes struct {
    // contains filtered or unexported fields

Bytes is a wrapper type for a two-dimensional byte slice ([][]byte).

func NewBytes Uses

func NewBytes(n int) *Bytes

NewBytes returns a Bytes struct with enough capacity for n zero-length []byte values. It is legal to call Set on the returned Bytes at this point, but Get is undefined until at least one element is Set.

func (*Bytes) AppendSlice Uses

func (b *Bytes) AppendSlice(src *Bytes, destIdx, srcStartIdx, srcEndIdx int)

AppendSlice appends srcStartIdx inclusive and srcEndIdx exclusive []byte values from src into the receiver starting at destIdx.

func (*Bytes) AppendVal Uses

func (b *Bytes) AppendVal(v []byte)

AppendVal appends the given []byte value to the end of the receiver. A nil value will be "converted" into an empty byte slice.

func (*Bytes) AssertOffsetsAreNonDecreasing Uses

func (b *Bytes) AssertOffsetsAreNonDecreasing(n uint64)

AssertOffsetsAreNonDecreasing asserts that all b.offsets[:n+1] are non-decreasing.

func (*Bytes) CopySlice Uses

func (b *Bytes) CopySlice(src *Bytes, destIdx, srcStartIdx, srcEndIdx int)

CopySlice copies srcStartIdx inclusive and srcEndIdx exclusive []byte values from src into the receiver starting at destIdx. Similar to the copy builtin, min(dest.Len(), src.Len()) values will be copied. Note that if the length of the receiver is greater than the length of the source, bytes will have to be physically moved. Consider the following example: dest Bytes: "helloworld", offsets: []int32{0, 5}, lengths: []int32{5, 5} src Bytes: "a", offsets: []int32{0}, lengths: []int32{1} If we copy src into the beginning of dest, we will have to move "world" so that the result is: result Bytes: "aworld", offsets: []int32{0, 1}, lengths: []int32{1, 5} Similarly, if "a", is instead "alongerstring", "world" would have to be shifted right.

func (*Bytes) Get Uses

func (b *Bytes) Get(i int) []byte

Get returns the ith []byte in Bytes. Note that the returned byte slice is unsafe for reuse if any write operation happens. NOTE: if ith element was never set in any way, the behavior of Get is undefined.

func (*Bytes) Len Uses

func (b *Bytes) Len() int

Len returns how many []byte values the receiver contains.

func (*Bytes) Reset Uses

func (b *Bytes) Reset()

Reset resets the underlying Bytes for reuse. Note that this zeroes out the underlying bytes but doesn't change the length (see #42054 for the discussion on why simply truncating and setting b.maxSetIndex to 0 is not sufficient). TODO(asubiotto): Move towards removing Set in favor of AppendVal. At that

point we can reset the length to 0.

func (*Bytes) Set Uses

func (b *Bytes) Set(i int, v []byte)

Set sets the ith []byte in Bytes. Overwriting a value that is not at the end of the Bytes is not allowed since it complicates memory movement to make/take away necessary space in the flat buffer. Note that a nil value will be "converted" into an empty byte slice.

func (*Bytes) SetLength Uses

func (b *Bytes) SetLength(l int)

SetLength sets the length of this Bytes. Note that it will panic if there is not enough capacity.

func (*Bytes) Size Uses

func (b *Bytes) Size() uintptr

Size returns the total size of the receiver in bytes.

func (*Bytes) Slice Uses

func (b *Bytes) Slice(start, end int) *Bytes

Slice returns a shallow copy of the receiver Bytes struct sliced according to start and end. NOTE: slicing is a lightweight operation, so the underlying memory will be shared between the receiver and a newly created Bytes struct. Beware that modification of the newly created slice can invalidate the receiver's data. Use with caution. TODO(yuzefovich): consider removing this entirely or making it a noop.

func (*Bytes) String Uses

func (b *Bytes) String() string

String is used for debugging purposes.

func (*Bytes) ToArrowSerializationFormat Uses

func (b *Bytes) ToArrowSerializationFormat(n int) ([]byte, []int32)

ToArrowSerializationFormat returns a bytes slice and offsets that are Arrow-compatible. n is the number of elements to serialize.

func (*Bytes) UpdateOffsetsToBeNonDecreasing Uses

func (b *Bytes) UpdateOffsetsToBeNonDecreasing(n uint64)

UpdateOffsetsToBeNonDecreasing makes sure that b.offsets[:n+1] are non-decreasing which is an invariant that we need to maintain. It must be called by the colexec.Operator that is modifying this Bytes before returning it as an output. A convenient place for this is Batch.SetLength() method - we assume that *always*, before returning a batch, the length is set on it.

func (*Bytes) Zero Uses

func (b *Bytes) Zero()

Zero zeroes out the underlying bytes. Note that this doesn't change the length.

type CopySliceArgs Uses

type CopySliceArgs struct {
    // Sel64 overrides Sel. Used when the amount of data being copied exceeds the
    // representation capabilities of a []uint16.
    Sel64 []uint64
    // SelOnDest, if true, uses the selection vector as a lens into the
    // destination as well as the source. Normally, when SelOnDest is false, the
    // selection vector is applied to the source vector, but the results are
    // copied densely into the destination vector.
    SelOnDest bool

CopySliceArgs represents the extension of SliceArgs that is passed in to Vec.Copy.

type MemBatch Uses

type MemBatch struct {
    // contains filtered or unexported fields

MemBatch is an in-memory implementation of Batch.

func (*MemBatch) AppendCol Uses

func (m *MemBatch) AppendCol(t coltypes.T)

AppendCol implements the Batch interface.

func (*MemBatch) ColVec Uses

func (m *MemBatch) ColVec(i int) Vec

ColVec implements the Batch interface.

func (*MemBatch) ColVecs Uses

func (m *MemBatch) ColVecs() []Vec

ColVecs implements the Batch interface.

func (*MemBatch) Length Uses

func (m *MemBatch) Length() uint16

Length implements the Batch interface.

func (*MemBatch) Reset Uses

func (m *MemBatch) Reset(types []coltypes.T, length int)

Reset implements the Batch interface.

func (*MemBatch) ResetInternalBatch Uses

func (m *MemBatch) ResetInternalBatch()

ResetInternalBatch implements the Batch interface.

func (*MemBatch) Selection Uses

func (m *MemBatch) Selection() []uint16

Selection implements the Batch interface.

func (*MemBatch) SetLength Uses

func (m *MemBatch) SetLength(n uint16)

SetLength implements the Batch interface.

func (*MemBatch) SetSelection Uses

func (m *MemBatch) SetSelection(b bool)

SetSelection implements the Batch interface.

func (*MemBatch) Width Uses

func (m *MemBatch) Width() int

Width implements the Batch interface.

type Nulls Uses

type Nulls struct {
    // contains filtered or unexported fields

Nulls represents a list of potentially nullable values using a bitmap. It is intended to be used alongside a slice (e.g. in the Vec interface) -- if the ith bit is off, then the ith element in that slice should be treated as NULL.

func NewNulls Uses

func NewNulls(len int) Nulls

NewNulls returns a new nulls vector, initialized with a length.

func (*Nulls) Copy Uses

func (n *Nulls) Copy() Nulls

Copy returns a copy of n which can be modified independently.

func (*Nulls) MaybeHasNulls Uses

func (n *Nulls) MaybeHasNulls() bool

MaybeHasNulls returns true if the column possibly has any null values, and returns false if the column definitely has no null values.

func (*Nulls) NullAt Uses

func (n *Nulls) NullAt(i uint16) bool

NullAt returns true if the ith value of the column is null.

func (*Nulls) NullAt64 Uses

func (n *Nulls) NullAt64(i uint64) bool

NullAt64 returns true if the ith value of the column is null.

func (*Nulls) NullBitmap Uses

func (n *Nulls) NullBitmap() []byte

NullBitmap returns the null bitmap.

func (*Nulls) Or Uses

func (n *Nulls) Or(n2 *Nulls) *Nulls

Or returns a new Nulls vector where NullAt(i) iff n1.NullAt(i) or n2.NullAt(i).

func (*Nulls) SetNull Uses

func (n *Nulls) SetNull(i uint16)

SetNull sets the ith value of the column to null.

func (*Nulls) SetNull64 Uses

func (n *Nulls) SetNull64(i uint64)

SetNull64 sets the ith value of the column to null.

func (*Nulls) SetNullBitmap Uses

func (n *Nulls) SetNullBitmap(bm []byte, size int)

SetNullBitmap sets the null bitmap. size corresponds to how many elements this bitmap represents. The bits past the end of this size will be set to valid.

func (*Nulls) SetNullRange Uses

func (n *Nulls) SetNullRange(start uint64, end uint64)

SetNullRange sets all the values in [start, end) to null.

func (*Nulls) SetNulls Uses

func (n *Nulls) SetNulls()

SetNulls sets the column to have only null values.

func (*Nulls) Slice Uses

func (n *Nulls) Slice(start uint64, end uint64) Nulls

Slice returns a new Nulls representing a slice of the current Nulls from [start, end).

func (*Nulls) Truncate Uses

func (n *Nulls) Truncate(start uint16)

Truncate sets all values with index greater than or equal to start to null.

func (*Nulls) UnsetNull Uses

func (n *Nulls) UnsetNull(i uint16)

UnsetNull unsets the ith value of the column.

func (*Nulls) UnsetNull64 Uses

func (n *Nulls) UnsetNull64(i uint64)

UnsetNull64 unsets the ith values of the column.

func (*Nulls) UnsetNullRange Uses

func (n *Nulls) UnsetNullRange(start, end uint64)

UnsetNullRange unsets all the nulls in the range [start, end). After using UnsetNullRange, n might not contain any null values, but maybeHasNulls could still be true.

func (*Nulls) UnsetNulls Uses

func (n *Nulls) UnsetNulls()

UnsetNulls sets the column to have no null values.

func (*Nulls) UnsetNullsAfter Uses

func (n *Nulls) UnsetNullsAfter(idx uint16)

UnsetNullsAfter sets all values with index greater than or equal to idx to non-null.

type SliceArgs Uses

type SliceArgs struct {
    // ColType is the type of both the destination and source slices.
    ColType coltypes.T
    // Src is the data being appended.
    Src Vec
    // Sel is an optional slice specifying indices to append to the destination
    // slice. Note that Src{Start,End}Idx apply to Sel.
    Sel []uint16
    // DestIdx is the first index that Append will append to.
    DestIdx uint64
    // SrcStartIdx is the index of the first element in Src that Append will
    // append.
    SrcStartIdx uint64
    // SrcEndIdx is the exclusive end index of Src. i.e. the element in the index
    // before SrcEndIdx is the last element appended to the destination slice,
    // similar to Src[SrcStartIdx:SrcEndIdx].
    SrcEndIdx uint64

SliceArgs represents the arguments passed in to Vec.Append and Nulls.set.

type Vec Uses

type Vec interface {
    // Type returns the type of data stored in this Vec.
    Type() coltypes.T

    // TODO(jordan): is a bitmap or slice of bools better?
    // Bool returns a bool list.
    Bool() []bool
    // Int16 returns an int16 slice.
    Int16() []int16
    // Int32 returns an int32 slice.
    Int32() []int32
    // Int64 returns an int64 slice.
    Int64() []int64
    // Float64 returns a float64 slice.
    Float64() []float64
    // Bytes returns a flat Bytes representation.
    Bytes() *Bytes
    // TODO(jordan): should this be [][]byte?
    // Decimal returns an apd.Decimal slice.
    Decimal() []apd.Decimal
    // Timestamp returns a time.Time slice.
    Timestamp() []time.Time

    // Col returns the raw, typeless backing storage for this Vec.
    Col() interface{}

    // SetCol sets the member column (in the case of mutable columns).

    // Append uses SliceArgs to append elements of a source Vec into this Vec.
    // It is logically equivalent to:
    // destVec = append(destVec[:args.DestIdx], args.Src[args.SrcStartIdx:args.SrcEndIdx])
    // An optional Sel slice can also be provided to apply a filter on the source
    // Vec.
    // Refer to the SliceArgs comment for specifics and TestAppend for examples.

    // Copy uses CopySliceArgs to copy elements of a source Vec into this Vec. It is
    // logically equivalent to:
    // copy(destVec[args.DestIdx:], args.Src[args.SrcStartIdx:args.SrcEndIdx])
    // An optional Sel slice can also be provided to apply a filter on the source
    // Vec.
    // Refer to the CopySliceArgs comment for specifics and TestCopy for examples.

    // Slice returns a new Vec representing a slice of the current Vec from
    // [start, end).
    Slice(colType coltypes.T, start uint64, end uint64) Vec

    // PrettyValueAt returns a "pretty"value for the idx'th value in this Vec.
    // It uses the reflect package and is not suitable for calling in hot paths.
    PrettyValueAt(idx uint16, colType coltypes.T) string

    // MaybeHasNulls returns true if the column possibly has any null values, and
    // returns false if the column definitely has no null values.
    MaybeHasNulls() bool

    // Nulls returns the nulls vector for the column.
    Nulls() *Nulls

    // SetNulls sets the nulls vector for this column.

    // SetLength sets the length of the slice that is underlying this Vec. Note
    // that the length of the batch which this Vec belongs to "takes priority".
    // contains filtered or unexported methods

Vec is an interface that represents a column vector that's accessible by Go native types.

func NewMemColumn Uses

func NewMemColumn(t coltypes.T, n int) Vec

NewMemColumn returns a new memColumn, initialized with a length.

Package coldata imports 8 packages (graph) and is imported by 25 packages. Updated 2019-11-12. Refresh now. Tools for package owners.