bigslice: github.com/grailbio/bigslice/sortio Index | Files

package sortio

import "github.com/grailbio/bigslice/sortio"

Package sortio provides facilities for sorting slice outputs and merging and reducing sorted record streams.

Index

Package Files

reader.go sort.go

func NewMergeReader Uses

func NewMergeReader(ctx context.Context, typ slicetype.Type, readers []sliceio.Reader) (sliceio.Reader, error)

NewMergeReader returns a new Reader that is sorted by its prefix columns. The readers to be merged must already be sorted.

func Reduce Uses

func Reduce(typ slicetype.Type, name string, readers []sliceio.Reader, combiner slicefunc.Func) sliceio.Reader

Reduce returns a Reader that merges and reduces a set of sorted (and possibly combined) readers. Reduce panics if the provided type is not reducable.

func SortReader Uses

func SortReader(ctx context.Context, spillTarget int, typ slicetype.Type, r sliceio.Reader) (sliceio.Reader, error)

SortReader sorts a Reader by its prefix columns. SortReader may spill to disk, in which case it targets spill file sizes of spillTarget (in bytes). Because the encoded size of objects is not known in advance, sortReader uses a "canary" batch size of ~16k rows in order to estimate the size of future reads. The estimate is revisited on every subsequent fill and adjusted if it is violated by more than 5%.

type FrameBuffer Uses

type FrameBuffer struct {
    // Frame is the buffer into which new data are read. The buffer is
    // always allocated externally and must be nonempty.
    frame.Frame
    // Reader is the slice reader from which the buffer is filled.
    sliceio.Reader
    // Index, Len is current index and length of the frame.
    Index, Len int

    // Off is the global offset of this framebuffer. It is added to
    // the index to compute the buffer's current position.
    Off int
}

A FrameBuffer is a buffered frame. The frame is filled from a reader, and maintains a current index and length.

func (*FrameBuffer) Fill Uses

func (f *FrameBuffer) Fill(ctx context.Context) error

Fill (re-) fills the FrameBuffer when it's empty. An error is returned if the underlying reader returns an error. EOF is returned if no more data are available.

func (FrameBuffer) Pos Uses

func (f FrameBuffer) Pos() int

Pos returns this frame buffer's current position.

type FrameBufferHeap Uses

type FrameBufferHeap struct {
    Buffers []*FrameBuffer
    // Less compares the current index of buffers i and j.
    LessFunc func(i, j int) bool
}

FrameBufferHeap implements a heap of FrameBuffers, ordered by the provided sorter.

func (*FrameBufferHeap) Len Uses

func (f *FrameBufferHeap) Len() int

func (*FrameBufferHeap) Less Uses

func (f *FrameBufferHeap) Less(i, j int) bool

func (*FrameBufferHeap) Pop Uses

func (f *FrameBufferHeap) Pop() interface{}

Pop removes the FrameBuffer with the smallest priority from the heap.

func (*FrameBufferHeap) Push Uses

func (f *FrameBufferHeap) Push(x interface{})

Push pushes a FrameBuffer onto the heap.

func (*FrameBufferHeap) Swap Uses

func (f *FrameBufferHeap) Swap(i, j int)

Package sortio imports 11 packages (graph) and is imported by 2 packages. Updated 2019-11-19. Refresh now. Tools for package owners.