Documentation
¶
Overview ¶
Package sortio provides facilities for sorting slice outputs and merging and reducing sorted record streams.
Index ¶
- func NewMergeReader(ctx context.Context, typ slicetype.Type, readers []sliceio.Reader) (sliceio.Reader, error)
- func Reduce(typ slicetype.Type, name string, readers []sliceio.Reader, ...) sliceio.Reader
- func SortReader(ctx context.Context, spillTarget int, typ slicetype.Type, r sliceio.Reader) (sliceio.Reader, error)
- type FrameBuffer
- type FrameBufferHeap
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewMergeReader ¶
func NewMergeReader(ctx context.Context, typ slicetype.Type, readers []sliceio.Reader) (sliceio.Reader, error)
NewMergeReader returns a new Reader that is sorted by its prefix columns. The readers to be merged must already be sorted.
func Reduce ¶
func Reduce(typ slicetype.Type, name string, readers []sliceio.Reader, combiner slicefunc.Func) sliceio.Reader
Reduce returns a Reader that merges and reduces a set of sorted (and possibly combined) readers. Reduce panics if the provided type is not reducable.
func SortReader ¶
func SortReader(ctx context.Context, spillTarget int, typ slicetype.Type, r sliceio.Reader) (sliceio.Reader, error)
SortReader sorts a Reader by its prefix columns. SortReader may spill to disk, in which case it targets spill file sizes of spillTarget (in bytes). Because the encoded size of objects is not known in advance, sortReader uses a "canary" batch size of ~16k rows in order to estimate the size of future reads. The estimate is revisited on every subsequent fill and adjusted if it is violated by more than 5%.
Types ¶
type FrameBuffer ¶
type FrameBuffer struct {
// Frame is the buffer into which new data are read. The buffer is
// always allocated externally and must be nonempty.
frame.Frame
// Reader is the slice reader from which the buffer is filled.
sliceio.Reader
// Index, Len is current index and length of the frame.
Index, Len int
// Off is the global offset of this framebuffer. It is added to
// the index to compute the buffer's current position.
Off int
}
A FrameBuffer is a buffered frame. The frame is filled from a reader, and maintains a current index and length.
func (*FrameBuffer) Fill ¶
func (f *FrameBuffer) Fill(ctx context.Context) error
Fill (re-) fills the FrameBuffer when it's empty. An error is returned if the underlying reader returns an error. EOF is returned if no more data are available.
func (FrameBuffer) Pos ¶
func (f FrameBuffer) Pos() int
Pos returns this frame buffer's current position.
type FrameBufferHeap ¶
type FrameBufferHeap struct {
Buffers []*FrameBuffer
// Less compares the current index of buffers i and j.
LessFunc func(i, j int) bool
}
FrameBufferHeap implements a heap of FrameBuffers, ordered by the provided sorter.
func (*FrameBufferHeap) Len ¶
func (f *FrameBufferHeap) Len() int
func (*FrameBufferHeap) Less ¶
func (f *FrameBufferHeap) Less(i, j int) bool
func (*FrameBufferHeap) Pop ¶
func (f *FrameBufferHeap) Pop() interface{}
Pop removes the FrameBuffer with the smallest priority from the heap.
func (*FrameBufferHeap) Push ¶
func (f *FrameBufferHeap) Push(x interface{})
Push pushes a FrameBuffer onto the heap.
func (*FrameBufferHeap) Swap ¶
func (f *FrameBufferHeap) Swap(i, j int)
Source Files