bigslice: github.com/grailbio/bigslice/sliceio Index | Files

package sliceio

import "github.com/grailbio/bigslice/sliceio"

Package sliceio provides utilities for managing I/O for Bigslice operations.

Index

Package Files

codec.go reader.go scanner.go spiller.go

Variables

var EOF = errors.New("EOF")

EOF is the error returned by Reader.Read when no more data is available. EOF is intended as a sentinel error: it signals a graceful end of output. If output terminates unexpectedly, a different error should be returned.

var SpillBatchSize = defaultChunksize

SpillBatchSize determines the amount of batching used in each spill file. A single read of a spill file produces this many rows. SpillBatchSize then trades off memory footprint for encoding size.

func ReadAll Uses

func ReadAll(ctx context.Context, r Reader, columns ...interface{}) error

ReadAll copies all elements from reader r into the provided column pointers. ReadAll is not tuned for performance and is intended for testing purposes.

func ReadFull Uses

func ReadFull(ctx context.Context, r Reader, f frame.Frame) (n int, err error)

ReadFull reads the full length of the frame. ReadFull reads short frames only on EOF.

type ClosingReader Uses

type ClosingReader struct {
    // contains filtered or unexported fields
}

ClosingReader closes the wrapped ReadCloser when Read returns any error.

func NewClosingReader Uses

func NewClosingReader(r ReadCloser) *ClosingReader

NewClosingReader returns a new ClosingReader for r.

func (*ClosingReader) Read Uses

func (c *ClosingReader) Read(ctx context.Context, out frame.Frame) (int, error)

Read implements sliceio.Reader.

type EmptyReader Uses

type EmptyReader struct{}

EmptyReader returns an EOF.

func (EmptyReader) Read Uses

func (EmptyReader) Read(ctx context.Context, f frame.Frame) (int, error)

type Encoder Uses

type Encoder struct {
    // contains filtered or unexported fields
}

An Encoder manages transmission of slices through an underlying io.Writer. The stream of slice values represented by batches of rows stored in column-major order. Streams can be read by a Decoder.

func NewEncoder Uses

func NewEncoder(w io.Writer) *Encoder

NewEncoder returns a a new Encoder that streams slices into the provided writer.

func (*Encoder) Encode Uses

func (e *Encoder) Encode(f frame.Frame) error

Encode encodes a batch of rows and writes the encoded output into the encoder's writer.

type PprofReader Uses

type PprofReader struct {
    Reader
    Label string
}

PprofReader executes Read in a labeled Context.

func (*PprofReader) Read Uses

func (r *PprofReader) Read(ctx context.Context, frame frame.Frame) (n int, err error)

type ReadCloser Uses

type ReadCloser interface {
    Reader
    io.Closer
}

ReadCloser groups the Read and Close methods.

func MultiReader Uses

func MultiReader(readers ...ReadCloser) ReadCloser

MultiReader returns a ReadCloser that's the logical concatenation of the provided input readers. Once every underlying ReadCloser has returned EOF, Read will return EOF, too. Non-EOF errors are returned immediately.

func NopCloser Uses

func NopCloser(r Reader) ReadCloser

type Reader Uses

type Reader interface {
    // Read reads a vector of records from the underlying Slice. Each
    // passed-in column should be a value containing a slice of column
    // values. The number of columns should match the number of columns
    // in the slice; their types should match the corresponding column
    // types of the slice. Each column should have the same slice
    // length.
    //
    // Read returns the total number of records read, or an error. When
    // no more records are available, Read returns EOF. Read may return
    // EOF when n > 0. In this case, n records were read, but no more
    // are available.
    //
    // Read should never reuse any allocated memory in the frame;
    // its callers should not mutate the data returned.
    //
    // Read should not be called concurrently.
    Read(ctx context.Context, frame frame.Frame) (int, error)
}

A Reader represents a stateful stream of records. Each call to Read reads the next set of available records.

func ErrReader Uses

func ErrReader(err error) Reader

ErrReader returns a reader that returns the provided error on every call to read. ErrReader panics if err is nil.

func FrameReader Uses

func FrameReader(frame frame.Frame) Reader

FrameReader returns a Reader that reads the provided Frame to completion.

func NewDecodingReader Uses

func NewDecodingReader(r io.Reader) Reader

NewDecodingReader returns a new Reader that decodes values from the provided stream. Since values are streamed in vectors, decoding reader must buffer values until they are read by the consumer.

type ReaderWithCloseFunc Uses

type ReaderWithCloseFunc struct {
    Reader
    CloseFunc func() error
}

ReaderWithCloseFunc is a ReadCloser that wraps an existing Reader and uses a provided function for its Close.

func (ReaderWithCloseFunc) Close Uses

func (r ReaderWithCloseFunc) Close() error

Close implements io.Closer.

type Scanner Uses

type Scanner struct {
    // contains filtered or unexported fields
}

A Scanner provides a convenient interface for reading records (e.g. from a Slice or a shard of a Slice). Successive calls to Scan (or Scanv) returns the next record (batch of records). Scanning stops when no more data are available or if an error is encountered. Scan returns true while it's safe to continue scanning. When scanning is complete, the user should inspect the scanner's error to see if scanning stopped because of an EOF or because another error occurred.

Callers should not mix calls to Scan and Scanv.

func NewScanner Uses

func NewScanner(typ slicetype.Type, r ReadCloser) *Scanner

NewScanner returns a new scanner of records of type typ from reader r.

func (*Scanner) Close Uses

func (s *Scanner) Close() error

Close releases resources used by the scanner. This must be called exactly once on the scanner returned by NewScanner.

func (*Scanner) Err Uses

func (s *Scanner) Err() error

Err returns any error that occurred while scanning.

func (*Scanner) Scan Uses

func (s *Scanner) Scan(ctx context.Context, out ...interface{}) bool

Scan the next record into the provided columns. Scanning fails if the columns do not match arity and type with the underlying data set. Scan returns true while no errors are encountered and there remains data to be scanned.

func (*Scanner) Scanv Uses

func (s *Scanner) Scanv(ctx context.Context, out ...interface{}) (int, bool)

Scanv scans a batch of elements into the provided column vectors. Each column should be a slice of the correct type. Scanv fails when the type or arity of the column vectors do not match the underlying dataset. The number of records scanned is returned together with a boolean indicating whether scanning should continue, as in Scan.

type Spiller Uses

type Spiller string

A Spiller manages a set of spill files.

func NewSpiller Uses

func NewSpiller(name string) (Spiller, error)

NewSpiller creates and returns a new spiller backed by a temporary directory. Spillers do not guarantee that the order of spillers returned matches the order of spills.

func (Spiller) Cleanup Uses

func (dir Spiller) Cleanup() error

Cleanup removes the spiller's temporary files. It is safe to call Cleanup after Readers(), but before reading is done.

func (Spiller) ClosingReaders Uses

func (dir Spiller) ClosingReaders() ([]Reader, error)

ClosingReaders returns a reader for each spiller file. The readers close the underlying file when Read returns a non-nil error (otherwise the underlying file resource will leak).

func (Spiller) Readers Uses

func (dir Spiller) Readers() ([]ReadCloser, error)

Readers returns a ReadCloser for each spiller file.

func (Spiller) Spill Uses

func (dir Spiller) Spill(frame frame.Frame) (int, error)

Spill spills the provided frame to a new file in the spiller. Spill returns the file's encoded size, or an error. The frame is encoded in batches of SpillBatchSize.

Package sliceio imports 22 packages (graph) and is imported by 7 packages. Updated 2019-11-19. Refresh now. Tools for package owners.