luci: Index | Files

package recordio

import ""

Package recordio implements a basic RecordIO reader and writer.

Each RecordIO frame begins with a Uvarint ( containing the size of the frame, followed by that many bytes of frame data.

The frame protocol does not handle data integrity; that is left to the outer protocol or medium which uses the frame.


Package Files

doc.go reader.go size.go writer.go


var ErrFrameTooLarge = fmt.Errorf("frame: frame size exceeds maximum")

ErrFrameTooLarge is an error that is returned if a frame that is larger than the maximum allowed size (not including the frame header) is read.

func FrameHeaderSize Uses

func FrameHeaderSize(s int64) int

FrameHeaderSize calculates the size of the RecordIO frame header for a given amount of data.

A RecordIO frame header is a Uvarint containing the length. Uvarint values contain 7 bits of size data and 1 continuation bit (see encoding/binary).

func Split Uses

func Split(data []byte) (records [][]byte, err error)

Split splits the supplied buffer into its component records.

This method implements zero-copy segmentation, so the individual records are slices of the original data set.

func WriteFrame Uses

func WriteFrame(w io.Writer, frame []byte) (int, error)

WriteFrame writes a single frame to an io.Writer.

type Reader Uses

type Reader interface {
    // ReadFrame reads the next frame, returning the frame's size and an io.Reader
    // for that frame's data. The io.Reader is restricted such that it cannot read
    // past the frame.
    // The frame must be fully read before another Reader call can be made.
    // Failure to do so will cause the Reader to become unsynchronized.
    ReadFrame() (int64, *io.LimitedReader, error)

    // ReadFrame returns the contents of the next frame. If there are no more
    // frames available, ReadFrame will return io.EOF.
    ReadFrameAll() ([]byte, error)

Reader reads individual frames from a frame-formatted input Reader.

func NewReader Uses

func NewReader(r io.Reader, maxSize int64) Reader

NewReader creates a new Reader which reads frame data from the supplied Reader instance.

If the Reader instance is also an io.ByteReader, its ReadByte method will be used directly.

type Writer Uses

type Writer interface {

    // Flush writes the buffered frame
    Flush() error

    // Reset clears the writer state and attaches it to a new inner Writer
    // instance.

Writer implements the io.Writer interface. Data written to the Writer is translated into a series of frames. Each frame is spearated by a call to Flush.

Frame boundaries are created by calling Flush. Flush will always write a frame, even if the frame's data size is zero.

Data written over consecutive Write calls belongs to the same frame. It is buffered until a frame boundary is created via Flush().

func NewWriter Uses

func NewWriter(w io.Writer) Writer

NewWriter creates a new Writer instance that data as frames to an underlying io.Writer.

Package recordio imports 4 packages (graph) and is imported by 19 packages. Updated 2020-10-29. Refresh now. Tools for package owners.