base: github.com/grailbio/base/recordio/internal Index | Files

package internal

import "github.com/grailbio/base/recordio/internal"

Index

Package Files

chunk.go magic.go

Constants

const NumMagicBytes = 8

NumMagicBytes is the size of a magic header stored at the beginning of every chunk.

Variables

var IEEECRC *crc32.Table

IEEECRC is used to compute IEEE CRC chunk checksums.

var MagicHeader = MagicBytes{0xd9, 0xe1, 0xd9, 0x5c, 0xc2, 0x16, 0x04, 0xf7}

MagicHeader is a constant random 8 bytes used to distinguish the header block of a v2 recordio file.

var MagicInvalid = MagicBytes{0xe4, 0xe7, 0x9a, 0xc1, 0xb3, 0xf6, 0xb7, 0xa2}

MagicInvalid is a sentinel. It is never stored in storage.

var MagicLegacyUnpacked = MagicBytes{0xfc, 0xae, 0x95, 0x31, 0xf0, 0xd9, 0xbd, 0x20}

MagicLegacy is the legacy "unpacked" block header.

var MagicPacked = MagicBytes{0x2e, 0x76, 0x47, 0xeb, 0x34, 0x07, 0x3c, 0x2e}

MagicPacked is the legacy "packed" block header, and the v2 data block header.

var MagicTrailer = MagicBytes{0xfe, 0xba, 0x1a, 0xd7, 0xcb, 0xdf, 0x75, 0x3a}

MagicTrailer is a constant random 8 bytes used to distinguish the trailer block of a v2 recordio file.

var MaxReadRecordSize = uint64(1 << 29)

MaxReadRecordSize defines a max size for a record when reading to avoid crashes for unreasonable requests.

func Seek Uses

func Seek(r io.ReadSeeker, off int64) error

Seek to "off". Returns nil iff the seek ptr moves to "off".

type ChunkScanner Uses

type ChunkScanner struct {
    // contains filtered or unexported fields
}

ChunkScanner reads a sequence of chunks and reconstructs a logical block. Thread compatible.

func NewChunkScanner Uses

func NewChunkScanner(r io.ReadSeeker, err *errors.Once) *ChunkScanner

NewChunkScanner creates a new chunk scanner. Any error is reported through "err".

func (*ChunkScanner) Block Uses

func (r *ChunkScanner) Block() (MagicBytes, [][]byte)

Block returns the current block contents.

REQUIRES: Last Scan() call returned true.

func (*ChunkScanner) LimitShard Uses

func (r *ChunkScanner) LimitShard(start, limit, nshard int)

LimitShard limits this scanner to scan the blocks belonging to a shard range [start,limit) out of [0, nshard). The shard range begins at the scanner's current offset, which must be on a block boundary. The file (beginning at the current scanner offset) is divided into n shards. Each shard scans blocks until the next segment. If a shard begins in the middle of a block, that block belongs to the previous shard.

func (*ChunkScanner) ReadLastBlock Uses

func (r *ChunkScanner) ReadLastBlock() (MagicBytes, [][]byte)

ReadLastBlock reads the trailer. Sets err if the trailer does not exist, or is corrupt. After the call, the read pointer is at an undefined position so the user must call Seek() explicitly.

func (*ChunkScanner) Scan Uses

func (r *ChunkScanner) Scan() bool

Scan reads the next block. It returns false on EOF or any error. AnB error is reported in r.Err()

func (*ChunkScanner) Seek Uses

func (r *ChunkScanner) Seek(off int64)

Seek moves the read pointer so that next Scan() will move to the block at the given file offset. Any error is reported in r.Err()

func (*ChunkScanner) Tell Uses

func (r *ChunkScanner) Tell() int64

Tell returns the file offset of the next block to be read. Any error is reported in r.Err()

type ChunkWriter Uses

type ChunkWriter struct {
    // contains filtered or unexported fields
}

ChunkWriter implements low-level block-write operations. It takes logical block and stores it as a sequence of chunks. Thread compatible.

func NewChunkWriter Uses

func NewChunkWriter(w io.Writer, err *errors.Once) *ChunkWriter

NewChunkWriter creates a new chunk writer. Any error is reported through "err".

func (*ChunkWriter) Len Uses

func (w *ChunkWriter) Len() int64

Len returns the number of bytes successfully written so far. The value is meaningful only when err.Err()==nil.

func (*ChunkWriter) Write Uses

func (w *ChunkWriter) Write(magic MagicBytes, payload []byte)

Write one block. An error is reported through w.err.

type MagicBytes Uses

type MagicBytes = [NumMagicBytes]byte

MagicBytes is stored in the first 8 bytes of any chunk.

Package internal imports 7 packages (graph) and is imported by 2 packages. Updated 2019-01-22. Refresh now. Tools for package owners.