Documentation
¶
Index ¶
Constants ¶
const ( // ChunkHeaderSize is the fixed header size for a chunk. ChunkHeaderSize = 28 // ChunkSize is the fixed size of a chunk, including its header. ChunkSize = 32 << 10 // MaxChunkPayloadSize is the maximum size of payload a chunk can carry. MaxChunkPayloadSize = ChunkSize - ChunkHeaderSize )
const NumMagicBytes = 8
NumMagicBytes is the size of a magic header stored at the beginning of every chunk.
Variables ¶
var IEEECRC *crc32.Table
IEEECRC is used to compute IEEE CRC chunk checksums.
var MagicHeader = MagicBytes{0xd9, 0xe1, 0xd9, 0x5c, 0xc2, 0x16, 0x04, 0xf7}
MagicHeader is a constant random 8 bytes used to distinguish the header block of a v2 recordio file.
var MagicInvalid = MagicBytes{0xe4, 0xe7, 0x9a, 0xc1, 0xb3, 0xf6, 0xb7, 0xa2}
MagicInvalid is a sentinel. It is never stored in storage.
var MagicLegacyUnpacked = MagicBytes{0xfc, 0xae, 0x95, 0x31, 0xf0, 0xd9, 0xbd, 0x20}
MagicLegacy is the legacy "unpacked" block header.
var MagicPacked = MagicBytes{0x2e, 0x76, 0x47, 0xeb, 0x34, 0x07, 0x3c, 0x2e}
MagicPacked is the legacy "packed" block header, and the v2 data block header.
var MagicTrailer = MagicBytes{0xfe, 0xba, 0x1a, 0xd7, 0xcb, 0xdf, 0x75, 0x3a}
MagicTrailer is a constant random 8 bytes used to distinguish the trailer block of a v2 recordio file.
var MaxReadRecordSize = uint64(1 << 29)
MaxReadRecordSize defines a max size for a record when reading to avoid crashes for unreasonable requests.
Functions ¶
Types ¶
type ChunkScanner ¶
type ChunkScanner struct {
// contains filtered or unexported fields
}
ChunkScanner reads a sequence of chunks and reconstructs a logical block. Thread compatible.
func NewChunkScanner ¶
func NewChunkScanner(r io.ReadSeeker, err *errors.Once) *ChunkScanner
NewChunkScanner creates a new chunk scanner. Any error is reported through "err".
func (*ChunkScanner) Block ¶
func (r *ChunkScanner) Block() (MagicBytes, [][]byte)
Block returns the current block contents.
REQUIRES: Last Scan() call returned true.
func (*ChunkScanner) LimitShard ¶
func (r *ChunkScanner) LimitShard(start, limit, nshard int)
LimitShard limits this scanner to scan the blocks belonging to a shard range [start,limit) out of [0, nshard). The shard range begins at the scanner's current offset, which must be on a block boundary. The file (beginning at the current scanner offset) is divided into n shards. Each shard scans blocks until the next segment. If a shard begins in the middle of a block, that block belongs to the previous shard.
func (*ChunkScanner) ReadLastBlock ¶
func (r *ChunkScanner) ReadLastBlock() (MagicBytes, [][]byte)
ReadLastBlock reads the trailer. Sets err if the trailer does not exist, or is corrupt. After the call, the read pointer is at an undefined position so the user must call Seek() explicitly.
func (*ChunkScanner) Scan ¶
func (r *ChunkScanner) Scan() bool
Scan reads the next block. It returns false on EOF or any error. AnB error is reported in r.Err()
func (*ChunkScanner) Seek ¶
func (r *ChunkScanner) Seek(off int64)
Seek moves the read pointer so that next Scan() will move to the block at the given file offset. Any error is reported in r.Err()
func (*ChunkScanner) Tell ¶
func (r *ChunkScanner) Tell() int64
Tell returns the file offset of the next block to be read. Any error is reported in r.Err()
type ChunkWriter ¶
type ChunkWriter struct {
// contains filtered or unexported fields
}
ChunkWriter implements low-level block-write operations. It takes logical block and stores it as a sequence of chunks. Thread compatible.
func NewChunkWriter ¶
func NewChunkWriter(w io.Writer, err *errors.Once) *ChunkWriter
NewChunkWriter creates a new chunk writer. Any error is reported through "err".
func (*ChunkWriter) Len ¶
func (w *ChunkWriter) Len() int64
Len returns the number of bytes successfully written so far. The value is meaningful only when err.Err()==nil.
func (*ChunkWriter) Write ¶
func (w *ChunkWriter) Write(magic MagicBytes, payload []byte)
Write one block. An error is reported through w.err.
type MagicBytes ¶
type MagicBytes = [NumMagicBytes]byte
MagicBytes is stored in the first 8 bytes of any chunk.
Source Files