README
¶
lengthprefixed
lengthprefixed implements a Reader and Writer for reading, and writing, data in length-prefixed frames.
This is useful for efficiently streaming, or storing, large quantities of serialized data structures together when they are not self-delimiting, e.g. protocol buffer messages.
A length-prefixed frame adds 9-18 bytes of overhead to user-provided data:
a varint representing length of user-provided data, and then an 8-byte
(uint64) checksum generated by the xxhash algorithm.
Frame Structure
Each frame contains three parts: length (4 bytes), chunk data (length bytes),
and checksum (8 bytes).
0 1-10 N N+8
+--------+================+----------+
| length | frame data | checksum |
+--------+================+----------+
The lengthprefixed API only requires that the user interact with the frame data. The length and checksum components are hidden from the user.
Usage
package main
import (
"bytes"
"fmt"
"github.com/dpirotte/lengthprefixed"
)
func main() {
var buf bytes.Buffer
w := lengthprefixed.NewWriter(&buf)
w.Write([]byte("This string would more likely be a protobuf message."))
r := lengthprefixed.NewReader(&buf)
b, err := r.ReadFrame()
if err != nil {
panic(err)
}
fmt.Println(string(b))
}
Documentation
¶
Overview ¶
Package lengthprefixed implements a Reader and Writer for reading, and writing, data in length-prefixed frames.
Each frame contains three parts: a varint representing the length of the data, N bytes of data, and 8 bytes (uint64) of checksum for the data.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrChecksum = errors.New("lengthprefixed: invalid checksum")
ErrChecksum is returned when frame data's computed checksum match the provided checksum
Functions ¶
This section is empty.
Types ¶
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
A Reader is a wrapper designed to read frames from an underlying io.Reader.
func (*Reader) ReadFrame ¶
ReadFrame returns the data component of length-prefixed frames. It first reads a varint from the io.Reader to determine how many bytes to return in the byteslice. Then, it reads that many bytes into a byteslice and checksums against the following int64 via xxhash. If the checksum matches, the byteslice will be returned to the caller. Otherwise, an ErrChecksum will be returned.
Note: The caller takes responsibility for watching for io.EOF, which will be bubbled up from the underlying reader.
Source Files
Directories