Documentation
¶
Overview ¶
Package recio enables high-performance buffered I/O of record-oriented streams.
It allows user-defined go types to be atomically read and written from io.Readers and to io.Writers by implementing simple interfaces. These types can encode to arbitrary forms, be it fixed or variable length, size-prefixed or character-delimited representations.
The package provides a BufferedReader and a BufferedWriter that rely on pre-allocated buffers, enabling streaming I/O with zero-allocation in the hot path.
Both BufferedReader and BufferedWriter offer an optional manual mode, where Read and Write calls that would result in a blocking operation return with an error (ErrMustFill, ErrMustFlush), giving the user an opportunity to implement custom logic around the Fill and Flush calls that perform the actual I/O. This allows for example to Flush a sink when a source is going to block on Read, thus providing unbuffered-like behavior without the performance hit.
LIMITATIONS: BufferedReader and BufferedWriter can only handle records that can be fit in their internal buffers in encoded form. Trying to read or write larger records will fail with err == ErrTooLarge. Record sizes have to fit in a 32 bits signed int to satisfy the interfaces. Thus records cannot exceed 2,147,483,647 bytes in length (approx. 2GB).
Index ¶
Constants ¶
This section is empty.
Variables ¶
Functions ¶
This section is empty.
Types ¶
type AtomicReader ¶
type AtomicReader struct {
// contains filtered or unexported fields
}
AtomicReader provides atomicity guarantees on records read from unreliable channels and storage media. It does so by wrapping a Reader and checking on each Read that the record's checksum matches its data.
func NewAtomicReader ¶
func NewAtomicReader(r Reader) (ar *AtomicReader)
NewAtomicReader creates a new AtomicReader wrapping the provided Reader.
type AtomicWriter ¶
type AtomicWriter struct {
// contains filtered or unexported fields
}
AtomicWriter provides atomicity guarantees on records written to unreliable channels and storage media. It does so by wrapping a Writer and appending checksums to records on write.
func NewAtomicWriter ¶
func NewAtomicWriter(w Writer) (aw *AtomicWriter)
NewAtomicWriter returns a new AtomicWriter wrapping the provided Writer.
type BufferedReader ¶
type BufferedReader struct {
// contains filtered or unexported fields
}
BufferedReader implements buffered decoding of records from an io.Reader.
func NewBufferedReader ¶
func NewBufferedReader(r io.Reader, size int, mode IOMode) (br *BufferedReader)
NewBufferedReader returns a new BufferedReader whose buffer has the specified size. If mode is set to ModeManual, Read operations that would trigger a blocking Fill from the underlying io.Reader will return with err == ErrMustFill. The caller must then call Fill manually before calling Read again. If mode is set to ModeAuto the reader will Fill its internal buffer transparently.
func (*BufferedReader) Fill ¶
func (br *BufferedReader) Fill() (err error)
Fill tries to fill the reader's internal buffer by reading from the underlying io.Reader.
func (*BufferedReader) Read ¶
func (br *BufferedReader) Read(v Decoder) (n int, err error)
Read decodes one record into v. If the reader's internal buffer does not contain enough data to decode a complete record, either it is automatically filled from the underlying io.Reader in auto mode, or Read returns with err == ErrMustFill in manual mode. Once all records have been read from the underlying io.Reader, Read fails with err == io.EOF. If EOF has been reached but the reader's internal buffer still contains a partial record, Read fails with err == io.ErrUnexpectedEOF. If a record cannot be entirely fit in the reader's internal buffer, Read fails with err == ErrTooLarge.
func (*BufferedReader) Reset ¶
func (br *BufferedReader) Reset(r io.Reader)
Reset discards any buffered data, resets all state, and switches the buffered reader to read from r.
type BufferedWriter ¶
type BufferedWriter struct {
// contains filtered or unexported fields
}
BufferedWriter implements buffered encoding of records to an io.Writer. After all records have been encoded, the caller should invoke the Flush method to guarantee all buffered data has been forwarded to the underlying io.Writer.
func NewBufferedWriter ¶
func NewBufferedWriter(w io.Writer, size int, mode IOMode) (bw *BufferedWriter)
NewBufferedWriter returns a new BufferedWriter whose buffer has the specified size. If mode is set to ModeManual, Write operations that would trigger a blocking Flush to the underlying io.Writer will return with err == ErrMustFlush. The caller must the call Flush manually before calling Write again. If mode is set to Auto (or 0) the writer will Flush its internal buffer transparently.
func (*BufferedWriter) Flush ¶
func (bw *BufferedWriter) Flush() (err error)
Flush writes any buffered data to the underlying io.Writer.
func (*BufferedWriter) Reset ¶
func (bw *BufferedWriter) Reset(w io.Writer)
Reset discards any unflushed data, resets all state, and switches the buffered writer to write to w.
func (*BufferedWriter) Write ¶
func (bw *BufferedWriter) Write(v Encoder) (n int, err error)
Write encodes one record to the writer's internal buffer. If the buffer does not have enough space left to encode the complete record, either it is automatically flushed to the underlying io.Writer in Auto mode, or Write returns with err == ErrMustFlush in manual mode. If a record cannot be entirely fit in the writer's internal buffer, Write fails with err == ErrTooLarge.
type Decoder ¶
Decoder is the interface that wraps the Decode method.
Decode decodes a record from p into the receiver. It returns the number of bytes decoded and any error that occured during decoding. Records are either completely decoded or not decoded at all. Thus even in case of error n must be either 0 or the full record size. This can allow callers to skip records that failed proper decoding but whose size is known. If p does not contain a complete record, Decode must return ErrShortBuffer. The caller will try to fill its buffer and retry decoding.
type EncodeDecoder ¶
type EncodeDecoder interface {
Encode(p []byte) (n int, err error)
Decode(p []byte) (n int, err error)
}
EncodeDecoder is the interface that wraps Encode and Decode methods.
See Encoder and Decoder interfaces for details.
type Encoder ¶
Encoder is the interface that wraps the Encode method.
Encode encodes the receiver to p. It returns the number of bytes encoded or any error that caused the encoding to fail. Records are either completely encoded or not encoded at all. Thus if any error is returned, n must be 0. If the encoded record does not fit in p, Encode must return ErrShortBuffer. The caller will flush its buffer to free space and retry encoding.
type IOMode ¶
type IOMode int
IOMode defines buffered reader and writers handling of blocking operations.
Source Files
Directories