wal

type DataCorruptionError ¶

type DataCorruptionError struct {
	// contains filtered or unexported fields
}

DataCorruptionError is an error that occures if data on disk was corrupted.

func (e DataCorruptionError) Cause() error

func (e DataCorruptionError) Error() string

type MetaMessage ¶

type MetaMessage struct {
	Height int64 `json:"h"`
}

Some lines are MetaMessages to denote new heights, etc. NOTE: The encoding must not contain the '#' character, so arbitrary strings will not work without some kind of escaping. TODO: consider alternative meta line schemas in the long run, or escape the '#' character in such a way that scanning randomly from any position in a file resumes correctly after the first and only likely corruption.

type NopWAL ¶

type NopWAL struct{}

func (NopWAL) FlushAndSync() error

func (NopWAL) SearchForHeight(height int64, options *WALSearchOptions) (rd io.ReadCloser, found bool, err error)

func (NopWAL) SetLogger(l log.Logger)

func (NopWAL) Start() error

func (NopWAL) Stop() error

func (NopWAL) Wait()

func (NopWAL) Write(m WALMessage) error

func (NopWAL) WriteMetaSync(m MetaMessage) error

func (NopWAL) WriteSync(m WALMessage) error

type TimedWALMessage ¶

type TimedWALMessage struct {
	Time time.Time  `json:"time"`
	Msg  WALMessage `json:"msg"`
}

TimedWALMessage wraps WALMessage and adds Time for debugging purposes.

type WAL ¶

type WAL interface {
	// config methods
	SetLogger(l log.Logger)

	// write methods
	Write(WALMessage) error
	WriteSync(WALMessage) error
	WriteMetaSync(MetaMessage) error
	FlushAndSync() error

	// search methods
	SearchForHeight(height int64, options *WALSearchOptions) (rd io.ReadCloser, found bool, err error)

	// service methods
	Start() error
	Stop() error
	Wait()
}

WAL is an interface for any write-ahead logger.

type WALMessage ¶

type WALMessage interface {
	AssertWALMessage()
}

type WALReader ¶

type WALReader struct {
	// contains filtered or unexported fields
}

A WALReader reads and decodes custom-encoded WAL messages from an input stream. See WALWriter for the format used.

It will also compare the checksums and make sure data size is equal to the length from the header. If that is not the case, error will be returned.

WALReader is itself an io.ReaderCloser, and it uses a bufio reader under the hood, which means it will usually end up reading more bytes than was actually returned via calls to ReadMessage().

func NewWALReader(rd io.Reader, maxSize int64) *WALReader

func (dec *WALReader) Close() (err error)

func (dec *WALReader) Read(p []byte) (n int, err error)

func (dec *WALReader) ReadMessage() (*TimedWALMessage, *MetaMessage, error)

type WALSearchOptions ¶

type WALSearchOptions struct {
	// IgnoreDataCorruptionErrors set to true will result in skipping data corruption errors.
	IgnoreDataCorruptionErrors bool
}

WALSearchOptions are optional arguments to SearchForHeight.

type WALWriter ¶

type WALWriter struct {
	// contains filtered or unexported fields
}

A WALWriter writes custom-encoded WAL messages to an output stream. Each binary WAL entry is length encoded, then crc encoded, then base64 encoded and delimited by newlines. The base64 encoding used is encodeStd, `ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/`.

Each WAL item is also newline delimited.

Meta lines are prefixed with a '#' (which is not a valid base64 character) denote meta information, such as new height. The purpose of base64 encoding is to enable backwards traversal of items (e.g. in search of some previous height).

Base64 is chosen to optimize the worst-case scenario for encoding delimited binary bytes while enabling metadata insertion and also supporting backwards traversal of binary blobs (to enable binary search, etc). In the future, base64 could be replaced with a similar encoding formula, and the crc function could change too, but otherwise the structure of the WAL should not change, including the rule that all metalines should start with '#' and that all items be delimited with a newline. This implementation enforces ASCII text characters, but other implementations may choose otherwise.

Format: base64(4 bytes CRC sum + length-prefixed amino bytes) + newline. e.g. ``` ABCDEFGHIJKLMNOPQRSTUV00 ABCDEFGHIJKLMNOPQRSTUV01 ABCDEFGHIJKLMNOPQRSTUV02 #{"h":"123"} ABCDEFGHIJKLMNOPQRSTUV03 ABCDEFGHIJKLMNOPQRSTUV04

func NewWALWriter(wr io.Writer, maxSize int64) *WALWriter

func (enc *WALWriter) Write(v TimedWALMessage) error

func (enc *WALWriter) WriteMeta(meta MetaMessage) error