Documentation
¶
Index ¶
- Constants
- Variables
- func Repair(logger api.Logger, dirPath string) error
- type LogRecordCRC
- type LogRecordHeader
- type LogRecordLength
- type LogRecordReader
- type Options
- type WriteAheadLogFile
- func Create(logger api.Logger, dirPath string, options *Options) (*WriteAheadLogFile, error)
- func InitializeAndReadAll(logger api.Logger, walDir string, options *Options) (writeAheadLog *WriteAheadLogFile, initialState [][]byte, err error)
- func Open(logger api.Logger, dirPath string, options *Options) (*WriteAheadLogFile, error)
Constants ¶
const ( FileSizeBytesDefault int64 = 64 * 1024 * 1024 // 64MB BufferSizeBytesDefault int64 = 1024 * 1024 // 1MB )
Variables ¶
Functions ¶
func Repair ¶
Repair tries to repair the last file of a WAL, in case an Open() comes back with a io.ErrUnexpectedEOF, which indicates the possibility to fix the WAL.
After a crash, the last log file in the WAL may be left in a state where an attempt tp reopen will result in the last Read() returning an error. This is because of several reasons: - we use pre-allocated / recycled files, the files have a "garbage" tail - the last write may have been torn - the failure might have occurred when the log file was being prepared (no anchor)
The Repair() tries to repair the last file of the wal by truncating after the last good record. Before doing so it copies the bad file to a side location for later analysis by operators.
logger: reference to a Logger implementation. dirPath: directory path of the WAL. return: an error if repair was not successful.
Types ¶
type LogRecordCRC ¶
type LogRecordCRC uint32
type LogRecordHeader ¶
type LogRecordHeader uint64
LogRecordHeader contains the LogRecordLength (lower 32 bits) and LogRecordCRC (upper 32 bits).
type LogRecordLength ¶
type LogRecordLength uint32
type LogRecordReader ¶
type LogRecordReader struct {
// contains filtered or unexported fields
}
func NewLogRecordReader ¶
func NewLogRecordReader(logger api.Logger, fileName string) (*LogRecordReader, error)
func (*LogRecordReader) CRC ¶
func (r *LogRecordReader) CRC() uint32
func (*LogRecordReader) Close ¶
func (r *LogRecordReader) Close() error
type Options ¶
func DefaultOptions ¶
func DefaultOptions() *Options
DefaultOptions returns the set of default options.
type WriteAheadLogFile ¶
type WriteAheadLogFile struct {
// contains filtered or unexported fields
}
WriteAheadLogFile is a simple implementation of a write ahead log (WAL).
The WAL is composed of a sequence of frames. Each frame contains: - a header (uint64) - payload: a record of type LogRecord, marshaled to bytes, and padded with zeros to 8B boundary.
The 64 bit header is made of two parts: - length of the marshaled LogRecord (not including pad bytes), in the lower 32 bits. - a crc32 of the data: marshaled record bytes + pad bytes, in the upper 32 bits.
The WAL is written to a sequence of files: <index>.wal, where index uint64=1,2,3...; represented in fixed-width hex format, e.g. 0000000000000001.wal
The WAL has two modes: append, and read.
When a WAL is first created, it is in append mode. When an existing WAL is opened, it is in read mode, and will change to append mode only after ReadAll() is invoked.
In append mode the WAL can accept Append() and TruncateTo() calls. The WAL must be closed after use to release all resources.
func Create ¶
Create will create a new WAL, if it does not exist, or an error if it already exists.
logger: reference to a Logger implementation. dirPath: directory path of the WAL. options: a structure containing Options, or nil, for default options.
return: pointer to a WAL, ErrWALAlreadyExists if WAL already exists or other errors
func InitializeAndReadAll ¶
func Open ¶
Open will open an existing WAL, if it exists, or an error if it does not exist.
After opening, the WAL is in read mode, and expects a call to ReadAll(). An attempt to write (e.g. Append, TruncateTo) will result in an error.
logger: reference to a Logger implementation. dirPath: directory path of the WAL. options: a structure containing Options, or nil, for default options.
return: pointer to a WAL, or an error
func (*WriteAheadLogFile) Append ¶
func (w *WriteAheadLogFile) Append(data []byte, truncateTo bool) error
Append a data item to the end of the WAL and indicate whether this entry is a truncation point.
The data item will be added to the log, and internally marked with a flag that indicates whether it is a truncation point. The log implementation may truncate all preceding data items, not including this one.
data: the data to be appended to the log. Cannot be nil or empty. truncateTo: whether all records preceding this one, but not including it, can be truncated from the log.
func (*WriteAheadLogFile) CRC ¶
func (w *WriteAheadLogFile) CRC() uint32
CRC returns the last CRC written to the log file.
func (*WriteAheadLogFile) Close ¶
func (w *WriteAheadLogFile) Close() error
Close the files and directory of the WAL, and release all resources.
func (*WriteAheadLogFile) ReadAll ¶
func (w *WriteAheadLogFile) ReadAll() ([][]byte, error)
ReadAll the data items from the latest truncation point to the end of the log. This method can be called only at the beginning of the WAL lifecycle, right after Open(). After a successful invocation the WAL moves to write mode, and is ready to Append().
In case of failure:
- an error of type io.ErrUnexpectedEOF is returned when the WAL can possibly be repaired by truncating the last log file after the last good record.
- all other errors indicate that the WAL is either
- is closed, or
- is in write mode, or
- is corrupted beyond the simple repair measure described above.
func (*WriteAheadLogFile) TruncateTo ¶
func (w *WriteAheadLogFile) TruncateTo() error
TruncateTo appends a control record in which the TruncateTo flag is true. This marks that every record prior to this one can be safely truncated from the log.
Source Files