wal: maze.io/x/wal Index | Examples | Files

package wal

import "maze.io/x/wal"

Package wal implements a Write-Ahead Log.

Writing logs

Each write-ahead log requires a dedicated directory for writing log files, writing to a directory containing other files is undefined behaviour. Two methods exist to open a new log, `Open` which uses `DefaultSize` or `OpenSize` which allows the user to specify a maximum log file size. The log file writer checks the log file size before writing and advances to the next log file if required, but only if the log is not empty, allowing for writes beyond the specified size.

Reading logs

It's the reader's duty to remember where they left off reading. This allows the wal package to return a read only scanner for each Write-Ahead Log.

Record size

The maximum log file record size is 4GiB. If larger chunk size is required, the caller has to write the contents in chunks of 4GiB and recombine them later. Writes that exceed this limit will result in an error.

Index

Examples

Package Files

doc.go file.go log.go memory.go record.go

Constants

const (
    // DefaultSize is the default maximum size for a log file.
    DefaultSize = 1 << 24 // 16 MB

    // SyncNever skips file syncing.
    SyncNever time.Duration = -1

    // SyncImmediately syncs after each Write method call.
    SyncImmediately time.Duration = 0
)

type Log Uses

type Log interface {
    // First entry in the log.
    First() (sequence uint64, exists bool)

    // Last entry in the log.
    Last() (sequence uint64, exists bool)

    // Scanner for the log.
    Scanner(start uint64) Scanner

    // Write new data to the log.
    Write(p []byte) (n int, err error)

    // Close the log.
    Close() error
}

Log is a Write-Ahead Log implementation. Logs are concurrency safe.

func Open Uses

func Open(dir string, syncInterval time.Duration) (Log, error)

Open a WAL log at the given directory. At most one log may exist in the directory.

Code:

// Open a write-ahead log in directory "log" with the default log size,
// syncing every 100ms if there has been a write.
w, err := wal.Open("log", 100*time.Millisecond)
if err != nil {
    panic(err)
}

// Make sure our WAL is closed.
defer w.Close()

// Start writing to the WAL...

func OpenSize Uses

func OpenSize(dir string, syncInterval time.Duration, size int64) (Log, error)

OpenSize opens a WAL log at the given directory. At most one log may exist in the directory. The supplied size indicates the maximum log file size. If the size <= 0, the DefaultSize shall be used. The sync interval specifies how often the log is synced to disk. If the sync interval == 0, the log is synced immediately after each write. If the sync interval < 0, the log is never synced.

Code:

// Open a write-ahead log in directory "log" with maximum log size of 1GB,
// syncing after each write.
w, err := wal.OpenSize("log", wal.SyncImmediately, 1<<30) // 1 GB log size
if err != nil {
    panic(err)
}

// Make sure our WAL is closed.
defer w.Close()

// Start writing to the WAL...

type Scanner Uses

type Scanner interface {
    // Scan advances the Scanner to the next log record, which will be
    // available through the Bytes method. After Scan returns false, the
    // Err method will return any error that occurred during scanning.
    Scan() bool

    // Bytes returns the most recent token generated by a call to Scan. The
    // underlying array may point to data that will be overwritten by a
    // subsequent call to Scan.
    Bytes() []byte

    // Err returns the last error occurred during scanning. If EOF is
    // returned, no new log entries are available currently.
    Err() error

    // Close the scanner and any logs it may be referencing.
    Close() error
}

Scanner can scan over log records. Scanners are not concurrency safe.

func Read Uses

func Read(dir string, start uint64) (Scanner, error)

Read a write-ahead log at the specified directory.

Code:

// Open a write-ahead log for reading in directory "log" and continue
// scanning at entry 1234.
r, err := wal.Read("log", 1234)
if err != nil {
    panic(err)
}

// Consume all entries that can be read.
for r.Scan() {
    // Consume r.Bytes() ...
}

// Check reader for fatal errors.
if err = r.Err(); err != nil {
    panic(err)
}

// Wait for new data to appear.
for {
    if r.Scan() {
        // Consume r.Bytes() ...
    } else if err = r.Err(); err != nil {
        // Handle fatal errors.
        panic(err)
    } else {
        // Wait a second and scan again.
        time.Sleep(time.Second)
    }
}

Package wal imports 14 packages (graph). Updated 2019-03-19. Refresh now. Tools for package owners.