writeaheadlog

package module
v0.0.0-...-17c1f03 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 7, 2020 License: MIT Imports: 17 Imported by: 21

README

A general purpose, high performance write-ahead-log

A write-ahead-log (WAL) ensures durability and atomicity for updating data on disk if used correctly. Instructions are marshaled and passed to the WAL before they are applied to disk. The WAL makes sure that the instructions are synced to the WAL file before the user applies them. That way the wal can notify the user about unfinished updates due to a sudden power outage. It is up to the caller to guarantee that the instructions are idempotent and consistent.

Usage

First the wal needs to be opened by calling New(string path). This will create a new WAL at the provided path or load an existing one. The latter will recover the WAL and return all the transactions that were not completed.

// Open the WAL.
recoveredTxns, wal, err := New(walPath)
if err != nil {
	return err
}

if len(recoveredTxns) != 0 {
	// Apparently the system crashed. Handle the unfinished updates
	// accordingly.
	applyUpdates(recoveredTxns)

	// After the recovery is complete we can signal the WAL that we are
	// done and that the updates were applied.
	// NOTE: This is optional. If for some reason an update cannot be
	// applied right away it may be skipped and applied later
	for _, txn := range recoveredTxns {
		if err := txn.SignalUpdatesApplied; err != nil {
			return err
		}
	}
}

The wal can then be used to create a Transaction like this using a set of updates:

// Create the WAL transaction.
tx, err := ca.wal.NewTransaction(updates)
if err != nil {
	return err
}

An Update consists of a Name, Version and Instructions. One transaction can hold multiple updates. After the Transaction is created the caller might want to do some kind of setup. Once completed the next step is to signal the WAL that the setup is complete.

// Signal completed setup and then wait for the commitment to finish. This
// will cause the WAL to call fsync on it's underlying file.
errChan := tx.SignalSetupComplete()
err = <-errChan
if err != nil {
	return err
}

This will write the updates to disk and commit them. The caller needs to wait on the returned channel to ensure a successful commit. After the commit it is safe for the caller to apply the updates to disk and once finished, signal the wal that it can now mark the transaction as applied and recycle the transaction.

// Signal that the updates were applied. This also causes the WAL to fsync and
// allows it to recycle used pages. The caller might run this in a goroutine
// but if the system crashes again before the call finishes the caller might
// receive the already applied instructions when recovering the WAL.
err = tx.SignalUpdatesApplied()
if err != nil {
	return err
}

Documentation

Overview

Package writeaheadlog defines and implements a general purpose, high performance write-ahead-log for performing ACID transactions to disk without sacrificing speed or latency more than fundamentally required.

Index

Constants

View Source
const (

	// MaxPayloadSize is the number of bytes that can fit into a single
	// page. For best performance, the number of pages written should be
	// minimized, so clients should try to keep the length of an Update's
	// Instructions field slightly below a multiple of MaxPayloadSize.
	MaxPayloadSize = pageSize - pageMetaSize
)

Variables

View Source
var (
	// NameDeleteUpdate is the name of an idempotent update that deletes a file or
	// folder from a given path on disk.
	NameDeleteUpdate = "DELETE"
	// NameTruncateUpdate is the name of an idempotent update that truncates a file
	// to have a certain size.
	NameTruncateUpdate = "TRUNCATE"
	// NameWriteAtUpdate is the name of an idempotent update that writes data to a
	// file at the specified offset. If the file doesn't exist it is created.
	NameWriteAtUpdate = "WRITEAT"
)

Functions

func ApplyDeleteUpdate

func ApplyDeleteUpdate(u Update) error

ApplyDeleteUpdate parses and applies a delete update.

func ApplyTruncateUpdate

func ApplyTruncateUpdate(u Update) error

ApplyTruncateUpdate parses and applies a truncate update.

func ApplyUpdates

func ApplyUpdates(updates ...Update) error

ApplyUpdates can be used to apply the common update types provided by the writeaheadlog. Since it potentially applies updates to many different files it's not optimized and opens and closes a file for each update. For optimal performance write a custom applyUpdates function.

func ApplyWriteAtUpdate

func ApplyWriteAtUpdate(u Update) error

ApplyWriteAtUpdate parses and applies a writeat update.

func New

func New(path string) ([]*Transaction, *WAL, error)

New will open a WAL. If the previous run did not shut down cleanly, a set of updates will be returned which got committed successfully to the WAL, but were never signaled as fully completed.

If no WAL exists, a new one will be created.

If in debugging mode, the WAL may return a series of updates multiple times, simulating multiple consecutive unclean shutdowns. If the updates are properly idempotent, there should be no functional difference between the multiple appearances and them just being loaded a single time correctly.

func NewWithOptions

func NewWithOptions(opts Options) ([]*Transaction, *WAL, error)

NewWithOptions opens a WAL like New but takes an Options struct as an additional argument to customize some of the WAL's behavior like logging.

Types

type Options

type Options struct {
	// dependencies are used to inject special behavior into the wal by providing
	// custom dependencies when the wal is created and calling deps.disrupt(setting).
	Deps           dependencies
	StaticLog      *log.Logger
	Path           string // path of the underlying logFile
	VerboseLogging bool
}

Options are a helper struct for creating a WAL. This allows for the API of the writeaheadlog to remain compatible by simply extending the Options struct with new fields as needed.

type Transaction

type Transaction struct {

	// Updates defines the set of updates that compose the transaction.
	Updates []Update
	// contains filtered or unexported fields
}

Transaction defines a series of updates that are to be performed atomically. In the event of an unclean shutdown, either all updates will have been saved together at full integrity, or none of the updates in the transaction will have been saved at all.

While multiple transactions can be safely open at the same time, multiple methods should not be called on the transaction at the same time - the WAL is thread-safe, but the transactions are not.

A Transaction is created by calling NewTransaction. Afterwards, the transactions SignalSetupComplete has to be called which returns a channel that is closed once the transaction is committed. Finally SignalUpdatesApplied needs to be called after the transaction was committed to signal a successful transaction and free used pages.

func (*Transaction) Append

func (t *Transaction) Append(updates []Update) <-chan error

Append appends additional updates to a transaction

func (*Transaction) SignalSetupComplete

func (t *Transaction) SignalSetupComplete() <-chan error

SignalSetupComplete will signal to the WAL that any required setup has completed, and that the WAL can safely commit to the transaction being applied atomically.

func (*Transaction) SignalUpdatesApplied

func (t *Transaction) SignalUpdatesApplied() error

SignalUpdatesApplied informs the WAL that it is safe to reuse t's pages.

type Update

type Update struct {
	// The name of the update type. When the WAL is loaded after an unclean
	// shutdown, any un-committed changes will be passed as Updates back to the
	// caller instantiating the WAL. The caller should determine what code to
	// run on the the update based on the name and version. The length of the
	// Name is capped to 255 bytes
	Name string

	// The marshalled data directing the update. The data is an opaque set of
	// instructions to follow that implement and idempotent change to a set of
	// persistent files. A series of unclean shutdowns in rapid succession could
	// mean that these instructions get followed multiple times, which means
	// idempotent instructions are required.
	Instructions []byte
}

Update defines a single update that can be sent to the WAL and saved atomically. Updates are sent to the wal in groups of one or more, and upon being signaled, will be saved to disk in a high-integrity, all- or-nothing fasion that follows ACID principles.

The name and version are optional, however best-practice code will make use of these fields.

When using the Update, it recommended that you typecast the Update type to another type which has methods on it for creating and applying the Update + instructions, including any special handling based on the version.

func DeleteUpdate

func DeleteUpdate(path string) Update

DeleteUpdate creates an update that deletes the file at the specified path.

func TruncateUpdate

func TruncateUpdate(path string, size int64) Update

TruncateUpdate is a helper function which creates a writeaheadlog update for truncating the specified file.

func WriteAtUpdate

func WriteAtUpdate(path string, index int64, data []byte) Update

WriteAtUpdate is a helper function which creates a writeaheadlog update for writing the specified data to the provided index of a file.

type WAL

type WAL struct {
	// contains filtered or unexported fields
}

WAL is a general purpose, high performance write-ahead-log for performing ACID transactions to disk without sacrificing speed or latency more than fundamentally required.

func (*WAL) Close

func (w *WAL) Close() error

Close closes the wal, frees used resources and checks for active transactions.

func (*WAL) CloseIncomplete

func (w *WAL) CloseIncomplete() (int64, error)

CloseIncomplete closes the WAL and reports the number of transactions that are still uncommitted.

func (*WAL) CreateAndApplyTransaction

func (w *WAL) CreateAndApplyTransaction(applyFunc func(...Update) error, updates ...Update) error

CreateAndApplyTransaction is a helper method which creates a transaction from a given set of updates and uses the supplied updateFunc to apply it. NOTE: Any error that occurs after writing the update that is about to be made to the WAL will cause this method to panic. This includes any error returned by `applyFunc`. This behavior is a safeguard against corruption.

func (*WAL) NewTransaction

func (w *WAL) NewTransaction(updates []Update) (*Transaction, error)

NewTransaction creates a transaction from a set of updates.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL