README
¶
WAL
Write-Ahead-Log (WAL) implemented with Go.
Benchmarks
$ go test -run=^$ -bench .
goos: darwin
goarch: amd64
pkg: github.com/ulysseses/wal
BenchmarkWrite_100B_Batch1-8 156 6534964 ns/op 0.02 MB/s
BenchmarkWrite_100B_Batch10-8 1850 631955 ns/op 0.18 MB/s
BenchmarkWrite_100B_Batch100-8 19870 64432 ns/op 1.80 MB/s
BenchmarkWrite_100B_Batch1000-8 150238 7196 ns/op 16.12 MB/s
BenchmarkWrite_100B_Batch5000-8 766251 1523 ns/op 76.16 MB/s
BenchmarkWrite_1000B_Batch1-8 183 6688114 ns/op 0.15 MB/s
BenchmarkWrite_1000B_Batch10-8 1725 656703 ns/op 1.55 MB/s
BenchmarkWrite_1000B_Batch100-8 17748 69383 ns/op 14.70 MB/s
BenchmarkWrite_1000B_Batch1000-8 142172 7986 ns/op 127.72 MB/s
BenchmarkWrite_1000B_Batch5000-8 414115 2496 ns/op 408.59 MB/s
BenchmarkWrite_5000B_Batch1-8 150 6744202 ns/op 0.74 MB/s
BenchmarkWrite_5000B_Batch10-8 1682 670651 ns/op 7.49 MB/s
BenchmarkWrite_5000B_Batch100-8 16718 70535 ns/op 71.17 MB/s
BenchmarkWrite_5000B_Batch1000-8 97143 11837 ns/op 424.10 MB/s
BenchmarkWrite_5000B_Batch5000-8 182594 5578 ns/op 900.03 MB/s
PASS
ok github.com/ulysseses/wal 27.732s
$
Documentation
¶
Overview ¶
Package wal implements a WAL. WAL is short for Write-Ahead-Log. Its purpose is durability. Before a write is acknowledged, it is copied to the WAL. If the machine crashes after a write to the WAL but before it is acknowledged, we can recover and apply the write by reading it from the WAL. If instead the machine crashes before the write to WAL, we can safely retry the write without fear of the write being applied twice (exactly once semantics).
A write may be pending in user-space buffers or the page cache. It must be synced to disk for it to be made durable. There are multiple ways to do this, though they make a trade-off between the level of durability and performance:
Sync after every write. This is the slowest option but offers the strongest durability.
Sync after N writes, where N is the batch size. This is faster since it calls sync less frequently, but you can lose at most N writes. However, if < N writes come in, there must be a mechanism to force-sync the batch at some point even though it hasn't reached N yet.
Sync at a regular T interval, where T is a time interval. This is faster since it calls sync less frequently, but you can lose un-synced writes. Another downside is that you must find an appropriate interval, dynamically/adaptively or via manual tuning.
Don't force sync. This is the fastest but most dangerous: the OS will asynchronously write to disk (i.e. choose when to flush the dirty page back to disk).
WAL is agnostic and lets the user decide which strategy is appropriate: call Write() and Sync() as one pleases.
Often we want to fetch all written records from an index onwards. To optimize this search pattern, WAL writes records to "segment" files that live in a "WAL" directory. Each segment has a maximum size, so when one is filled up, WAL will automatically close that file and begin a new one. Segment files are formatted as "{seq}-{index}.wal", where seq is the `seq`-th segment file, and the first record in that segment is the `index`-th overall record. Thus, we don't have to read every segment in the directory to find the appropriate records.
Index ¶
Constants ¶
const ( // SegmentSizeBytes is the preallocated size of each segment file. // The actual size might actually be larger than this. In general, the // default value should be used, but this is defined as an exported variable // so that tests can set a different segment size. SegmentSizeBytes = 64 * 1000 * 1000 // SegExt is the segment file extension. SegExt = ".seg" // ScratchSuffix is the suffix of the scratch directory for the write-ahead-log. ScratchSuffix = ".tmp" )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type WAL ¶
type WAL struct {
// contains filtered or unexported fields
}
WAL is a write-ahead-log.
func (*WAL) Sync ¶
Sync persists accumulated writes from both the user-land buffer and kernel page cache to disk.
Source Files