lldb

type ACIDFiler0 ¶

type ACIDFiler0 struct {
	*RollbackFiler
	// contains filtered or unexported fields
}

ACIDFiler0 is a very simple, synchronous implementation of 2PC. It uses a single write ahead log file to provide the structural atomicity (BeginUpdate/EndUpdate/Rollback) and durability (DB can be recovered from WAL if a crash occurred).

ACIDFiler0 is a Filer.

NOTE: Durable synchronous 2PC involves three fsyncs in this implementation (WAL, DB, zero truncated WAL). Where possible, it's recommended to collect transactions for, say one second before performing the two phase commit as the typical performance for rotational hard disks is about few tens of fsyncs per second atmost. For an example of such collective transaction approach please see the colecting FSM STT in Dbm's documentation[1].

[1]: http://godoc.org/github.com/cznic/exp/dbm

func NewACIDFiler(db Filer, wal *os.File, opts ...WALOption) (r *ACIDFiler0, err error)

func (a ACIDFiler0) PeakWALSize() int64

type AllocStats ¶

type AllocStats struct {
	Handles     int64           // total valid handles in use
	Compression int64           // number of compressed blocks
	TotalAtoms  int64           // total number of atoms == AllocAtoms + FreeAtoms
	AllocBytes  int64           // bytes allocated (after decompression, if/where used)
	AllocAtoms  int64           // atoms allocated/used, including relocation atoms
	Relocations int64           // number of relocated used blocks
	FreeAtoms   int64           // atoms unused
	AllocMap    map[int64]int64 // allocated block size in atoms -> count of such blocks
	FreeMap     map[int64]int64 // free block size in atoms -> count of such blocks
}

AllocStats record statistics about a Filer. It can be optionally filled by Allocator.Verify, if successful.

type Allocator ¶

type Allocator struct {
	Compress bool // enables content compression
	// contains filtered or unexported fields
}

Allocator implements "raw" storage space management (allocation and deallocation) for a low level of a DB engine. The storage is an abstraction provided by a Filer.

The terms MUST or MUST NOT, if/where used in the documentation of Allocator, written in all caps as seen here, are a requirement for any possible alternative implementations aiming for compatibility with this one.

Filer file ¶

A Filer file, or simply 'file', is a linear, contiguous sequence of blocks. Blocks may be either free (currently unused) or allocated (currently used). Some blocks may eventually become virtual in a sense as they may not be realized in the storage (sparse files).

Free Lists Table ¶

File starts with a FLT. This table records heads of 14 doubly linked free lists. The zero based index (I) vs minimal size of free blocks in that list, except the last one which registers free blocks of size 4112+ atoms:

MinSize == 2^I

For example 0 -> 1, 1 -> 2, ... 12 -> 4096.

Each entry in the FLT is 8 bytes in netwtork order, MSB MUST be zero, ie. the slot value is effectively only 7 bytes. The value is the handle of the head of the respective doubly linked free list. The FLT size is 14*8 == 112(0x70) bytes. If the free blocks list for any particular size is empty, the respective FLT slot is zero. Sizes of free blocks in one list MUST NOT overlap with sizes of free lists in other list. For example, even though a free block of size 2 technically is of minimal size >= 1, it MUST NOT be put to the list for slot 0 (minimal size 1), but in slot 1( minimal size 2).

slot 0:		sizes [1, 2)
slot 1:		sizes [2, 4)
slot 2:		sizes [4, 8)
...
slot 11:	sizes [2048, 4096)
slot 12:	sizes [4096, 4112)
slot 13:	sizes [4112, inf)

The last FLT slot collects all free blocks bigger than its minimal size. That still respects the 'no overlap' invariant.

File blocks ¶

A block is a linear, contiguous sequence of atoms. The first and last atoms of a block provide information about, for example, whether the block is free or used, what is the size of the block, etc. Details are discussed elsewhere. The first block of a file starts immediately after FLT, ie. at file offset 112(0x70).

Block atoms ¶

An atom is a fixed size piece of a block (and thus of a file too); it is 16 bytes long. A consequence is that for a valid file:

filesize == 0 (mod 16)

The first atom of the first block is considered to be atom #1.

Block handles ¶

A handle is an integer referring to a block. The reference is the number of the atom the block starts with. Put in other way:

handle == offset/16 - 6
offset == 16 * (handle + 6)

`offset` is the offset of the first byte of the block, measured in bytes - as in fseek(3). Handle has type `int64`, but only the lower 7 bytes may be nonzero while referring to a block, both in code as well as when persisted in the the file's internal bookkeeping structures - see 'Block types' bellow. So a handle is effectively only `uint56`. This also means that the maximum usable size of a file is 2^56 atoms. That is 2^60 bytes == 1 exabyte (10^18 bytes).

Nil handles ¶

A handle with numeric value of '0' refers to no block.

Zero padding ¶

A padding is used to round-up a block size to be a whole number of atoms. Any padding, if present, MUST be all zero bytes. Note that the size of padding is in [0, 15].

Content wiping ¶

When a block is deallocated, its data content is not wiped as the added overhead may be substantial while not necessarily needed. Client code should however overwrite the content of any block having sensitive data with eg. zeros (good compression) - before deallocating the block.

Block tags ¶

Every block is tagged in its first byte (a head tag) and last byte (tail tag). Block types are:

1. Short content used block (head tags 0x00-0xFB)
2. Long content used block (head tag 0xFC)
3. Relocated used block (head tag 0xFD)
4. Short, single atom, free block (head tag 0xFE)
5. Long free block (head tag 0xFF)

Note: Relocated used block, 3. above (head tag 0xFD) MUST NOT refer to blocks other then 1. or 2. above (head tags 0x00-0xFC).

Content blocks ¶

Used blocks (head tags 0x00-0xFC) tail tag distinguish used/unused block and if content is compressed or not.

Content compression ¶

The tail flag of an used block is one of

CC == 0 // Content is not compressed.
CC == 1 // Content is in zappy compression format.

If compression of written content is enabled, there are two cases: If compressed size < original size then the compressed content should be written if it will save at least one atom of the block. If compressed size >= original size then the compressed content should not be used.

It's recommended to use compression. For example the BTrees implementation assumes compression is used. Using compression may cause a slowdown in some cases while it may as well cause a speedup.

Short content block ¶

Short content block carries content of length between N == 0(0x00) and N == 251(0xFB) bytes.

|<-first atom start  ...  last atom end->|
+---++--   ...   --+--   ...   --++------+
| 0 ||    1...     |  0x*...0x*E || 0x*F |
+---++--   ...   --+--   ...   --++------+
| N ||   content   |   padding   ||  CC  |
+---++--   ...   --+--   ...   --++------+

A == (N+1)/16 + 1        // The number of atoms in the block [1, 16]
padding == 15 - (N+1)%16 // Length of the zero padding

Long content block ¶

Long content block carries content of length between N == 252(0xFC) and N == 65787(0x100FB) bytes.

|<-first atom start    ...     last atom end->|
+------++------+-- ... --+--  ...   --++------+
|  0   || 1..2 |   3...  | 0x*...0x*E || 0x*F |
+------++------+-- ... --+--  ...   --++------+
| 0xFC ||  M   | content |  padding   ||  CC  |
+------++------+-- ... --+--  ...   --++------+

A == (N+3)/16 + 1        // The number of atoms in the block [16, 4112]
M == N % 0x10000         // Stored as 2 bytes in network byte order
padding == 15 - (N+3)%16 // Length of the zero padding

Relocated used block ¶

Relocated block allows to permanently assign a handle to some content and resize the content anytime afterwards without having to update all the possible existing references; the handle can be constant while the content size may be dynamic. When relocating a block, any space left by the original block content, above this single atom block, MUST be reclaimed.

Relocations MUST point only to a used short or long block == blocks with tags 0x00...0xFC.

+------++------+---------++----+
|  0   || 1..7 | 8...14  || 15 |
+------++------+---------++----+
| 0xFD ||  H   | padding || 0  |
+------++------+---------++----+

H is the handle of the relocated block in network byte order.

Free blocks ¶

Free blocks are the result of space deallocation. Free blocks are organized in one or more doubly linked lists, abstracted by the FLT interface. Free blocks MUST be "registered" by putting them in such list. Allocator MUST reuse a big enough free block, if such exists, before growing the file size. When a free block is created by deallocation or reallocation it MUST be joined with any adjacently existing free blocks before "registering". If the resulting free block is now a last block of a file, the free block MUST be discarded and the file size MUST be truncated accordingly instead. Put differently, there MUST NOT ever be a free block at the file end.

A single free atom ¶

Is an unused block of size 1 atom.

+------++------+--------++------+
|  0   || 1..7 | 8...14 ||  15  |
+------++------+--------++------+
| 0xFE ||  P   |   N    || 0xFE |
+------++------+--------++------+

P and N, stored in network byte order, are the previous and next free block handles in the doubly linked list to which this free block belongs.

A long unused block ¶

Is an unused block of size > 1 atom.

+------++------+-------+---------+- ... -+----------++------+
|  0   || 1..7 | 8..14 | 15...21 |       | Z-7..Z-1 ||  Z   |
+------++------+-------+---------+- ... -+----------++------+
| 0xFF ||  S   |   P   |    N    | Leak  |    S     || 0xFF |
+------++------+-------+---------+- ... -+----------++------+

Z == 16 * S - 1

S is the size of this unused block in atoms. P and N are the previous and next free block handles in the doubly linked list to which this free block belongs. Leak contains any data the block had before deallocating this block. See also the subtitle 'Content wiping' above. S, P and N are stored in network byte order. Large free blocks may trigger a consideration of file hole punching of the Leak field - for some value of 'large'.

Note: Allocator methods vs CRUD[1]:

Alloc	[C]reate
Get	[R]ead
Realloc	[U]pdate
Free	[D]elete

Note: No Allocator method returns io.EOF.

[1]: http://en.wikipedia.org/wiki/Create,_read,_update_and_delete

func NewAllocator(f Filer, opts *Options) (a *Allocator, err error)

func (a *Allocator) Alloc(b []byte) (handle int64, err error)

func (a *Allocator) CacheStats() (buffersUsed, buffersTotal int, bytesUsed, bytesTotal, hits, misses int64)

func (a *Allocator) Free(handle int64) (err error)

func (a *Allocator) Get(buf []byte, handle int64) (b []byte, err error)

func (a *Allocator) Realloc(handle int64, b []byte) (err error)

func (a *Allocator) Verify(bitmap Filer, log func(error) bool, stats *AllocStats) (err error)

type BTree ¶

type BTree struct {
	// contains filtered or unexported fields
}

BTree is a B+tree[1][2], i.e. a variant which speeds up enumeration/iteration of the BTree. According to its origin it can be volatile (backed only by memory) or non-volatile (backed by a non-volatile Allocator).

The specific implementation of BTrees in this package are B+trees with delayed split/concatenation (discussed in e.g. [3]).

Note: No BTree methods returns io.EOF for physical Filer reads/writes. The io.EOF is returned only by bTreeEnumerator methods to indicate "no more K-V pair".

[1]: http://en.wikipedia.org/wiki/B+tree
[2]: http://zgking.com:8080/home/donghui/publications/books/dshandbook_BTree.pdf
[3]: http://people.cs.aau.dk/~simas/aalg06/UbiquitBtree.pdf

func CreateBTree(store *Allocator, collate func(a, b []byte) int) (bt *BTree, handle int64, err error)

func NewBTree(collate func(a, b []byte) int) *BTree

func OpenBTree(store *Allocator, collate func(a, b []byte) int, handle int64) (bt *BTree, err error)

func (t *BTree) Clear() (err error)

func (t *BTree) Delete(key []byte) (err error)

func (t *BTree) DeleteAny() (empty bool, err error)

func (t *BTree) Dump(w io.Writer) (err error)

func (t *BTree) Extract(buf, key []byte) (value []byte, err error)

func (t *BTree) First() (key, value []byte, err error)

func (t *BTree) Get(buf, key []byte) (value []byte, err error)

func (t *BTree) Handle() int64

func (t *BTree) IndexSeek(key []byte, c func(a, b []byte) int) (enum *BTreeEnumerator, hit bool, err error)

func (t *BTree) IsMem() (r bool)

func (t *BTree) Last() (key, value []byte, err error)

func (t *BTree) Put(buf, key []byte, upd func(key, old []byte) (new []byte, write bool, err error)) (old []byte, written bool, err error)

tree.Set(k, v)

tree.Put(k, func(k, v []byte){ return v, true }([]byte, bool))

func (t *BTree) Seek(key []byte) (enum *BTreeEnumerator, hit bool, err error)

func (t *BTree) SeekFirst() (enum *BTreeEnumerator, err error)

func (t *BTree) SeekLast() (enum *BTreeEnumerator, err error)

func (t *BTree) Set(key, value []byte) (err error)

type BTreeEnumerator ¶

type BTreeEnumerator struct {
	// contains filtered or unexported fields
}

BTreeEnumerator captures the state of enumerating a tree. It is returned from the Seek* methods. The enumerator is aware of any mutations made to the tree in the process of enumerating it and automatically resumes the enumeration.

func (e *BTreeEnumerator) Next() (key, value []byte, err error)

func (e *BTreeEnumerator) Prev() (key, value []byte, err error)

type ErrDecodeScalars ¶

type ErrDecodeScalars struct {
	B []byte // Data being decoded
	I int    // offending offset
}

ErrDecodeScalars is possibly returned from DecodeScalars

func (e *ErrDecodeScalars) Error() string

type ErrILSEQ ¶

type ErrILSEQ struct {
	Type ErrType
	Off  int64
	Arg  int64
	Arg2 int64
	Arg3 int64
	Name string
	More interface{}
}

ErrILSEQ reports a corrupted file format. Details in fields according to Type.

func (e *ErrILSEQ) Error() string

type ErrINVAL ¶

type ErrINVAL struct {
	Src string
	Val interface{}
}

ErrINVAL reports invalid values passed as parameters, for example negative offsets where only non-negative ones are allowed or read from the DB.

func (e *ErrINVAL) Error() string

type ErrPERM ¶

type ErrPERM struct {
	Src string
}

ErrPERM is for example reported when a Filer is closed while BeginUpdate(s) are not balanced with EndUpdate(s)/Rollback(s) or when EndUpdate or Rollback is invoked which is not paired with a BeginUpdate.

func (e *ErrPERM) Error() string

type ErrType ¶

type ErrType int

ErrTag represents an ErrILSEQ kind.

const (
	ErrOther ErrType = iota

	ErrAdjacentFree          // Adjacent free blocks (.Off and .Arg)
	ErrDecompress            // Used compressed block: corrupted compression
	ErrExpFreeTag            // Expected a free block tag, got .Arg
	ErrExpUsedTag            // Expected a used block tag, got .Arg
	ErrFLT                   // Free block is invalid or referenced multiple times
	ErrFLTLoad               // FLT truncated to .Off, need size >= .Arg
	ErrFLTSize               // Free block size (.Arg) doesn't belong to its list min size: .Arg2
	ErrFileSize              // File .Name size (.Arg) != 0 (mod 16)
	ErrFreeChaining          // Free block, .prev.next doesn't point back to this block
	ErrFreeTailBlock         // Last block is free
	ErrHead                  // Head of a free block list has non zero Prev (.Arg)
	ErrInvalidRelocTarget    // Reloc doesn't target (.Arg) a short or long used block
	ErrInvalidWAL            // Corrupted write ahead log. .Name: file name, .More: more
	ErrLongFreeBlkTooLong    // Long free block spans beyond EOF, size .Arg
	ErrLongFreeBlkTooShort   // Long free block must have at least 2 atoms, got only .Arg
	ErrLongFreeNextBeyondEOF // Long free block .Next (.Arg) spans beyond EOF
	ErrLongFreePrevBeyondEOF // Long free block .Prev (.Arg) spans beyond EOF
	ErrLongFreeTailTag       // Expected a long free block tail tag, got .Arg
	ErrLostFreeBlock         // Free block is not in any FLT list
	ErrNullReloc             // Used reloc block with nil target
	ErrRelocBeyondEOF        // Used reloc points (.Arg) beyond EOF
	ErrShortFreeTailTag      // Expected a short free block tail tag, got .Arg
	ErrSmall                 // Request for a free block (.Arg) returned a too small one (.Arg2) at .Off
	ErrTailTag               // Block at .Off has invalid tail CC (compression code) tag, got .Arg
	ErrUnexpReloc            // Unexpected reloc block referred to from reloc block .Arg
	ErrVerifyPadding         // Used block has nonzero padding
	ErrVerifyTailSize        // Long free block size .Arg but tail size .Arg2
	ErrVerifyUsedSpan        // Used block size (.Arg) spans beyond EOF
)

type Filer ¶

type Filer interface {
	// BeginUpdate increments the "nesting" counter (initially zero). Every
	// call to BeginUpdate must be eventually "balanced" by exactly one of
	// EndUpdate or Rollback. Calls to BeginUpdate may nest.
	BeginUpdate() error

	// Analogous to os.File.Close().
	Close() error

	// EndUpdate decrements the "nesting" counter. If it's zero after that
	// then assume the "storage" has reached structural integrity (after a
	// batch of partial updates). If a Filer implements some support for
	// that (write ahead log, journal, etc.) then the appropriate actions
	// are to be taken for nesting == 0. Invocation of an unbalanced
	// EndUpdate is an error.
	EndUpdate() error

	// Analogous to os.File.Name().
	Name() string

	// PunchHole deallocates space inside a "file" in the byte range
	// starting at off and continuing for size bytes. The actual hole
	// created by PunchHole may be smaller than requested. The Filer size
	// (as reported by `Size()` does not change when hole punching, even
	// when punching the end of a file off.  In contrast to the Linux
	// implementation of FALLOC_FL_PUNCH_HOLE in `fallocate`(2); a Filer is
	// free not only to ignore `PunchHole()` (implement it as a nop), but
	// additionally no guarantees about the content of the hole, when
	// eventually read back, are required, i.e.  any data, not only zeros,
	// can be read from the "hole", including just anything what was left
	// there - with all of the possible security problems.
	PunchHole(off, size int64) error

	// As os.File.ReadAt. Note: `off` is an absolute "file pointer"
	// address and cannot be negative even when a Filer is a InnerFiler.
	ReadAt(b []byte, off int64) (n int, err error)

	// Rollback cancels and undoes the innermost pending update level.
	// Rollback decrements the "nesting" counter.  If a Filer implements
	// some support for keeping structural integrity (write ahead log,
	// journal, etc.) then the appropriate actions are to be taken.
	// Invocation of an unbalanced Rollback is an error.
	Rollback() error

	// Analogous to os.File.FileInfo().Size().
	Size() (int64, error)

	// Analogous to os.Sync().
	Sync() (err error)

	// Analogous to os.File.Truncate().
	Truncate(size int64) error

	// Analogous to os.File.WriteAt(). Note: `off` is an absolute "file
	// pointer" address and cannot be negative even when a Filer is a
	// InnerFiler.
	WriteAt(b []byte, off int64) (n int, err error)
}

A Filer is a []byte-like model of a file or similar entity. It may optionally implement support for structural transaction safety. In contrast to a file stream, a Filer is not sequentially accessible. ReadAt and WriteAt are always "addressed" by an offset and are assumed to perform atomically. A Filer is not safe for concurrent access, it's designed for consumption by the other objects in package, which should use a Filer from one goroutine only or via a mutex. BeginUpdate, EndUpdate and Rollback must be either all implemented by a Filer for structural integrity - or they should be all no-ops; where/if that requirement is relaxed.

If a Filer wraps another Filer implementation, it usually invokes the same methods on the "inner" one, after some possible argument translations etc. If a Filer implements the structural transactions handling methods (BeginUpdate, EndUpdate and Rollback) as no-ops _and_ wraps another Filer: it then still MUST invoke those methods on the inner Filer. This is important for the case where a RollbackFiler exists somewhere down the chain. It's also important for an Allocator - to know when it must invalidate its FLT cache.

type InnerFiler ¶

type InnerFiler struct {
	// contains filtered or unexported fields
}

A InnerFiler is a Filer with added addressing/size translation.

func NewInnerFiler(outer Filer, off int64) *InnerFiler

inner := NewInnerFiler(outer, 10)

inner.WriteAt([]byte{42}, 4)

outer.WriteAt([]byte{42}, 14)

outer.WriteAt([]byte{17}, 9)

inner.WriteAt([]byte{17}, -1)

inner.Size() == outer.Size() - off,

inner.Truncate(7)
outer.Size() == 17

func (f *InnerFiler) BeginUpdate() error

func (f *InnerFiler) Close() (err error)

func (f *InnerFiler) EndUpdate() error

func (f *InnerFiler) Name() string

func (f *InnerFiler) PunchHole(off, size int64) error

func (f *InnerFiler) ReadAt(b []byte, off int64) (n int, err error)

func (f *InnerFiler) Rollback() error

func (f *InnerFiler) Size() (int64, error)

func (f *InnerFiler) Sync() (err error)

func (f *InnerFiler) Truncate(size int64) error

func (f *InnerFiler) WriteAt(b []byte, off int64) (n int, err error)

type MemFiler ¶

type MemFiler struct {
	// contains filtered or unexported fields
}

MemFiler is a memory backed Filer. It implements BeginUpdate, EndUpdate and Rollback as no-ops. MemFiler is not automatically persistent, but it has ReadFrom and WriteTo methods.

func NewMemFiler() *MemFiler

func (f *MemFiler) BeginUpdate() error

func (f *MemFiler) Close() (err error)

func (f *MemFiler) EndUpdate() (err error)

func (f *MemFiler) Name() string

func (f *MemFiler) PunchHole(off, size int64) (err error)

func (f *MemFiler) ReadAt(b []byte, off int64) (n int, err error)

func (f *MemFiler) ReadFrom(r io.Reader) (n int64, err error)

func (f *MemFiler) Rollback() (err error)

func (f *MemFiler) Size() (int64, error)

func (f *MemFiler) Sync() error

func (f *MemFiler) Truncate(size int64) (err error)

func (f *MemFiler) WriteAt(b []byte, off int64) (n int, err error)

func (f *MemFiler) WriteTo(w io.Writer) (n int64, err error)

type OSFile ¶

type OSFile interface {
	Name() string
	Stat() (fi os.FileInfo, err error)
	Sync() (err error)
	Truncate(size int64) (err error)
	io.Closer
	io.Reader
	io.ReaderAt
	io.Seeker
	io.Writer
	io.WriterAt
}

OSFile is an os.File like minimal set of methods allowing to construct a Filer.

type OSFiler ¶

type OSFiler struct {
	// contains filtered or unexported fields
}

OSFiler is like a SimpleFileFiler but based on an OSFile.

func NewOSFiler(f OSFile) (r *OSFiler)

func (f *OSFiler) BeginUpdate() (err error)

func (f *OSFiler) Close() (err error)

func (f *OSFiler) EndUpdate() (err error)

func (f *OSFiler) Name() string

func (f *OSFiler) PunchHole(off, size int64) (err error)

func (f *OSFiler) ReadAt(b []byte, off int64) (n int, err error)

func (f *OSFiler) Rollback() (err error)

func (f *OSFiler) Size() (n int64, err error)

func (f *OSFiler) Sync() (err error)

func (f *OSFiler) Truncate(size int64) (err error)

func (f *OSFiler) WriteAt(b []byte, off int64) (n int, err error)

type Options ¶

type Options struct{}

Options are passed to the NewAllocator to amend some configuration. The compatibility promise is the same as of struct types in the Go standard library - introducing changes can be made only by adding new exported fields, which is backward compatible as long as client code uses field names to assign values of imported struct types literals.

NOTE: No options are currently defined.

type RollbackFiler ¶

type RollbackFiler struct {
	// contains filtered or unexported fields
}

RollbackFiler is a Filer implementing structural transaction handling. Structural transactions should be small and short lived because all non committed data are held in memory until committed or discarded by a Rollback.

While using RollbackFiler, every intended update of the wrapped Filler, by WriteAt, Truncate or PunchHole, _must_ be made within a transaction. Attempts to do it outside of a transaction will return ErrPERM. OTOH, invoking ReadAt outside of a transaction is not a problem.

No nested transactions: All updates within a transaction are held in memory. On a matching EndUpdate the updates held in memory are actually written to the wrapped Filer.

Nested transactions: Correct data will be seen from RollbackFiler when any level of a nested transaction is rollbacked. The actual writing to the wrapped Filer happens only when the outer most transaction nesting level is closed.

Invoking Rollback is an alternative to EndUpdate. It discards all changes made at the current transaction level and returns the "state" (possibly not yet persisted) of the Filer to what it was before the corresponding BeginUpdate.

During an open transaction, all reads (using ReadAt) are "dirty" reads, seeing the uncommitted changes made to the Filer's data.

Lldb databases should be based upon a RollbackFiler.

With a wrapped MemFiler one gets transactional memory. With, for example a wrapped disk based SimpleFileFiler it protects against at least some HW errors - if Rollback is properly invoked on such failures and/or if there's some WAL or 2PC or whatever other safe mechanism based recovery procedure used by the client.

The "real" writes to the wrapped Filer (or WAL instead) go through the writerAt supplied to NewRollbackFiler.

List of functions/methods which are recommended to be wrapped in a BeginUpdate/EndUpdate structural transaction:

Allocator.Alloc
Allocator.Free
Allocator.Realloc

CreateBTree
RemoveBTree
BTree.Clear
BTree.Delete
BTree.DeleteAny
BTree.Clear
BTree.Extract
BTree.Get (it can mutate the DB)
BTree.Put
BTree.Set

NOTE: RollbackFiler is a generic solution intended to wrap Filers provided by this package which do not implement any of the transactional methods. RollbackFiler thus _does not_ invoke any of the transactional methods of its wrapped Filer.

RollbackFiler is safe for concurrent use by multiple goroutines.

func NewRollbackFiler(f Filer, checkpoint func(sz int64) error, writerAt io.WriterAt) (r *RollbackFiler, err error)

func (r *RollbackFiler) BeginUpdate() (err error)

func (r *RollbackFiler) Close() (err error)

func (r *RollbackFiler) EndUpdate() (err error)

func (r *RollbackFiler) Name() string

func (r *RollbackFiler) PunchHole(off, size int64) error

func (r *RollbackFiler) ReadAt(b []byte, off int64) (n int, err error)

func (r *RollbackFiler) Rollback() (err error)

func (r *RollbackFiler) Size() (sz int64, err error)

func (r *RollbackFiler) Sync() error

func (r *RollbackFiler) Truncate(size int64) error

func (r *RollbackFiler) WriteAt(b []byte, off int64) (n int, err error)

type SimpleFileFiler ¶

type SimpleFileFiler struct {
	// contains filtered or unexported fields
}

SimpleFileFiler is an os.File backed Filer intended for use where structural consistency can be reached by other means (SimpleFileFiler is for example wrapped in eg. an RollbackFiler or ACIDFiler0) or where persistence is not required (temporary/working data sets).

SimpleFileFiler is the most simple os.File backed Filer implementation as it does not really implement BeginUpdate and EndUpdate/Rollback in any way which would protect the structural integrity of data. If misused e.g. as a real database storage w/o other measures, it can easily cause data loss when, for example, a power outage occurs or the updating process terminates abruptly.

func NewSimpleFileFiler(f *os.File) *SimpleFileFiler

func (f *SimpleFileFiler) BeginUpdate() error

func (f *SimpleFileFiler) Close() (err error)

func (f *SimpleFileFiler) EndUpdate() (err error)

func (f *SimpleFileFiler) Name() string

func (f *SimpleFileFiler) PunchHole(off, size int64) (err error)

func (f *SimpleFileFiler) ReadAt(b []byte, off int64) (n int, err error)

func (f *SimpleFileFiler) Rollback() (err error)

func (f *SimpleFileFiler) Size() (int64, error)

func (f *SimpleFileFiler) Sync() error

func (f *SimpleFileFiler) Truncate(size int64) (err error)

func (f *SimpleFileFiler) WriteAt(b []byte, off int64) (n int, err error)

type WALOption ¶ added in v1.1.0

type WALOption func(*walOptions) error

WALOption amends WAL properties.

func MinWAL(min int64) WALOption