file: github.com/cznic/file Index | Examples | Files

package file

import "github.com/cznic/file"

Package file handles write-ahead logs and space management of os.File-like entities.

Changelog

2017-09-09: Write ahead log support - initial release.

Index

Examples

Package Files

file.go wal.go

Constants

const (
    // AllocAlign defines File offsets of allocations are 0 (mod AllocAlign).
    AllocAlign = 16
    // LowestAllocationOffset is the offset of the first allocation of an empty File.
    LowestAllocationOffset = szFile + szPage
)

type Allocator Uses

type Allocator struct {
    // contains filtered or unexported fields
}

Allocator manages allocation of file blocks within a File.

Allocator methods are not safe for concurrent use by multiple goroutines. Callers must provide their own synchronization when it's used concurrently by multiple goroutines.

func NewAllocator Uses

func NewAllocator(f File) (*Allocator, error)

NewAllocator returns a newly created Allocator managing f or an eror, if any. Allocator never touches the first 16 bytes within f.

func (*Allocator) Alloc Uses

func (a *Allocator) Alloc(size int64) (int64, error)

Alloc allocates a file block large enough for storing size bytes and returns its offset or an error, if any.

func (*Allocator) Calloc Uses

func (a *Allocator) Calloc(size int64) (int64, error)

Calloc is like Alloc but the allocated file block is zeroed up to size.

func (*Allocator) Close Uses

func (a *Allocator) Close() error

Close flushes and closes the allocator and its underlying File.

func (*Allocator) Flush Uses

func (a *Allocator) Flush() error

Flush writes the allocator metadata to its backing File.

Note: Close calls Flush automatically.

func (*Allocator) Free Uses

func (a *Allocator) Free(off int64) error

Free recycles the allocated file block at off.

func (*Allocator) Realloc Uses

func (a *Allocator) Realloc(off, size int64) (int64, error)

Realloc changes the size of the file block allocated at off, which must have been returned from Alloc or Realloc, to size and returns the offset of the relocated file block or an error, if any. The contents will be unchanged in the range from the start of the region up to the minimum of the old and new sizes. Realloc(off, 0) is equal to Free(off). If the file block was moved, a Free(off) is done.

func (*Allocator) SetAutoFlush Uses

func (a *Allocator) SetAutoFlush(v bool)

SetAutoFlush turns on/off automatic flushing of allocator's metadata. When the argument is true Flush is called automatically whenever the metadata are chaneged. When the argument is false, Flush is called automatically only on Close.

The default allocator state has auto flushing turned on.

func (*Allocator) SetFile Uses

func (a *Allocator) SetFile(f File) error

SetFile sets the allocator's backing File.

func (*Allocator) UsableSize Uses

func (a *Allocator) UsableSize(off int64) (int64, error)

UsableSize reports the size of the file block allocated at off, which must have been returned from Alloc or Realloc. The allocated file block size can be larger than the size originally requested from Alloc or Realloc.

func (*Allocator) Verify Uses

func (a *Allocator) Verify(opt *VerifyOptions) error

Verify audits the correctness of the allocator and its backing File.

type File Uses

type File interface {
    Close() error
    ReadAt(p []byte, off int64) (n int, err error)
    Stat() (os.FileInfo, error)
    Sync() error
    Truncate(int64) error
    WriteAt(p []byte, off int64) (n int, err error)
}

File is an os.File-like entity.

Note: *os.File implements File.

func Map Uses

func Map(f *os.File) (File, error)

Map returns a File backed by memory mapping f or an error, if any. The Close method of the result must be eventually called to avoid resource leaks.

func Mem Uses

func Mem(name string) (File, error)

Mem returns a volatile File backed only by process memory or an error, if any. The Close method of the result must be eventually called to avoid resource leaks.

type VerifyOptions Uses

type VerifyOptions struct {
    Allocs    int64 // Number of allocations in use.
    Pages     int64 // Number of pages.
    UsedPages int64 // Number of pages in use.
}

VerifyOptions optionally provide more information from Verify.

type WAL Uses

type WAL struct {
    DoSync bool // Secure commits with fsync.
    F      File // The f argument of NewWAL for convenience. R/O
    W      File // The w argument of NewWAL for convenience. R/O
    // contains filtered or unexported fields
}

WAL implements a write ahead log of F using W. Call wal.F.ReadAt to perform 'read-commited' reads. Call wal.ReadAt to perform 'read uncommitted' reads. Call wal.W.ReadAt to read the write ahead log itself.

Concurrency

wal.ReadAt (read uncommitted) is safe for concurrent use by multiple goroutines so multiple readers are fine, but multiple readers and a single writer is not. However, wal.F.ReadAt (read committed) is safe to run concurrently with any WAL method except Commit. In a typical DB isolation scenario the setup is something like

var wal *file.WAL
var mu sync.RWMutex		// The mutex associated with wal.

// in another goroutine
mu.RLock()			// read isolated, concurrently with other readers
n, err := wal.F.ReadAt(buf, off) // note the F
...
// reads are isolated only until the next RUnlock.
mu.RUnlock()
// wal.Commit and mutating of F is now possible.
...

// in another goroutine (writers serialization not considered in this example)
n, err := wal.WriteAt(buf, off)
...

// and eventually somewhere
mu.Lock()
err := wal.Commit()
...
mu.Unlock()
...

No other WAL method is safe for concurrent use by multiple goroutines or concurrently with ReadAt. Callers must provide their own synchronization when used concurrently by multiple goroutines.

Logical page

The WAL is divided in logical pages. Every page is 1<<pageLog bytes, pageLog being the argument of NewWAL.

Journal page

Journal page holds the uncommitted logical page of F. Journal page is prefixed with a big-endian encoded int64 offset into F. If the offset is negative then the journal page is considered to be all zeros regardless of its actual content.

Journal allocating and size

One journal page is appended to W on first write to any logical page. Total size of W is thus skip (argument of NewWAL) + N * (1<<pageLog + 8), where N is the number of write-touched logical pages. Commit adds a small amount of metadata at the end of W. The size and content of the meta data are not part of the API. However, future changes, if any, of the meta data size/content will continue to support journal files written by the previous versions.

Additionally, one map[int64]int64 item is used for every allocated journal page.

Crash before Commit

If the process writing to WAL crashes before commit, W is invalid and it's not possible to continue the interrupted operation as it is not known at what point the crash occurred. NewWAL will reject invalid WAL file and will not delete it.

Crash during a Commit

If the WAL metadata has not yet been written and the W file has not yet been synced then the situation is the same as crashing before commit.

Once Commit writes the metadata to W and W was synced, the transaction is secured and journal replay starts.

Journal replay

Journal replay transfers all write-touched pages from W to F. Journal replay starts when Commit completes writing W metadata and syncing W. When the transfer is successfully completed, F is synced, W is emptied and synced, in that order.

Crash during journal replay

If journal replay has started but not completed due to a crash then W is valid and non empty. If NewWAL is passed a valid, non empty W in its w argument, NewWAL restarts journal replay.

Code:

const pageLog = 1

dir, err := ioutil.TempDir("", "file-example-")
if err != nil {
    panic(err)
}

defer os.RemoveAll(dir)

f, err := os.Create(filepath.Join(dir, "f"))
if err != nil {
    panic(err)
}

w, err := os.Create(filepath.Join(dir, "w"))
if err != nil {
    panic(err)
}

db, err := NewWAL(f, w, 0, pageLog)
if err != nil {
    panic(err)
}

write := func(b []byte, off int64) {
    if _, err := db.WriteAt(b, off); err != nil {
        panic(err)
    }

    fi, err := w.Stat()
    if err != nil {
        panic(err)
    }

    fmt.Printf("---- db.WriteAt(%#v, %v)\n", b, off)
    fmt.Printf("journal pages %v\n", len(db.m))
    fmt.Printf("journal size %v\n", fi.Size())
}

read := func() {
    b := make([]byte, 1024)
    n, err := f.ReadAt(b, 0)
    if n == 0 && err != io.EOF {
        panic(err.Error())
    }

    fmt.Printf("Read   committed: |% x|\n", b[:n])
    if n, err = db.ReadAt(b, 0); n == 0 && err != io.EOF {
        panic(err.Error())
    }

    fmt.Printf("Read uncommitted: |% x|\n", b[:n])
}

commit := func() {
    if err := db.Commit(); err != nil {
        panic(err)
    }

    fmt.Printf("---- Commit()\n")
    fmt.Printf("journal pages %v\n", len(db.m))
    read()
}

fmt.Printf("logical page size  %v\n", 1<<pageLog)
fmt.Printf("journal page size %v\n", 1<<pageLog+8)
write([]byte{1, 2, 3}, 0)
read()
commit()
write([]byte{0xff}, 1)
read()
commit()

Output:

logical page size  2
journal page size 10
---- db.WriteAt([]byte{0x1, 0x2, 0x3}, 0)
journal pages 2
journal size 20
Read   committed: ||
Read uncommitted: |01 02 03|
---- Commit()
journal pages 0
Read   committed: |01 02 03|
Read uncommitted: |01 02 03|
---- db.WriteAt([]byte{0xff}, 1)
journal pages 1
journal size 10
Read   committed: |01 02 03|
Read uncommitted: |01 ff 03|
---- Commit()
journal pages 0
Read   committed: |01 ff 03|
Read uncommitted: |01 ff 03|

func NewWAL Uses

func NewWAL(f, w File, skip int64, pageLog int) (r *WAL, err error)

NewWAL returns a newly created WAL or an error, if any. The f argument is the File to which the writes to WAL will eventually be committed. The w argument is a File used to collect the writes to the WAL before commit. The skip argument offsets the usage of w by skip bytes allowing for bookkeeping, header etc. The pageLog argument is the binary logarithm of WAL page size. Passing pageLog less than 1 will panic. The f and w arguments must not represent the same entity.

If w contains a valid, non empty write-ahead log, it's first committed to f and emptied. If w contains an invalid or unreadable write ahead log, the function returns an error.

func (*WAL) Close Uses

func (w *WAL) Close() error

Close does nothing and returns nil.

Close implements File.

func (*WAL) Commit Uses

func (w *WAL) Commit() error

Commit transfers all writes to w collected so far into w.F and empties w and w.W or returns an error, if any. If the program crashes during committing a subsequent NewWAL call with the same files f and w will re-initiate the commit operation.

The WAL is ready for reuse if Commit returns nil.

func (*WAL) ReadAt Uses

func (w *WAL) ReadAt(b []byte, off int64) (n int, err error)

ReadAt performs a read-uncommitted operation on w. The semantics are those of io.ReaderAt.ReadAt. Call w.F.ReadAt to perform a read-committed operation.

ReadAt implements File.

func (*WAL) Rollback Uses

func (w *WAL) Rollback() error

Rollback empties the WAL journal without transferring it to F.

func (*WAL) Stat Uses

func (w *WAL) Stat() (os.FileInfo, error)

Stat implements File. The size reported by the result is that of w.F _after_ w.Commit will be eventually successfully performed.

Stat implements File.

func (*WAL) Sync Uses

func (w *WAL) Sync() error

Sync executes w.W.Sync.

Sync implements File.

func (*WAL) Truncate Uses

func (w *WAL) Truncate(size int64) error

Truncate changes the size of the File represented by w to the size argument. Size of w.F is not changed. Truncate instead changes w's metadata.

Truncate implements File.

func (*WAL) WriteAt Uses

func (w *WAL) WriteAt(b []byte, off int64) (n int, err error)

WriteAt performs a write operation on w. The semantics are those of io.WriteAT.WriteAt. WriteAt does not write to w.F. Instead the writes are collected in w.W until committed.

WriteAt implements File.

Package file imports 12 packages (graph) and is imported by 9 packages. Updated 2018-11-30. Refresh now. Tools for package owners.