README
¶
file
Package file handles write-ahead logs and space management of os.File-like entities.
Build status
available at https://modern-c.appspot.com/-/builder/?importpath=modernc.org%2ffile
Installation
$ go get modernc.org/file
Documentation: godoc.org/modernc.org/file
Documentation
¶
Overview ¶
Package file handles write-ahead logs and space management of os.File-like entities.
Build status ¶
available at https://modern-c.appspot.com/-/builder/?importpath=modernc.org%2ffile
Changelog ¶
2017-09-09: Write ahead log support - initial release.
Index ¶
- Constants
- type Allocator
- func (a *Allocator) Alloc(size int64) (int64, error)
- func (a *Allocator) Calloc(size int64) (int64, error)
- func (a *Allocator) Close() error
- func (a *Allocator) Flush() error
- func (a *Allocator) Free(off int64) error
- func (a *Allocator) Realloc(off, size int64) (int64, error)
- func (a *Allocator) SetAutoFlush(v bool)
- func (a *Allocator) SetFile(f File) error
- func (a *Allocator) UsableSize(off int64) (int64, error)
- func (a *Allocator) Verify(opt *VerifyOptions) error
- type File
- type VerifyOptions
- type WAL
- func (w *WAL) Close() error
- func (w *WAL) Commit() error
- func (w *WAL) ReadAt(b []byte, off int64) (n int, err error)
- func (w *WAL) Rollback() error
- func (w *WAL) Stat() (os.FileInfo, error)
- func (w *WAL) Sync() error
- func (w *WAL) Truncate(size int64) error
- func (w *WAL) WriteAt(b []byte, off int64) (n int, err error)
Examples ¶
Constants ¶
const ( // AllocAlign defines File offsets of allocations are 0 (mod AllocAlign). AllocAlign = 16 // LowestAllocationOffset is the offset of the first allocation of an empty File. LowestAllocationOffset = szFile + szPage )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Allocator ¶
type Allocator struct {
// contains filtered or unexported fields
}
Allocator manages allocation of file blocks within a File.
Allocator methods are not safe for concurrent use by multiple goroutines. Callers must provide their own synchronization when it's used concurrently by multiple goroutines.
func NewAllocator ¶
NewAllocator returns a newly created Allocator managing f or an eror, if any. Allocator never touches the first 16 bytes within f.
func (*Allocator) Alloc ¶
Alloc allocates a file block large enough for storing size bytes and returns its offset or an error, if any.
func (*Allocator) Flush ¶
Flush writes the allocator metadata to its backing File.
Note: Close calls Flush automatically.
func (*Allocator) Realloc ¶
Realloc changes the size of the file block allocated at off, which must have been returned from Alloc or Realloc, to size and returns the offset of the relocated file block or an error, if any. The contents will be unchanged in the range from the start of the region up to the minimum of the old and new sizes. Realloc(off, 0) is equal to Free(off). If the file block was moved, a Free(off) is done.
func (*Allocator) SetAutoFlush ¶
SetAutoFlush turns on/off automatic flushing of allocator's metadata. When the argument is true Flush is called automatically whenever the metadata are chaneged. When the argument is false, Flush is called automatically only on Close.
The default allocator state has auto flushing turned on.
func (*Allocator) UsableSize ¶
UsableSize reports the size of the file block allocated at off, which must have been returned from Alloc or Realloc. The allocated file block size can be larger than the size originally requested from Alloc or Realloc.
func (*Allocator) Verify ¶
func (a *Allocator) Verify(opt *VerifyOptions) error
Verify audits the correctness of the allocator and its backing File.
type File ¶
type File interface {
Close() error
ReadAt(p []byte, off int64) (n int, err error)
Stat() (os.FileInfo, error)
Sync() error
Truncate(int64) error
WriteAt(p []byte, off int64) (n int, err error)
}
File is an os.File-like entity.
Note: *os.File implements File.
func Map ¶
Map returns a File backed by memory mapping f or an error, if any. The Close method of the result must be eventually called to avoid resource leaks.
This function returns (f, nil) on Windows because there some File methods, like Truncate, produce errors in the current implementation:
The requested operation cannot be performed on a file with a user-mapped section open.
Windows expert needed to fix this.
type VerifyOptions ¶
type VerifyOptions struct {
Allocs int64 // Number of allocations in use.
Pages int64 // Number of pages.
UsedPages int64 // Number of pages in use.
}
VerifyOptions optionally provide more information from Verify.
type WAL ¶
type WAL struct {
DoSync bool // Secure commits with fsync.
F File // The f argument of NewWAL for convenience. R/O
W File // The w argument of NewWAL for convenience. R/O
// contains filtered or unexported fields
}
WAL implements a write ahead log of F using W. Call wal.F.ReadAt to perform 'read-commited' reads. Call wal.ReadAt to perform 'read uncommitted' reads. Call wal.W.ReadAt to read the write ahead log itself.
Concurrency ¶
wal.ReadAt (read uncommitted) is safe for concurrent use by multiple goroutines so multiple readers are fine, but multiple readers and a single writer is not. However, wal.F.ReadAt (read committed) is safe to run concurrently with any WAL method except Commit. In a typical DB isolation scenario the setup is something like
var wal *file.WAL var mu sync.RWMutex // The mutex associated with wal. // in another goroutine mu.RLock() // read isolated, concurrently with other readers n, err := wal.F.ReadAt(buf, off) // note the F ... // reads are isolated only until the next RUnlock. mu.RUnlock() // wal.Commit and mutating of F is now possible. ... // in another goroutine (writers serialization not considered in this example) n, err := wal.WriteAt(buf, off) ... // and eventually somewhere mu.Lock() err := wal.Commit() ... mu.Unlock() ...
No other WAL method is safe for concurrent use by multiple goroutines or concurrently with ReadAt. Callers must provide their own synchronization when used concurrently by multiple goroutines.
Logical page ¶
The WAL is divided in logical pages. Every page is 1<<pageLog bytes, pageLog being the argument of NewWAL.
Journal page ¶
Journal page holds the uncommitted logical page of F. Journal page is prefixed with a big-endian encoded int64 offset into F. If the offset is negative then the journal page is considered to be all zeros regardless of its actual content.
Journal allocating and size ¶
One journal page is appended to W on first write to any logical page. Total size of W is thus skip (argument of NewWAL) + N * (1<<pageLog + 8), where N is the number of write-touched logical pages. Commit adds a small amount of metadata at the end of W. The size and content of the meta data are not part of the API. However, future changes, if any, of the meta data size/content will continue to support journal files written by the previous versions.
Additionally, one map[int64]int64 item is used for every allocated journal page.
Crash before Commit ¶
If the process writing to WAL crashes before commit, W is invalid and it's not possible to continue the interrupted operation as it is not known at what point the crash occurred. NewWAL will reject invalid WAL file and will not delete it.
Crash during a Commit ¶
If the WAL metadata has not yet been written and the W file has not yet been synced then the situation is the same as crashing before commit.
Once Commit writes the metadata to W and W was synced, the transaction is secured and journal replay starts.
Journal replay ¶
Journal replay transfers all write-touched pages from W to F. Journal replay starts when Commit completes writing W metadata and syncing W. When the transfer is successfully completed, F is synced, W is emptied and synced, in that order.
Crash during journal replay ¶
If journal replay has started but not completed due to a crash then W is valid and non empty. If NewWAL is passed a valid, non empty W in its w argument, NewWAL restarts journal replay.
Example ¶
const pageLog = 1
dir, err := ioutil.TempDir("", "file-example-")
if err != nil {
panic(err)
}
defer os.RemoveAll(dir)
f, err := os.Create(filepath.Join(dir, "f"))
if err != nil {
panic(err)
}
w, err := os.Create(filepath.Join(dir, "w"))
if err != nil {
panic(err)
}
db, err := NewWAL(f, w, 0, pageLog)
if err != nil {
panic(err)
}
write := func(b []byte, off int64) {
if _, err := db.WriteAt(b, off); err != nil {
panic(err)
}
fi, err := w.Stat()
if err != nil {
panic(err)
}
fmt.Printf("---- db.WriteAt(%#v, %v)\n", b, off)
fmt.Printf("journal pages %v\n", len(db.m))
fmt.Printf("journal size %v\n", fi.Size())
}
read := func() {
b := make([]byte, 1024)
n, err := f.ReadAt(b, 0)
if n == 0 && err != io.EOF {
panic(err.Error())
}
fmt.Printf("Read committed: |% x|\n", b[:n])
if n, err = db.ReadAt(b, 0); n == 0 && err != io.EOF {
panic(err.Error())
}
fmt.Printf("Read uncommitted: |% x|\n", b[:n])
}
commit := func() {
if err := db.Commit(); err != nil {
panic(err)
}
fmt.Printf("---- Commit()\n")
fmt.Printf("journal pages %v\n", len(db.m))
read()
}
fmt.Printf("logical page size %v\n", 1<<pageLog)
fmt.Printf("journal page size %v\n", 1<<pageLog+8)
write([]byte{1, 2, 3}, 0)
read()
commit()
write([]byte{0xff}, 1)
read()
commit()
Output: logical page size 2 journal page size 10 ---- db.WriteAt([]byte{0x1, 0x2, 0x3}, 0) journal pages 2 journal size 20 Read committed: || Read uncommitted: |01 02 03| ---- Commit() journal pages 0 Read committed: |01 02 03| Read uncommitted: |01 02 03| ---- db.WriteAt([]byte{0xff}, 1) journal pages 1 journal size 10 Read committed: |01 02 03| Read uncommitted: |01 ff 03| ---- Commit() journal pages 0 Read committed: |01 ff 03| Read uncommitted: |01 ff 03|
func NewWAL ¶
NewWAL returns a newly created WAL or an error, if any. The f argument is the File to which the writes to WAL will eventually be committed. The w argument is a File used to collect the writes to the WAL before commit. The skip argument offsets the usage of w by skip bytes allowing for bookkeeping, header etc. The pageLog argument is the binary logarithm of WAL page size. Passing pageLog less than 1 will panic. The f and w arguments must not represent the same entity.
If w contains a valid, non empty write-ahead log, it's first committed to f and emptied. If w contains an invalid or unreadable write ahead log, the function returns an error.
func (*WAL) Commit ¶
Commit transfers all writes to w collected so far into w.F and empties w and w.W or returns an error, if any. If the program crashes during committing a subsequent NewWAL call with the same files f and w will re-initiate the commit operation.
The WAL is ready for reuse if Commit returns nil.
func (*WAL) ReadAt ¶
ReadAt performs a read-uncommitted operation on w. The semantics are those of io.ReaderAt.ReadAt. Call w.F.ReadAt to perform a read-committed operation.
ReadAt implements File.
func (*WAL) Stat ¶
Stat implements File. The size reported by the result is that of w.F _after_ w.Commit will be eventually successfully performed.
Stat implements File.
Source Files