vfs

package
v0.15.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 1, 2024 License: MIT Imports: 20 Imported by: 0

README

Go SQLite VFS API

This package implements the SQLite OS Interface (aka VFS).

It replaces the default SQLite VFS with a pure Go implementation.

It also exposes interfaces that should allow you to implement your own custom VFSes.

Since it is a from scratch reimplementation, there are naturally some ways it deviates from the original.

The main differences are file locking and WAL mode support.

File Locking

POSIX advisory locks, which SQLite uses on Unix, are broken by design.

On Linux and macOS, this module uses OFD locks to synchronize access to database files. OFD locks are fully compatible with POSIX advisory locks.

On BSD Unixes, this module uses BSD locks. On BSD, these locks are fully compatible with POSIX advisory locks. However, concurrency is reduced with BSD locks (BEGIN IMMEDIATE behaves the same as BEGIN EXCLUSIVE).

On Windows, this module uses LockFileEx and UnlockFileEx, like SQLite.

On all other platforms, file locking is not supported, and you must use nolock=1 (or immutable=1) to open database files.
To use the database/sql driver with nolock=1 you must disable connection pooling by calling db.SetMaxOpenConns(1).

You can use vfs.SupportsFileLocking to check if your platform supports file locking.

Write-Ahead Logging

On 64-bit Linux and macOS, this module uses mmap to implement shared-memory for the WAL-index, like SQLite.

To allow mmap to work, each connection needs to reserve up to 4GB of address space.
To limit the amount of address space each connection needs, use WithMemoryLimitPages.

On Windows and BSD, WAL support is limited. EXCLUSIVE locking mode can be set to create, read, and write WAL databases.
To use EXCLUSIVE locking mode with the database/sql driver you must disable connection pooling by calling db.SetMaxOpenConns(1).

On all other platforms, where file locking is not supported, WAL mode does not work.

You can use vfs.SupportsSharedMemory to check if your platform supports shared memory.

Batch-Atomic Write

On 64-bit Linux, this module supports batch-atomic writes with the F2FS filesystem.

Build tags

The VFS can be customized with a few build tags:

  • sqlite3_flock forces the use of BSD locks; it can be used on macOS to test the BSD locking implementation.
  • sqlite3_nosys prevents importing x/sys; disables locking and shared memory on all platforms.
  • sqlite3_noshm disables shared memory on all platforms.

Documentation

Overview

Package vfs wraps the C SQLite VFS API.

Index

Constants

View Source
const SupportsFileLocking = true

SupportsFileLocking is false on platforms that do not support file locking. To open a database file on those platforms, you need to use the nolock or immutable URI parameters.

View Source
const SupportsSharedMemory = true

SupportsSharedMemory is false on platforms that do not support shared memory. To use WAL without shared-memory, you need to set EXCLUSIVE locking mode.

Variables

This section is empty.

Functions

func ExportHostFunctions

func ExportHostFunctions(env wazero.HostModuleBuilder) wazero.HostModuleBuilder

ExportHostFunctions is an internal API users need not call directly.

ExportHostFunctions registers the required VFS host functions with the provided env module.

func Register

func Register(name string, vfs VFS)

Register registers a VFS. Empty and "os" are reserved names.

https://sqlite.org/c3ref/vfs_find.html

func Unregister

func Unregister(name string)

Unregister unregisters a VFS.

https://sqlite.org/c3ref/vfs_find.html

Types

type AccessFlag

type AccessFlag uint32

AccessFlag is a flag for the VFS Access method.

https://sqlite.org/c3ref/c_access_exists.html

const (
	ACCESS_EXISTS    AccessFlag = 0
	ACCESS_READWRITE AccessFlag = 1 /* Used by PRAGMA temp_store_directory */
	ACCESS_READ      AccessFlag = 2 /* Unused */
)

type DeviceCharacteristic

type DeviceCharacteristic uint32

DeviceCharacteristic is a flag retuned by the File DeviceCharacteristics method.

https://sqlite.org/c3ref/c_iocap_atomic.html

const (
	IOCAP_ATOMIC                DeviceCharacteristic = 0x00000001
	IOCAP_ATOMIC512             DeviceCharacteristic = 0x00000002
	IOCAP_ATOMIC1K              DeviceCharacteristic = 0x00000004
	IOCAP_ATOMIC2K              DeviceCharacteristic = 0x00000008
	IOCAP_ATOMIC4K              DeviceCharacteristic = 0x00000010
	IOCAP_ATOMIC8K              DeviceCharacteristic = 0x00000020
	IOCAP_ATOMIC16K             DeviceCharacteristic = 0x00000040
	IOCAP_ATOMIC32K             DeviceCharacteristic = 0x00000080
	IOCAP_ATOMIC64K             DeviceCharacteristic = 0x00000100
	IOCAP_SAFE_APPEND           DeviceCharacteristic = 0x00000200
	IOCAP_SEQUENTIAL            DeviceCharacteristic = 0x00000400
	IOCAP_UNDELETABLE_WHEN_OPEN DeviceCharacteristic = 0x00000800
	IOCAP_POWERSAFE_OVERWRITE   DeviceCharacteristic = 0x00001000
	IOCAP_IMMUTABLE             DeviceCharacteristic = 0x00002000
	IOCAP_BATCH_ATOMIC          DeviceCharacteristic = 0x00004000
)

type File

type File interface {
	Close() error
	ReadAt(p []byte, off int64) (n int, err error)
	WriteAt(p []byte, off int64) (n int, err error)
	Truncate(size int64) error
	Sync(flags SyncFlag) error
	Size() (int64, error)
	Lock(lock LockLevel) error
	Unlock(lock LockLevel) error
	CheckReservedLock() (bool, error)
	SectorSize() int
	DeviceCharacteristics() DeviceCharacteristic
}

A File represents an open file in the OS interface layer.

Use sqlite3.ErrorCode or sqlite3.ExtendedErrorCode to return specific error codes to SQLite. In particular, sqlite3.BUSY is necessary to correctly implement lock methods.

https://sqlite.org/c3ref/io_methods.html

type FileBatchAtomicWrite

type FileBatchAtomicWrite interface {
	File
	BeginAtomicWrite() error
	CommitAtomicWrite() error
	RollbackAtomicWrite() error
}

FileBatchAtomicWrite extends File to implement the SQLITE_FCNTL_BEGIN_ATOMIC_WRITE, SQLITE_FCNTL_COMMIT_ATOMIC_WRITE and SQLITE_FCNTL_ROLLBACK_ATOMIC_WRITE file control opcodes.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlbeginatomicwrite

type FileCheckpoint added in v0.15.0

type FileCheckpoint interface {
	File
	CheckpointDone() error
	CheckpointStart() error
}

FileCheckpoint extends File to implement the SQLITE_FCNTL_CKPT_START and SQLITE_FCNTL_CKPT_DONE file control opcodes.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlckptstart

type FileChunkSize added in v0.15.0

type FileChunkSize interface {
	File
	ChunkSize(size int)
}

FileChunkSize extends File to implement the SQLITE_FCNTL_CHUNK_SIZE file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlchunksize

type FileCommitPhaseTwo

type FileCommitPhaseTwo interface {
	File
	CommitPhaseTwo() error
}

FileCommitPhaseTwo extends File to implement the SQLITE_FCNTL_COMMIT_PHASETWO file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlcommitphasetwo

type FileHasMoved

type FileHasMoved interface {
	File
	HasMoved() (bool, error)
}

FileHasMoved extends File to implement the SQLITE_FCNTL_HAS_MOVED file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlhasmoved

type FileLockState

type FileLockState interface {
	File
	LockState() LockLevel
}

FileLockState extends File to implement the SQLITE_FCNTL_LOCKSTATE file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntllockstate

type FileOverwrite added in v0.9.0

type FileOverwrite interface {
	File
	Overwrite() error
}

FileOverwrite extends File to implement the SQLITE_FCNTL_OVERWRITE file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntloverwrite

type FilePersistentWAL added in v0.13.0

type FilePersistentWAL interface {
	File
	PersistentWAL() bool
	SetPersistentWAL(bool)
}

FilePersistentWAL extends File to implement the SQLITE_FCNTL_PERSIST_WAL file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlpersistwal

type FilePowersafeOverwrite

type FilePowersafeOverwrite interface {
	File
	PowersafeOverwrite() bool
	SetPowersafeOverwrite(bool)
}

FilePowersafeOverwrite extends File to implement the SQLITE_FCNTL_POWERSAFE_OVERWRITE file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlpowersafeoverwrite

type FilePragma added in v0.15.0

type FilePragma interface {
	File
	Pragma(name, value string) (string, error)
}

FilePragma extends File to implement the SQLITE_FCNTL_PRAGMA file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlpragma

type FileSharedMemory added in v0.15.0

type FileSharedMemory interface {
	File
	SharedMemory() SharedMemory
}

FileSharedMemory extends File to possibly implement shared-memory for the WAL-index. The same shared-memory instance must be returned for the entire life of the file. It's OK for SharedMemory to return nil.

type FileSizeHint

type FileSizeHint interface {
	File
	SizeHint(size int64) error
}

FileSizeHint extends File to implement the SQLITE_FCNTL_SIZE_HINT file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlsizehint

type Filename added in v0.15.0

type Filename struct {
	// contains filtered or unexported fields
}

Filename is used by SQLite to pass filenames to the Open method of a VFS.

https://sqlite.org/c3ref/filename.html

func OpenFilename added in v0.15.0

func OpenFilename(ctx context.Context, mod api.Module, id uint32, flags OpenFlag) *Filename

OpenFilename is an internal API users should not call directly.

func (*Filename) Database added in v0.15.0

func (n *Filename) Database() string

Database returns the name of the corresponding database file.

https://sqlite.org/c3ref/filename_database.html

func (*Filename) DatabaseFile added in v0.15.0

func (n *Filename) DatabaseFile() File

DatabaseFile returns the main database File corresponding to a journal.

https://sqlite.org/c3ref/database_file_object.html

func (*Filename) Journal added in v0.15.0

func (n *Filename) Journal() string

Journal returns the name of the corresponding rollback journal file.

https://sqlite.org/c3ref/filename_database.html

func (*Filename) String added in v0.15.0

func (n *Filename) String() string

String returns this filename as a string.

func (*Filename) URIParameter added in v0.15.0

func (n *Filename) URIParameter(key string) string

URIParameter returns the value of a URI parameter.

https://sqlite.org/c3ref/uri_boolean.html

func (*Filename) URIParameters added in v0.15.0

func (n *Filename) URIParameters() url.Values

URIParameters obtains values for URI parameters.

https://sqlite.org/c3ref/uri_boolean.html

func (*Filename) WAL added in v0.15.0

func (n *Filename) WAL() string

Journal returns the name of the corresponding WAL file.

https://sqlite.org/c3ref/filename_database.html

type LockLevel

type LockLevel uint32

LockLevel is a value used with File Lock and Unlock methods.

https://sqlite.org/c3ref/c_lock_exclusive.html

const (
	// No locks are held on the database.
	// The database may be neither read nor written.
	// Any internally cached data is considered suspect and subject to
	// verification against the database file before being used.
	// Other processes can read or write the database as their own locking
	// states permit.
	// This is the default state.
	LOCK_NONE LockLevel = 0 /* xUnlock() only */

	// The database may be read but not written.
	// Any number of processes can hold SHARED locks at the same time,
	// hence there can be many simultaneous readers.
	// But no other thread or process is allowed to write to the database file
	// while one or more SHARED locks are active.
	LOCK_SHARED LockLevel = 1 /* xLock() or xUnlock() */

	// A RESERVED lock means that the process is planning on writing to the
	// database file at some point in the future but that it is currently just
	// reading from the file.
	// Only a single RESERVED lock may be active at one time,
	// though multiple SHARED locks can coexist with a single RESERVED lock.
	// RESERVED differs from PENDING in that new SHARED locks can be acquired
	// while there is a RESERVED lock.
	LOCK_RESERVED LockLevel = 2 /* xLock() only */

	// A PENDING lock means that the process holding the lock wants to write to
	// the database as soon as possible and is just waiting on all current
	// SHARED locks to clear so that it can get an EXCLUSIVE lock.
	// No new SHARED locks are permitted against the database if a PENDING lock
	// is active, though existing SHARED locks are allowed to continue.
	LOCK_PENDING LockLevel = 3 /* internal use only */

	// An EXCLUSIVE lock is needed in order to write to the database file.
	// Only one EXCLUSIVE lock is allowed on the file and no other locks of any
	// kind are allowed to coexist with an EXCLUSIVE lock.
	// In order to maximize concurrency, SQLite works to minimize the amount of
	// time that EXCLUSIVE locks are held.
	LOCK_EXCLUSIVE LockLevel = 4 /* xLock() only */
)

type OpenFlag

type OpenFlag uint32

OpenFlag is a flag for the VFS Open method.

https://sqlite.org/c3ref/c_open_autoproxy.html

const (
	OPEN_READONLY      OpenFlag = 0x00000001 /* Ok for sqlite3_open_v2() */
	OPEN_READWRITE     OpenFlag = 0x00000002 /* Ok for sqlite3_open_v2() */
	OPEN_CREATE        OpenFlag = 0x00000004 /* Ok for sqlite3_open_v2() */
	OPEN_DELETEONCLOSE OpenFlag = 0x00000008 /* VFS only */
	OPEN_EXCLUSIVE     OpenFlag = 0x00000010 /* VFS only */
	OPEN_AUTOPROXY     OpenFlag = 0x00000020 /* VFS only */
	OPEN_URI           OpenFlag = 0x00000040 /* Ok for sqlite3_open_v2() */
	OPEN_MEMORY        OpenFlag = 0x00000080 /* Ok for sqlite3_open_v2() */
	OPEN_MAIN_DB       OpenFlag = 0x00000100 /* VFS only */
	OPEN_TEMP_DB       OpenFlag = 0x00000200 /* VFS only */
	OPEN_TRANSIENT_DB  OpenFlag = 0x00000400 /* VFS only */
	OPEN_MAIN_JOURNAL  OpenFlag = 0x00000800 /* VFS only */
	OPEN_TEMP_JOURNAL  OpenFlag = 0x00001000 /* VFS only */
	OPEN_SUBJOURNAL    OpenFlag = 0x00002000 /* VFS only */
	OPEN_SUPER_JOURNAL OpenFlag = 0x00004000 /* VFS only */
	OPEN_NOMUTEX       OpenFlag = 0x00008000 /* Ok for sqlite3_open_v2() */
	OPEN_FULLMUTEX     OpenFlag = 0x00010000 /* Ok for sqlite3_open_v2() */
	OPEN_SHAREDCACHE   OpenFlag = 0x00020000 /* Ok for sqlite3_open_v2() */
	OPEN_PRIVATECACHE  OpenFlag = 0x00040000 /* Ok for sqlite3_open_v2() */
	OPEN_WAL           OpenFlag = 0x00080000 /* VFS only */
	OPEN_NOFOLLOW      OpenFlag = 0x01000000 /* Ok for sqlite3_open_v2() */
)

type SharedMemory added in v0.15.0

type SharedMemory interface {
	io.Closer
	// contains filtered or unexported methods
}

SharedMemory is a shared-memory WAL-index implementation. Use NewSharedMemory to create a shared-memory.

func NewSharedMemory added in v0.15.0

func NewSharedMemory(path string, flags OpenFlag) SharedMemory

NewSharedMemory returns a shared-memory WAL-index backed by a file with the given path. It will return nil if shared-memory is not supported, or not appropriate for the given flags. Only OPEN_MAIN_DB databases may need a WAL-index. You must ensure all concurrent accesses to a database use shared-memory instances created with the same path.

type SyncFlag

type SyncFlag uint32

SyncFlag is a flag for the File Sync method.

https://sqlite.org/c3ref/c_sync_dataonly.html

const (
	SYNC_NORMAL   SyncFlag = 0x00002
	SYNC_FULL     SyncFlag = 0x00003
	SYNC_DATAONLY SyncFlag = 0x00010
)

type VFS

type VFS interface {
	Open(name string, flags OpenFlag) (File, OpenFlag, error)
	Delete(name string, syncDir bool) error
	Access(name string, flags AccessFlag) (bool, error)
	FullPathname(name string) (string, error)
}

A VFS defines the interface between the SQLite core and the underlying operating system.

Use sqlite3.ErrorCode or sqlite3.ExtendedErrorCode to return specific error codes to SQLite.

https://sqlite.org/c3ref/vfs.html

func Find

func Find(name string) VFS

Find returns a VFS given its name. If there is no match, nil is returned. If name is empty, the default VFS is returned.

https://sqlite.org/c3ref/vfs_find.html

type VFSFilename added in v0.15.0

type VFSFilename interface {
	VFS
	OpenFilename(name *Filename, flags OpenFlag) (File, OpenFlag, error)
}

VFSFilename extends VFS with the ability to use Filename objects for opening files.

https://sqlite.org/c3ref/filename.html

Directories

Path Synopsis
Package adiantum wraps an SQLite VFS to offer encryption at rest.
Package adiantum wraps an SQLite VFS to offer encryption at rest.
Package memdb implements the "memdb" SQLite VFS.
Package memdb implements the "memdb" SQLite VFS.
Package readervfs implements an SQLite VFS for immutable databases.
Package readervfs implements an SQLite VFS for immutable databases.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL