persist

package
v1.5.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 16, 2021 License: MIT Imports: 20 Imported by: 66

README

Persist

The persist package contains functionality for storing persisted data.

Subsystems

AppendOnly

Key Files

The AppendOnly Persistence System is responsible for persistence files maintained in an append-only fashion and ensures safe and performant ACID operations. An append-only structure is used with the length in bytes encoded in the metadata.

Inbound Complexities

  • NewAppendOnlyPersist initializes the persistence file, either creating or loading it
  • Write appends the information to the persistence file
BoltDB

Key Files

TODO

  • fill out module explanation
JSON

Key Files

TODO

  • fill out module explanation
Log

Key Files

TODO

  • fill out module explanation
Persist

Key Files

TODO

  • fill out module explanation

Documentation

Index

Constants

View Source
const (
	// DefaultDiskPermissionsTest when creating files or directories in tests.
	DefaultDiskPermissionsTest = 0750

	// FixedMetadataSize is the size of the FixedMetadata header in bytes
	FixedMetadataSize = 32
)
View Source
const (
	// MetadataPageSize is the number of bytes set aside for the metadata page
	// on disk. It is the length of a disk sector so that we can ensure that a
	// metadata disk write is ACID with one write and sync and we don't have to
	// worry about the persist data ever crossing the first disk sector.
	MetadataPageSize uint64 = 4096
)

Variables

View Source
var (
	// ErrWrongHeader is the wrong header error.
	ErrWrongHeader = errors.New("wrong header")
	// ErrWrongVersion is the wrong version error.
	ErrWrongVersion = errors.New("wrong version")
)
View Source
var (
	// ErrBadFilenameSuffix indicates that SaveJSON or LoadJSON was called using
	// a filename that has a bad suffix. This prevents users from trying to use
	// this package to manage the temp files - this package will manage them
	// automatically.
	ErrBadFilenameSuffix = errors.New("filename suffix not allowed")

	// ErrBadHeader indicates that the file opened is not the file that was
	// expected.
	ErrBadHeader = errors.New("wrong header")

	// ErrBadVersion indicates that the version number of the file is not
	// compatible with the current codebase.
	ErrBadVersion = errors.New("incompatible version")

	// ErrFileInUse is returned if SaveJSON or LoadJSON is called on a file
	// that's already being manipulated in another thread by the persist
	// package.
	ErrFileInUse = errors.New("another thread is saving or loading this file")
)
View Source
var (
	// MetadataVersionv150 is a common metadata version specifier to avoid
	// types.Specifier conflicts
	MetadataVersionv150 = types.NewSpecifier("v1.5.0\n")

	// MetadataVersionv156 is a common metadata version specifier to avoid
	// types.Specifier conflicts
	MetadataVersionv156 = types.NewSpecifier("v1.5.6\n")
)

Functions

func LoadJSON added in v1.2.1

func LoadJSON(meta Metadata, object interface{}, filename string) error

LoadJSON will load a persisted json object from disk.

func RandomSuffix added in v1.0.0

func RandomSuffix() string

RandomSuffix returns a 20 character base32 suffix for a filename. There are 100 bits of entropy, and a very low probability of colliding with existing files unintentionally.

func RemoveFile added in v1.3.1

func RemoveFile(filename string) error

RemoveFile removes an atomic file from disk, along with any uncommitted or temporary files.

func SaveJSON added in v1.2.1

func SaveJSON(meta Metadata, object interface{}, filename string) error

SaveJSON will save a json object to disk in a durable, atomic way. The resulting file will have a checksum of the data as the third line. If manually editing files, the checksum line can be replaced with the 8 characters "manual". This will cause the reader to accept the checksum even though the file has been changed.

func UID added in v1.5.0

func UID() string

UID returns a hexadecimal encoded string that can be used as an unique ID.

Types

type AppendOnlyPersist added in v1.4.9

type AppendOnlyPersist struct {
	// contains filtered or unexported fields
}

AppendOnlyPersist is the object responsible for creating, loading, and updating append-only persist files.

func NewAppendOnlyPersist added in v1.4.9

func NewAppendOnlyPersist(dir, file string, metadataHeader, metadataVersion types.Specifier) (*AppendOnlyPersist, io.Reader, error)

NewAppendOnlyPersist creates a new AppendOnlyPersist object and initializes the persistence file.

func (*AppendOnlyPersist) Close added in v1.4.9

func (aop *AppendOnlyPersist) Close() error

Close closes the persist file, freeing the resource and preventing further writes.

func (*AppendOnlyPersist) FilePath added in v1.4.9

func (aop *AppendOnlyPersist) FilePath() string

FilePath returns the filepath of the persist file.

func (*AppendOnlyPersist) PersistLength added in v1.4.9

func (aop *AppendOnlyPersist) PersistLength() uint64

PersistLength returns the length of the persist data.

func (*AppendOnlyPersist) Write added in v1.4.9

func (aop *AppendOnlyPersist) Write(b []byte) (int, error)

Write implements the io.Writer interface. It updates the persist file, appending the changes to the persist file on disk.

type BoltDatabase added in v1.0.0

type BoltDatabase struct {
	Metadata
	*bolt.DB
}

BoltDatabase is a persist-level wrapper for the bolt database, providing extra information such as a version number.

func OpenDatabase added in v1.0.0

func OpenDatabase(md Metadata, filename string) (*BoltDatabase, error)

OpenDatabase opens a database and validates its metadata.

func (*BoltDatabase) Close added in v1.0.0

func (db *BoltDatabase) Close() error

Close closes the database.

type FixedMetadata added in v1.4.2

type FixedMetadata struct {
	Header  types.Specifier
	Version types.Specifier
}

FixedMetadata contains the header and version of the data being stored as a fixed-length byte-array.

func VerifyMetadataHeader added in v1.4.2

func VerifyMetadataHeader(r io.Reader, expected FixedMetadata) (FixedMetadata, error)

VerifyMetadataHeader will take in a reader and an expected metadata header, if the file's header has a different header or version it will return the corresponding error and the actual metadata header

type Logger added in v1.0.0

type Logger struct {
	*log.Logger
}

Logger is a wrapper for log.Logger.

func NewFileLogger added in v1.0.0

func NewFileLogger(logFilename string) (*Logger, error)

NewFileLogger returns a logger that logs to logFilename. The file is opened in append mode, and created if it does not exist.

func NewLogger added in v1.0.0

func NewLogger(w io.Writer) (*Logger, error)

NewLogger returns a logger that can be closed. Calls should not be made to the logger after 'Close' has been called.

type Metadata added in v0.3.3

type Metadata struct {
	Header  string
	Version string
}

Metadata contains the header and version of the data being stored.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL