storage

package
v0.0.0-...-f1c747c Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 11, 2017 License: Apache-2.0 Imports: 11 Imported by: 0

README

Storage layer

The interface, various concrete implementations, and any associated components live here. Currently, there is only one storage implementation:

  • MySQL/MariaDB, which lives in mysql/.

The design is such that both LogStorage and MapStorage models reuse a shared TreeStorage model which can store arbitrary nodes in a tree.

Anyone poking around in here should be aware that there are some subtle wrinkles introduced by the fact that Log trees grow upwards (i.e. the Log considers nodes at level 0 to be the leaves), and in contrast the Map considers the leaves to be at level 255 (and the root at 0), this is based on the HStar2 algorithm.

TreeStorage

Nodes

Nodes within the tree are each given a unique NodeID (see storage/types.go), this ID can be thought of as the binary path (0 for left, 1 for right) from the root of the tree down to the node in question (or, equivalently, as the binary representation of the node's horizonal index into the tree layer at the depth of the node.)

TODO(al): pictures!

Subtrees

The TreeStorage model does not, in fact, store all the internal nodes of the tree; it divides the tree into subtrees of depth 8 and stores the data for each subtree as a single unit. Within these subtrees, only the (subtree-relative) "leaf" nodes are actually written to disk, the internal structure of the subtrees is re-calculated when the subtree is read from disk.

Doing this compaction saves a considerable amout of on-disk space, and at least for the MySQL storage implementation, results in a ~20% speed increase.

History

Updates to the tree storage are performed in a batched fashion (i.e. some unit of update which provides a self-consistent view of the tree - e.g.:

  • n append leaf operations along with internal node updates for the LogStorage, and tagged with their sequence number.
  • n set value operations along with internal node updates for the MapStorage.

These batched updates are termed treeRevisions, and nodes updated within each revision are tagged with a monotonically incrementing sequence number.

In this fashion, the storage model records all historical revisions of the tree.

To perform lookups at a particular treeRevision, the TreeStorage simply requests nodes from disk which are associated with the given NodeID and whose treeRevsions are <= the desired revision.

Currently there's no mechanism to safely garbage collect obsolete nodes so storage grows without bound. This will be addressed at some point in the future.

Updates to the tree

The current treeRevision is defined to be the one referenced by the latest Signed [Map|Log] Head (if there is no SignedHead, then the current treeRevision is -1.)

Updates to the tree are performed in the future (i.e. at currentTreeRevision + 1), and, to allow for reasonable performance, are not required to be atomic (i.e. a tree update may partially fail), however for this to work, higher layers using the storage layer must guarantee that failed tree updates are later re-tried with either precisely the same set of node chages, or a superset there-of, in order to ensure integrity of the tree.

We intend to enforce this contract within the treeStorage layer at some point in the future.

LogStorage

TODO(al): flesh this out

LogStorage builds upon TreeStorage and additionally provides a means of storing log leaves, and SignedTreeHeads, and an API for sequencing new leaves into the tree.

MapStorage

TODO(al): flesh this out

MapStorage builds upon TreeStorage and additionally provides a means of storing map values, and SignedMapHeads.

Documentation

Overview

Package storage provides general interfaces to Trillian storage layers.

Index

Constants

View Source
const (
	DuplicateLeaf = iota
)

Integer types to distinguish storage errors that might need to be mapped at a higher level.

Variables

This section is empty.

Functions

func NewTreeID

func NewTreeID() (int64, error)

NewTreeID generates a random, positive, non-zero tree ID.

func ValidateTreeForCreation

func ValidateTreeForCreation(tree *trillian.Tree) error

ValidateTreeForCreation returns nil if tree is valid for insertion, error otherwise. See the documentation on trillian.Tree for reference on which values are valid.

func ValidateTreeForUpdate

func ValidateTreeForUpdate(storedTree, newTree *trillian.Tree) error

ValidateTreeForUpdate returns nil if newTree is valid for update, error otherwise. The newTree is compared to the storedTree to determine if readonly fields have been changed. It's assumed that storage-generated fields, such as update_time, have not yet changed when this method is called. See the documentation on trillian.Tree for reference on which fields may be changed and what is considered valid for each of them.

Types

type AdminReader

type AdminReader interface {
	// GetTree returns the tree corresponding to treeID or an error.
	GetTree(ctx context.Context, treeID int64) (*trillian.Tree, error)

	// ListTreeIDs returns the IDs of all trees in storage.
	// Note that there's no authorization restriction on the IDs returned,
	// so it should be used with caution in production code.
	ListTreeIDs(ctx context.Context) ([]int64, error)

	// ListTrees returns all trees in storage.
	// Note that there's no authorization restriction on the trees returned,
	// so it should be used with caution in production code.
	ListTrees(ctx context.Context) ([]*trillian.Tree, error)
}

AdminReader provides a read-only interface for tree data.

type AdminStorage

type AdminStorage interface {
	// Snapshot starts a read-only transaction.
	// A transaction must be explicitly committed before the data read by it
	// is considered consistent.
	Snapshot(ctx context.Context) (ReadOnlyAdminTX, error)

	// Begin starts a read/write transaction.
	// A transaction must be explicitly committed before the data read by it
	// is considered consistent.
	Begin(ctx context.Context) (AdminTX, error)
}

AdminStorage represents the persistent storage of tree data.

type AdminTX

type AdminTX interface {
	ReadOnlyAdminTX
	AdminWriter
}

AdminTX is a transaction capable of read and write operations in the AdminStorage.

type AdminWriter

type AdminWriter interface {
	// CreateTree inserts the specified tree in storage, returning a tree
	// with all storage-generated fields set.
	// Note that treeID and timestamps will be automatically generated by
	// the storage layer, thus may be ignored by the implementation.
	// Remaining fields must be set to valid values.
	// Returns an error if the tree is invalid or creation fails.
	CreateTree(ctx context.Context, tree *trillian.Tree) (*trillian.Tree, error)

	// UpdateTree updates the specified tree in storage, returning a tree
	// with all storage-generated fields set.
	// updateFunc is called to perform the desired tree modifications. Refer
	// to trillian.Tree for details on which fields are mutable and what is
	// considered valid.
	// Returns an error if the tree is invalid or the update cannot be
	// performed.
	UpdateTree(ctx context.Context, treeID int64, updateFunc func(*trillian.Tree)) (*trillian.Tree, error)
}

AdminWriter provides a write-only interface for tree data.

type DatabaseChecker

type DatabaseChecker interface {
	// CheckDatabaseAccessible returns nil if the database is accessible, error otherwise.
	CheckDatabaseAccessible(context.Context) error
}

DatabaseChecker performs connectivity checks on the database.

type Error

type Error struct {
	ErrType int
	Detail  string
	Cause   error
}

Error is a typed error that the storage layer can return to give callers information about the error to decide how to handle it.

func (Error) Error

func (s Error) Error() string

Error formats the internal details of an Error including the original cause.

type Getter

type Getter interface {
	// Get retrieves the values associates with the keyHashes, if any, at the
	// specified revision.
	// Setting revision to -1 will fetch the latest revision.
	// The returned array of MapLeaves will only contain entries for which values
	// exist.  i.e. requesting a set of unknown keys would result in a
	// zero-length array being returned.
	Get(ctx context.Context, revision int64, keyHashes [][]byte) ([]trillian.MapLeaf, error)
}

Getter allows access to the values stored in the map.

type LeafDequeuer

type LeafDequeuer interface {
	// DequeueLeaves will return between [0, limit] leaves from the queue.
	// Leaves which have been dequeued within a Rolled-back Tx will become available for dequeing again.
	// Leaves queued more recently than the cutoff time will not be returned. This allows for
	// guard intervals to be configured.
	DequeueLeaves(ctx context.Context, limit int, cutoffTime time.Time) ([]*trillian.LogLeaf, error)
	UpdateSequencedLeaves(ctx context.Context, leaves []*trillian.LogLeaf) error
}

LeafDequeuer provides an interface for reading previously queued leaves for integration into the tree.

type LeafQueuer

type LeafQueuer interface {
	// QueueLeaves enqueues leaves for later integration into the tree.
	// If error is nil, the returned slice of leaves will be the same size as the
	// input, and each entry will hold:
	//  - the existing leaf entry if a duplicate has been submitted
	//  - nil otherwise.
	// Duplicates are only reported if the underlying tree does not permit duplicates, and are
	// considered duplicate if their leaf.LeafIdentityHash matches.
	QueueLeaves(ctx context.Context, leaves []*trillian.LogLeaf, queueTimestamp time.Time) ([]*trillian.LogLeaf, error)
}

LeafQueuer provides a write-only interface for the queueing (but not necessarily integration) of leaves.

type LeafReader

type LeafReader interface {
	// GetSequencedLeafCount returns the total number of leaves that have been integrated into the
	// tree via sequencing.
	GetSequencedLeafCount(ctx context.Context) (int64, error)
	// GetLeavesByIndex returns leaf metadata and data for a set of specified sequenced leaf indexes.
	GetLeavesByIndex(ctx context.Context, leaves []int64) ([]*trillian.LogLeaf, error)
	// GetLeavesByHash looks up sequenced leaf metadata and data by their Merkle leaf hash. If the
	// tree permits duplicate leaves callers must be prepared to handle multiple results with the
	// same hash but different sequence numbers. If orderBySequence is true then the returned data
	// will be in ascending sequence number order.
	GetLeavesByHash(ctx context.Context, leafHashes [][]byte, orderBySequence bool) ([]*trillian.LogLeaf, error)
}

LeafReader provides a read only interface to stored tree leaves

type LogMetadata

type LogMetadata interface {
	// GetActiveLogs returns a list of the IDs of all the logs that are configured in storage
	GetActiveLogIDs(ctx context.Context) ([]int64, error)
	// GetActiveLogIDsWithPendingWork returns a list of IDs of logs that have
	// pending queued leaves that need to be integrated into the log.
	GetActiveLogIDsWithPendingWork(ctx context.Context) ([]int64, error)
}

LogMetadata provides access to information about the logs in storage

type LogRootReader

type LogRootReader interface {
	// LatestSignedLogRoot returns the most recent SignedLogRoot, if any.
	LatestSignedLogRoot(ctx context.Context) (trillian.SignedLogRoot, error)
}

LogRootReader provides an interface for reading SignedLogRoots.

type LogRootWriter

type LogRootWriter interface {
	// StoreSignedLogRoot stores a freshly created SignedLogRoot.
	StoreSignedLogRoot(ctx context.Context, root trillian.SignedLogRoot) error
}

LogRootWriter provides an interface for storing new SignedLogRoots.

type LogStorage

type LogStorage interface {
	ReadOnlyLogStorage

	// BeginForTree starts a transaction for the specified treeID.
	// Either Commit or Rollback must be called when the caller is finished with
	// the returned object, and values read through it should only be propagated
	// if Commit returns without error.
	BeginForTree(ctx context.Context, treeID int64) (LogTreeTX, error)
}

LogStorage should be implemented by concrete storage mechanisms which want to support Logs.

type LogTreeTX

LogTreeTX is the transactional interface for reading/updating a Log. It extends the basic TreeTX interface with Log specific methods. After a call to Commit or Rollback implementations must be in a clean state and have released any resources owned by the LogTX. A LogTreeTX can only modify the tree specified in its creation.

type MapRootReader

type MapRootReader interface {
	// GetSignedMapRoot returns the SignedMapRoot associated with the
	// specified revision.
	GetSignedMapRoot(ctx context.Context, revision int64) (trillian.SignedMapRoot, error)
	// LatestSignedMapRoot returns the most recently created SignedMapRoot.
	LatestSignedMapRoot(ctx context.Context) (trillian.SignedMapRoot, error)
}

MapRootReader provides access to the map roots.

type MapRootWriter

type MapRootWriter interface {
	// StoreSignedMapRoot stores root.
	StoreSignedMapRoot(ctx context.Context, root trillian.SignedMapRoot) error
}

MapRootWriter allows the storage of new SignedMapRoots

type MapStorage

type MapStorage interface {
	ReadOnlyMapStorage
	// BeginForTree starts a new Map transaction.
	// Either Commit or Rollback must be called when the caller is finished with
	// the returned object, and values read through it should only be propagated
	// if Commit returns without error.
	BeginForTree(ctx context.Context, treeID int64) (MapTreeTX, error)
}

MapStorage should be implemented by concrete storage mechanisms which want to support Maps

type MapTreeTX

type MapTreeTX interface {
	TreeTX
	MapRootReader
	MapRootWriter
	Getter
	Setter
}

MapTreeTX is the transactional interface for reading/modifying a Map. It extends the basic TreeTX interface with Map specific methods. After a call to Commit or Rollback implementations must be in a clean state and have released any resources owned by the MapTX. A MapTreeTX can only read from the tree specified in its creation.

type MockAdminStorage

type MockAdminStorage struct {
	// contains filtered or unexported fields
}

Mock of AdminStorage interface

func NewMockAdminStorage

func NewMockAdminStorage(ctrl *gomock.Controller) *MockAdminStorage

func (*MockAdminStorage) Begin

func (_m *MockAdminStorage) Begin(_param0 context.Context) (AdminTX, error)

func (*MockAdminStorage) EXPECT

func (_m *MockAdminStorage) EXPECT() *_MockAdminStorageRecorder

func (*MockAdminStorage) Snapshot

func (_m *MockAdminStorage) Snapshot(_param0 context.Context) (ReadOnlyAdminTX, error)

type MockAdminTX

type MockAdminTX struct {
	// contains filtered or unexported fields
}

Mock of AdminTX interface

func NewMockAdminTX

func NewMockAdminTX(ctrl *gomock.Controller) *MockAdminTX

func (*MockAdminTX) Close

func (_m *MockAdminTX) Close() error

func (*MockAdminTX) Commit

func (_m *MockAdminTX) Commit() error

func (*MockAdminTX) CreateTree

func (_m *MockAdminTX) CreateTree(_param0 context.Context, _param1 *trillian.Tree) (*trillian.Tree, error)

func (*MockAdminTX) EXPECT

func (_m *MockAdminTX) EXPECT() *_MockAdminTXRecorder

func (*MockAdminTX) GetTree

func (_m *MockAdminTX) GetTree(_param0 context.Context, _param1 int64) (*trillian.Tree, error)

func (*MockAdminTX) IsClosed

func (_m *MockAdminTX) IsClosed() bool

func (*MockAdminTX) ListTreeIDs

func (_m *MockAdminTX) ListTreeIDs(_param0 context.Context) ([]int64, error)

func (*MockAdminTX) ListTrees

func (_m *MockAdminTX) ListTrees(_param0 context.Context) ([]*trillian.Tree, error)

func (*MockAdminTX) Rollback

func (_m *MockAdminTX) Rollback() error

func (*MockAdminTX) UpdateTree

func (_m *MockAdminTX) UpdateTree(_param0 context.Context, _param1 int64, _param2 func(*trillian.Tree)) (*trillian.Tree, error)

type MockLogStorage

type MockLogStorage struct {
	// contains filtered or unexported fields
}

Mock of LogStorage interface

func NewMockLogStorage

func NewMockLogStorage(ctrl *gomock.Controller) *MockLogStorage

func (*MockLogStorage) BeginForTree

func (_m *MockLogStorage) BeginForTree(_param0 context.Context, _param1 int64) (LogTreeTX, error)

func (*MockLogStorage) CheckDatabaseAccessible

func (_m *MockLogStorage) CheckDatabaseAccessible(_param0 context.Context) error

func (*MockLogStorage) EXPECT

func (_m *MockLogStorage) EXPECT() *_MockLogStorageRecorder

func (*MockLogStorage) Snapshot

func (_m *MockLogStorage) Snapshot(_param0 context.Context) (ReadOnlyLogTX, error)

func (*MockLogStorage) SnapshotForTree

func (_m *MockLogStorage) SnapshotForTree(_param0 context.Context, _param1 int64) (ReadOnlyLogTreeTX, error)

type MockLogTreeTX

type MockLogTreeTX struct {
	// contains filtered or unexported fields
}

Mock of LogTreeTX interface

func NewMockLogTreeTX

func NewMockLogTreeTX(ctrl *gomock.Controller) *MockLogTreeTX

func (*MockLogTreeTX) Close

func (_m *MockLogTreeTX) Close() error

func (*MockLogTreeTX) Commit

func (_m *MockLogTreeTX) Commit() error

func (*MockLogTreeTX) DequeueLeaves

func (_m *MockLogTreeTX) DequeueLeaves(_param0 context.Context, _param1 int, _param2 time.Time) ([]*trillian.LogLeaf, error)

func (*MockLogTreeTX) EXPECT

func (_m *MockLogTreeTX) EXPECT() *_MockLogTreeTXRecorder

func (*MockLogTreeTX) GetActiveLogIDs

func (_m *MockLogTreeTX) GetActiveLogIDs(_param0 context.Context) ([]int64, error)

func (*MockLogTreeTX) GetActiveLogIDsWithPendingWork

func (_m *MockLogTreeTX) GetActiveLogIDsWithPendingWork(_param0 context.Context) ([]int64, error)

func (*MockLogTreeTX) GetLeavesByHash

func (_m *MockLogTreeTX) GetLeavesByHash(_param0 context.Context, _param1 [][]byte, _param2 bool) ([]*trillian.LogLeaf, error)

func (*MockLogTreeTX) GetLeavesByIndex

func (_m *MockLogTreeTX) GetLeavesByIndex(_param0 context.Context, _param1 []int64) ([]*trillian.LogLeaf, error)

func (*MockLogTreeTX) GetMerkleNodes

func (_m *MockLogTreeTX) GetMerkleNodes(_param0 context.Context, _param1 int64, _param2 []NodeID) ([]Node, error)

func (*MockLogTreeTX) GetSequencedLeafCount

func (_m *MockLogTreeTX) GetSequencedLeafCount(_param0 context.Context) (int64, error)

func (*MockLogTreeTX) IsOpen

func (_m *MockLogTreeTX) IsOpen() bool

func (*MockLogTreeTX) LatestSignedLogRoot

func (_m *MockLogTreeTX) LatestSignedLogRoot(_param0 context.Context) (trillian.SignedLogRoot, error)

func (*MockLogTreeTX) QueueLeaves

func (_m *MockLogTreeTX) QueueLeaves(_param0 context.Context, _param1 []*trillian.LogLeaf, _param2 time.Time) ([]*trillian.LogLeaf, error)

func (*MockLogTreeTX) ReadRevision

func (_m *MockLogTreeTX) ReadRevision() int64

func (*MockLogTreeTX) Rollback

func (_m *MockLogTreeTX) Rollback() error

func (*MockLogTreeTX) SetMerkleNodes

func (_m *MockLogTreeTX) SetMerkleNodes(_param0 context.Context, _param1 []Node) error

func (*MockLogTreeTX) StoreSignedLogRoot

func (_m *MockLogTreeTX) StoreSignedLogRoot(_param0 context.Context, _param1 trillian.SignedLogRoot) error

func (*MockLogTreeTX) UpdateSequencedLeaves

func (_m *MockLogTreeTX) UpdateSequencedLeaves(_param0 context.Context, _param1 []*trillian.LogLeaf) error

func (*MockLogTreeTX) WriteRevision

func (_m *MockLogTreeTX) WriteRevision() int64

type MockMapStorage

type MockMapStorage struct {
	// contains filtered or unexported fields
}

Mock of MapStorage interface

func NewMockMapStorage

func NewMockMapStorage(ctrl *gomock.Controller) *MockMapStorage

func (*MockMapStorage) BeginForTree

func (_m *MockMapStorage) BeginForTree(_param0 context.Context, _param1 int64) (MapTreeTX, error)

func (*MockMapStorage) CheckDatabaseAccessible

func (_m *MockMapStorage) CheckDatabaseAccessible(_param0 context.Context) error

func (*MockMapStorage) EXPECT

func (_m *MockMapStorage) EXPECT() *_MockMapStorageRecorder

func (*MockMapStorage) Snapshot

func (_m *MockMapStorage) Snapshot(_param0 context.Context) (ReadOnlyMapTX, error)

func (*MockMapStorage) SnapshotForTree

func (_m *MockMapStorage) SnapshotForTree(_param0 context.Context, _param1 int64) (ReadOnlyMapTreeTX, error)

type MockMapTreeTX

type MockMapTreeTX struct {
	// contains filtered or unexported fields
}

Mock of MapTreeTX interface

func NewMockMapTreeTX

func NewMockMapTreeTX(ctrl *gomock.Controller) *MockMapTreeTX

func (*MockMapTreeTX) Close

func (_m *MockMapTreeTX) Close() error

func (*MockMapTreeTX) Commit

func (_m *MockMapTreeTX) Commit() error

func (*MockMapTreeTX) EXPECT

func (_m *MockMapTreeTX) EXPECT() *_MockMapTreeTXRecorder

func (*MockMapTreeTX) Get

func (_m *MockMapTreeTX) Get(_param0 context.Context, _param1 int64, _param2 [][]byte) ([]trillian.MapLeaf, error)

func (*MockMapTreeTX) GetMerkleNodes

func (_m *MockMapTreeTX) GetMerkleNodes(_param0 context.Context, _param1 int64, _param2 []NodeID) ([]Node, error)

func (*MockMapTreeTX) GetSignedMapRoot

func (_m *MockMapTreeTX) GetSignedMapRoot(_param0 context.Context, _param1 int64) (trillian.SignedMapRoot, error)

func (*MockMapTreeTX) IsOpen

func (_m *MockMapTreeTX) IsOpen() bool

func (*MockMapTreeTX) LatestSignedMapRoot

func (_m *MockMapTreeTX) LatestSignedMapRoot(_param0 context.Context) (trillian.SignedMapRoot, error)

func (*MockMapTreeTX) ReadRevision

func (_m *MockMapTreeTX) ReadRevision() int64

func (*MockMapTreeTX) Rollback

func (_m *MockMapTreeTX) Rollback() error

func (*MockMapTreeTX) Set

func (_m *MockMapTreeTX) Set(_param0 context.Context, _param1 []byte, _param2 trillian.MapLeaf) error

func (*MockMapTreeTX) SetMerkleNodes

func (_m *MockMapTreeTX) SetMerkleNodes(_param0 context.Context, _param1 []Node) error

func (*MockMapTreeTX) StoreSignedMapRoot

func (_m *MockMapTreeTX) StoreSignedMapRoot(_param0 context.Context, _param1 trillian.SignedMapRoot) error

func (*MockMapTreeTX) WriteRevision

func (_m *MockMapTreeTX) WriteRevision() int64

type MockReadOnlyAdminTX

type MockReadOnlyAdminTX struct {
	// contains filtered or unexported fields
}

Mock of ReadOnlyAdminTX interface

func NewMockReadOnlyAdminTX

func NewMockReadOnlyAdminTX(ctrl *gomock.Controller) *MockReadOnlyAdminTX

func (*MockReadOnlyAdminTX) Close

func (_m *MockReadOnlyAdminTX) Close() error

func (*MockReadOnlyAdminTX) Commit

func (_m *MockReadOnlyAdminTX) Commit() error

func (*MockReadOnlyAdminTX) EXPECT

func (_m *MockReadOnlyAdminTX) EXPECT() *_MockReadOnlyAdminTXRecorder

func (*MockReadOnlyAdminTX) GetTree

func (_m *MockReadOnlyAdminTX) GetTree(_param0 context.Context, _param1 int64) (*trillian.Tree, error)

func (*MockReadOnlyAdminTX) IsClosed

func (_m *MockReadOnlyAdminTX) IsClosed() bool

func (*MockReadOnlyAdminTX) ListTreeIDs

func (_m *MockReadOnlyAdminTX) ListTreeIDs(_param0 context.Context) ([]int64, error)

func (*MockReadOnlyAdminTX) ListTrees

func (_m *MockReadOnlyAdminTX) ListTrees(_param0 context.Context) ([]*trillian.Tree, error)

func (*MockReadOnlyAdminTX) Rollback

func (_m *MockReadOnlyAdminTX) Rollback() error

type MockReadOnlyLogTX

type MockReadOnlyLogTX struct {
	// contains filtered or unexported fields
}

Mock of ReadOnlyLogTX interface

func NewMockReadOnlyLogTX

func NewMockReadOnlyLogTX(ctrl *gomock.Controller) *MockReadOnlyLogTX

func (*MockReadOnlyLogTX) Close

func (_m *MockReadOnlyLogTX) Close() error

func (*MockReadOnlyLogTX) Commit

func (_m *MockReadOnlyLogTX) Commit() error

func (*MockReadOnlyLogTX) EXPECT

func (_m *MockReadOnlyLogTX) EXPECT() *_MockReadOnlyLogTXRecorder

func (*MockReadOnlyLogTX) GetActiveLogIDs

func (_m *MockReadOnlyLogTX) GetActiveLogIDs(_param0 context.Context) ([]int64, error)

func (*MockReadOnlyLogTX) GetActiveLogIDsWithPendingWork

func (_m *MockReadOnlyLogTX) GetActiveLogIDsWithPendingWork(_param0 context.Context) ([]int64, error)

func (*MockReadOnlyLogTX) Rollback

func (_m *MockReadOnlyLogTX) Rollback() error

type MockReadOnlyLogTreeTX

type MockReadOnlyLogTreeTX struct {
	// contains filtered or unexported fields
}

Mock of ReadOnlyLogTreeTX interface

func NewMockReadOnlyLogTreeTX

func NewMockReadOnlyLogTreeTX(ctrl *gomock.Controller) *MockReadOnlyLogTreeTX

func (*MockReadOnlyLogTreeTX) Close

func (_m *MockReadOnlyLogTreeTX) Close() error

func (*MockReadOnlyLogTreeTX) Commit

func (_m *MockReadOnlyLogTreeTX) Commit() error

func (*MockReadOnlyLogTreeTX) EXPECT

func (_m *MockReadOnlyLogTreeTX) EXPECT() *_MockReadOnlyLogTreeTXRecorder

func (*MockReadOnlyLogTreeTX) GetLeavesByHash

func (_m *MockReadOnlyLogTreeTX) GetLeavesByHash(_param0 context.Context, _param1 [][]byte, _param2 bool) ([]*trillian.LogLeaf, error)

func (*MockReadOnlyLogTreeTX) GetLeavesByIndex

func (_m *MockReadOnlyLogTreeTX) GetLeavesByIndex(_param0 context.Context, _param1 []int64) ([]*trillian.LogLeaf, error)

func (*MockReadOnlyLogTreeTX) GetMerkleNodes

func (_m *MockReadOnlyLogTreeTX) GetMerkleNodes(_param0 context.Context, _param1 int64, _param2 []NodeID) ([]Node, error)

func (*MockReadOnlyLogTreeTX) GetSequencedLeafCount

func (_m *MockReadOnlyLogTreeTX) GetSequencedLeafCount(_param0 context.Context) (int64, error)

func (*MockReadOnlyLogTreeTX) IsOpen

func (_m *MockReadOnlyLogTreeTX) IsOpen() bool

func (*MockReadOnlyLogTreeTX) LatestSignedLogRoot

func (_m *MockReadOnlyLogTreeTX) LatestSignedLogRoot(_param0 context.Context) (trillian.SignedLogRoot, error)

func (*MockReadOnlyLogTreeTX) ReadRevision

func (_m *MockReadOnlyLogTreeTX) ReadRevision() int64

func (*MockReadOnlyLogTreeTX) Rollback

func (_m *MockReadOnlyLogTreeTX) Rollback() error

type MockReadOnlyMapTreeTX

type MockReadOnlyMapTreeTX struct {
	// contains filtered or unexported fields
}

Mock of ReadOnlyMapTreeTX interface

func NewMockReadOnlyMapTreeTX

func NewMockReadOnlyMapTreeTX(ctrl *gomock.Controller) *MockReadOnlyMapTreeTX

func (*MockReadOnlyMapTreeTX) Close

func (_m *MockReadOnlyMapTreeTX) Close() error

func (*MockReadOnlyMapTreeTX) Commit

func (_m *MockReadOnlyMapTreeTX) Commit() error

func (*MockReadOnlyMapTreeTX) EXPECT

func (_m *MockReadOnlyMapTreeTX) EXPECT() *_MockReadOnlyMapTreeTXRecorder

func (*MockReadOnlyMapTreeTX) Get

func (_m *MockReadOnlyMapTreeTX) Get(_param0 context.Context, _param1 int64, _param2 [][]byte) ([]trillian.MapLeaf, error)

func (*MockReadOnlyMapTreeTX) GetMerkleNodes

func (_m *MockReadOnlyMapTreeTX) GetMerkleNodes(_param0 context.Context, _param1 int64, _param2 []NodeID) ([]Node, error)

func (*MockReadOnlyMapTreeTX) GetSignedMapRoot

func (_m *MockReadOnlyMapTreeTX) GetSignedMapRoot(_param0 context.Context, _param1 int64) (trillian.SignedMapRoot, error)

func (*MockReadOnlyMapTreeTX) IsOpen

func (_m *MockReadOnlyMapTreeTX) IsOpen() bool

func (*MockReadOnlyMapTreeTX) LatestSignedMapRoot

func (_m *MockReadOnlyMapTreeTX) LatestSignedMapRoot(_param0 context.Context) (trillian.SignedMapRoot, error)

func (*MockReadOnlyMapTreeTX) ReadRevision

func (_m *MockReadOnlyMapTreeTX) ReadRevision() int64

func (*MockReadOnlyMapTreeTX) Rollback

func (_m *MockReadOnlyMapTreeTX) Rollback() error

type Node

type Node struct {
	NodeID       NodeID
	Hash         []byte
	NodeRevision int64
}

Node represents a single node in a Merkle tree.

type NodeID

type NodeID struct {
	// path is effectively a BigEndian bit set, with path[0] being the MSB
	// (identifying the root child), and successive bits identifying the lower
	// level children down to the leaf.
	Path []byte
	// PrefixLenBits is the number of MSB in Path which are considered part of
	// this NodeID.
	//
	// e.g. if Path contains two bytes, and PrefixLenBits is 9, then the 8 bits
	// in Path[0] are included, along with the lowest bit of Path[1]
	PrefixLenBits int
	PathLenBits   int
}

NodeID uniquely identifies a Node within a versioned MerkleTree.

func NewEmptyNodeID

func NewEmptyNodeID(maxLenBits int) NodeID

NewEmptyNodeID creates a new zero-length NodeID with sufficient underlying capacity to store a maximum of maxLenBits.

func NewNodeIDForTreeCoords

func NewNodeIDForTreeCoords(depth int64, index int64, maxPathBits int) (NodeID, error)

NewNodeIDForTreeCoords creates a new NodeID for a Tree node with a specified depth and index. This method is used exclusively by the Log, and, since the Log model grows upwards from the leaves, we modify the provided coords accordingly.

depth is the Merkle tree level: 0 = leaves, and increases upwards towards the root.

index is the horizontal index into the tree at level depth, so the returned NodeID will be zero padded on the right by depth places.

func NewNodeIDFromHash

func NewNodeIDFromHash(h []byte) NodeID

NewNodeIDFromHash creates a new NodeID for the given Hash.

func NewNodeIDWithPrefix

func NewNodeIDWithPrefix(prefix uint64, prefixLenBits, nodeIDLenBits, maxLenBits int) NodeID

NewNodeIDWithPrefix creates a new NodeID of nodeIDLen bits with the prefixLen MSBs set to prefix.

func (*NodeID) Bit

func (n *NodeID) Bit(i int) uint

Bit returns 1 if the ith bit is true, and false otherwise.

func (*NodeID) CoordString

func (n *NodeID) CoordString() string

CoordString returns a string representation assuming that the NodeID represents a tree coordinate. Using this on a NodeID for a sparse Merkle tree will give incorrect results. Intended for debugging purposes, the format could change.

func (*NodeID) Equivalent

func (n *NodeID) Equivalent(other NodeID) bool

Equivalent return true iff the other represents the same path prefix as this NodeID.

func (*NodeID) SetBit

func (n *NodeID) SetBit(i int, b uint)

SetBit sets the ith bit to true if b is non-zero, and false otherwise.

func (*NodeID) Siblings

func (n *NodeID) Siblings() []NodeID

Siblings returns the siblings of the given node.

func (*NodeID) String

func (n *NodeID) String() string

String returns a string representation of the binary value of the NodeID. The left-most bit is the MSB (i.e. nearer the root of the tree).

type NodeReader

type NodeReader interface {
	// GetMerkleNodes looks up the set of nodes identified by ids, at treeRevision, and returns them.
	GetMerkleNodes(ctx context.Context, treeRevision int64, ids []NodeID) ([]Node, error)
}

NodeReader provides a read-only interface into the stored tree nodes.

type NodeWriter

type NodeWriter interface {
	// SetMerkleNodes stores the provided nodes, at the transaction's writeRevision.
	SetMerkleNodes(ctx context.Context, nodes []Node) error
}

NodeWriter provides a write interface into the stored tree nodes.

type PopulateSubtreeFunc

type PopulateSubtreeFunc func(*storagepb.SubtreeProto) error

PopulateSubtreeFunc is a function which knows how to re-populate a subtree from just its leaf nodes.

type PrepareSubtreeWriteFunc

type PrepareSubtreeWriteFunc func(*storagepb.SubtreeProto) error

PrepareSubtreeWriteFunc is a function that carries out any required tree type specific manipulation of a subtree before it's written to storage

type ReadOnlyAdminTX

type ReadOnlyAdminTX interface {
	AdminReader

	// Commit applies the operations performed to the underlying storage, or
	// returns an error.
	// A commit must be performed before any reads from storage are
	// considered consistent.
	Commit() error

	// Rollback aborts any performed operations, or returns an error.
	// See Close() for a way to automatically manage transactions.
	Rollback() error

	// IsClosed returns true if the transaction is closed.
	// A transaction is closed when either Commit() or Rollback() are
	// called.
	IsClosed() bool

	// Close rolls back the transaction if it's not yet closed.
	// It's advisable to call "defer tx.Close()" after the creation of
	// transaction to ensure that it's always rolled back if not explicitly
	// committed.
	Close() error
}

ReadOnlyAdminTX is a transaction capable only of read operations in the AdminStorage.

type ReadOnlyLogStorage

type ReadOnlyLogStorage interface {
	DatabaseChecker

	// Snapshot starts a read-only transaction not tied to any particular tree.
	Snapshot(ctx context.Context) (ReadOnlyLogTX, error)

	// SnapshotForTree starts a read-only transaction for the specified treeID.
	// Commit must be called when the caller is finished with the returned object,
	// and values read through it should only be propagated if Commit returns
	// without error.
	SnapshotForTree(ctx context.Context, treeID int64) (ReadOnlyLogTreeTX, error)
}

ReadOnlyLogStorage represents a narrowed read-only view into a LogStorage.

type ReadOnlyLogTX

type ReadOnlyLogTX interface {
	LogMetadata

	// Commit ensures the data read by the TX is consistent in the database. Only after Commit the
	// data read should be regarded as valid.
	Commit() error

	// Rollback discards the read-only TX.
	Rollback() error

	// Close attempts to Rollback the TX if it's open, it's a noop otherwise.
	Close() error
}

ReadOnlyLogTX provides a read-only view into log data. A ReadOnlyLogTX, unlike ReadOnlyLogTreeTX, is not tied to a particular tree.

type ReadOnlyLogTreeTX

type ReadOnlyLogTreeTX interface {
	ReadOnlyTreeTX
	LeafReader
	LogRootReader
}

ReadOnlyLogTreeTX provides a read-only view into the Log data. A ReadOnlyLogTreeTX can only read from the tree specified in its creation.

type ReadOnlyMapStorage

type ReadOnlyMapStorage interface {
	DatabaseChecker

	// Snapshot starts a read-only transaction not tied to any particular tree.
	Snapshot(ctx context.Context) (ReadOnlyMapTX, error)

	// SnapshotForTree starts a new read-only transaction.
	// Commit must be called when the caller is finished with the returned object,
	// and values read through it should only be propagated if Commit returns
	// without error.
	SnapshotForTree(ctx context.Context, treeID int64) (ReadOnlyMapTreeTX, error)
}

ReadOnlyMapStorage provides a narrow read-only view into a MapStorage.

type ReadOnlyMapTX

type ReadOnlyMapTX interface {
	// Commit ensures the data read by the TX is consistent in the database. Only after Commit the
	// data read should be regarded as valid.
	Commit() error

	// Rollback discards the read-only TX.
	Rollback() error

	// Close attempts to Rollback the TX if it's open, it's a noop otherwise.
	Close() error
}

ReadOnlyMapTX provides a read-only view into log data. A ReadOnlyMapTX, unlike ReadOnlyMapTreeTX, is not tied to a particular tree.

type ReadOnlyMapTreeTX

type ReadOnlyMapTreeTX interface {
	ReadOnlyTreeTX
	MapRootReader
	Getter
}

ReadOnlyMapTreeTX provides a read-only view into the Map data. A ReadOnlyMapTreeTX can only read from the tree specified in its creation.

type ReadOnlyTreeTX

type ReadOnlyTreeTX interface {
	NodeReader

	// ReadRevision returns the tree revision that was current at the time this
	// transaction was started.
	ReadRevision() int64

	// Commit attempts to commit any reads performed under this transaction.
	Commit() error

	// Rollback aborts this transaction.
	Rollback() error

	// Close attempts to Rollback the TX if it's open, it's a noop otherwise.
	Close() error

	// Open indicates if this transaction is open. An open transaction is one for which
	// Commit() or Rollback() has never been called. Implementations must do all clean up
	// in these methods so transactions are assumed closed regardless of the reported success.
	IsOpen() bool
}

ReadOnlyTreeTX represents a read-only transaction on a TreeStorage. A ReadOnlyTreeTX can only modify the tree specified in its creation.

type Setter

type Setter interface {
	// Set sets key to leaf
	Set(ctx context.Context, keyHash []byte, value trillian.MapLeaf) error
}

Setter allows the setting of key->value pairs on the map.

type TreeTX

type TreeTX interface {
	ReadOnlyTreeTX
	NodeWriter

	// WriteRevision returns the tree revision that any writes through this TreeTX will be stored at.
	WriteRevision() int64
}

TreeTX represents an in-process tree-modifying transaction. The transaction must end with a call to Commit or Rollback. After a call to Commit or Rollback, all operations on the transaction will fail. After a call to Commit or Rollback implementations must be in a clean state and have released any resources owned by the TreeTX. A TreeTX can only modify the tree specified in its creation.

Directories

Path Synopsis
Package cache provides subtree caching functionality.
Package cache provides subtree caching functionality.
Package memory provides a simple in-process implementation of the tree- and log-storage interfaces.
Package memory provides a simple in-process implementation of the tree- and log-storage interfaces.
Package mysql provides a MySQL-based storage layer implementation.
Package mysql provides a MySQL-based storage layer implementation.
Package storagepb is a generated protocol buffer package.
Package storagepb is a generated protocol buffer package.
Package testonly holds test-specific code for Trillian storage layers.
Package testonly holds test-specific code for Trillian storage layers.
tools
fetch_leaves
The fetch_leaves program retrieves leaves from a tree.
The fetch_leaves program retrieves leaves from a tree.
log_client
The log_client binary retrieves leaves from a log.
The log_client binary retrieves leaves from a log.
queue_leaves
The queue_leaves binary queues a number of leaves for a log from a given start point with predictable hashes.
The queue_leaves binary queues a number of leaves for a log from a given start point with predictable hashes.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL