go-datastore: github.com/ipfs/go-datastore Index | Files | Directories

package datastore

import "github.com/ipfs/go-datastore"

Index

Package Files

basic_ds.go batch.go datastore.go key.go

Variables

var ErrBatchUnsupported = errors.New("this datastore does not support batching")

ErrBatchUnsupported is returned if the by Batch if the Datastore doesn't actually support batching.

var ErrNotFound error = &dsError{error: errors.New("datastore: key not found"), isNotFound: true}

ErrNotFound is returned by Get and GetSize when a datastore does not map the given key to a value.

func DiskUsage Uses

func DiskUsage(d Datastore) (uint64, error)

DiskUsage checks if a Datastore is a PersistentDatastore and returns its DiskUsage(), otherwise returns 0.

func GetBackedHas Uses

func GetBackedHas(ds Read, key Key) (bool, error)

GetBackedHas provides a default Datastore.Has implementation. It exists so Datastore.Has implementations can use it, like so:

func (*d SomeDatastore) Has(key Key) (exists bool, err error) {

return GetBackedHas(d, key)

}

func GetBackedSize Uses

func GetBackedSize(ds Read, key Key) (int, error)

GetBackedSize provides a default Datastore.GetSize implementation. It exists so Datastore.GetSize implementations can use it, like so:

func (*d SomeDatastore) GetSize(key Key) (size int, err error) {

return GetBackedSize(d, key)

}

func NamespaceType Uses

func NamespaceType(namespace string) string

NamespaceType is the first component of a namespace. `foo` in `foo:bar`

func NamespaceValue Uses

func NamespaceValue(namespace string) string

NamespaceValue returns the last component of a namespace. `baz` in `f:b:baz`

type Batch Uses

type Batch interface {
    Write

    Commit() error
}

func NewBasicBatch Uses

func NewBasicBatch(ds Datastore) Batch

type Batching Uses

type Batching interface {
    Datastore

    Batch() (Batch, error)
}

Batching datastores support deferred, grouped updates to the database. `Batch`es do NOT have transactional semantics: updates to the underlying datastore are not guaranteed to occur in the same iota of time. Similarly, batched updates will not be flushed to the underlying datastore until `Commit` has been called. `Txn`s from a `TxnDatastore` have all the capabilities of a `Batch`, but the reverse is NOT true.

type CheckedDatastore Uses

type CheckedDatastore interface {
    Datastore

    Check() error
}

CheckedDatastore is an interface that should be implemented by datastores which may need checking on-disk data integrity.

type Datastore Uses

type Datastore interface {
    Read
    Write
    io.Closer
}

Datastore represents storage for any key-value pair.

Datastores are general enough to be backed by all kinds of different storage: in-memory caches, databases, a remote datastore, flat files on disk, etc.

The general idea is to wrap a more complicated storage facility in a simple, uniform interface, keeping the freedom of using the right tools for the job. In particular, a Datastore can aggregate other datastores in interesting ways, like sharded (to distribute load) or tiered access (caches before databases).

While Datastores should be written general enough to accept all sorts of values, some implementations will undoubtedly have to be specific (e.g. SQL databases where fields should be decomposed into columns), particularly to support queries efficiently. Moreover, certain datastores may enforce certain types of values (e.g. requiring an io.Reader, a specific struct, etc) or serialization formats (JSON, Protobufs, etc).

IMPORTANT: No Datastore should ever Panic! This is a cross-module interface, and thus it should behave predictably and handle exceptional conditions with proper error reporting. Thus, all Datastore calls may return errors, which should be checked by callers.

type GCDatastore Uses

type GCDatastore interface {
    Datastore

    CollectGarbage() error
}

GCDatastore is an interface that should be implemented by datastores which don't free disk space by just removing data from them.

type Key Uses

type Key struct {
    // contains filtered or unexported fields
}

A Key represents the unique identifier of an object. Our Key scheme is inspired by file systems and Google App Engine key model.

Keys are meant to be unique across a system. Keys are hierarchical, incorporating more and more specific namespaces. Thus keys can be deemed 'children' or 'ancestors' of other keys::

Key("/Comedy")
Key("/Comedy/MontyPython")

Also, every namespace can be parametrized to embed relevant object information. For example, the Key `name` (most specific namespace) could include the object type::

Key("/Comedy/MontyPython/Actor:JohnCleese")
Key("/Comedy/MontyPython/Sketch:CheeseShop")
Key("/Comedy/MontyPython/Sketch:CheeseShop/Character:Mousebender")

func EntryKeys Uses

func EntryKeys(e []dsq.Entry) []Key

EntryKeys

func KeyWithNamespaces Uses

func KeyWithNamespaces(ns []string) Key

KeyWithNamespaces constructs a key out of a namespace slice.

func NewKey Uses

func NewKey(s string) Key

NewKey constructs a key from string. it will clean the value.

func RandomKey Uses

func RandomKey() Key

RandomKey returns a randomly (uuid) generated key.

RandomKey()
NewKey("/f98719ea086343f7b71f32ea9d9d521d")

func RawKey Uses

func RawKey(s string) Key

RawKey creates a new Key without safety checking the input. Use with care.

func (Key) BaseNamespace Uses

func (k Key) BaseNamespace() string

BaseNamespace returns the "base" namespace of this key (path.Base(filename))

NewKey("/Comedy/MontyPython/Actor:JohnCleese").BaseNamespace()
"Actor:JohnCleese"

func (Key) Bytes Uses

func (k Key) Bytes() []byte

Bytes returns the string value of Key as a []byte

func (Key) Child Uses

func (k Key) Child(k2 Key) Key

Child returns the `child` Key of this Key.

NewKey("/Comedy/MontyPython").Child(NewKey("Actor:JohnCleese"))
NewKey("/Comedy/MontyPython/Actor:JohnCleese")

func (Key) ChildString Uses

func (k Key) ChildString(s string) Key

ChildString returns the `child` Key of this Key -- string helper.

NewKey("/Comedy/MontyPython").ChildString("Actor:JohnCleese")
NewKey("/Comedy/MontyPython/Actor:JohnCleese")

func (*Key) Clean Uses

func (k *Key) Clean()

Clean up a Key, using path.Clean.

func (Key) Equal Uses

func (k Key) Equal(k2 Key) bool

Equal checks equality of two keys

func (Key) Instance Uses

func (k Key) Instance(s string) Key

Instance returns an "instance" of this type key (appends value to namespace).

NewKey("/Comedy/MontyPython/Actor").Instance("JohnClesse")
NewKey("/Comedy/MontyPython/Actor:JohnCleese")

func (Key) IsAncestorOf Uses

func (k Key) IsAncestorOf(other Key) bool

IsAncestorOf returns whether this key is a prefix of `other`

NewKey("/Comedy").IsAncestorOf("/Comedy/MontyPython")
true

func (Key) IsDescendantOf Uses

func (k Key) IsDescendantOf(other Key) bool

IsDescendantOf returns whether this key contains another as a prefix.

NewKey("/Comedy/MontyPython").IsDescendantOf("/Comedy")
true

func (Key) IsTopLevel Uses

func (k Key) IsTopLevel() bool

IsTopLevel returns whether this key has only one namespace.

func (Key) Less Uses

func (k Key) Less(k2 Key) bool

Less checks whether this key is sorted lower than another.

func (Key) List Uses

func (k Key) List() []string

List returns the `list` representation of this Key.

NewKey("/Comedy/MontyPython/Actor:JohnCleese").List()
["Comedy", "MontyPythong", "Actor:JohnCleese"]

func (Key) MarshalJSON Uses

func (k Key) MarshalJSON() ([]byte, error)

MarshalJSON implements the json.Marshaler interface, keys are represented as JSON strings

func (Key) Name Uses

func (k Key) Name() string

Name returns the "name" of this key (field of last namespace).

NewKey("/Comedy/MontyPython/Actor:JohnCleese").Name()
"JohnCleese"

func (Key) Namespaces Uses

func (k Key) Namespaces() []string

Namespaces returns the `namespaces` making up this Key.

NewKey("/Comedy/MontyPython/Actor:JohnCleese").Namespaces()
["Comedy", "MontyPython", "Actor:JohnCleese"]

func (Key) Parent Uses

func (k Key) Parent() Key

Parent returns the `parent` Key of this Key.

NewKey("/Comedy/MontyPython/Actor:JohnCleese").Parent()
NewKey("/Comedy/MontyPython")

func (Key) Path Uses

func (k Key) Path() Key

Path returns the "path" of this key (parent + type).

NewKey("/Comedy/MontyPython/Actor:JohnCleese").Path()
NewKey("/Comedy/MontyPython/Actor")

func (Key) Reverse Uses

func (k Key) Reverse() Key

Reverse returns the reverse of this Key.

NewKey("/Comedy/MontyPython/Actor:JohnCleese").Reverse()
NewKey("/Actor:JohnCleese/MontyPython/Comedy")

func (Key) String Uses

func (k Key) String() string

Strings is the string value of Key

func (Key) Type Uses

func (k Key) Type() string

Type returns the "type" of this key (value of last namespace).

NewKey("/Comedy/MontyPython/Actor:JohnCleese").Type()
"Actor"

func (*Key) UnmarshalJSON Uses

func (k *Key) UnmarshalJSON(data []byte) error

UnmarshalJSON implements the json.Unmarshaler interface, keys will parse any value specified as a key to a string

type KeySlice Uses

type KeySlice []Key

KeySlice attaches the methods of sort.Interface to []Key, sorting in increasing order.

func (KeySlice) Len Uses

func (p KeySlice) Len() int

func (KeySlice) Less Uses

func (p KeySlice) Less(i, j int) bool

func (KeySlice) Swap Uses

func (p KeySlice) Swap(i, j int)

type LogBatch Uses

type LogBatch struct {
    Name string
    // contains filtered or unexported fields
}

LogBatch logs all accesses through the batch.

func (*LogBatch) Commit Uses

func (d *LogBatch) Commit() (err error)

Commit implements Batch.Commit

func (*LogBatch) Delete Uses

func (d *LogBatch) Delete(key Key) (err error)

Delete implements Batch.Delete

func (*LogBatch) Put Uses

func (d *LogBatch) Put(key Key, value []byte) (err error)

Put implements Batch.Put

type LogDatastore Uses

type LogDatastore struct {
    Name string
    // contains filtered or unexported fields
}

LogDatastore logs all accesses through the datastore.

func NewLogDatastore Uses

func NewLogDatastore(ds Datastore, name string) *LogDatastore

NewLogDatastore constructs a log datastore.

func (*LogDatastore) Batch Uses

func (d *LogDatastore) Batch() (Batch, error)

func (*LogDatastore) Check Uses

func (d *LogDatastore) Check() error

func (*LogDatastore) Children Uses

func (d *LogDatastore) Children() []Datastore

Children implements Shim

func (*LogDatastore) Close Uses

func (d *LogDatastore) Close() error

func (*LogDatastore) CollectGarbage Uses

func (d *LogDatastore) CollectGarbage() error

func (*LogDatastore) Delete Uses

func (d *LogDatastore) Delete(key Key) (err error)

Delete implements Datastore.Delete

func (*LogDatastore) DiskUsage Uses

func (d *LogDatastore) DiskUsage() (uint64, error)

DiskUsage implements the PersistentDatastore interface.

func (*LogDatastore) Get Uses

func (d *LogDatastore) Get(key Key) (value []byte, err error)

Get implements Datastore.Get

func (*LogDatastore) GetSize Uses

func (d *LogDatastore) GetSize(key Key) (size int, err error)

GetSize implements Datastore.GetSize

func (*LogDatastore) Has Uses

func (d *LogDatastore) Has(key Key) (exists bool, err error)

Has implements Datastore.Has

func (*LogDatastore) Put Uses

func (d *LogDatastore) Put(key Key, value []byte) (err error)

Put implements Datastore.Put

func (*LogDatastore) Query Uses

func (d *LogDatastore) Query(q dsq.Query) (dsq.Results, error)

Query implements Datastore.Query

func (*LogDatastore) Scrub Uses

func (d *LogDatastore) Scrub() error

type MapDatastore Uses

type MapDatastore struct {
    // contains filtered or unexported fields
}

MapDatastore uses a standard Go map for internal storage.

func NewMapDatastore Uses

func NewMapDatastore() (d *MapDatastore)

NewMapDatastore constructs a MapDatastore. It is _not_ thread-safe by default, wrap using sync.MutexWrap if you need thread safety (the answer here is usually yes).

func (*MapDatastore) Batch Uses

func (d *MapDatastore) Batch() (Batch, error)

func (*MapDatastore) Close Uses

func (d *MapDatastore) Close() error

func (*MapDatastore) Delete Uses

func (d *MapDatastore) Delete(key Key) (err error)

Delete implements Datastore.Delete

func (*MapDatastore) Get Uses

func (d *MapDatastore) Get(key Key) (value []byte, err error)

Get implements Datastore.Get

func (*MapDatastore) GetSize Uses

func (d *MapDatastore) GetSize(key Key) (size int, err error)

GetSize implements Datastore.GetSize

func (*MapDatastore) Has Uses

func (d *MapDatastore) Has(key Key) (exists bool, err error)

Has implements Datastore.Has

func (*MapDatastore) Put Uses

func (d *MapDatastore) Put(key Key, value []byte) (err error)

Put implements Datastore.Put

func (*MapDatastore) Query Uses

func (d *MapDatastore) Query(q dsq.Query) (dsq.Results, error)

Query implements Datastore.Query

type NullDatastore Uses

type NullDatastore struct {
}

NullDatastore stores nothing, but conforms to the API. Useful to test with.

func NewNullDatastore Uses

func NewNullDatastore() *NullDatastore

NewNullDatastore constructs a null datastoe

func (*NullDatastore) Batch Uses

func (d *NullDatastore) Batch() (Batch, error)

func (*NullDatastore) Close Uses

func (d *NullDatastore) Close() error

func (*NullDatastore) Delete Uses

func (d *NullDatastore) Delete(key Key) (err error)

Delete implements Datastore.Delete

func (*NullDatastore) Get Uses

func (d *NullDatastore) Get(key Key) (value []byte, err error)

Get implements Datastore.Get

func (*NullDatastore) GetSize Uses

func (d *NullDatastore) GetSize(key Key) (size int, err error)

Has implements Datastore.GetSize

func (*NullDatastore) Has Uses

func (d *NullDatastore) Has(key Key) (exists bool, err error)

Has implements Datastore.Has

func (*NullDatastore) Put Uses

func (d *NullDatastore) Put(key Key, value []byte) (err error)

Put implements Datastore.Put

func (*NullDatastore) Query Uses

func (d *NullDatastore) Query(q dsq.Query) (dsq.Results, error)

Query implements Datastore.Query

type PersistentDatastore Uses

type PersistentDatastore interface {
    Datastore

    // DiskUsage returns the space used by a datastore, in bytes.
    DiskUsage() (uint64, error)
}

PersistentDatastore is an interface that should be implemented by datastores which can report disk usage.

type Read Uses

type Read interface {
    // Get retrieves the object `value` named by `key`.
    // Get will return ErrNotFound if the key is not mapped to a value.
    Get(key Key) (value []byte, err error)

    // Has returns whether the `key` is mapped to a `value`.
    // In some contexts, it may be much cheaper only to check for existence of
    // a value, rather than retrieving the value itself. (e.g. HTTP HEAD).
    // The default implementation is found in `GetBackedHas`.
    Has(key Key) (exists bool, err error)

    // GetSize returns the size of the `value` named by `key`.
    // In some contexts, it may be much cheaper to only get the size of the
    // value rather than retrieving the value itself.
    GetSize(key Key) (size int, err error)

    // Query searches the datastore and returns a query result. This function
    // may return before the query actually runs. To wait for the query:
    //
    //   result, _ := ds.Query(q)
    //
    //   // use the channel interface; result may come in at different times
    //   for entry := range result.Next() { ... }
    //
    //   // or wait for the query to be completely done
    //   entries, _ := result.Rest()
    //   for entry := range entries { ... }
    //
    Query(q query.Query) (query.Results, error)
}

Read is the read-side of the Datastore interface.

type ScrubbedDatastore Uses

type ScrubbedDatastore interface {
    Datastore

    Scrub() error
}

ScrubbedDatastore is an interface that should be implemented by datastores which want to provide a mechanism to check data integrity and/or error correction.

type Shim Uses

type Shim interface {
    Datastore

    Children() []Datastore
}

Shim is a datastore which has a child.

type TTL Uses

type TTL interface {
    PutWithTTL(key Key, value []byte, ttl time.Duration) error
    SetTTL(key Key, ttl time.Duration) error
    GetExpiration(key Key) (time.Time, error)
}

TTL encapulates the methods that deal with entries with time-to-live.

type TTLDatastore Uses

type TTLDatastore interface {
    Datastore
    TTL
}

TTLDatastore is an interface that should be implemented by datastores that support expiring entries.

type Txn Uses

type Txn interface {
    Read
    Write

    // Commit finalizes a transaction, attempting to commit it to the Datastore.
    // May return an error if the transaction has gone stale. The presence of an
    // error is an indication that the data was not committed to the Datastore.
    Commit() error
    // Discard throws away changes recorded in a transaction without committing
    // them to the underlying Datastore. Any calls made to Discard after Commit
    // has been successfully called will have no effect on the transaction and
    // state of the Datastore, making it safe to defer.
    Discard()
}

Txn extends the Datastore type. Txns allow users to batch queries and mutations to the Datastore into atomic groups, or transactions. Actions performed on a transaction will not take hold until a successful call to Commit has been made. Likewise, transactions can be aborted by calling Discard before a successful Commit has been made.

type TxnDatastore Uses

type TxnDatastore interface {
    Datastore

    NewTransaction(readOnly bool) (Txn, error)
}

TxnDatastore is an interface that should be implemented by datastores that support transactions.

type Write Uses

type Write interface {
    // Put stores the object `value` named by `key`.
    //
    // The generalized Datastore interface does not impose a value type,
    // allowing various datastore middleware implementations (which do not
    // handle the values directly) to be composed together.
    //
    // Ultimately, the lowest-level datastore will need to do some value checking
    // or risk getting incorrect values. It may also be useful to expose a more
    // type-safe interface to your application, and do the checking up-front.
    Put(key Key, value []byte) error

    // Delete removes the value for given `key`. If the key is not in the
    // datastore, this method returns no error.
    Delete(key Key) error
}

Write is the write-side of the Datastore interface.

Directories

PathSynopsis
autobatchPackage autobatch provides a go-datastore implementation that automatically batches together writes by holding puts in memory until a certain threshold is met.
delayedPackage delayed wraps a datastore allowing to artificially delay all operations.
examplesPackage fs is a simple Datastore implementation that stores keys as directories and files, mirroring the key.
failstorePackage failstore implements a datastore which can produce custom failures on operations by calling a user-provided error function.
keytransformPackage keytransform introduces a Datastore Shim that transforms keys before passing them to its child.
mountPackage mount provides a Datastore that has other Datastores mounted at various key prefixes and is threadsafe
namespacePackage namespace introduces a namespace Datastore Shim, which basically mounts the entire child datastore under a prefix.
query
retrystorePackage retrystore provides a datastore wrapper which allows to retry operations.
sync
test

Package datastore imports 9 packages (graph) and is imported by 256 packages. Updated 2019-11-11. Refresh now. Tools for package owners.