pachyderm: Index | Files

package collection

import ""


Package Files

collection.go errors.go range.go transaction.go types.go


const (
    DefaultPrefix string = "pachyderm/1.7.0"

defaultLimit was experimentally determined to be the highest value that could work (It gets scaled down for specific collections if they trip the max-message size.)


var DefaultOptions = &Options{etcd.SortByCreateRevision, etcd.SortDescend, false}

DefaultOptions are the default sort options when iterating through etcd key/values.

var (
    // ErrNotClaimed is an error used to indicate that a different requester beat
    // the current requester to a key claim.
    ErrNotClaimed = errors.Errorf("NOT_CLAIMED")

func IsErrExists Uses

func IsErrExists(e error) bool

IsErrExists determines if an error is an ErrExists error

func IsErrMalformedValue Uses

func IsErrMalformedValue(e error) bool

IsErrMalformedValue determines if an error is an ErrMalformedValue error

func IsErrNotFound Uses

func IsErrNotFound(e error) bool

IsErrNotFound determines if an error is an ErrNotFound error

func NewDryrunSTM Uses

func NewDryrunSTM(ctx context.Context, c *v3.Client, apply func(STM) error) error

NewDryrunSTM intiates a new STM operation, but the final commit is skipped. It uses a serializable model.

func NewSTM Uses

func NewSTM(ctx context.Context, c *v3.Client, apply func(STM) error) (*v3.TxnResponse, error)

NewSTM intiates a new STM operation. It uses a serializable model.

type Collection Uses

type Collection interface {
    // Path returns the full etcd path of the given key in the collection
    Path(string) string
    // ReadWrite enables reads and writes on a collection in a
    // transactional manner.  Specifically, all writes are applied
    // atomically, and writes are only applied if reads have not been
    // invalidated at the end of the transaction.  Basically, it's
    // software transactional memory.  See this blog post for details:
    ReadWrite(stm STM) ReadWriteCollection
    // ReadWriteInt is the same as ReadWrite except that it operates on
    // integral items, as opposed to protobuf items
    ReadWriteInt(stm STM) ReadWriteIntCollection
    // For read-only operatons, use the ReadOnly for better performance
    ReadOnly(ctx context.Context) ReadonlyCollection
    // Claim attempts to claim a key and run the passed in callback with
    // the context for the claim.
    Claim(ctx context.Context, key string, val proto.Message, f func(context.Context) error) error

Collection implements helper functions that makes common operations on top of etcd more pleasant to work with. It's called collection because most of our data is modelled as collections, such as repos, commits, branches, etc.

func NewCollection Uses

func NewCollection(etcdClient *etcd.Client, prefix string, indexes []*Index, template proto.Message, keyCheck func(string) error, valCheck func(proto.Message) error) Collection

NewCollection creates a new collection.

type ErrExists Uses

type ErrExists struct {
    Type string
    Key  string

ErrExists indicates that a key was found to exist when it was expected not to.

func (ErrExists) Error Uses

func (e ErrExists) Error() string

type ErrMalformedValue Uses

type ErrMalformedValue struct {
    Type string
    Key  string
    Val  string

ErrMalformedValue indicates that a value was malformed, such as when it was supposed to be parseable as an int but wasn't.

func (ErrMalformedValue) Error Uses

func (e ErrMalformedValue) Error() string

type ErrNotFound Uses

type ErrNotFound struct {
    Type string
    Key  string

ErrNotFound indicates that a key was not found when it was expected to exist.

func (ErrNotFound) Error Uses

func (e ErrNotFound) Error() string

type Index Uses

type Index struct {
    Field string
    Multi bool
    // contains filtered or unexported fields

Index specifies a secondary index on a collection.

Indexes are created in a transactional manner thanks to etcd's transactional support.

A secondary index for collection "foo" on field "bar" will reside under the path `/foo__index_bar`. Each item under the path is in turn a directory whose name is the value of the field `bar`. For instance, if you have a object in collection `foo` whose `bar` field is `test`, then you will see a directory at path `/foo__index_bar/test`.

Under that directory, you have keys that point to items in the collection. For instance, if the aforementioned object has the key "buzz", then you will see an item at `/foo__index_bar/test/buzz`. The value of this item is empty. Thus, to get all items in collection `foo` whose values of field `bar` is `test`, we issue a query for all items under `foo__index_bar/test`.

Multi specifies whether this is a multi-index. A multi-index is an index on a field that's a slice. The item is then indexed on each element of the slice.

type Options Uses

type Options struct {
    Target   etcd.SortTarget
    Order    etcd.SortOrder
    SelfSort bool

Options are the sort options when iterating through etcd key/values. Currently implemented sort targets are CreateRevision and ModRevision. The sorting can be done in the calling process by setting SelfSort to true.

type ReadWriteCollection Uses

type ReadWriteCollection interface {
    Get(key string, val proto.Message) error
    Put(key string, val proto.Message) error
    // TTL returns the amount of time that 'key' will continue to exist in the
    // collection, or '0' if 'key' will remain in the collection indefinitely
    TTL(key string) (int64, error)
    // PutTTL is the same as Put except that the object is removed after
    // TTL seconds.
    // WARNING: using PutTTL with a collection that has secondary indices
    // can result in inconsistency, as the indices are removed at roughly
    // but not exactly the same time as the documents.
    PutTTL(key string, val proto.Message, ttl int64) error
    // Update reads the current value associated with 'key', calls 'f' to update
    // the value, and writes the new value back to the collection. 'key' must be
    // present in the collection, or a 'Not Found' error is returned
    Update(key string, val proto.Message, f func() error) error
    // Upsert is like Update but 'key' is not required to be present
    Upsert(key string, val proto.Message, f func() error) error
    Create(key string, val proto.Message) error
    Delete(key string) error
    DeleteAllPrefix(prefix string)

ReadWriteCollection is a collection interface that supports read,write and delete operations.

type ReadWriteIntCollection Uses

type ReadWriteIntCollection interface {
    Create(key string, val int) error
    Get(key string) (int, error)
    Increment(key string) error
    IncrementBy(key string, n int) error
    Decrement(key string) error
    DecrementBy(key string, n int) error
    Delete(key string) error

ReadWriteIntCollection is a ReadonlyCollection interface specifically for ints.

type ReadonlyCollection Uses

type ReadonlyCollection interface {
    Get(key string, val proto.Message) error
    GetByIndex(index *Index, indexVal interface{}, val proto.Message, opts *Options, f func(key string) error) error
    // GetBlock is like Get but waits for the key to exist if it doesn't already.
    GetBlock(key string, val proto.Message) error
    // TTL returns the number of seconds that 'key' will continue to exist in the
    // collection, or '0' if 'key' will remain in the collection indefinitely
    TTL(key string) (int64, error)
    List(val proto.Message, opts *Options, f func(key string) error) error
    ListRev(val proto.Message, opts *Options, f func(key string, createRev int64) error) error
    ListPrefix(prefix string, val proto.Message, opts *Options, f func(string) error) error
    Count() (int64, error)
    Watch(opts (watch.Watcher, error)
    WatchF(f func(*watch.Event) error, opts error
    WatchOne(key string, opts (watch.Watcher, error)
    WatchOneF(key string, f func(*watch.Event) error, opts error
    WatchByIndex(index *Index, val interface{}) (watch.Watcher, error)

ReadonlyCollection is a collection interface that only supports read ops.

type STM Uses

type STM interface {
    // Get returns the value for a key and inserts the key in the txn's read set.
    // If Get fails, it aborts the transaction with an error, never returning.
    Get(key string) (string, error)
    // Put adds a value for a key to the write set.
    Put(key, val string, ttl int64, ptr uintptr) error
    // Rev returns the revision of a key in the read set.
    Rev(key string) int64
    // Del deletes a key.
    Del(key string)
    // TTL returns the remaining time to live for 'key', or 0 if 'key' has no TTL
    TTL(key string) (int64, error)
    // DelAll deletes all keys with the given prefix
    // Note that the current implementation of DelAll is incomplete.
    // To use DelAll safely, do not issue any Get/Put operations after
    // DelAll is called.
    DelAll(key string)
    Context() context.Context
    // SetSafePutCheck sets the bit pattern to check if a put is safe.
    SetSafePutCheck(key string, ptr uintptr)
    // IsSafePut checks against the bit pattern for a key to see if it is safe to put.
    IsSafePut(key string, ptr uintptr) bool
    // contains filtered or unexported methods

STM is an interface for software transactional memory.

Package collection imports 23 packages (graph) and is imported by 21 packages. Updated 2020-08-13. Refresh now. Tools for package owners.