manipulate: Index | Files | Directories

package manipulate

import ""

Package manipulate provides everything needed to perform CRUD operations on an based data model.

The main interface is Manipulator. This interface provides various methods for creation, modification, retrieval and so on. TransactionalManipulator, which is an extension of the Manipulator add methods to manage transactions, like Commit and Abort.

A Manipulator works with some elemental.Identifiables.

The storage engine used by a Manipulator is abstracted. By default manipulate provides implementations for Rest API over HTTP or websocket, Mongo DB, Memory and a mock Manipulator for unit testing. You can of course create your own implementation.

Each method of a Manipulator is taking a manipulate.Context as argument. The context is used to pass additional informations like a Filter or some Parameters.

Example for creating an object:

// Create a User from a generated Elemental model.
user := models.NewUser()
user.FullName, user.Login := "Antoine Mercadal", "primalmotion"

// Create Mongo Manipulator.
m := manipmongo.NewMongoManipulator([]{""}, "test", "db-username", "db-password", "db-authsource", 512)

// Then create the User.
m.Create(nil, user)

Example for retreving an object:

// Create a Context with a filter.
ctx := manipulate.NewContextWithFilter(

// Retrieve the users matching the filter.
var users models.UserLists
m.RetrieveMany(ctx, models.UserIdentity, &users)


Package Files

context.go doc.go errors.go filter.go iter.go manipulate.go options.go retry.go transaction.go

func IsCannotBuildQueryError Uses

func IsCannotBuildQueryError(err error) bool

IsCannotBuildQueryError returns true if the given error is am ErrCannotBuildQuery.

func IsCannotCommitError Uses

func IsCannotCommitError(err error) bool

IsCannotCommitError returns true if the given error is am ErrCannotCommit.

func IsCannotCommunicateError Uses

func IsCannotCommunicateError(err error) bool

IsCannotCommunicateError returns true if the given error is am ErrCannotCommunicate.

func IsCannotExecuteQueryError Uses

func IsCannotExecuteQueryError(err error) bool

IsCannotExecuteQueryError returns true if the given error is am ErrCannotExecuteQuery.

func IsCannotMarshalError Uses

func IsCannotMarshalError(err error) bool

IsCannotMarshalError returns true if the given error is am ErrCannotMarshal.

func IsCannotUnmarshalError Uses

func IsCannotUnmarshalError(err error) bool

IsCannotUnmarshalError returns true if the given error is am ErrCannotUnmarshal.

func IsConstraintViolationError Uses

func IsConstraintViolationError(err error) bool

IsConstraintViolationError returns true if the given error is am ErrConstraintViolation.

func IsDisconnectedError Uses

func IsDisconnectedError(err error) bool

IsDisconnectedError returns true if the given error is am ErrDisconnected.

func IsLockedError Uses

func IsLockedError(err error) bool

IsLockedError returns true if the given error is am ErrLocked.

func IsMultipleObjectsFoundError Uses

func IsMultipleObjectsFoundError(err error) bool

IsMultipleObjectsFoundError returns true if the given error is am ErrMultipleObjectsFound.

func IsNotImplementedError Uses

func IsNotImplementedError(err error) bool

IsNotImplementedError returns true if the given error is am ErrNotImplemented.

func IsObjectNotFoundError Uses

func IsObjectNotFoundError(err error) bool

IsObjectNotFoundError returns true if the given error is am ErrObjectNotFound.

func IsTLSError Uses

func IsTLSError(err error) bool

IsTLSError returns true if the given error is am ErrTLS.

func IsTooManyRequestsError Uses

func IsTooManyRequestsError(err error) bool

IsTooManyRequestsError returns true if the given error is am ErrTooManyRequests.

func IsTransactionNotFoundError Uses

func IsTransactionNotFoundError(err error) bool

IsTransactionNotFoundError returns true if the given error is am ErrTransactionNotFound.

func Iter Uses

func Iter(
    ctx context.Context,
    m Manipulator,
    mctx Context,
    identifiablesTemplate elemental.Identifiables,
    blockSize int,
) (elemental.Identifiables, error)

Iter is a helper function for IterFunc.

It will simply iterates on the object with identity of the given elemental.Identifiables. Not that this function cannot populate the data in the identifiable parameter. Instead It will return the destination.

Always pass an empty elemental.Identifiables to this function

For more information, please check IterFunc documentation.


dest, err := Iter(context.Background(), m, mctx, model.ThingsList{}, 100)

func IterFunc Uses

func IterFunc(
    ctx context.Context,
    manipulator Manipulator,
    identifiablesTemplate elemental.Identifiables,
    mctx Context,
    iteratorFunc func(block elemental.Identifiables) error,
    blockSize int,
) error

IterFunc calls RetrieveMany on the given Manipulator, and will retrieve the data by block of the given blockSize.

IterFunc will naturally ends and return when there is no more data to pull.

For each retrieved block, the given func will be called with the current data block. If the function returns an error, the error is returned to the caller of IterFunc and the iteration stops.

The given context will be used if the underlying manipulator honors it. Be careful to NOT pass a filter matching objects then updating the objects to not match anynmore. This would shift pagination and will produce unexpected results. To do so, prefer using manipulate.IterUntilFunc

The given manipulate.Context will be used to retry any failed batch recovery.

The identifiablesTemplate parameter is must be an empty elemental.Identifiables that will be used to hold the data block. It is reset at every iteration. Do not rely on it to be filled once IterFunc is complete.

Finally, if the given blockSize is <= 0, then it will use the default that is 1000.

func IterUntilFunc Uses

func IterUntilFunc(
    ctx context.Context,
    manipulator Manipulator,
    identifiablesTemplate elemental.Identifiables,
    mctx Context,
    iteratorFunc func(block elemental.Identifiables) error,
    blockSize int,
) error

IterUntilFunc works as IterFunc but pagination will not increase. It will always retrieve the first page with a size of given blockSize.

The goal of this function is to be used with a filter, then update (or delete) the objects that match until no more are matching.

func Retry Uses

func Retry(ctx context.Context, manipulateFunc func() error, onRetryFunc func(int, error) error) error

Retry is deprecated and only calls manipulateFunc for backward compatibility.

type BufferedManipulator Uses

type BufferedManipulator interface {

A BufferedManipulator is a Manipulator with a local cache

type Context Uses

type Context interface {
    Count() int
    SetCount(count int)
    Filter() *elemental.Filter
    Finalizer() FinalizerFunc
    Version() int
    TransactionID() TransactionID
    Page() int
    PageSize() int
    After() string
    Limit() int
    Next() string
    Override() bool
    Recursive() bool
    Namespace() string
    Credentials() (string, string)
    Parameters() url.Values
    Parent() elemental.Identifiable
    ExternalTrackingID() string
    ExternalTrackingType() string
    Order() []string
    Context() context.Context
    Derive(...ContextOption) Context
    Fields() []string
    ReadConsistency() ReadConsistency
    WriteConsistency() WriteConsistency
    Messages() []string
    ClientIP() string
    RetryFunc() RetryFunc
    RetryRatio() int64


A Context holds all information regarding a particular manipulate operation.

func NewContext Uses

func NewContext(ctx context.Context, options ...ContextOption) Context

NewContext creates a context with the given ContextOption.

type ContextOption Uses

type ContextOption func(Context)

ContextOption represents an option can can be passed to NewContext.

func ContextOptionAfter Uses

func ContextOptionAfter(from string, limit int) ContextOption

ContextOptionAfter sets the lazy pagination option of the context.

func ContextOptionClientIP Uses

func ContextOptionClientIP(clientIP string) ContextOption

ContextOptionClientIP sets the optional headers for the request.

func ContextOptionCredentials Uses

func ContextOptionCredentials(username, password string) ContextOption

ContextOptionCredentials sets user name and password for this context.

func ContextOptionFields Uses

func ContextOptionFields(fields []string) ContextOption

ContextOptionFields sets the list of fields to include in the reply.

func ContextOptionFilter Uses

func ContextOptionFilter(f *elemental.Filter) ContextOption

ContextOptionFilter sets the filter.

func ContextOptionFinalizer Uses

func ContextOptionFinalizer(f FinalizerFunc) ContextOption

ContextOptionFinalizer sets the create finalizer option of the context.

func ContextOptionIdempotencyKey Uses

func ContextOptionIdempotencyKey(key string) ContextOption

ContextOptionIdempotencyKey sets a custom idempotency key.

func ContextOptionNamespace Uses

func ContextOptionNamespace(n string) ContextOption

ContextOptionNamespace sets the namespace.

func ContextOptionOpaque Uses

func ContextOptionOpaque(o map[string]interface{}) ContextOption

ContextOptionOpaque sets a opaque data. Their interpretation depends on the manipulator implementation.

func ContextOptionOrder Uses

func ContextOptionOrder(orders ...string) ContextOption

ContextOptionOrder sets the ordering option of the context.

func ContextOptionOverride Uses

func ContextOptionOverride(o bool) ContextOption

ContextOptionOverride sets the override option of the context.

func ContextOptionPage Uses

func ContextOptionPage(n, size int) ContextOption

ContextOptionPage sets the pagination option of the context.

func ContextOptionParameters Uses

func ContextOptionParameters(p url.Values) ContextOption

ContextOptionParameters sets the parameters option of the context.

func ContextOptionParent Uses

func ContextOptionParent(i elemental.Identifiable) ContextOption

ContextOptionParent sets the parent option of the context.

func ContextOptionReadConsistency Uses

func ContextOptionReadConsistency(consistency ReadConsistency) ContextOption

ContextOptionReadConsistency sets the desired read consistency of the request.

func ContextOptionRecursive Uses

func ContextOptionRecursive(r bool) ContextOption

ContextOptionRecursive sets the recursive option of the context.

func ContextOptionRetryFunc Uses

func ContextOptionRetryFunc(f RetryFunc) ContextOption

ContextOptionRetryFunc sets the retry function. This function will be called on every communication error, and will be passed the try number and the error. If it itself return an error, retrying will stop and that error will be returned from the manipulator operation.

func ContextOptionRetryRatio Uses

func ContextOptionRetryRatio(r int64) ContextOption

ContextOptionRetryRatio sets the retry ratio.

RetryRatio divides the remaining time unitl context deadline to perfrom single retry query down to a minimum defined by manipulator implementations (typically 20s). The default value is 4.

For example if the context has a timeout of 2m, each retry will use a sub context with a timeout of 30s.

func ContextOptionToken Uses

func ContextOptionToken(token string) ContextOption

ContextOptionToken sets the token for this request.

func ContextOptionTracking Uses

func ContextOptionTracking(identifier, typ string) ContextOption

ContextOptionTracking sets the opentracing tracking option of the context.

func ContextOptionTransactionID Uses

func ContextOptionTransactionID(tid TransactionID) ContextOption

ContextOptionTransactionID sets the parameters option of the context.

func ContextOptionVersion Uses

func ContextOptionVersion(v int) ContextOption

ContextOptionVersion sets the version option of the context.

func ContextOptionWriteConsistency Uses

func ContextOptionWriteConsistency(consistency WriteConsistency) ContextOption

ContextOptionWriteConsistency sets the desired write consistency of the request.

type ErrCannotBuildQuery Uses

type ErrCannotBuildQuery struct {
    // contains filtered or unexported fields

ErrCannotBuildQuery represents query building error.

func NewErrCannotBuildQuery Uses

func NewErrCannotBuildQuery(message string) ErrCannotBuildQuery

NewErrCannotBuildQuery returns a new ErrCannotBuildQuery.

func (ErrCannotBuildQuery) Error Uses

func (e ErrCannotBuildQuery) Error() string

type ErrCannotCommit Uses

type ErrCannotCommit struct {
    // contains filtered or unexported fields

ErrCannotCommit represents commit execution error.

func NewErrCannotCommit Uses

func NewErrCannotCommit(message string) ErrCannotCommit

NewErrCannotCommit returns a new ErrCannotCommit.

func (ErrCannotCommit) Error Uses

func (e ErrCannotCommit) Error() string

type ErrCannotCommunicate Uses

type ErrCannotCommunicate struct {
    // contains filtered or unexported fields

ErrCannotCommunicate represents a failure in backend communication.

func NewErrCannotCommunicate Uses

func NewErrCannotCommunicate(message string) ErrCannotCommunicate

NewErrCannotCommunicate returns a new ErrCannotCommunicate.

func (ErrCannotCommunicate) Error Uses

func (e ErrCannotCommunicate) Error() string

type ErrCannotExecuteQuery Uses

type ErrCannotExecuteQuery struct {
    // contains filtered or unexported fields

ErrCannotExecuteQuery represents query execution error.

func NewErrCannotExecuteQuery Uses

func NewErrCannotExecuteQuery(message string) ErrCannotExecuteQuery

NewErrCannotExecuteQuery returns a new ErrCannotExecuteQuery.

func (ErrCannotExecuteQuery) Error Uses

func (e ErrCannotExecuteQuery) Error() string

type ErrCannotMarshal Uses

type ErrCannotMarshal struct {
    // contains filtered or unexported fields

ErrCannotMarshal represents marshaling error.

func NewErrCannotMarshal Uses

func NewErrCannotMarshal(message string) ErrCannotMarshal

NewErrCannotMarshal returns a new ErrCannotMarshal.

func (ErrCannotMarshal) Error Uses

func (e ErrCannotMarshal) Error() string

type ErrCannotUnmarshal Uses

type ErrCannotUnmarshal struct {
    // contains filtered or unexported fields

ErrCannotUnmarshal represents unmarshaling error.

func NewErrCannotUnmarshal Uses

func NewErrCannotUnmarshal(message string) ErrCannotUnmarshal

NewErrCannotUnmarshal returns a new ErrCannotUnmarshal.

func (ErrCannotUnmarshal) Error Uses

func (e ErrCannotUnmarshal) Error() string

type ErrConstraintViolation Uses

type ErrConstraintViolation struct {
    // contains filtered or unexported fields

ErrConstraintViolation represents a failure to find a transaction.

func NewErrConstraintViolation Uses

func NewErrConstraintViolation(message string) ErrConstraintViolation

NewErrConstraintViolation returns a new ErrConstraintViolation.

func (ErrConstraintViolation) Error Uses

func (e ErrConstraintViolation) Error() string

type ErrDisconnected Uses

type ErrDisconnected struct {
    // contains filtered or unexported fields

ErrDisconnected represents an error due user disconnection.

func NewErrDisconnected Uses

func NewErrDisconnected(message string) ErrDisconnected

NewErrDisconnected returns a new ErrDisconnected.

func (ErrDisconnected) Error Uses

func (e ErrDisconnected) Error() string

type ErrLocked Uses

type ErrLocked struct {
    // contains filtered or unexported fields

ErrLocked represents the error returned when the server api is locked..

func NewErrLocked Uses

func NewErrLocked(message string) ErrLocked

NewErrLocked returns a new ErrCannotCommunicate.

func (ErrLocked) Error Uses

func (e ErrLocked) Error() string

type ErrMultipleObjectsFound Uses

type ErrMultipleObjectsFound struct {
    // contains filtered or unexported fields

ErrMultipleObjectsFound represents too many object found error.

func NewErrMultipleObjectsFound Uses

func NewErrMultipleObjectsFound(message string) ErrMultipleObjectsFound

NewErrMultipleObjectsFound returns a new ErrMultipleObjectsFound.

func (ErrMultipleObjectsFound) Error Uses

func (e ErrMultipleObjectsFound) Error() string

type ErrNotImplemented Uses

type ErrNotImplemented struct {
    // contains filtered or unexported fields

ErrNotImplemented represents a non implemented function.

func NewErrNotImplemented Uses

func NewErrNotImplemented(message string) ErrNotImplemented

NewErrNotImplemented returns a new ErrNotImplemented.

func (ErrNotImplemented) Error Uses

func (e ErrNotImplemented) Error() string

type ErrObjectNotFound Uses

type ErrObjectNotFound struct {
    // contains filtered or unexported fields

ErrObjectNotFound represents object not found error.

func NewErrObjectNotFound Uses

func NewErrObjectNotFound(message string) ErrObjectNotFound

NewErrObjectNotFound returns a new ErrObjectNotFound.

func (ErrObjectNotFound) Error Uses

func (e ErrObjectNotFound) Error() string

type ErrTLS Uses

type ErrTLS struct {
    // contains filtered or unexported fields

ErrTLS represents the error returned when there is a TLS error.

func NewErrTLS Uses

func NewErrTLS(message string) ErrTLS

NewErrTLS returns a new ErrTLS.

func (ErrTLS) Error Uses

func (e ErrTLS) Error() string

type ErrTooManyRequests Uses

type ErrTooManyRequests struct {
    // contains filtered or unexported fields

ErrTooManyRequests represents the error returned when the server api is locked.

func NewErrTooManyRequests Uses

func NewErrTooManyRequests(message string) ErrTooManyRequests

NewErrTooManyRequests returns a new ErrTooManyRequests.

func (ErrTooManyRequests) Error Uses

func (e ErrTooManyRequests) Error() string

type ErrTransactionNotFound Uses

type ErrTransactionNotFound struct {
    // contains filtered or unexported fields

ErrTransactionNotFound represents a failure to find a transaction.

func NewErrTransactionNotFound Uses

func NewErrTransactionNotFound(message string) ErrTransactionNotFound

NewErrTransactionNotFound returns a new ErrTransactionNotFound.

func (ErrTransactionNotFound) Error Uses

func (e ErrTransactionNotFound) Error() string

type Filter Uses

type Filter = elemental.Filter

Filter is an alias of elemental.Filter

func NewFilter Uses

func NewFilter() *Filter

NewFilter returns a new Filter using the aliased type.

func NewFilterFromString Uses

func NewFilterFromString(filter string) (*Filter, error)

NewFilterFromString returns a new NewFilterFromString using the aliased type.

type FilterKeyComposer Uses

type FilterKeyComposer = elemental.FilterKeyComposer

FilterKeyComposer is an alias of elemental.FilterKeyComposer

func NewFilterComposer Uses

func NewFilterComposer() FilterKeyComposer

NewFilterComposer returns a new FilterKeyComposer using the aliased type.

type FilterParser Uses

type FilterParser = elemental.FilterParser

FilterParser is an alias of elemental.FilterParser

func NewFilterParser Uses

func NewFilterParser(input string) *FilterParser

NewFilterParser returns a new NewFilterParser using the aliased type.

type FinalizerFunc Uses

type FinalizerFunc func(o elemental.Identifiable) error

A FinalizerFunc is the type of a function that can be used as a creation finalizer. This is only supported by manipulators that generate an ID to let a chance to the user. to now the intended ID before actually creating the object.

type FlushableManipulator Uses

type FlushableManipulator interface {

    // Flush flushes and empties the cache.
    Flush(ctx context.Context) error

A FlushableManipulator is a manipulator that can flush its content to somewhere, like a file.

type Manipulator Uses

type Manipulator interface {

    // RetrieveMany retrieves the a list of objects with the given elemental.Identity and put them in the given dest.
    RetrieveMany(mctx Context, dest elemental.Identifiables) error

    // Retrieve retrieves one or multiple elemental.Identifiables.
    // In order to be retrievable, the elemental.Identifiable needs to have their Identifier correctly set.
    Retrieve(mctx Context, object elemental.Identifiable) error

    // Create creates a the given elemental.Identifiables.
    Create(mctx Context, object elemental.Identifiable) error

    // Update updates one or multiple elemental.Identifiables.
    // In order to be updatable, the elemental.Identifiable needs to have their Identifier correctly set.
    Update(mctx Context, object elemental.Identifiable) error

    // Delete deletes one or multiple elemental.Identifiables.
    // In order to be deletable, the elemental.Identifiable needs to have their Identifier correctly set.
    Delete(mctx Context, object elemental.Identifiable) error

    // DeleteMany deletes all objects of with the given identity or
    // all the ones matching the filter in the given context.
    DeleteMany(mctx Context, identity elemental.Identity) error

    // Count returns the number of objects with the given identity.
    Count(mctx Context, identity elemental.Identity) (int, error)

Manipulator is the interface of a storage backend.

type ReadConsistency Uses

type ReadConsistency string

ReadConsistency represents the desired consistency of the request. Not all driver may implement this.

const (
    ReadConsistencyDefault   ReadConsistency = "default"
    ReadConsistencyNearest   ReadConsistency = "nearest"
    ReadConsistencyEventual  ReadConsistency = "eventual"
    ReadConsistencyMonotonic ReadConsistency = "monotonic"
    ReadConsistencyStrong    ReadConsistency = "strong"
    ReadConsistencyWeakest   ReadConsistency = "weakest"

Various values for Consistency

type RetryFunc Uses

type RetryFunc func(RetryInfo) error

A RetryFunc is a function that can be called during an auto retry. The current manipulate.Context is given, a Stringer interface containing, more info about the current request, the error that caused the retry and the try number. If this function returns an error, the retry procedure will be interupted and this error will be returns to the caller of the operation.

type RetryInfo Uses

type RetryInfo interface {
    Err() error
    Context() Context
    Try() int

A RetryInfo is the interface that can be passed to RetryFunc that will contain retry information. Content will depend on the manipulator implementation.

type Subscriber Uses

type Subscriber interface {

    // Start connects to the websocket and starts collecting events
    // until the given context is canceled or any non communication error is
    // received. The eventual error will be received in the Errors() channel.
    // If not nil, the given push config will be applied right away.
    Start(context.Context, *elemental.PushConfig)

    // UpdateFilter updates the current push config.

    // Events returns the events channel.
    Events() chan *elemental.Event

    // Errors returns the errors channel.
    Errors() chan error

    // Status returns the status channel.
    Status() chan SubscriberStatus

A Subscriber is the interface to control a push event subscription.

type SubscriberStatus Uses

type SubscriberStatus int

SubscriberStatus is the type of a subscriber status.

const (
    SubscriberStatusInitialConnection SubscriberStatus = iota + 1

Various values of SubscriberEvent.

type TokenManager Uses

type TokenManager interface {

    // Issues isses a new token.
    Issue(context.Context) (string, error)

    // Run runs the token renewal job and published the new token in the
    // given channel.
    Run(ctx context.Context, tokenCh chan string)

A TokenManager issues an renew tokens periodically.

type TransactionID Uses

type TransactionID string

TransactionID is the type used to define a transcation ID of a store

func NewTransactionID Uses

func NewTransactionID() TransactionID

NewTransactionID returns a new transaction ID.

type TransactionalManipulator Uses

type TransactionalManipulator interface {

    // Commit commits the given TransactionID.
    Commit(id TransactionID) error

    // Abort aborts the give TransactionID. It returns true if
    // a transaction has been effectively aborted, otherwise it returns false.
    Abort(id TransactionID) bool


A TransactionalManipulator is a Manipulator that handles transactions.

type WriteConsistency Uses

type WriteConsistency string

WriteConsistency represents the desired consistency of the request. Not all driver may implement this.

const (
    WriteConsistencyDefault   WriteConsistency = "default"
    WriteConsistencyNone      WriteConsistency = "none"
    WriteConsistencyStrong    WriteConsistency = "strong"
    WriteConsistencyStrongest WriteConsistency = "strongest"

Various values for Consistency


maniphttpPackage maniphttp provides a ReST backed Manipulator.
manipmemoryPackage manipmemory provides a go-memdb backed Manipulator.
manipmongoPackage manipmongo provides a MongoDB backed TransactionalManipulator.
manipmongo/internalPackage internal is a generated GoMock package.
maniptestPackage maniptest contains a Mockable TransactionalManipulator.
manipvortexPackage manipvortex contains a Manipulator that can be used as cache in front of another Manipulator.

Package manipulate imports 5 packages (graph) and is imported by 8 packages. Updated 2021-01-19. Refresh now. Tools for package owners.