gae: Index | Files

package txnBuf

import ""

Package txnBuf contains a transaction buffer filter for the datastore service.

By default, datastore transactions take a snapshot of the entity group as soon as you Get or Put into it. All subsequent Get (and query) operations reflect the state of the ORIGINAL transaction snapshot, regardless of any Put/Delete operations you've done since the beginning of the transaction.

If you've installed this transaction buffer, then:

- All mutations will be reflected in all read operations (see LIMITATIONS).
  Without the buffer, read operations always observe the state of the
  entity group(s) at the time that the transaction started.

- All mutation operations will be buffered until the close of the
  transaction. This can help reduce the transaction size, and thus avoid
  the transaction size limit (currently 10MB). Multiple puts to the same
  entity will not increase the transaction size multiple times.

- Transactions inside of an existing transaction will add to their outer
  transaction if they don't cause the outer transaction to exceed its
  size budget.

- If an inner transaction would cause the OUTERMOST transaction to exceed
  the appengine-imposed 10MB transaction size limit, an error will be
  returned from the inner transaction, instead of adding it into the
  outermost transaction. This only applies to the first level of inner
  transactions, and does not apply to recursive transactions. The reason
  for this is that it's entirely feasible for inner transactions to
  temporarially exceed the limit, but still only commit an outer
  transaction which is under the limit. An example of this would be having
  one inner-inner transaction add a lot of large entities and then having
  a subsequent inner-inner transaction delete some of those entities.

LIMITATIONS (only inside of a transaction)

- KeysOnly/Projection/Count queries are supported, but may incur additional

  These query types are implemented via projection queries, but will
  project all order-by fields in addition to any specified in the original

- Distinct Projection queries do all 'distinct' deduplication in-memory.
  This could make them substantially more expensive than their native

- Metadata entities (e.g. `__entity_group__`) will reflect their values as
  they were at the beginning of the transaction, and will not increment
  as you write inside of the transaction.

- Query cursors are not supported. Since the cursor format for the
  in-memory datastore implementation isn't compatible with the production
  cursors, it would be pretty tricky to make it so that cursors were
  viable outside the transaction as well as inside of it while also having
  it accurately reflect the 'merged' query results.

- No parallel access* to datastore while in a transaction; all nested
  operations are serialized. This is done for simplicity and correctness.

  * The exception is that callbacks inside of
  a Run/GetMulti/DeleteMulti/PutMulti query MAY read/write the current
  transaction. Modifications to the datastore during query executions will
  not affect the query results (e.g. the query has snapshot consistency
  from the moment that it begins iteration). Note, however, that datastore
  operations within the callback are still synchronized. This behavior is
  so that the user is not forced to buffer all of the query results before
  doing work with them, but can treat the query like a stream of events,
  if they so choose.

- The changing of namespace inside of a transaction is undefined... This is
  just generally a terrible idea anyway, but I thought it was worth

- Currently, the soft transactions are not directly accessible using the
  CurrentTransaction interface; it returns the wrapped datastore's
  transaction. While this is still correct, it could definitely be made
  more useful by adding transaction buffer metadata to the returned


Package Files

context.go doc.go ds.go ds_txn.go query_merger.go state.go


const DefaultSizeBudget = int64((10 * 1000 * 1000) * 0.95)

DefaultSizeBudget is the size budget for the root transaction.

Because our estimation algorithm isn't entirely correct, we take 5% off the limit for encoding and estimate inaccuracies.

10MB taken on 2015/09/24:

const DefaultWriteCountBudget = 500

DefaultWriteCountBudget is the maximum number of entities that can be written in a single call.

This is not known to be documented, and has instead been extracted from a datastore error message.

const XGTransactionGroupLimit = 25

XGTransactionGroupLimit is the number of transaction groups to allow in an XG transaction.

25 taken on 2015/09/24:


var ErrTooManyRoots = errors.New(
    "operating on too many entity groups in nested transaction")

ErrTooManyRoots is returned when executing an operation which would cause the transaction to exceed it's allotted number of entity groups.

var ErrTransactionTooLarge = errors.New(
    "applying the transaction would make the parent transaction too large")

ErrTransactionTooLarge is returned when applying an inner transaction would cause an outer transaction to become too large.

func FilterRDS Uses

func FilterRDS(c context.Context) context.Context

FilterRDS installs a transaction buffer datastore filter in the context.

Package txnBuf imports 11 packages (graph) and is imported by 2 packages. Updated 2018-08-15. Refresh now. Tools for package owners.