aetools: Index | Files | Directories

package aetools

import ""

Package aetools helps writting, testing and analysing Google App Engine applications.

The aetools package implements a simple API to export the entity data from Datastore as a JSON stream, as well as load a JSON stream back into the Datastore. This can be used as a simple way to express state into a unit test, to backup a development environment state that can be shared with team members, or to make quick batch changes to data offline, like setting up configuration entities via Remote API.

The goal is to provide both an API and a set of executable tools that uses that API, allowing for maximum flexibility.

Load and Dump Data Format

The functions Load, LoadJSON, Dump and DumpJSON operate using JSON data that represents datastore entities. Each entity is mapped to a JSON Object, where each entity property name is an Object atribute, and each property value is the corresponding Object atribute value.

The property value is encoded using a JSON primitive, when possible. When the primitives are not sufficient to represent the property value, a JSON Object with the attributes "type" and "value" is used. The "type" attribute is a Datastore type, and value is a json-primitive serialization of that value. For instance, Blobs are encoded as a base64 JSON string, and time.Time values are encoded using the time.RFC3339 layout, also as strings.

Datastore Keys are aways encoded as a JSON Array that represents the Key Path, including ancestors, but without the application ID. This is done to allow the entity key to be more readable and to be application independent. Currently, they don't support namespaces.

Multiple properties are represented as a JSON Array of values described above. Unindexed properties are aways JSON objects with the "indexed" attribute set to false.

This format is intended to make use of the JSON types as much as possible, so an entity can be easily represented as a text file, suitable for read or SCM checkin.

The exported data format can also be used as an alternative way to export from Datastore, and then load the results right into other service, such as Google BigQuery or MongoDB.

The Web Bundle

The package aetools/bundle contains a sample webapp to help you manage and stream datastore entities into BigQuery. The bundle uses the aetools/bigquerysync functions to infer an usefull schema from datastore statistics, and sync your entity data into BigQuery.

The Remote API CLI

The command aetools/remote_api is a Remote API client that exposes the Load and Dump functions to make backup and restore of development environment state quick and easy. This tool can also help setting up Q.A. or Production apps, but should be used with care.


Package Files

doc.go entity.go funcs.go marshal.go


const (
    // DateTimeFormat is used to store and load time.Time objects
    DateTimeFormat = time.RFC3339


var (
    // ErrInvalidRootElement is returned when the root element is not a valid JSON Array.
    ErrInvalidRootElement = errors.New("aetools: root object is not an array")
    // ErrInvalidElementType is retunred when the element is not a JSON Object.
    ErrInvalidElementType = errors.New("aetools: element is not a JSON object")
    // ErrInvalidPropertiesElement is returned when the field to be decoded is not valid.
    ErrInvalidPropertiesElement = errors.New("aetools: element's properties field is invalid")
    // ErrNoKeyElement is returned for an entity with missing key information.
    ErrNoKeyElement = errors.New("aetools: element's key field is not present")
    // ErrInvalidKeyElement is returned when the key is not properly encoded.
    ErrInvalidKeyElement = errors.New("aetools: element's key field is invalid")
var (
    // LoadSync is an aetools.Options that enforces data to sync
    // after it get loaded into the datastore.
    // This Options will cause a datastore.Get to happen
    // for each entity loaded.
    LoadSync = &Options{
        GetAfterPut: true,

func Dump Uses

func Dump(c context.Context, w io.Writer, o *Options) error

Dump exports entities from the context c using the specified Options o and writing the generated JSON representations to the io.Writer w. You can configure how the dump will run by using the Options parameter. If there is an error generating the output, or writting to the writer, it is returned. This method may return an error after writting bytes to w: the output is not buffered.

func DumpEntity Uses

func DumpEntity(c context.Context, w io.Writer, keyString string, o *Options) error

DumpEntity export a single entity from the context

func DumpJSON Uses

func DumpJSON(c context.Context, o *Options) (string, error)

DumpJSON is a convenient wrapper that captures the generated JSON from Dump in memory, and return it as a string. If Dump returns an error, an empty string and the error are returned.

func EncodeEntities Uses

func EncodeEntities(entities []Entity, w io.Writer) error

EncodeEntities serializes the parameter into a JSON string.

func KeyPath Uses

func KeyPath(k *datastore.Key) string

KeyPath returns a representation of a Key as a string with each value in the key path separated by a coma. The key representation has all ancestors, but has no information about namespaces or ancestors.

func Load Uses

func Load(c context.Context, r io.Reader, o *Options) error

Load reads the JSON representation of entities from the io.Reader "r", and stores them in the Datastore using the given context.Context. The Options parameter allows you to configure how the dump will work. If there is any parsing erros, improper format, or datastore failures during the process, that error is returned and processing stops. The error may be returned after some entities were loaded: there is no parsing cache.

func LoadJSON Uses

func LoadJSON(c context.Context, s string, o *Options) error

LoadJSON is a convenient wrapper to call Load using a JSON string in memory, wrapped by a strings.Reader. The error result from Load, if any, is returned.

type DumpOptions Uses

type DumpOptions struct {

DumpOptions is deprecated. Use Options instead.

type Entity Uses

type Entity struct {
    Key        *datastore.Key
    Properties datastore.PropertyList

Entity is a small wrapper around datastore.PropertyList to also hold a *datastore.Key.

func DecodeEntities Uses

func DecodeEntities(c context.Context, r io.Reader) ([]Entity, error)

DecodeEntities deserielizes the parameter from a JSON string

func (*Entity) Add Uses

func (e *Entity) Add(p datastore.Property)

Add append p to the Properties attribute.

func (*Entity) Get Uses

func (e *Entity) Get(name string) interface{}

Get returns the property value of the given name. If no property with that name exists, returns nil. It also returns nil if the property Value attribute is nil.

func (*Entity) GetBool Uses

func (e *Entity) GetBool(name string) bool

GetBool returns the string value of the named property, and returns the zero value (false) if the property is not found, if its value is nil, or if its type is not bool.

func (*Entity) GetFloat Uses

func (e *Entity) GetFloat(name string) float64

GetFloat returns the string value of the named property, and returns the zero value (0.0) if the property is not found, if its value is nil or if its type is not float32 or float64.

func (*Entity) GetInt Uses

func (e *Entity) GetInt(name string) int64

GetInt returns the int value of the named property, and returns the zero value (0) if the property is not found, if its value is nil or if its type is not int, int32 or int64.

func (*Entity) GetString Uses

func (e *Entity) GetString(name string) string

GetString returns the string value of the named property, and returns the zero value ("") if the property is not found, if its value is nil or if its type is not string.

func (*Entity) Load Uses

func (e *Entity) Load(p []datastore.Property) error

Load decodes all properties into Entity.Propertioes object, implementing the datastore.PropertyLoadSaver interface.

func (*Entity) Map Uses

func (e *Entity) Map() (map[string]interface{}, error)

Map converts the entity data into a JSON compatible map. Usefull to manually encode the entity using different marshallers than the JSON built-in.

func (*Entity) MarshalJSON Uses

func (e *Entity) MarshalJSON() ([]byte, error)

MarshalJSON implements the json.Marshaller interface by dumping the entity key and properties.

func (*Entity) Save Uses

func (e *Entity) Save() ([]datastore.Property, error)

Save encodes all properties from Entity.Properties object, implmenting the datastore.PropetyLoadSaver interface.

type Options Uses

type Options struct {
    // GetAfterPut indicates if we must force the Datastore to load
    // entities to be visible for non-ancestor queries, by issuing a
    // Get by key.
    // Not used when loading.
    GetAfterPut bool

    // The size for batch operations when loading/dumping
    BatchSize int

    // Kind is used to specify the kind when dumping.
    // Not used when loading.
    Kind string

    // PrettyPrint is used to specify if the dump should beaultify the output.
    // Not used when loading.
    PrettyPrint bool

Options allows callees to specify parameters to the Load function.


aeremoteCommand aeremote is a simple Remote API client to download and upload data on Google App Engine Datastore.
bigquerysyncPackage bigquerysync allow the AppEngine Datastore to be synced with Google BigQuery.
bundlePackage bundle is a self-contained appengine module with aetools.
vmproxyPackage vmproxy provides tools to proxy App Engine requests to on-demand Compute Engine instances.

Package aetools imports 16 packages (graph) and is imported by 6 packages. Updated 2018-08-10. Refresh now. Tools for package owners.