gae: go.chromium.org/gae/service/datastore/serialize Index | Files

package serialize

import "go.chromium.org/gae/service/datastore/serialize"

Package serialize provides methods for reading and writing concatenable, bytewise-sortable forms of the datatypes defined in the datastore package.

Index

Package Files

binary_tools.go doc.go invertible.go serialize.go

Constants

const MaxIndexColumns = 64

MaxIndexColumns is the maximum number of sort columns (e.g. sort orders) that ReadIndexDefinition is willing to deserialize. 64 was chosen as a likely-astronomical number.

const ReadKeyNumToksReasonableLimit = 50

ReadKeyNumToksReasonableLimit is the maximum number of Key tokens that ReadKey is willing to read for a single key.

const ReadPropertyMapReasonableLimit uint64 = 30000

ReadPropertyMapReasonableLimit sets a limit on the number of rows and number of properties per row which can be read by ReadPropertyMap. The total number of Property objects readable by this method is this number squared (e.g. Limit rows * Limit properties)

Variables

var WritePropertyMapDeterministic = false

WritePropertyMapDeterministic allows tests to make WritePropertyMap deterministic.

func Increment Uses

func Increment(bstr []byte) ([]byte, bool)

Increment attempts to increment a copy of bstr as if adding 1 to an integer.

If it overflows, the returned []byte will be all 0's, and the overflow bool will be true.

func Invert Uses

func Invert(bs []byte) []byte

Invert simply inverts all the bytes in bs.

func Join Uses

func Join(itms ...[]byte) []byte

Join is a convenience invocation of bytes.Join(itms, nil)

func ReadGeoPoint Uses

func ReadGeoPoint(buf ReadBuffer) (gp ds.GeoPoint, err error)

ReadGeoPoint reads a GeoPoint from the buffer.

func ReadIndexColumn Uses

func ReadIndexColumn(buf ReadBuffer) (c ds.IndexColumn, err error)

ReadIndexColumn reads an IndexColumn from the buffer.

func ReadIndexDefinition Uses

func ReadIndexDefinition(buf ReadBuffer) (i ds.IndexDefinition, err error)

ReadIndexDefinition reads an IndexDefinition from the buffer.

func ReadKey Uses

func ReadKey(buf ReadBuffer, context KeyContext, inKC ds.KeyContext) (ret *ds.Key, err error)

ReadKey deserializes a key from the buffer. The value of context must match the value of context that was passed to WriteKey when the key was encoded. If context == WithoutContext, then the appid and namespace parameters are used in the decoded Key. Otherwise they're ignored.

func ReadKeyTok Uses

func ReadKeyTok(buf ReadBuffer) (ret ds.KeyTok, err error)

ReadKeyTok reads a KeyTok from the buffer. You usually want ReadKey instead of this.

func ReadProperty Uses

func ReadProperty(buf ReadBuffer, context KeyContext, kc ds.KeyContext) (p ds.Property, err error)

ReadProperty reads a Property from the buffer. `context` and `kc` behave the same way they do for ReadKey, but only have an effect if the decoded property has a Key value.

func ReadPropertyMap Uses

func ReadPropertyMap(buf ReadBuffer, context KeyContext, kc ds.KeyContext) (pm ds.PropertyMap, err error)

ReadPropertyMap reads a PropertyMap from the buffer. `context` and friends behave the same way that they do for ReadKey.

func ReadTime Uses

func ReadTime(buf ReadBuffer) (time.Time, error)

ReadTime reads a time.Time from the buffer.

func ToBytes Uses

func ToBytes(i interface{}) []byte

ToBytes serializes i to a byte slice, if it's one of the type supported by this library. If an error is encountered (e.g. `i` is not a supported type), this method panics.

Key types will be serialized using the 'WithoutContext' option (e.g. their encoded forms will not contain AppID or Namespace).

func ToBytesErr Uses

func ToBytesErr(i interface{}) ([]byte, error)

ToBytesErr serializes i to a byte slice, if it's one of the type supported by this library, otherwise it returns an error.

Key types will be serialized using the 'WithoutContext' option (e.g. their encoded forms will not contain AppID or Namespace).

func ToBytesWithContext Uses

func ToBytesWithContext(i interface{}) []byte

ToBytesWithContext serializes i to a byte slice, if it's one of the type supported by this library. If an error is encountered (e.g. `i` is not a supported type), this method panics.

Key types will be serialized using the 'WithContext' option (e.g. their encoded forms will not contain AppID or Namespace).

func ToBytesWithContextErr Uses

func ToBytesWithContextErr(i interface{}) ([]byte, error)

ToBytesWithContextErr serializes i to a byte slice, if it's one of the type supported by this library, otherwise it returns an error.

Key types will be serialized using the 'WithContext' option (e.g. their encoded forms will contain AppID and Namespace).

func WriteGeoPoint Uses

func WriteGeoPoint(buf WriteBuffer, gp ds.GeoPoint) (err error)

WriteGeoPoint writes a GeoPoint to the buffer.

func WriteIndexColumn Uses

func WriteIndexColumn(buf WriteBuffer, c ds.IndexColumn) (err error)

WriteIndexColumn writes an IndexColumn to the buffer.

func WriteIndexDefinition Uses

func WriteIndexDefinition(buf WriteBuffer, i ds.IndexDefinition) (err error)

WriteIndexDefinition writes an IndexDefinition to the buffer

func WriteIndexProperty Uses

func WriteIndexProperty(buf WriteBuffer, context KeyContext, p ds.Property) error

WriteIndexProperty writes a Property to the buffer as its native index type. `context` behaves the same way that it does for WriteKey, but only has an effect if `p` contains a Key as its IndexValue.

func WriteKey Uses

func WriteKey(buf WriteBuffer, context KeyContext, k *ds.Key) (err error)

WriteKey encodes a key to the buffer. If context is WithContext, then this encoded value will include the appid and namespace of the key.

func WriteKeyTok Uses

func WriteKeyTok(buf WriteBuffer, tok ds.KeyTok) (err error)

WriteKeyTok writes a KeyTok to the buffer. You usually want WriteKey instead of this.

func WriteProperty Uses

func WriteProperty(buf WriteBuffer, context KeyContext, p ds.Property) error

WriteProperty writes a Property to the buffer. `context` behaves the same way that it does for WriteKey, but only has an effect if `p` contains a Key as its IndexValue.

func WritePropertyMap Uses

func WritePropertyMap(buf WriteBuffer, context KeyContext, pm ds.PropertyMap) (err error)

WritePropertyMap writes an entire PropertyMap to the buffer. `context` behaves the same way that it does for WriteKey.

If WritePropertyMapDeterministic is true, then the rows will be sorted by property name before they're serialized to buf (mostly useful for testing, but also potentially useful if you need to make a hash of the property data).

Write skips metadata keys.

func WriteTime Uses

func WriteTime(buf WriteBuffer, t time.Time) error

WriteTime writes a time.Time to the buffer.

The supplied time is rounded via datastore.RoundTime and written as a microseconds-since-epoch integer to comform to datastore storage standards.

type InvertibleBuffer Uses

type InvertibleBuffer interface {
    WriteBuffer
    SetInvert(inverted bool)
}

InvertibleBuffer is just like Buffer, except that it also has a stateful Invert() method, which will cause all reads and writes to/from it to be inverted (e.g. every byte XOR 0xFF).

Implementing queries requires manipulating the index entries (e.g. synthesizing them, parsing them, etc.). In particular, when you have a reverse-sorted field (e.g. high to low instead of low to high), it's achieved by having all the bits inverted.

All the serialization formats include delimiter information, which the parsers only know to parse non-inverted. If we don't have this buffer, we'd basically have to invert every byte in the []byte array when we're trying to decode a reverse-ordered field (including the bytes of all fields after the one we intend to parse) so that the parser can consume as many bytes as it needs (and it only knows the number of bytes it needs as it decodes them). This InvertibleBuffer lets that happen on the fly without having to flip the whole []byte.

If you know you need it, you'll know it's the right thing. If you're not sure then you definitely don't need it!

func Invertible Uses

func Invertible(b WriteBuffer) InvertibleBuffer

Invertible returns an InvertibleBuffer based on the Buffer.

type KeyContext Uses

type KeyContext bool

KeyContext controls whether the various Write and Read serializtion routines should encode the context of Keys (read: the appid and namespace). Frequently the appid and namespace of keys are known in advance and so there's no reason to redundantly encode them.

const (
    WithContext    KeyContext = true
    WithoutContext            = false
)

With- and WithoutContext indicate if the serialization method should include context for Keys. See KeyContext for more information.

type ReadBuffer Uses

type ReadBuffer interface {
    Len() int

    Read([]byte) (int, error)
    ReadByte() (byte, error)
}

ReadBuffer is the interface which corresponds to the subset of *bytes.Reader that this package requires.

type SerializedPmap Uses

type SerializedPmap map[string]SerializedPslice

SerializedPmap maps from

prop name -> [<serialized DSProperty>, ...]

includes special values '__key__' and '__ancestor__' which contains all of the ancestor entries for this key.

func PropertyMapPartially Uses

func PropertyMapPartially(k *ds.Key, pm ds.PropertyMap) (ret SerializedPmap)

PropertyMapPartially turns a regular PropertyMap into a SerializedPmap. Essentially all the []Property's become SerializedPslice, using cmpbin and datastore/serialize's encodings.

type SerializedPslice Uses

type SerializedPslice [][]byte

SerializedPslice is all of the serialized DSProperty values in ASC order.

func PropertySlice Uses

func PropertySlice(vals ds.PropertySlice) SerializedPslice

PropertySlice serializes a single row of a DSProperty map.

It does not differentiate between single- and multi- properties.

func (SerializedPslice) Len Uses

func (s SerializedPslice) Len() int

func (SerializedPslice) Less Uses

func (s SerializedPslice) Less(i, j int) bool

func (SerializedPslice) Swap Uses

func (s SerializedPslice) Swap(i, j int)

type WriteBuffer Uses

type WriteBuffer interface {
    ReadBuffer

    String() string
    Bytes() []byte

    Grow(int)

    Write([]byte) (int, error)
    WriteByte(c byte) error
    WriteString(s string) (int, error)
}

WriteBuffer is the interface which corresponds to the subset of *bytes.Buffer that this package requires.

Package serialize imports 9 packages (graph) and is imported by 9 packages. Updated 2018-08-15. Refresh now. Tools for package owners.