stalecucumber: github.com/hydrogen18/stalecucumber Index | Files | Directories

package stalecucumber

import "github.com/hydrogen18/stalecucumber"

This package reads and writes pickled data. The format is the same as the Python "pickle" module.

Protocols 0,1,2 are implemented. These are the versions written by the Python 2.x series. Python 3 defines newer protocol versions, but can write the older protocol versions so they are readable by this package.

To read data, see stalecucumber.Unpickle.

To write data, see stalecucumber.NewPickler.

TLDR

Read a pickled string or unicode object

pickle.dumps("foobar")
---
var somePickledData io.Reader
mystring, err := stalecucumber.String(stalecucumber.Unpickle(somePickledData))

Read a pickled integer

pickle.dumps(42)
---
var somePickledData io.Reader
myint64, err := stalecucumber.Int(stalecucumber.Unpickle(somePickledData))

Read a pickled list of numbers into a structure

pickle.dumps([8,8,2005])
---
var somePickledData io.Reader
numbers := make([]int64,0)

err := stalecucumber.UnpackInto(&numbers).From(stalecucumber.Unpickle(somePickledData))

Read a pickled dictionary into a structure

pickle.dumps({"apple":1,"banana":2,"cat":"hello","Dog":42.0})
---
var somePickledData io.Reader
mystruct := struct{
	Apple int
	Banana uint
	Cat string
	Dog float32}{}

err := stalecucumber.UnpackInto(&mystruct).From(stalecucumber.Unpickle(somePickledData))

Pickle a struct

buf := new(bytes.Buffer)
mystruct := struct{
		Apple int
		Banana uint
		Cat string
		Dog float32}{}

err := stalecucumber.NewPickler(buf).Pickle(mystruct)

Recursive objects

You can pickle recursive objects like so

a = {}
a["self"] = a
pickle.dumps(a)

Python's pickler is intelligent enough not to emit an infinite data structure when a recursive object is pickled.

I recommend against pickling recursive objects in the first place, but this library handles unpickling them without a problem. The result of unpickling the above is map[interface{}]interface{} with a key "a" that contains a reference to itself.

Attempting to unpack the result of the above python code into a structure with UnpackInto would either fail or recurse forever.

Unpickling Python objects

The Python Pickle module can pickle most Python objects. By default, some Python objects such as the set type and bytearray type are automatically supported by this library.

To support unpickling custom Python objects, you need to implement a resolver. A resolver meets the PythonResolver interface, which is just this function

Resolve(module string, name string, args []interface{})

The module and name are the class name. So if you have a class called "Foo" in the module "bar" the first argument would be "bar" and the second would be "Foo". You can pass in your custom resolver by calling

UnpickleWithResolver(buf, yourCustomResolver)

The third argument of the Resolve function is originally a Python tuple, so it is slice of anything. For most user defined objects this is just a Python dictionary. However, if a Python object implements the __reduce__ method it could be anything.

If your resolver can't identify the type named by module & string, just return stalecucumber.ErrUnresolvablePythonGlobal. Otherwise convert the args into whatever you want and return that as the value from the function with nil for the error.

To avoid reimplementing the same logic over and over, you can chain resolvers together. You can use your resolver in addition to the default resolver by doing the following

UnpickleWithResolver(reader, MakePythonResolverChain(customResolver, PythonBuiltinResolver{}))

Protocol Performance

If the version of Python you are using supports protocol version 1 or 2, you should always specify that protocol version. By default the "pickle" and "cPickle" modules in Python write using protocol 0. Protocol 0 requires much more space to represent the same values and is much slower to parse.

Unsupported Opcodes

The pickle format is incredibly flexible and as a result has some features that are impractical or unimportant when implementing a reader in another language.

Each set of opcodes is listed below by protocol version with the impact.

Protocol 0

PERSID

This opcode is used to reference concrete definitions of objects between a pickler and an unpickler by an ID number. The pickle protocol doesn't define what a persistent ID means.

This opcode is unlikely to ever be supported by this package.

Protocol 1

OBJ

This opcodes is used in recreating pickled python objects. That is currently not supported by this package.

This opcode will supported in a future revision to this package that allows the unpickling of instances of Python classes.

BINPERSID

This opcode is equivalent to PERSID in protocol 0 and won't be supported for the same reason.

Protocol 2

NEWOBJ

This opcodes is used in recreating pickled python objects. That is currently not supported by this package.

This opcode will supported in a future revision to this package that allows the unpickling of instances of Python classes.

EXT1
EXT2
EXT4

These opcodes allow using a registry of popular objects that are pickled by name, typically classes. It is envisioned that through a global negotiation and registration process, third parties can set up a mapping between ints and object names.

These opcodes are unlikely to ever be supported by this package.

Index

Package Files

helpers.go jump_list.go opcodes.go pickle_machine.go pickle_writer.go populate_jump_list.go protocol_0.go protocol_1.go protocol_2.go python_builtin_resolver.go python_resolver.go python_types.go sentinel.go tuple.go unpack.go

Constants

const BININT_MAX = (1 << 31) - 1
const BININT_MIN = 0 - BININT_MAX
const OPCODE_APPEND = 0x61
const OPCODE_APPENDS = 0x65
const OPCODE_BINFLOAT = 0x47
const OPCODE_BINGET = 0x68
const OPCODE_BININT = 0x4a
const OPCODE_BININT1 = 0x4b
const OPCODE_BININT2 = 0x4d
const OPCODE_BINPERSID = 0x51
const OPCODE_BINPUT = 0x71
const OPCODE_BINSTRING = 0x54
const OPCODE_BINUNICODE = 0x58
const OPCODE_BUILD = 0x62
const OPCODE_DICT = 0x64
const OPCODE_DUP = 0x32
const OPCODE_EMPTY_DICT = 0x7d
const OPCODE_EMPTY_LIST = 0x5d
const OPCODE_EMPTY_TUPLE = 0x29
const OPCODE_EXT1 = 0x82
const OPCODE_EXT2 = 0x83
const OPCODE_EXT4 = 0x84
const OPCODE_FLOAT = 0x46
const OPCODE_GET = 0x67
const OPCODE_GLOBAL = 0x63
const OPCODE_INST = 0x69
const OPCODE_INT = 0x49
const OPCODE_LIST = 0x6c
const OPCODE_LONG = 0x4c
const OPCODE_LONG1 = 0x8a
const OPCODE_LONG4 = 0x8b
const OPCODE_LONG_BINGET = 0x6a
const OPCODE_LONG_BINPUT = 0x72
const OPCODE_MARK = 0x28
const OPCODE_NEWFALSE = 0x89
const OPCODE_NEWOBJ = 0x81
const OPCODE_NEWTRUE = 0x88
const OPCODE_NONE = 0x4e
const OPCODE_OBJ = 0x6f
const OPCODE_PERSID = 0x50
const OPCODE_POP = 0x30
const OPCODE_POP_MARK = 0x31
const OPCODE_PROTO = 0x80
const OPCODE_PUT = 0x70
const OPCODE_REDUCE = 0x52
const OPCODE_SETITEM = 0x73
const OPCODE_SETITEMS = 0x75
const OPCODE_SHORT_BINSTRING = 0x55
const OPCODE_STOP = 0x2e
const OPCODE_STRING = 0x53
const OPCODE_TUPLE = 0x74
const OPCODE_TUPLE1 = 0x85
const OPCODE_TUPLE2 = 0x86
const OPCODE_TUPLE3 = 0x87
const OPCODE_UNICODE = 0x56
const PICKLE_TAG = "pickle"

Variables

var ErrEmptyInterfaceNotPickleable = errors.New("The empty interface is not pickleable")
var ErrInputTruncated = errors.New("Input to the pickle machine was truncated")
var ErrMarkNotFound = errors.New("Mark could not be found on the stack")
var ErrNilPointer = errors.New("Destination cannot be a nil pointer")
var ErrNoResult = errors.New("Input did not place a value onto the stack")
var ErrNotPointer = errors.New("Destination must be a pointer type")
var ErrOpcodeInvalid = errors.New("Opcode is invalid")
var ErrOpcodeNotImplemented = errors.New("Input encountered opcode that is not implemented")
var ErrOpcodeStopped = errors.New("STOP opcode found")
var ErrStackTooSmall = errors.New("Stack is too small to perform requested operation")
var ErrTargetTypeMismatch = errors.New("Target type does not match source type")
var ErrTargetTypeNotPointer = errors.New("Target type must be a pointer to unpack this value")
var ErrTargetTypeOverflow = errors.New("Value overflows target type")
var ErrTypeNotPickleable = errors.New("Can't pickle this type")
var ErrUnresolvablePythonGlobal = errors.New("Unresolvable Python global value")

func Big Uses

func Big(v interface{}, err error) (*big.Int, error)

This helper attempts to convert the return value of Unpickle into a *big.Int.

If Unpickle returns an error that error is returned immediately.

If the value cannot be converted an error is returned.

func Bool Uses

func Bool(v interface{}, err error) (bool, error)

This helper attempts to convert the return value of Unpickle into a bool.

If Unpickle returns an error that error is returned immediately.

If the value cannot be converted an error is returned.

func Dict Uses

func Dict(v interface{}, err error) (map[interface{}]interface{}, error)

This helper attempts to convert the return value of Unpickle into a map[interface{}]interface{}.

If Unpickle returns an error that error is returned immediately.

If the value cannot be converted an error is returned.

func DictString Uses

func DictString(v interface{}, err error) (map[string]interface{}, error)

This helper attempts to convert the return value of Unpickle into a map[string]interface{}.

If Unpickle returns an error that error is returned immediately.

If the value cannot be converted an error is returned.

func Float Uses

func Float(v interface{}, err error) (float64, error)

This helper attempts to convert the return value of Unpickle into a float64.

If Unpickle returns an error that error is returned immediately.

If the value cannot be converted an error is returned.

func Int Uses

func Int(v interface{}, err error) (int64, error)

This helper attempts to convert the return value of Unpickle into a int64.

If Unpickle returns an error that error is returned immediately.

If the value cannot be converted an error is returned.

func ListOrTuple Uses

func ListOrTuple(v interface{}, err error) ([]interface{}, error)

This helper attempts to convert the return value of Unpickle into a []interface{}.

If Unpickle returns an error that error is returned immediately.

If the value cannot be converted an error is returned.

func Set Uses

func Set(v interface{}, err error) (map[interface{}]bool, error)

This helper attempts to convert the return value of Unpickle into a map[interface{}]bool.

If Unpickle returns an error that error is returned immediately.

If the value cannot be converted an error is returned.

func String Uses

func String(v interface{}, err error) (string, error)

This helper attempts to convert the return value of Unpickle into a string.

If Unpickle returns an error that error is returned immediately.

If the value cannot be converted an error is returned.

func UnpackInto Uses

func UnpackInto(dest interface{}) unpacker

func Unpickle Uses

func Unpickle(reader io.Reader) (interface{}, error)

Unpickle a value from a reader. This function takes a reader and attempts to read a complete pickle program from it. This is normally the output of the function "pickle.dump" from Python.

The returned type is interface{} because unpickling can generate any type. Use a helper function to convert to another type without an additional type check.

This function returns an error if the reader fails, the pickled data is invalid, or if the pickled data contains an unsupported opcode. See unsupported opcodes in the documentation of this package for more information.

Type Conversions

Types conversion Python types to Go types is performed as followed

int -> int64
string -> string
unicode -> string
float -> float64
long -> big.Int from the "math/big" package
lists -> []interface{}
tuples -> []interface{}
dict -> map[interface{}]interface{}

The following values are converted from Python to the Go types

True & False -> bool
None -> stalecucumber.PickleNone, sets pointers to nil

Helper Functions

The following helper functions were inspired by the github.com/garyburd/redigo package. Each function takes the result of Unpickle as its arguments. If unpickle fails it does nothing and returns that error. Otherwise it attempts to convert to the appropriate type. If type conversion fails it returns an error

String - string from Python string or unicode
Int - int64 from Python int or long
Bool - bool from Python True or False
Big - *big.Int from Python long
ListOrTuple - []interface{} from Python Tuple or List
Float - float64 from Python float
Dict - map[interface{}]interface{} from Python dictionary
DictString -
	map[string]interface{} from Python dictionary.
	Keys must all be of type unicode or string.

Unpacking into structures

If the pickled object is a python dictionary that has only unicode and string objects for keys, that object can be unpickled into a struct in Go by using the "UnpackInto" function. The "From" receiver on the return value accepts the result of "Unpickle" as its actual parameters.

The keys of the python dictionary are assigned to fields in a structure. Structures may specify the tag "pickle" on fields. The value of this tag is taken as the key name of the Python dictionary value to place in this field. If no field has a matching "pickle" tag the fields are looked up by name. If the first character of the key is not uppercase, it is uppercased. If a field matching that name is found, the value in the python dictionary is unpacked into the value of the field within the structure.

A list of python dictionaries can be unpickled into a slice of structures in Go.

A homogeneous list of python values can be unpickled into a slice in Go with the appropriate element type.

A nested python dictionary is unpickled into nested structures in Go. If a field is of type map[interface{}]interface{} it is of course unpacked into that as well.

By default UnpackInto skips any missing fields and fails if a field's type is not compatible with the object's type.

This behavior can be changed by setting "AllowMissingFields" and "AllowMismatchedFields" on the return value of UnpackInto before calling From.

func UnpickleWithResolver Uses

func UnpickleWithResolver(reader io.Reader, resolver PythonResolver) (interface{}, error)

type PickleMachine Uses

type PickleMachine struct {
    Stack  []interface{}
    Memo   []interface{}
    Reader io.Reader
    // contains filtered or unexported fields
}

This struct is current exposed but not useful. It is likely to be hidden in the near future.

func (*PickleMachine) Opcode_Invalid Uses

func (pm *PickleMachine) Opcode_Invalid() error

type PickleMachineError Uses

type PickleMachineError struct {
    Err       error
    StackSize int
    MemoSize  int
    Opcode    uint8
}

This type is returned whenever Unpickle encounters an error in pickled data.

func (PickleMachineError) Error Uses

func (pme PickleMachineError) Error() string

type PickleMark Uses

type PickleMark struct{}

This type is used internally to represent a concept known as a mark on the Pickle Machine's stack. Oddly formed pickled data could return this value as the result of Unpickle. In normal usage this type is needed only internally.

func (PickleMark) String Uses

func (_ PickleMark) String() string

type PickleNone Uses

type PickleNone struct{}

This type is used to represent the Python object "None"

func (PickleNone) String Uses

func (_ PickleNone) String() string

type PickleTuple Uses

type PickleTuple []interface{}

func NewTuple Uses

func NewTuple(v ...interface{}) PickleTuple

type Pickler Uses

type Pickler struct {
    W io.Writer
    // contains filtered or unexported fields
}

func NewPickler Uses

func NewPickler(writer io.Writer) *Pickler

This type is used to pickle data.Picklers are created by calling NewPickler. Each call to Pickle writes a complete pickle program to the underlying io.Writer object.

Its safe to assign W to other values in between calls to Pickle.

Failures return the underlying error or an instance of PicklingError.

Data is always written using Pickle Protocol 2. This format is compatible with Python 2.3 and all newer version.

Type Conversions

Type conversion from Go types to Python types is as follows

uint8,uint16,int8,int16,int32 -> Python int
int,int64,uint,uint64 -> Python int if it fits, otherwise Python Long
string -> Python unicode
slices, arrays -> Python list
maps -> Python dict
bool -> Python True and False
big.Int -> Python Long
struct -> Python dict

Structs are pickled using their field names unless a tag is present on the field specifying the name. For example

type MyType struct {
	FirstField int
	SecondField int `pickle:"meow"`
}

This struct would be pickled into a dictionary with two keys: "FirstField" and "meow".

Embedded structs are marshalled as a nested dictionary. Exported types are never pickled.

Pickling Tuples

There is no equivalent type to Python's tuples in Go. You may not need to use tuples at all. For example, consider the following Python code

a, b, c = pickle.load(data_in)

This code tries to set to the variables "a", "b", and "c" from the result of unpickling. In this case it does not matter if the source type is a Python list or a Python tuple.

If you really need to write tuples, call NewTuple and pass the data in as the arguments. This special type exists to inform stalecucumber.Pickle that a tuple should be pickled.

func (*Pickler) Pickle Uses

func (p *Pickler) Pickle(v interface{}) (int, error)

type PicklingError Uses

type PicklingError struct {
    V   interface{}
    Err error
}

func (PicklingError) Error Uses

func (pe PicklingError) Error() string

type PythonBuiltinResolver Uses

type PythonBuiltinResolver struct{}

func (PythonBuiltinResolver) Resolve Uses

func (this PythonBuiltinResolver) Resolve(module string, name string, args []interface{}) (interface{}, error)

type PythonResolver Uses

type PythonResolver interface {
    Resolve(module string, name string, args []interface{}) (interface{}, error)
}

A type to convert to a GLOBAL opcode to something meaningful in golang

type PythonResolverChain Uses

type PythonResolverChain []PythonResolver

func MakePythonResolverChain Uses

func MakePythonResolverChain(args ...PythonResolver) PythonResolverChain

func (PythonResolverChain) Resolve Uses

func (this PythonResolverChain) Resolve(module string, name string, args []interface{}) (interface{}, error)

type UnbuildableValueError Uses

type UnbuildableValueError struct {
    Value interface{}
}

func (UnbuildableValueError) Error Uses

func (this UnbuildableValueError) Error() string

type UnpackingError Uses

type UnpackingError struct {
    Source      interface{}
    Destination reflect.Value
    Err         error
}

func (UnpackingError) Error Uses

func (ue UnpackingError) Error() string

This type is returned when a call to From() fails. Setting "AllowMissingFields" and "AllowMismatchedFields" on the result of "UnpackInto" controls if this error is returned or not.

type UnparseablePythonGlobalError Uses

type UnparseablePythonGlobalError struct {
    Args    interface{}
    Message string
}

func (UnparseablePythonGlobalError) Error Uses

func (this UnparseablePythonGlobalError) Error() string

type UnreducibleValueError Uses

type UnreducibleValueError struct {
    Value interface{}
}

func (UnreducibleValueError) Error Uses

func (this UnreducibleValueError) Error() string

type WrongTypeError Uses

type WrongTypeError struct {
    Result  interface{}
    Request string
}

This type is returned whenever a helper cannot convert the result of Unpickle into the desired type.

func (WrongTypeError) Error Uses

func (wte WrongTypeError) Error() string

Directories

PathSynopsis
struct_export_test

Package stalecucumber imports 10 packages (graph) and is imported by 28 packages. Updated 2018-02-26. Refresh now. Tools for package owners.