msgpack

package module
v0.0.0-...-6b0aecf Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 31, 2013 License: BSD-3-Clause Imports: 17 Imported by: 0

README

MsgPack library for Go.

Implements:

[http://wiki.msgpack.org/display/MSGPACK/Format+specification]

To install:

go get github.com/ugorji/go-msgpack

It provides features similar to encoding packages in the standard library (ie json, xml, gob, etc).

Supports:

  • Standard Marshal/Unmarshal interface.
  • Support for all exported fields (including anonymous fields)
  • Standard field renaming via tags
  • Encoding from any value (struct, slice, map, primitives, pointers, interface{}, etc)
  • Decoding into pointer to any non-nil value (struct, slice, map, int, float32, bool, string, etc)
  • Decoding into a nil interface{}
  • Handles time.Time transparently (stores time as 2 element array: seconds since epoch and nanosecond offset)
  • Provides a Server and Client Codec so msgpack can be used as communication protocol for net/rpc.

Usage


    dec = msgpack.NewDecoder(r, nil)  
    err = dec.Decode(&v)  
    
    enc = msgpack.NewEncoder(w)  
    err = enc.Encode(v)  
    
    //methods below are convenience methods over functions above.  
    data, err = msgpack.Marshal(v)  
    err = msgpack.Unmarshal(data, &v, nil)  
    
    //RPC Server
    conn, err := listener.Accept()
    rpcCodec := msgpack.NewRPCServerCodec(conn, nil)
    rpc.ServeCodec(rpcCodec)

    //RPC Communication (client side)
    conn, err = net.Dial("tcp", "localhost:5555")  
    rpcCodec := msgpack.NewRPCClientCodec(conn, nil)  
    client := rpc.NewClientWithCodec(rpcCodec)  

Documentation

Overview

MsgPack library for Go.

Implements:

http://wiki.msgpack.org/display/MSGPACK/Format+specification

It provides features similar to encoding packages in the standard library (ie json, xml, gob, etc).

Supports:

  • Standard Marshal/Unmarshal interface.
  • Standard field renaming via tags
  • Encoding from any value (struct, slice, map, primitives, pointers, interface{}, etc)
  • Decoding into pointer to any non-nil value (struct, slice, map, int, float32, bool, string, etc)
  • Decoding into a nil interface{}
  • Handles time.Time transparently
  • Provides a Server and Client Codec so msgpack can be used as communication protocol for net/rpc. Also includes an option for msgpack-rpc: http://wiki.msgpack.org/display/MSGPACK/RPC+specification

Usage

dec = msgpack.NewDecoder(r, nil)
err = dec.Decode(&v)

enc = msgpack.NewEncoder(w)
err = enc.Encode(v)

//methods below are convenience methods over functions above.
data, err = msgpack.Marshal(v)
err = msgpack.Unmarshal(data, &v, nil)

//RPC Server
conn, err := listener.Accept()
rpcCodec := msgpack.NewRPCServerCodec(conn, nil)
rpc.ServeCodec(rpcCodec)

//RPC Communication (client side)
conn, err = net.Dial("tcp", "localhost:5555")
rpcCodec := msgpack.NewRPCClientCodec(conn, nil)
client := rpc.NewClientWithCodec(rpcCodec)

RPC

An RPC Client and Server Codec is implemented, so that msgpack can be used with the standard net/rpc package. It supports both a basic net/rpc serialization, and the custom format defined at http://wiki.msgpack.org/display/MSGPACK/RPC+specification

Index

Constants

View Source
const (
	ContainerRawBytes = ContainerType('b')
	ContainerList     = ContainerType('a')
	ContainerMap      = ContainerType('m')
)
View Source
const DefaultFlightLimitSize = 100

If not set in a constructor, this will limit the number of "in-flight" requests that will be processed before blocking For single connections, this should be set high to avoid blocking. When pooling clients, a lower limit may be desireable to limit the number of requests that will fail when an error occurs.

Variables

View Source
var DefaultDecoderContainerResolver = SimpleDecoderContainerResolver{
	MapType:                 nil,
	SliceType:               nil,
	BytesStringLiteral:      true,
	BytesStringSliceElement: true,
	BytesStringMapValue:     true,
}

Default DecoderContainerResolver used when a nil parameter is passed to NewDecoder(). Sample Usage:

opts := msgpack.DefaultDecoderContainerResolver // makes a copy
opts.BytesStringLiteral = false // change some options
err := msgpack.NewDecoder(r, &opts).Decode(&v)

Functions

func Marshal

func Marshal(v interface{}) (b []byte, err error)

Marshal is a convenience function which encodes v to a stream of bytes. It delegates to Encoder.Encode.

func NewCustomRPCClientCodec

func NewCustomRPCClientCodec(conn io.ReadWriteCloser, opts DecoderContainerResolver) rpc.ClientCodec

NewCustomRPCClientCodec uses msgpack serialization for rpc communication from client side, but uses a custom protocol defined at http://wiki.msgpack.org/display/MSGPACK/RPC+specification

func NewCustomRPCServerCodec

func NewCustomRPCServerCodec(conn io.ReadWriteCloser, opts DecoderContainerResolver) rpc.ServerCodec

NewCustomRPCServerCodec uses msgpack serialization for rpc communication from server side, but uses a custom protocol defined at http://wiki.msgpack.org/display/MSGPACK/RPC+specification

func NewRPCClientCodec

func NewRPCClientCodec(conn io.ReadWriteCloser, opts DecoderContainerResolver) rpc.ClientCodec

NewRPCClientCodec uses basic msgpack serialization for rpc communication from client side.

Sample Usage:

conn, err = net.Dial("tcp", "localhost:5555")
codec, err := msgpack.NewRPCClientCodec(conn, nil)
client := rpc.NewClientWithCodec(codec)
... (see rpc package for how to use an rpc client)

func NewRPCServerCodec

func NewRPCServerCodec(conn io.ReadWriteCloser, opts DecoderContainerResolver) rpc.ServerCodec

NewRPCServerCodec uses basic msgpack serialization for rpc communication from the server side.

func Unmarshal

func Unmarshal(data []byte, v interface{}, dam DecoderContainerResolver) error

Unmarshal is a convenience function which decodes a stream of bytes into v. It delegates to Decoder.Decode.

Types

type Client

type Client struct {
	ClientClosedNotification func(*Client)
	// contains filtered or unexported fields
}

The Client class reimplements the Msgpack-RPC protocol with pipelining support which is not supported by the Go net/rpc impelementation

In other words, many requests can be sent and replies may be sent before getting a reply and replies may come back out of order

func NewClient

func NewClient(conn net.Conn) *Client

func NewClientWithOptions

func NewClientWithOptions(conn net.Conn, dopts DecoderContainerResolver) *Client

func NewClientWithOptionsAndFlightLimit

func NewClientWithOptionsAndFlightLimit(conn net.Conn, dopts DecoderContainerResolver, inFlight int) *Client

func (*Client) Close

func (c *Client) Close()

Close now, dumping any connections

func (*Client) Closed

func (c *Client) Closed() *ClientClosedError

Use this to check if a client has closed and get the underlying error

func (*Client) Send

func (c *Client) Send(method string, params []interface{}) (interface{}, interface{})

Send a request and wait for a response. Under the hood, this uses SendAsync so the client can still be used by other goroutines without blocking @return the result data or error if there is an error

if the error is of type ClientClosedError, then the client has had a fatal error and can no longer be used.

func (*Client) SendAsync

func (c *Client) SendAsync(method string, params []interface{}, replyChan chan ClientResponse) (msgid uint32, err *ClientClosedError)

Send a request but don't wait for the response. This call may block if the in-flight limit is reached

@param replyChan this channel may be new for every request, or a shared channel may be used. Be sure to

use a buffered channel or immediately wait (or have a goroutine already waiting) for the response because
the message will be thrown away if the reply cannot be sent immediately to the channel

@return message id of the request. it is useful for demultiplexing requests if using a single reply channel. err will

always be nil unless the Client has closed due to an error.  If you get a ClientClosedError

func (*Client) ShutdownGracefully

func (c *Client) ShutdownGracefully(timeout time.Duration, done chan int)

This will close the client but attempt to finish all in-flight requests first The client will closed forefully after the timeout expires

func (*Client) String

func (c *Client) String() string

type ClientClosedError

type ClientClosedError struct {
	Cause error
}

func (*ClientClosedError) Error

func (cce *ClientClosedError) Error() string

func (*ClientClosedError) String

func (cce *ClientClosedError) String() string

type ClientResponse

type ClientResponse struct {
	Msgid  uint32
	Error  interface{}
	Result interface{}
}

type ContainerType

type ContainerType byte

type Decoder

type Decoder struct {
	// contains filtered or unexported fields
}

A Decoder reads and decodes an object from an input stream in the msgpack format.

func NewDecoder

func NewDecoder(r io.Reader, dam DecoderContainerResolver) (d *Decoder)

NewDecoder returns a Decoder for decoding a stream of bytes into an object. If nil DecoderContainerResolver is passed, we use DefaultDecoderContainerResolver

func (*Decoder) Decode

func (d *Decoder) Decode(v interface{}) (err error)

Decode decodes the stream from reader and stores the result in the value pointed to by v.

If v is a pointer to a non-nil value, we will decode the stream into that value (if the value type and the stream match. For example: integer in stream must go into int type (int8...int64), etc

If you do not know what type of stream it is, pass in a pointer to a nil interface. We will decode and store a value in that nil interface.

time.Time is handled transparently, by (en)decoding (to)from a []int64{Seconds since Epoch, Nanoseconds offset}.

Sample usages:

// Decoding into a non-nil typed value
var f float32
err = msgpack.NewDecoder(r, nil).Decode(&f)

// Decoding into nil interface
var v interface{}
dec := msgpack.NewDecoder(r, nil)
err = dec.Decode(&v)

// To configure default options, see DefaultDecoderContainerResolver usage.
// or write your own DecoderContainerResolver

func (*Decoder) DecodeValue

func (d *Decoder) DecodeValue(rv reflect.Value) (err error)

DecodeValue decodes the stream into a reflect.Value. The reflect.Value must be a pointer. See Decoder.Decode documentation. (Decode internally calls DecodeValue).

type DecoderContainerResolver

type DecoderContainerResolver interface {
	// DecoderContainer is used to get a proper reflect.Value when decoding
	// a msgpack map, array or raw bytes (for which the stream defines the length and
	// corresponding containerType) into a nil interface{}.
	//
	// This may be within the context of a container: ([]interface{} or map[XXX]interface{}),
	// or just a top-level literal.
	//
	// The parentcontainer and parentkey define the context
	//   - If decoding into a map, they will be the map and the key in the map (a reflect.Value)
	//   - If decoding into a slice, they will be the slice and the index into the slice (an int)
	//   - Else they will be Invalid/nil
	//
	// Custom code can use this callback to determine how specifically to decode something.
	// A simple implementation exists which just uses some options to do it
	// (see SimpleDecoderContainerResolver).
	DecoderContainer(parentcontainer reflect.Value, parentkey interface{},
		length int, ct ContainerType) (val reflect.Value)
}

DecoderContainerResolver has the DecoderContainer method for getting a usable reflect.Value when decoding a container (map, array, raw bytes) from a stream into a nil interface{}.

type Encoder

type Encoder struct {
	// contains filtered or unexported fields
}

An Encoder writes an object to an output stream in the msgpack format.

func NewEncoder

func NewEncoder(w io.Writer) (e *Encoder)

NewDecoder returns an Encoder for encoding an object.

func (*Encoder) Encode

func (e *Encoder) Encode(v interface{}) (err error)

Encode writes an object into a stream in the MsgPack format.

time.Time is handled transparently, by (en)decoding (to)from a []int64{Seconds since Epoch, Nanoseconds offset}.

Struct values encode as maps. Each exported struct field is encoded unless:

  • the field's tag is "-", or
  • the field is empty and its tag specifies the "omitempty" option.

The empty values are false, 0, any nil pointer or interface value, and any array, slice, map, or string of length zero.

Anonymous fields are encoded inline if no msgpack tag is present. Else they are encoded as regular fields.

The object's default key string is the struct field name but can be specified in the struct field's tag value. The "msgpack" key in struct field's tag value is the key name, followed by an optional comma and options.

To set an option on all fields (e.g. omitempty on all fields), you can create a field called _struct, and set flags on it.

Examples:

type MyStruct struct {
    _struct bool    `msgpack:",omitempty"`   //set omitempty for every field
    Field1 string   `msgpack:"-"`            //skip this field
    Field2 int      `msgpack:"myName"`       //Use key "myName" in encode stream
    Field3 int32    `msgpack:",omitempty"`   //use key "Field3". Omit if empty.
    Field4 bool     `msgpack:"f4,omitempty"` //use key "f4". Omit if empty.
    ...
}

func (*Encoder) EncodeValue

func (e *Encoder) EncodeValue(rv reflect.Value) (err error)

EncodeValue encodes a reflect.Value.

type SimpleDecoderContainerResolver

type SimpleDecoderContainerResolver struct {
	// If decoding into a nil interface{} and we detect a map in the stream,
	// we create a map of the type specified. It defaults to creating a
	// map[interface{}]interface{} if not specified.
	MapType reflect.Type
	// If decoding into a nil interface{} and we detect a slice/array in the stream,
	// we create a slice of the type specified. It defaults to creating a
	// []interface{} if not specified.
	SliceType reflect.Type
	// convert to a string if raw bytes are detected while decoding
	// into a interface{},
	BytesStringLiteral bool
	// convert to a string if raw bytes are detected while decoding
	// into a []interface{},
	BytesStringSliceElement bool
	// convert to a string if raw bytes are detected while decoding
	// into a value in a map[XXX]interface{},
	BytesStringMapValue bool
}

SimpleDecoderContainerResolver is a simple DecoderContainerResolver which uses some simple options to determine how to decode into a nil interface{}. Most applications will work fine with just this.

func (SimpleDecoderContainerResolver) DecoderContainer

func (d SimpleDecoderContainerResolver) DecoderContainer(
	parentcontainer reflect.Value, parentkey interface{},
	length int, ct ContainerType) (rvn reflect.Value)

DecoderContainer supports common cases for decoding into a nil interface{} depending on the context.

When decoding into a nil interface{}, the following rules apply as we have to make assumptions about the specific types you want.

  • Maps are decoded as map[interface{}]interface{} unless you provide a default map type when creating your decoder. option: MapType
  • Lists are always decoded as []interface{} unless you provide a default slice type when creating your decoder. option: SliceType
  • raw bytes are decoded into []byte or string depending on setting of: option: BytesStringMapValue (if within a map value, use this setting) option: BytesStringSliceElement (else if within a slice, use this setting) option: BytesStringLiteral (else use this setting)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL