jsonpack

package module
v1.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 19, 2020 License: MIT Imports: 9 Imported by: 0

README

GoDoc Go Report Card License

Fast and space efficiency JSON serialization golang library. It is a schema oriented design which leverages schema definition to encode JSON document into compact binary encoded format, and decodes back into JSON document.

Introduction

When we want to exchange data between services or over network, the JSON is most popular format to do it. In golang world, the most convenient way is using official encoding/json package to marshal and unmarshal JSON document from/to struct or map. For usual RESTful web service scenario, JSON format is quite convenience and representative, but for real-time message exchanging or other scenarios that has small footprint data and low-latency requirement, JSON is a bit too heavy weight, not only data footprint is not space saving, but also has heavy loading in encode/decode procedure.

So if we want to a compact, small footprint data for exchanging over network, and also leverages the convenience of JSON, we need an encoding format that removes "property name" and other notations likes ':', '[', '{'...etc from original JSON document, and leaves "value" only.

To achieve this goal, we need a schematic to define our JSON document and provide enough information for serialization engine to know sequence and data type of every properties in document. it's the reason why JSONPack is a schema oriented design library.

Key Features

  • Similar Marshal / Unmarshal API to standard encoding/json package.
  • Space saving encoded format, the size of encoded data is similar to Protocol Buffers, can be 30-80% of original JSON document, depends on data.
  • Blazing fast, provides about 3.x decoding speed compared to protobuf and many times than other JSON packages.
  • Memory saving design, avoids any un-necessary memory allocations, suitable for embedded environment.
  • Has production ready javascript implementation Buffer Plus, can be used in node.js and Web browser environment.
  • No need to write schema definition by hand, jsonpack will generate schema definition from golang struct automatically.

How to Get

go get github.com/arloliu/jsonpack

Usage

Example of add schema definition

import "github.com/arloliu/jsonpack"

type Info struct {
	Name string `json:"name"`
	Area uint32 `json:"area"`
	// omit this field
	ExcludeField string `-`
}

jsonPack := jsonpack.NewJSONPack()
sch, err := jsonPack.AddSchema("Info", Info{}, jsonpack.LittleEndian)

Example of encoding data with Info struct

infoStruct := &Info{
    Name: "example name",
    Area: 888,
}
// encodedResult1 contains encoded data,
encodedResult1, err := jsonPack.Marshal("Info", infoStruct)

Example of encoding data with golang map

infoMap := map[string]interface{} {
	"name": "example name",
	"area": 888,
}

encodedResult2, err := jsonPack.Marshal("Info", infoMap)

Example of decoding data

decodeInfoStruct = Info{}
err := jsonPack.Decode("Info", encodedResult1, &decodeInfoStruct)

decodeInfoMap = make(map[string]interface{})
err := jsonPack.Decode("Info", encodedResult2, &decodeInfoMap)

Benchmark

The benchmark result is a important reference but not always suitable for every scenarios.

Test environment: Intel i7-9700K CPU@3.60GHz.

Benchmark code is here

Sorts from fastest to slowest in the following.

Encode from golang map

ns/op allocation bytes allocation times
jsonpack 1933 ns/op 752 B/op 2 allocs/op
jsoniter 10134 ns/op 3320 B/op 46 allocs/op
std. json 23560 ns/op 8610 B/op 171 allocs/op
goccy 75298 ns/op 82639 B/op 651 allocs/op

Decode into golang map

ns/op allocation bytes allocation times
jsonpack 6461 ns/op 6512 B/op 96 allocs/op
jsoniter 17436 ns/op 9666 B/op 290 allocs/op
std. json 18949 ns/op 8864 B/op 228 allocs/op
goccy 19985 ns/op 15900 B/op 316 allocs/op

Encode from golang struct

ns/op allocation bytes allocation times
jsonpack 1834 ns/op 800 B/op 3 allocs/op
protobuf 1972 ns/op 896 B/op 1 allocs/op
goccy 2166 ns/op 1280 B/op 1 allocs/op
jsoniter 3372 ns/op 1296 B/op 3 allocs/op
std. json 3578 ns/op 1280 B/op 1 allocs/op

Decode into golang struct

ns/op allocation bytes allocation times
jsonpack 1475 ns/op 96 B/op 2 allocs/op
goccy 3284 ns/op 2215 B/op 5 allocs/op
jsoniter 4680 ns/op 1072 B/op 79 allocs/op
protobuf 5075 ns/op 3152 B/op 84 allocs/op
std. json 18378 ns/op 1232 B/op 69 allocs/op

The benchmark result indicates jsonpack keeps constant performance on both encoding and encoding side, and keeps very low memory allocation size and times.

Explanation of Benchmark Result

The benchmark result also delivers some important messages.

  1. The performance of operating with golang map sucks.
  2. The pointer of type likes *string, *int32 cost twice of time.
  3. Golang reflection is expensive, reflect.ValueOf(...) is very expensive.

Performance Tips

According to the explanation of benchmarking and some tests during the implementation stage of this library, there are some suggestions in the following.

  1. It's better to use struct instead of map for encode/decode if possible.
  2. When declares struct fields, declares it as value instead of pointer to value gives performance gain.
  3. Designs a small and compacted structure, uses number type to represent enum. fields instead of declares it as string type.

Documentation

Overview

Package jsonpack is a fast and space efficiency JSON serialization golang library. It is a schema oriented design which leverages schema definition to encode JSON document into compact binary encoded format, and decodes back into JSON document.

Introduction

When we want to exchange data between services or over network, the JSON is most popular format to do it. In golang world, the most convenient way is using official `encoding/json` package to marshal and unmarshal JSON document from/to struct or map. For usual RESTful web service scenario, JSON format is quite convenience and representative, but for real-time message exchanging or other scenarios that has small footprint data and low-letency requirement, JSON is a bit too heavyweight, not only data footprint is not space saving, but also has heavy loading in encode/decode procedure.

So if we want to a compact, small footprint data for exchanging over network, and also leverages the convenience of JSON, we need an encoding format that removes "property name" and other notations likes ':', '[', '{'...etc from original JSON document, and leaves "value" only.

To achieve this goal, we need a schematic to define our JSON document and provide enough information for serialization engine to know sequence and data type of every properties in document. it's the reason why jsonpack is a schema oriented design library.

Key Features

* Similar Marshal / Unmarshal API to standard `encoding/json` package.

* Space saving encoded format, the size of encoded data is similar to Protocol Buffers, can be 30-80% of original JSON document, depends on data.

* Blazing fast, provides about 3.x decoding speed compared to `protobuf` and many times than other JSON packages.

* Memory saving design, avoids any un-necessary memory allocations, suitable for embedded environment.

* Has production ready javascript implementation https://github.com/arloliu/buffer-plus, can be used in node.js and Web browser environment.

* No need to write schema definition by hand, `jsonpack` will generate schema definition from golang struct automatically.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type ByteOrder 

type ByteOrder int

ByteOrder represents the byte order of numeric type that will be encoded to and decoded from. Possible values are LittleEndian and BigEndian 

const (
	// little endian byte order.
	LittleEndian ByteOrder = 0
	// big endian byte order.
	BigEndian ByteOrder = 1
)

byte order for decode/encode,

type CompileError 

type CompileError struct {
	Name string // schema name
	Err  error  // actual error
}

CompileError represents an error from calling AddSchema method, it indicates there has an error occurs in compiling procedure of schema definition.

func (*CompileError) Error 

func (e *CompileError) Error() string

func (*CompileError) Unwrap 

func (e *CompileError) Unwrap() error

Unwrap returns the underlying error.

type DecodeError 

type DecodeError struct {
	Name string // schema name
	Err  error  // actual error
}

DecodeError represents an error from calling Decode or Unmarshal methods.

func (*DecodeError) Error 

func (e *DecodeError) Error() string

func (*DecodeError) Unwrap 

func (e *DecodeError) Unwrap() error

Unwrap returns the underlying error.

type EncodeError 

type EncodeError struct {
	Name string // schema name
	Err  error  // actual error
}

EncodeError represents an error from calling Encode or Marshal methods.

func (*EncodeError) Error 

func (e *EncodeError) Error() string

func (*EncodeError) Unwrap 

func (e *EncodeError) Unwrap() error

Unwrap returns the underlying error.

type JSONPack 

type JSONPack struct {
	// contains filtered or unexported fields
}

JSONPack provides top-level operations for schema.

func NewJSONPack 

func NewJSONPack() *JSONPack

NewJSONPack returns a new jsonpack instance.

func (*JSONPack) AddSchema 

func (p *JSONPack) AddSchema(schemaName string, v ...interface{}) (*Schema, error)

AddSchema compiles schema definition and stores compiled result in internal schema manager.

It's a variadic function which accepts two input argument forms in the following.

AddSchema(schemaName string, v interface{})

The v is schema definition which want to compile. The value of v can be a JSON document with []byte/string type, a map represents JSON format of schema definition, or a SchemaDef struct which represents a schema definition.

Example of add new schema from JSON document: 

schDef := `
{
	"type": "object",
	"properties": {
		"name": {"type": "string"},
		"area": {"type": "uint32le"}
	},
	"order": ["name", "area"]
}
`

jsonPack := jsonpack.NewJSONPack()
sch, err := jsonPack.AddSchema("info", schDef)

Example of adding new schema from a map of schema definition: 

schDef := map[string]interface{}{
	"type": "object",
	"properties": map[string]interface{}{
		"name": map[string]interface{}{"type": "string"},
		"area": map[string]interface{}{"type": "uint32le"},
	},
	"order": []string{"name", "area"},
}

jsonPack := jsonpack.NewJSONPack()
sch, err := jsonPack.AddSchema("info", schDef)

Example of adding new schema from SchemaDef struct: 

schDef := jsonpack.SchemaDef{
	Type: "object",
	Properties: map[string]*jsonpack.SchemaDef{
		"name": {Type: "string"},
		"area": {Type: "uint32le"},
	},
	Order: []string{"name", "area"},
}
jsonPack := jsonpack.NewJSONPack()
sch, err := jsonPack.AddSchema("info", schDef)

AddSchema(schemaName string, v interface{}, byteOrder jsonpack.ByteOrder)

For fast prototyping, AddSchema method supports generate schema definition from existing struct without writing schema definition by hand.

In this scenario, the value of v is the target struct which to be generated, and byteOrder parameter indicates the byte order, can be either jsonpack.LittleEndian or jsonpack.BigEndian, it defaults to little-endian if not specified.

This method supports struct tag, use the same format as standard encoding/json excepts "omitempty" option, the "omitempty" option will be ignored.

Example of adding new schema and build schema definition from struct: 

type Info struct {
	Name string `json:"name"`
	// "omitempty" option will be ignore, so this field will be not be omitted
	Area uint32 `json:"area,omitempty"`
	ExcludeField string `-` // this field is ignored
}

jsonPack := jsonpack.NewJSONPack()
sch, err := jsonPack.AddSchema("Info", Info{}, jsonpack.BigEndian)

func (*JSONPack) Decode 

func (p *JSONPack) Decode(schemaName string, data []byte, v interface{}) error

Decode is a wrapper of Schema.Decode, it returns *SchemaNonExistError error if schema not found.

func (*JSONPack) Encode 

func (p *JSONPack) Encode(schemaName string, v interface{}) ([]byte, error)

Encode is a wrapper of Schema.Encode, it returns *SchemaNonExistError error if schema not found.

func (*JSONPack) EncodeTo 

func (p *JSONPack) EncodeTo(schemaName string, v interface{}, dataPtr *[]byte) error

EncodeTo is a wrapper of Schema.EncodeTo, it returns *SchemaNonExistError error if schema not found.

func (*JSONPack) GetAllSchemaDefTexts 

func (p *JSONPack) GetAllSchemaDefTexts() map[string][]byte

GetAllSchemaDefTexts returns a map which contains all existed schema text definitions, key of map it schema name, and value of map is text format of schema definition which presented as []byte.

func (*JSONPack) GetAllSchemaDefs 

func (p *JSONPack) GetAllSchemaDefs() map[string]*SchemaDef

GetAllSchemaDefs returns a map which contains all existed schema definitions, key of map it schema name, and value of map is *SchemaDef.

func (*JSONPack) GetAllSchemas 

func (p *JSONPack) GetAllSchemas() map[string]*Schema

GetAllSchemas returns a map which contains all existed schema instances, the key of map it schema name, and value of map is *Schema.

func (*JSONPack) GetSchema 

func (p *JSONPack) GetSchema(schemaName string) *Schema

GetSchema returns a schema instance by schemaName, returns nil if schema not found.

func (*JSONPack) GetSchemaDef 

func (p *JSONPack) GetSchemaDef(schemaName string) (*SchemaDef, error)

GetSchemaDef is a wrapper of Schema.GetSchemaDef, it gets a Schema instance by schemaName, it returns *SchemaNonExistError error if schema not found.

func (*JSONPack) GetSchemaDefText 

func (p *JSONPack) GetSchemaDefText(schemaName string) ([]byte, error)

GetSchemaDefText is a wrapper of Schema.GetSchemaDefText, it returns *SchemaNonExistError error if schema not found.

func (*JSONPack) Marshal 

func (p *JSONPack) Marshal(schemaName string, v interface{}) ([]byte, error)

Marshal is an alias to Encode function, provides familiar interface of standard json package.

func (*JSONPack) RemoveSchema 

func (p *JSONPack) RemoveSchema(schemaName string) error

RemoveSchema removes schema by schemaName, it returns *SchemaNonExistError error if schema not found.

func (*JSONPack) Reset 

func (p *JSONPack) Reset()

Reset removes all schema instances

func (*JSONPack) Unmarshal 

func (p *JSONPack) Unmarshal(schemaName string, data []byte, v interface{}) error

Unmarshal is an alias to Decode function, provides familiar interface of standard json package.

type NotImplementedError 

type NotImplementedError struct {
	Name string // operation name that not implemented by handler
}

NotImplementedError is returned when the operation in handler doesn't implemented.

func (*NotImplementedError) Error 

func (e *NotImplementedError) Error() string

type Schema 

type Schema struct {
	// schema name
	Name string
	// contains filtered or unexported fields
}

Schema represents a compiled jsonpack schema instance which created by jsonpack.AddSchema function

func (*Schema) Decode 

func (s *Schema) Decode(data []byte, v interface{}) (err error)

Decode reads encoded data with compiled schema definition and stores the result in the value pointed to v.

If type of v is not a pointer type that pointed to a map or struct, Decode function will return DecodeError.

The valid type of v is either a *map[string]interface{} or a pointer to the struct which added by AddSchema function.

Example of decoding data into map with "Info" schema 

decodeInfoMap = make(map[string]interface{})
err := jsonPack.Decode("Info", encodedData, &decodeInfoMap)

Example of decoding data into Info struct instance with "Info" schema 

decodeInfoStruct = Info{}
err := jsonPack.Decode("Info", encodedData, &decodeInfoStruct)

func (*Schema) Encode 

func (s *Schema) Encode(d interface{}) ([]byte, error)

Encode returns packed binary data of v with compiled schema definition. The compiled schema definition is specified by schemaName and return data will be nil if error occurs.

The type of v can be a map[string]interface{} which represents valid JSON data, or a struct instance which added by AddSchema function.

The return data contains encoded binary data that can then be decoded by Decode function.

Before encode data into jsonpack encoding format, we need to create a jsonpack instance and create a schema. 

// the Info struct we want to decode
type Info struct {
	Name string `json:"name"`
	Area uint32 `json:"area"`
	// omit this field
	ExcludeField string `-`
}
// create a new jsonpack instance
jsonPack := jsonpack.NewJSONPack()
// create schema with Info struct
sch, err := jsonPack.AddSchema("Info", Info{}, jsonpack.LittleEndian)

If we want to encode a map with Info schema. 

data := map[string]interface{} {
	"name": "example name",
	"area": uint32(888),
}

// encodes data into encodedResult
encodedResult, err := sch.Encode(data)

Or if we want to encode a Info struct instance with Info schema. 

data := &Info{
	Name: "example name",
	Area: 888,
}

// encodes data into encodedResult
encodedResult, err := sch.Encode(infoStruct)

func (*Schema) EncodeTo 

func (s *Schema) EncodeTo(d interface{}, dataPtr *[]byte) (err error)

EncodeTo is similar to Encode function, but passing a pointer to []byte to store encoded data instead of returning new allocated []byte encoded data.

This method is useful with buffer pool for saving memory allocation usage and improving performance.

Caution: the encoder might re-allocate and grow the slice if necessary, the length and capacity of slice might be changed.

func (*Schema) GetSchemaDef 

func (s *Schema) GetSchemaDef() (*SchemaDef, error)

GetSchemaDef returns a schema definition instance, returns nil and error if error occurs.

func (*Schema) GetSchemaDefText 

func (s *Schema) GetSchemaDefText() []byte

GetSchemaDefText returns JSON document of schema definition that represented as []byte type.

func (*Schema) Marshal 

func (s *Schema) Marshal(d interface{}) ([]byte, error)

Marshal is an alias to Encode function, provides familiar interface of json package

func (*Schema) SetEncodeBufSize 

func (s *Schema) SetEncodeBufSize(size int64)

SetEncodeBufSize sets default allocation byte size of encode buffer.

The encode buffer will be allocated with this value when calling Encode/Marshall method, and will be re-allocate and growed automatically if necessary. The encoder will also adjust this value by latest encoded result.

Sets a proper default allocation size might help to reduce re-allocation frequency and saves memory usage.

func (*Schema) Unmarshal 

func (s *Schema) Unmarshal(data []byte, v interface{}) (err error)

Unmarshal is an alias to Decode function, provides familiar interface of json package

type SchemaDef 

type SchemaDef struct {
	Type       string                `json:"type"`
	Properties map[string]*SchemaDef `json:"properties,omitempty"`
	Items      *SchemaDef            `json:"items,omitempty"`
	Order      []string              `json:"order,omitempty"`
}

SchemaDef represents a schema definition that defines the structure of JSON document.

Example: 

schDef := SchemaDef{
	Type: "object",
	Properties: map[string]*jsonpack.SchemaDef{
		"name": {Type: "string"},
		"area": {Type: "uint32le"},
	},
	Order: []string{"name", "area"},
}

type SchemaNonExistError 

type SchemaNonExistError struct {
	Name string // schema name
}

SchemaNonExistError indicates an error that occurs when pre-compiled schema definition does not exist.

func (*SchemaNonExistError) Error 

func (e *SchemaNonExistError) Error() string

type StructFieldNonExistError 

type StructFieldNonExistError struct {
	Name  string // name of structure
	Field string // field of structure that expects to be existed
}

StructFieldNonExistError indicates an error that occurs when structure doesn't have required field.

func (*StructFieldNonExistError) Error 

func (e *StructFieldNonExistError) Error() string

type TypeAssertionError 

type TypeAssertionError struct {
	Data         interface{} // the data that failed on type assertion
	ExpectedType string      // expected data type
}

TypeAssertionError indicates an error that occurs when doing data type asserrion.

func (*TypeAssertionError) Error 

func (e *TypeAssertionError) Error() string

type UnknownTypeError 

type UnknownTypeError struct {
	DataType string // data type that un-supported by schema definition
}

UnknownTypeError is returned when un-supported data type found, it happens in AddSchema method.

func (*UnknownTypeError) Error 

func (e *UnknownTypeError) Error() string

type WrongTypeError 

type WrongTypeError struct {
	DataType string // wrong type of data
}

WrongTypeError is returned when wrong data type found in data, it happens in Encode/Decode methods.

func (*WrongTypeError) Error 

func (e *WrongTypeError) Error() string

Source Files

View all Source files

Directories

Path Synopsis
Package buffer provides a easy way to manipulate byte slices.
 Package buffer provides a easy way to manipulate byte slices.
cmd
jsonpack-parser
jsonpack-parser is a tool to generate jsonpack schema definition from specified package and struct name.
 jsonpack-parser is a tool to generate jsonpack schema definition from specified package and struct name.
internal
logger

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.