dynobuffers

package module
v0.0.0-...-2836dd4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 30, 2020 License: MIT Imports: 13 Imported by: 0

README

codecov

Dyno Buffers

Codegen-less wrapper for FlatBuffers with get\set by name feature

Features

  • Uses FlatBuffer to read\write values from\to byte array converting to the required type described in the Scheme.
  • No codegen, no compilers, no (de)serialization. Just fields description and get\set by name.
  • In contrast to FlatBuffers tracks if the field was unset or initially not set
  • Supported types
    • int32, int64, float32, float64, bool, string, byte
    • nested objects
    • arrays
  • Scheme versioning
    • Any data written with Scheme of any version will be correctly read using Scheme of any other version
      • Written in old Scheme, read in New Scheme -> nil result on new field read, field considered as unset
      • Written in new Scheme, read in old Scheme -> no errors

Limitations

  • Only 2 cases of scheme modification are allowed: field rename and append fields to the end. This is necessary to have ability to read byte buffers in Scheme of any version
  • Written in New -> read in Old -> write in Old -> New fields are lost (todo)

Installation

go get github.com/untillpro/dynobuffers

Usage

  • Describe Scheme
    • By yaml. Field types:
      • int -> int32
      • long -> int64
      • float -> float32
      • double -> float64
      • bool -> bool
      • string -> string
      • byte -> byte
      var schemeStr = `
name: string
price: float
quantity: int
Id: long # first letter is capital -> field is mandatory
`
scheme, err := dynobuffers.YamlToScheme(schemeStr)
if err != nil {
	panic(err)
}
    • Manually 
      scheme := dynobuffers.NewScheme()
scheme.AddField("name", dynobuffers.FieldTypeString, false)
scheme.AddField("price", dynobuffers.FieldTypeFloat, false)
scheme.AddField("quantity", dynobuffers.FieldTypeInt, false)
scheme.AddField("id", dynobuffers.FieldTypeLong, true)
  • Create empty Dyno Buffer using Scheme 
    b := dynobuffers.NewBuffer(scheme)
    • panics if nil provided
  • Set\modify fields according to the Scheme 
    b.Set("price", float32(0.123))
b.Set("name", nil) // unset field
  • To bytes array 
    bytes := b.ToBytes()
  • To JSON key-value 
    jsonStr := b.ToJSON()
    Note: arrays of byte are encoded to base64 strings
  • To map key-value (JSON-compatible) 
    jsonMap := b.ToJSONMap()
  • Read Buffer from bytes using Scheme 
    b = dynobuffers.ReadBuffer(bytes, scheme)
    • panics if nil Scheme provided
  • Work with Buffer 
    value, ok := b.GetFloat32("price") // read typed. !ok -> field is unset or no such field in the scheme. Works faster and takes less memory allocations than Get()
b.Get("price") // read untyped. nil -> field is unset or no such field in the scheme
b.Set("price", nil) // set to nil means unset
bytes = b.ToBytes()
  • Load data from JSON key-value and to bytes array 
    bytes, err := b.ApplyJSONAndToBytes([]byte(`{"name": "str", "price": 0.123, "fld": null}`))
if err != nil {
	panic(err)
}
    • value is nil and field is mandatory -> error
    • value type and field type are incompatible (e.g. string provided for numeric field) -> error
    • value type and field type differs but value fits into field (e.g. float64(255) fits into float, double, int, long, byte; float64(256) does not fit into byte etc) -> ok
    • no such field in the scheme -> error
    • array element value is nil -> error (not supported)
  • Check if a field exists in the scheme and is set to non-nil 
    b.HasValue("name")
  • See dynobuffers_test.go for usage examples

Nested objects

  • Declare scheme
    • by yaml 
      var schemeStr = `
name: string
Nested: 
  nes1: int
  Nes2: int
Id: long
`
schemeRoot := dynobuffers.YamlToScheme(schemeStr)
    • manually 
      schemeNested := dynobuffers.NewScheme()
schemeNested.AddField("nes1", dynobuffers.FieldTypeInt, false)
schemeNested.AddField("nes2", dynobuffers.FieldTypeInt, true)
schemeRoot := dynobuffers.NewScheme()
schemeRoot.AddField("name", dynobuffers.FieldTypeString, false)
schemeRoot.AddNested("nested", schemeNested, true)
schemeRoot.AddField("id", dynobuffers.FieldTypeLong, true)
  • Create Buffer, fill and to bytes 
    bRoot := dynobuffers.NewBuffer(schemeRoot)
bNested := dynobuffers.NewBuffer(schemeNested)
bNested.Set("nes1", 1)
bNested.Set("nes2", 2)
bRoot.Set("name", "str")
bRoot.Set("nested", bNested)
bytes, err := bRoot.ToBytes()
if err != nil {
	panic(err)
}
  • Read from bytes, modify and to bytes again 
    bRoot = dynobuffers.ReadBuffer(bytes, schemeRoot)
bRoot.Set("name", "str modified")
bNested := bRoot.Get("nested").(*Buffer)
bNested.Set("nes2", 3)
bytes, err := bRoot.ToBytes()
if err != nil {
	panic(err)
}
bRoot = dynobuffers.ReadBuffer(bytes, scheme)
// note: bNested is obsolete here. Need to obtain it again from bRoot
  • See dynobuffers_test.go for usage examples

Arrays

  • Declare scheme
    • by yaml. Append .. to field name to make it array 
      var schemeStr = `
name: string
Nested..: 
  nes1: int
  Nes2: int
Ids..: long
`
schemeRoot := dynobuffers.YamlToScheme(schemeStr)
    • manually 
      schemeNested := dynobuffers.NewScheme()
schemeNested.AddField("nes1", dynobuffers.FieldTypeInt, false)
schemeNested.AddField("nes2", dynobuffers.FieldTypeInt, true)
schemeRoot := dynobuffers.NewScheme()
schemeRoot.AddField("name", dynobuffers.FieldTypeString, false)
schemeRoot.AddNestedArray("nested", schemeNested, true)
schemeRoot.AddArray("ids", dynobuffers.FieldTypeLong, true)
  • Create Buffer, fill and to bytes 
    bRoot := dynobuffers.NewBuffer(schemeRoot)
buffersNested := make([]*Buffer, 2)

bNested := dynobuffers.NewBuffer(schemeNested)
bNested.Set("nes1", 1)
bNested.Set("nes2", 2)
buffersNested = append(buffersNested, bNested)

bNested = dynobuffers.NewBuffer(schemeNested)
bNested.Set("nes1", 3)
bNested.Set("nes2", 4)
buffersNested = append(buffersNested, bNested)

ids := []int64{5,6}
bRoot.Set("name", "str")
bRoot.Set("nested", buffersNested)
bRoot.Set("ids", ids)
bytes, err := bRoot.ToBytes()
if err != nil {
	panic(err)
}
  • Read array
    • By index 
      bRoot = dynobuffers.ReadBuffer(bytes, schemeRoot)
assert.Equal(t, int64(5), bRoot.GetByIndex("ids", 0))
assert.Equal(t, int32(1), bRoot.GetByIndex("nested", 0).(*Buffer).Get("nes1"))
    • Read filled array of non-objects 
      bRoot = dynobuffers.ReadBuffer(bytes, schemeRoot)
arr := b.Get("ids").([]int64)
    • Read array of objects using iterator 
      bRoot = dynobuffers.ReadBuffer(bytes, schemeRoot)
arr := bRoot.Get("nested").(*dynobuffers.ObjectArray)
for arr.Next() {
	// arr.Buffer is switched on each arr.Next()
	assert.Equal(t, int32(1), arr.Buffer.Get("nes1"))
}
  • Modify array and to bytes
    • Set 
      bRoot = dynobuffers.ReadBuffer(bytes, schemeRoot)

ids := []int64{5,6}
bRoot.Set("ids", ids)

arr := bRoot.Get("nested").(*dynobuffers.ObjectArray)
arr.Next()
arr.Buffer.Set("nes1", -1)
bytes, err := bRoot.ToBytes()
if err != nil {
	panic(err)
}
bRoot = dynobuffers.ReadBuffer(bytes, scheme)
// note: elements of `buffers` array are obsolete here. Need to obtain the array again from bRoot
    • Append 
      bRoot = dynobuffers.ReadBuffer(bytes, schemeRoot)
bRoot.Append("ids", []int32{7,8}) // if wasn't set then equivalent to Set()
bytes, err := bRoot.ToBytes()
if err != nil {
	panic(err)
}
  • Nils as array elements are not supported
  • Byte arrays are decoded to JSON as base64 strings
  • Byte array value could be set from either byte array and base64-encoded string
  • See dynobuffers_test.go for usage examples

Benchmarks

Reading Many Fields

  • cd benchmarks
  • go test -bench=ReadAllArticleFields -benchmem
goos: windows
goarch: amd64
pkg: github.com/untillpro/dynobuffers/benchmarks
Benchmark_ReadAllArticleFields_Avro-8                      71046             23394 ns/op           11257 B/op        149 allocs/op
Benchmark_ReadAllArticleFields_Dyno_Untyped-8             195801              6437 ns/op             808 B/op         84 allocs/op
Benchmark_ReadAllArticleFields_Dyno_Typed-8               313376              3776 ns/op               0 B/op          0 allocs/op
Benchmark_ReadAllArticleFields_Flat-8                    1000000              1132 ns/op               0 B/op          0 allocs/op
Benchmark_ReadAllArticleFields_Json-8                      14431             87331 ns/op           14145 B/op        603 allocs/op

Reading Few of Many Fields

  • cd benchmarks
  • go test -bench=ReadFewArticleFields -benchmem
goos: windows
goarch: amd64
pkg: github.com/untillpro/dynobuffers/benchmarks
Benchmark_ReadFewArticleFields_Avro-8              98437             19311 ns/op           11257 B/op        149 allocs/op
Benchmark_ReadFewArticleFields_Dyno_Typed-8     18520500                62.2 ns/op             0 B/op          0 allocs/op
Benchmark_ReadFewArticleFields_Flat-8           60032416                19.8 ns/op             0 B/op          0 allocs/op
Benchmark_ReadFewArticleFields_Json-8              15333             83824 ns/op           11073 B/op        603 allocs/op

Reading Few Fields

  • cd benchmarks
  • go test -bench=ReadSimple -benchmem
goos: windows
goarch: amd64
pkg: github.com/untillpro/dynobuffers/benchmarks
Benchmark_ReadSimple_Avro-8              2038466              1193 ns/op             744 B/op         10 allocs/op
Benchmark_ReadSimple_Dyno_Typed-8       20017480                55.2 ns/op             0 B/op          0 allocs/op
Benchmark_ReadSimple_Flat-8             59981404                20.2 ns/op             0 B/op          0 allocs/op
Benchmark_ReadSimple_Flat_String-8      29275360                40.7 ns/op             0 B/op          0 allocs/op
Benchmark_ReadSimple_Json-8               545769              2423 ns/op             776 B/op         21 allocs/op

NOTE: DynoBuffers allocs caused by string types

Documentation

Index

Constants

View Source 
const DefaultBufferSize = 10

Variables

View Source 
var BufferPool = sync.Pool{
	New: func() interface{} {
		return &Buffer{
			builder:        flatbuffers.NewBuilder(0),
			modifiedFields: make([]*modifiedField, DefaultBufferSize),
		}
	},
}
View Source 
var BuffersPool = sync.Pool{
	New: func() interface{} {
		return &BuffersSlice{
			Slice: make([]*Buffer, DefaultBufferSize),
		}
	},
}
View Source 
var BuilderPool = sync.Pool{
	New: func() interface{} { return flatbuffers.NewBuilder(0) },
}
View Source 
var OffsetPool = sync.Pool{
	New: func() interface{} {
		return &OffsetSlice{
			Slice: make([]offset, DefaultBufferSize),
		}
	},
}
View Source 
var StringsPool = sync.Pool{
	New: func() interface{} {
		return &StringsSlice{
			Slice: make([]string, DefaultBufferSize),
		}
	},
}
View Source 
var UOffsetPool = sync.Pool{
	New: func() interface{} {
		return &UOffsetSlice{
			Slice: make([]flatbuffers.UOffsetT, DefaultBufferSize),
		}
	},
}

Functions

func ApplyMapBuffer 

func ApplyMapBuffer(jsonMap []byte) error

Types

type Buffer 

type Buffer struct {
	Scheme *Scheme
	// contains filtered or unexported fields
}

Buffer is wrapper for FlatBuffers

func NewBuffer 

func NewBuffer(Scheme *Scheme) *Buffer

NewBuffer creates new empty Buffer

func ReadBuffer 

func ReadBuffer(bytes []byte, Scheme *Scheme) *Buffer

ReadBuffer creates Buffer from bytes using provided Scheme

func (*Buffer) Append 

func (b *Buffer) Append(name string, toAppend interface{})

Append s.e.

func (*Buffer) ApplyJSONAndToBytes 

func (b *Buffer) ApplyJSONAndToBytes(jsonBytes []byte) ([]byte, error)

ApplyJSONAndToBytes sets field values described by provided json and returns new FlatBuffer byte array See `ApplyMap` for details

func (*Buffer) ApplyMap 

func (b *Buffer) ApplyMap(data map[string]interface{}) error

func (*Buffer) Builder 

func (b *Buffer) Builder() *flatbuffers.Builder

Builder s.e.

func (*Buffer) Get 

func (b *Buffer) Get(name string) interface{}

Get returns stored field value by name. field is scalar -> scalar is returned field is an array of scalars -> []T is returned field is a nested object -> *dynobuffers.Buffer is returned field is an array of nested objects -> *dynobuffers.ObjectArray is returned field is not set, set to nil or no such field in the Scheme -> nil

func (*Buffer) GetBool 

func (b *Buffer) GetBool(name string) (bool, bool)

GetBool returns bool value by name and if the Scheme contains the field and if the value was set to non-nil

func (*Buffer) GetByIndex 

func (b *Buffer) GetByIndex(name string, index int) interface{}

GetByIndex returns array field element by its index no such field, index out of bounds, array field is not set or unset -> nil

func (*Buffer) GetByte 

func (b *Buffer) GetByte(name string) (byte, bool)

GetByte returns byte value by name and if the Scheme contains the field and if the value was set to non-nil

func (*Buffer) GetBytes 

func (b *Buffer) GetBytes() []byte

GetBytes returns underlying byte buffer

func (*Buffer) GetDouble 

func (b *Buffer) GetDouble(name string) (float64, bool)

GetDouble returns float64 value by name and if the Scheme contains the field and if the value was set to non-nil

func (*Buffer) GetFloat 

func (b *Buffer) GetFloat(name string) (float32, bool)

GetFloat returns float32 value by name and if the Scheme contains the field and if the value was set to non-nil

func (*Buffer) GetInt 

func (b *Buffer) GetInt(name string) (int32, bool)

GetInt returns int32 value by name and if the Scheme contains the field and the value was set to non-nil

func (*Buffer) GetLong 

func (b *Buffer) GetLong(name string) (int64, bool)

GetLong returns int64 value by name and if the Scheme contains the field and if the value was set to non-nil

func (*Buffer) GetNames 

func (b *Buffer) GetNames() []string

GetNames returns list of field names which values are non-nil in current buffer Set() fields are not considered fields of nested objects are not considered

func (*Buffer) GetString 

func (b *Buffer) GetString(name string) (string, bool)

GetString returns string value by name and if the Scheme contains the field and if the value was set to non-nil

func (*Buffer) HasValue 

func (b *Buffer) HasValue(name string) bool

HasValue returns if specified field exists in the scheme and its value is set to non-nil

func (*Buffer) Release 

func (b *Buffer) Release()

func (*Buffer) ReleaseFields 

func (b *Buffer) ReleaseFields()

func (*Buffer) Reset 

func (b *Buffer) Reset(bytes []byte)

Reset sets current underlying byte array and clears modified fields. Useful for *Buffer instance reuse Note: bytes must match the Buffer's scheme

func (*Buffer) Set 

func (b *Buffer) Set(name string, value interface{})

Set sets field value by name. Underlying byte array is not modified. Call ToBytes() to get modified byte array

func (*Buffer) ToBytes 

func (b *Buffer) ToBytes() ([]byte, error)

ToBytes returns new FlatBuffer byte array with fields modified by Set() and fields which initially had values Note: initial byte array and current modifications are kept

func (*Buffer) ToBytesWithBuilder 

func (b *Buffer) ToBytesWithBuilder(builder *flatbuffers.Builder) ([]byte, error)

ToBytesWithBuilder same as ToBytes but uses builder builder.Reset() is invoked

func (*Buffer) ToJSON 

func (b *Buffer) ToJSON() []byte

ToJSON returns JSON key->value string

func (*Buffer) ToJSONMap 

func (b *Buffer) ToJSONMap() map[string]interface{}

ToJSONMap returns map[string]interface{} representation of the buffer compatible to json numeric field types are kept (not float64 as json does)

type BuffersSlice 

type BuffersSlice struct {
	Slice []*Buffer
}

type Field 

type Field struct {
	Name string
	Ft   FieldType

	IsMandatory bool
	FieldScheme *Scheme // != nil for FieldTypeObject only

	IsArray bool
	// contains filtered or unexported fields
}

Field describes a Scheme field

func (*Field) QualifiedName 

func (f *Field) QualifiedName() string

QualifiedName returns ownerScheme.fieldName

type FieldType 

type FieldType int

FieldType s.e. 

const (
	// FieldTypeUnspecified - wrong type
	FieldTypeUnspecified FieldType = iota
	// FieldTypeObject field is nested Scheme
	FieldTypeObject
	// FieldTypeInt int32
	FieldTypeInt
	// FieldTypeLong int64
	FieldTypeLong
	// FieldTypeFloat float32
	FieldTypeFloat
	// FieldTypeDouble float64
	FieldTypeDouble
	// FieldTypeString variable length
	FieldTypeString
	// FieldTypeBool s.e.
	FieldTypeBool
	// FieldTypeByte byte
	FieldTypeByte
)

type ObjectArray 

type ObjectArray struct {
	Buffer *Buffer
	Len    int
	// contains filtered or unexported fields
}

ObjectArray used to iterate over array of nested objects

func (*ObjectArray) Next 

func (oa *ObjectArray) Next() bool

Next proceeds to a next nested object in the array. If true then .Buffer represents the next element

func (*ObjectArray) Value 

func (oa *ObjectArray) Value() interface{}

Value returns *dynobuffers.Buffer instance as current element

type OffsetSlice 

type OffsetSlice struct {
	Slice []offset
}

type Scheme 

type Scheme struct {
	Name      string
	FieldsMap map[string]*Field
	Fields    []*Field
}

Scheme describes fields and theirs order in byte array

func MapSliceToScheme 

func MapSliceToScheme(mapSlice yaml.MapSlice) (*Scheme, error)

MapSliceToScheme s.e.

func NewScheme 

func NewScheme() *Scheme

NewScheme creates new empty Scheme

func YamlToScheme 

func YamlToScheme(yamlStr string) (*Scheme, error)

YamlToScheme creates Scheme by provided yaml `fieldName: yamlFieldType` Field types:

  • `int` -> `int32`
  • `long` -> `int64`
  • `float` -> `float32`
  • `double` -> `float64`
  • `bool` -> `bool`
  • `string` -> `string`
  • `byte` -> `byte`

Field name starts with the capital letter -> field is mandatory Field name ends with `..` -> field is an array See [dynobuffers_test.go](dynobuffers_test.go) for examples

func (*Scheme) AddArray 

func (s *Scheme) AddArray(name string, elementType FieldType, isMandatory bool)

AddArray adds array field

func (*Scheme) AddField 

func (s *Scheme) AddField(name string, ft FieldType, isMandatory bool)

AddField adds field

func (*Scheme) AddFieldC 

func (s *Scheme) AddFieldC(name string, ft FieldType, nested *Scheme, isMandatory bool, IsArray bool)

AddFieldC adds new finely-tuned field

func (*Scheme) AddNested 

func (s *Scheme) AddNested(name string, nested *Scheme, isMandatory bool)

AddNested adds nested object field

func (*Scheme) AddNestedArray 

func (s *Scheme) AddNestedArray(name string, nested *Scheme, isMandatory bool)

AddNestedArray adds array of nested objects field

func (*Scheme) GetNestedScheme 

func (s *Scheme) GetNestedScheme(nestedObjectField string) *Scheme

GetNestedScheme returns Scheme of nested object if the field has FieldTypeObject type, nil otherwise

func (*Scheme) MarshalYAML 

func (s *Scheme) MarshalYAML() (interface{}, error)

MarshalYAML marshals Scheme to yaml. Needs to conform to yaml.Marshaler interface

func (*Scheme) UnmarshalYAML 

func (s *Scheme) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML unmarshals Scheme from yaml. Needs to conform to yaml.Unmarshaler interface

type StringsSlice 

type StringsSlice struct {
	Slice []string
}

type UOffsetSlice 

type UOffsetSlice struct {
	Slice []flatbuffers.UOffsetT
}

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.