MusGo
MusGo is an extremely fast serializer based on code generation. It supports
validation, different encodings, aliases, pointers, and private fields.
Fail Fast Serializer
When unmarshalling, MusGo fails fast with validation error as soon as it
realizes that the data is invalid. This leaves most of the data untouched and
saves system resources. More info you can find in Validation
section.
Go to the MusGen documentation.
Architecture
MusGen performs the main task of the code
generation. MusGo, on the other hand, is tasked with creating the type
description needed for MusGen + persists
the generated code.
Tests
It is also MusGo's responsibility to test the code generated by MusGen
for Golang. You can find these tests in the musgen_..._test.go
files and
corresponding test data in testdata/musgen. The test coverage
of the generated code is about 80%.
MusGo itself is pretty well tested, test coverage is about 90%.
Benchmarks
github.com/alecthomas/go_serialization_benchmarks
Backward compatibility
Go to the MusGen documentation.
Versioning
Go to the MusGen documentation.
How to use
First, you should download and install Go, version 1.18 or later.
Create in your home directory a foo
folder with the following structure:
foo/
|‒‒‒gen/
| |‒‒‒mus.go
|‒‒‒validator/
| |‒‒‒validator.go
|‒‒‒foo.go
foo.go
//go:generate go run gen/mus.go
package foo
type Foo struct {
num int `mus:"validator.Positive"` // Private fields are supported
// too. It will be checked with Positive validator while unmarshalling.
arr []int `mus:",,validator.Positive"` // Every slice element will be checked
// with Positive validator.
Alias StringAlias // Alias types are supported too.
Bool bool `mus:"-"` // This field will be skiped.
}
type StringAlias string
validator/validator.go
package validator
import "errors"
var ErrNegative error = errors.New("negative")
func Positive(n int) error {
if n < 0 {
return ErrNegative
}
return nil
}
gen/mus.go
//go:build ignore
package main
import (
"foo"
"reflect"
"github.com/ymz-ncnk/musgo/v2"
)
func main() {
// MusGo can generate code for struct or alias types.
musGo, err := musgo.New()
if err != nil {
panic(err)
}
// You should "Generate" for all involved custom types.
unsafe := false // To generate safe code.
var alias foo.StringAlias
// Alias types don't support tags, so to set up validator we use
// GenerateAliasAs() method.
conf := musgo.DefAliasConf
conf.MaxLength = 5 // Restricts length of StringAlias values to 5 characters.
err = musGo.GenerateAliasAs(reflect.TypeOf(alias), conf)
if err != nil {
panic(err)
}
// reflect.Type could be created without the explicit variable.
err = musGo.Generate(reflect.TypeOf((*foo.Foo)(nil)).Elem(), unsafe)
if err != nil {
panic(err)
}
}
Run from the command line:
$ cd ~/foo
$ go mod init foo
$ go get github.com/ymz-ncnk/musgo/v2
$ go get github.com/ymz-ncnk/muserrs
$ go generate
Now you can see Foo.mus.go
and StringAlias.mus.go
files in the foo
folder. Pay attention to the location of the generated files. The data type and
the code generated for it must be in the same package. Let's write some tests.
Create a foo_test.go
file:
foo/
|‒‒‒...
|‒‒‒foo_test.go
foo_test.go
package foo
import (
"foo/validator"
"reflect"
"testing"
"github.com/ymz-ncnk/muserrs"
)
func TestFooSerialization(t *testing.T) {
foo := Foo{
num: 5,
arr: []int{4, 2},
Alias: StringAlias("hello"),
Bool: true,
}
buf := make([]byte, foo.SizeMUS())
foo.MarshalMUS(buf)
afoo := Foo{}
_, err := afoo.UnmarshalMUS(buf)
if err != nil {
t.Error(err)
}
foo.Bool = false
if !reflect.DeepEqual(foo, afoo) {
t.Error("something went wrong")
}
}
func TestFooValidation(t *testing.T) {
t.Run("Validator", func(t *testing.T) {
var (
foo = Foo{
num: -11,
arr: []int{1, 2},
Alias: "hello",
}
want = muserrs.NewFieldError("num", validator.ErrNegative)
)
buf := make([]byte, foo.SizeMUS())
foo.MarshalMUS(buf)
afoo := Foo{}
_, err := afoo.UnmarshalMUS(buf)
if err == nil {
t.Error("validation error expected")
}
if err.Error() != want.Error() {
t.Fatalf("unexpected error, want '%v' actual '%v'", want, err)
}
})
t.Run("Element validator", func(t *testing.T) {
var (
foo = Foo{
num: 3,
arr: []int{1, -12, 2},
Alias: "hello",
}
want = muserrs.NewFieldError("arr", muserrs.NewSliceError(1,
validator.ErrNegative))
)
buf := make([]byte, foo.SizeMUS())
foo.MarshalMUS(buf)
afoo := Foo{}
_, err := afoo.UnmarshalMUS(buf)
if err == nil {
t.Error("validation error expected")
}
if err.Error() != want.Error() {
t.Fatalf("unexpected error, want '%v' actual '%v'", want, err)
}
})
t.Run("Max length", func(t *testing.T) {
var (
foo = Foo{
num: 8,
arr: []int{1, 2},
Alias: "hello world",
}
want = muserrs.NewFieldError("Alias", muserrs.ErrMaxLengthExceeded)
)
buf := make([]byte, foo.SizeMUS())
foo.MarshalMUS(buf)
afoo := Foo{}
_, err := afoo.UnmarshalMUS(buf)
if err == nil {
t.Error("validation error expected")
}
if err.Error() != want.Error() {
t.Fatalf("unexpected error, want '%v' actual '%v'", want, err)
}
})
}
More advanced usage you can find at https://github.com/ymz-ncnk/musgotry.
When encoding multiple values, it is impractical to create a new buffer each
time, it takes too long. Instead, you can use the same buffer for each Marshal:
...
buf := make([]byte, FixedLength)
for foo := range foos {
if foo.Size() > len(buf) {
return errors.New("buf is too small")
}
i = foo.MarshalMUS(buf)
err = handle(buf[:i])
...
}
To gain more performance, the recover()
function can be used:
...
defer func() {
if r := recover(); r != nil {
return errors.New("buf is too small")
}
}()
buf := make([]byte, FixedLength)
for _, foo := range foos {
i = foo.MarshalMUS(buf)
err = handle(buf[:i])
...
}
It will intercept every panic, so use it with careful.
Supported Types
Supports following types:
bool
byte
int
int8
int16
int32
int64
uint
uint8
uint16
uint32
uint64
float32
float64
string
array
slice
map
struct
alias
Pointers are supported as well. But aliases to pointer types are not, Go
doesn't allow methods for such types.
Private fields
You could encode and decode private fields too.
Unsafe code
You could generate fast unsafe code. Read more about it in the
MusGen documentation.
Nil pointers support
Take a note, that nil pointers are encoded with zeros:
type PtrInt struct {
Value *int
}
var ptr PtrInt
// Buf will not equal to an empty slice here.
buf := make([]byte, ptr.SizeMUS())
ptr.MarshalMUS(buf) // buf == []{0}
There is no any reason, for example, to persist a buf of zeros, instead you can
persist an empty slice:
...
var buf []byte
if ptr.Value == nil {
buf = []byte{}
} else {
buf = make([]byte, ptr.SizeMUS())
ptr.MarshalMUS(buf)
}
persist(buf)
And on Unmarhsal:
...
var ptr PtrInt
if len(buf) != 0 {
_, err = ptr.UnamrshalMUS(buf)
...
}
Validation
For every structure field you can set up validators using the
mus:"Validator,MaxLength,ElemValidator,KeyValidator"
tag , where:
- Validator - it's a name of the function that will validate the current
field.
- MaxLength - if the field is a string, array, slice or map, MaxLength will
restrict its length. Must be a positive number.
- ElemValidator - it's a name of the function that will validate field
elements, if the field is an array, slice or map.
- KeyValidator - it's a name of the function that will validate field keys,
if the field is a map.
All tag items, except MaxLength, must have the "package.FunctionName" or
"FunctionName" format.
Decoding(and encoding) is performed in order, from the first field to the last
one. That's why, it will stop with a validation error on the first not valid
field. There is no practical reason for decoding the rest of the structure when
we already know that it is not valid.
For an alias type, you can set up validators with help of the
MusGo.GenerateAliasAs()
method.
Validators
Validator is a function with the following signature func (value Type) error
,
where Type
is a type of the value to which the validator is applied.
A few examples:
// Validator for the field.
type Foo struct {
Field string `mus:"StrValidator"`
}
func StrValidator(str string) error {...}
// ElemValidator for the slice field.
type Bar struct {
Field []string `mus:",,StrValidator"`
}
// KeyValidator for the map field.
type Zoo struct {
Field map[string]int `mus:",,,StrValidator"`
}
// Validator for the field of a custom pointer type.
type Far struct {
Field *Foo `mus:FooValidator`
}
func FooValidator(foo *Foo) error {...}
// Validator for the alias field.
type Ror []string
type Pac struct {
Field Ror `mus:RorValidator` // you can't set MaxLength or
// ElemValidator here, they should be applied for the Ror type.
}
func RorValidator(ror Ror) error {...}
Errors
Often validation errors are wrapped by one of the predefined error
(from the MusErrs):
- FieldError - happens when the field validation failed. Contains the field name
and cause.
- SliceError - happens when the validation of the slice element failed. Contains
the element index and cause.
- ArrayError - happens when the validation of the array element failed. Contains
the element index and cause.
- MapKeyError - happens when the validation of the map key failed. Contains the
key and cause.
- MapValueError - happens when the validation of the map value failed. Contains
the key, value and cause.
Encodings
All uint
, int
and float
types support Varint
and Raw
encodings. By
default Varint
is used. You can choose Raw
encoding using the #raw
in
mus:"Validator#raw,MaxLength,ElemValidator#raw,KeyValidator#raw"
tag.
For example:
// Set up the Raw encoding without a validator for the field.
type Foo struct {
Field uint64 `mus:"#raw"`
}
// Set up the validator and Raw encoding for the field.
type Foo struct {
Field uint64 `mus:"Positive#raw"`
}
Raw
encoding has better speed and worse size. Only on large numbers
(> 2^48 in uint representation) it has same or lesser size as Varint
.
For an alias type, you can set up encoding with help of the
MusGo.GenerateAliasAs()
method.
Single number serialization
If all you want is to serialize a single number you can use: