refmt

package module
v0.89.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 10, 2022 License: MIT Imports: 6 Imported by: 16

README

refmt

refmt is a serialization and object-mapping library.

Why?

Mostly because I have some types which I need to encode in two different ways, and that needs to not suck, and that totally sucks with most serialization libraries I've used. Also, I need to serialize things in different formats, e.g. sometimes JSON and other times CBOR, and that needs to work without me wrestling two different object-serial libraries and configs.

More broadly, I want a single library that can handle my serialization -- with the possibility of different setups on the same types -- and if it can do general object traversals, e.g. a deepcopy, that also seems like... just something that should be natural.

So it seems like there should be some way to define token streams... and a way to define converting objects to and from token streams... and a way to covert token streams to and from serialized bytes... and all of these should be pretty separate!

Thusly was this library thrust into the world: refmt/tok to define the token stream, and refmt/obj to define how to map objects to tokens and back, and refmt/json and refmt/cbor as ways to exchange tokens with serialized formats.

All of these formats can mix-n-match freely, because they communicate values as the standard token stream. Voilà:

  • pair obj.Marshaller with json.Encoder to get a json serializer.
  • pair cbor.Decoder with obj.Unmarshaller to get a cbor deserializer.
  • pair cbor.Decoder with json.Encoder to get a cbor->json streaming transcoder!
  • pair obj.Marshaller with obj.Unmarshaller to get a deep-copy system! (Try it with two different types: marshalling a struct and unmarshalling into a freeform map!)

Along the way, we've added a powerful system for defining how exactly the refmt/obj tools should treat your structures: the Atlas system (defined in the refmt/obj/atlas package). Atlases can be used to customize how struct fields are handled, how map keys are sorted, and even define conversions between completely different kinds of objects: serialize arrays as strings, or turn stringy enums into bitfields, no problem. By default, refmt will generate atlases automatically for your structs and types, just like the standard library json marshallers do; if you want more control, atlases give you the power.

An Atlas can be given to each obj.Marshaller and obj.Unmarshaller when it is constructed. This allows great variation in how you wish to handle types -- more than one mapping can be defined for the same concrete type! (This is a killer feature if you need to support multiple versions of an API, for example: you can define 'v1' and 'v2' types, each with their own structs to unmarshal user requests into; then in the backend implement another Marshal/Unmarshal with different atlases which translates the 'v1' requests to 'v2' types, and you only have to implement business logic against the latest types!)

Atlases are significantly more convenient to use than defining custom JSONMarshal() methods. Atlases attach to the type they concern. This means you can use atlases to define custom serialization even for types in packages you can't modify! Atlases also behave better in complex situations: for example, if you have a TypeFoo struct and you wish to serialize it as a string, you don't have to write a custom marshaller for every type that contains a TypeFoo field. Leaking details of custom serialization into the types that contain the interesting objects is a common pitfall when getting into advanced usage of other marshalling libraries; refmt has no such issue.

tl;dr:

  • you can swap out atlases for custom serialization on any type;
  • and that works for serialization modes even deep in other structures;
  • and even on types you don't own!!
  • at the same time, you can swap out a cbor encoder for a json encoder, or anything else you want;
  • and the mapper part doesn't care -- no complex interactions between the layers.

Come to refmt. It's nicer here.

Where do I start?

If you're already using json.Marshal: switch to json.Marshal (yes, I'm not kidding; just switch your imports!).

If you're already using json.NewEncoder().Encode(): switch to json.NewMarshaller().Marshal().

If you want to get more serial-flexible: try using refmt.Marshal(json.EncodeOptions{}, obj)... then switch to cbor.EncodeOptions{} and see what happens!

If you want to use Atlas to get fancy: go take a peek at the example*.go files in this repo! :)

Happy hacking!

Documentation

Overview

Refmt is a serialization and object-mapping library.

Look to the README on github for more high-level information.

This top-level package exposes simple helper methods for most common operations. You can also compose custom marshallers/unmarshallers and serializer/deserializers by constructing a `refmt.TokenPump` with components from the packages beneath this one. For example, the `refmt.JsonEncode` helper method can be replicated by combining an `obj.Marshaller` with a `json.Encoder`.

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

func Clone 

func Clone(src, dst interface{}) error

func CloneAtlased 

func CloneAtlased(src, dst interface{}, atl atlas.Atlas) error

func Marshal 

func Marshal(opts EncodeOptions, v interface{}) ([]byte, error)

func MarshalAtlased 

func MarshalAtlased(opts EncodeOptions, v interface{}, atl atlas.Atlas) ([]byte, error)

func MustClone 

func MustClone(src, dst interface{})

func MustCloneAtlased 

func MustCloneAtlased(src, dst interface{}, atl atlas.Atlas)

func Unmarshal 

func Unmarshal(opts DecodeOptions, data []byte, v interface{}) error

func UnmarshalAtlased 

func UnmarshalAtlased(opts DecodeOptions, data []byte, v interface{}, atl atlas.Atlas) error

Types

type Cloner 

type Cloner interface {
	Clone(src, dst interface{}) error
}

func NewCloner 

func NewCloner(atl atlas.Atlas) Cloner

type DecodeOptions 

type DecodeOptions interface {
	IsDecodeOptions() // marker method.
}

type EncodeOptions 

type EncodeOptions interface {
	IsEncodeOptions() // marker method.
}

type Marshaller 

type Marshaller interface {
	Marshal(v interface{}) error
}

func NewMarshaller 

func NewMarshaller(opts EncodeOptions, wr io.Writer) Marshaller
Example (Json) 
package main

import (
	"bytes"
	"fmt"

	"github.com/polydawn/refmt"
	"github.com/polydawn/refmt/json"
)

func main() {
	var buf bytes.Buffer
	encoder := refmt.NewMarshaller(json.EncodeOptions{}, &buf)
	err := encoder.Marshal(map[string]interface{}{
		"x": "a",
		"y": 1,
	})
	fmt.Println(buf.String())
	fmt.Printf("%v\n", err)

}

Output:

{"x":"a","y":1}
<nil>
Example (Json_prettyprint) 
package main

import (
	"bytes"
	"fmt"

	"github.com/polydawn/refmt"
	"github.com/polydawn/refmt/json"
)

func main() {
	var buf bytes.Buffer
	encoder := refmt.NewMarshaller(json.EncodeOptions{
		Line:   []byte{'\n'},
		Indent: []byte{'\t'},
	}, &buf)
	err := encoder.Marshal(map[string]interface{}{
		"x": "a",
		"y": 1,
	})
	fmt.Println(buf.String())
	fmt.Printf("%v\n", err)

}

Output:

{
	"x": "a",
	"y": 1
}

<nil>

func NewMarshallerAtlased 

func NewMarshallerAtlased(opts EncodeOptions, wr io.Writer, atl atlas.Atlas) Marshaller
Example (AutogeneratedAtlas) 
package main

import (
	"bytes"
	"fmt"

	"github.com/polydawn/refmt"
	"github.com/polydawn/refmt/json"
	"github.com/polydawn/refmt/obj/atlas"
)

func main() {
	type MyType struct {
		X string
		Y int
	}

	MyType_AtlasEntry := atlas.BuildEntry(MyType{}).
		StructMap().Autogenerate().
		Complete()

	atl := atlas.MustBuild(
		MyType_AtlasEntry,
		// this is a vararg... stack more entries here!
	)

	var buf bytes.Buffer
	encoder := refmt.NewMarshallerAtlased(json.EncodeOptions{}, &buf, atl)
	err := encoder.Marshal(MyType{"a", 1})
	fmt.Println(buf.String())
	fmt.Printf("%v\n", err)

}

Output:

{"x":"a","y":1}
<nil>
Example (CustomStructMapping) 
package main

import (
	"bytes"
	"fmt"

	"github.com/polydawn/refmt"
	"github.com/polydawn/refmt/json"
	"github.com/polydawn/refmt/obj/atlas"
)

func main() {
	type MyType struct {
		X string
		Y int
	}

	MyType_AtlasEntry := atlas.BuildEntry(MyType{}).
		StructMap().
		AddField("X", atlas.StructMapEntry{SerialName: "overrideName"}).
		// and no "Y" mapping at all!
		Complete()

	atl := atlas.MustBuild(
		MyType_AtlasEntry,
		// this is a vararg... stack more entries here!
	)

	var buf bytes.Buffer
	encoder := refmt.NewMarshallerAtlased(json.EncodeOptions{}, &buf, atl)
	err := encoder.Marshal(MyType{"a", 1})
	fmt.Println(buf.String())
	fmt.Printf("%v\n", err)

}

Output:

{"overrideName":"a"}
<nil>
Example (TransformFuncs) 
package main

import (
	"bytes"
	"fmt"
	"strings"

	"github.com/polydawn/refmt"
	"github.com/polydawn/refmt/json"
	"github.com/polydawn/refmt/obj/atlas"
)

func main() {
	type MyType struct {
		X string
		Y string
		Z string
	}

	MyType_AtlasEntry := atlas.BuildEntry(MyType{}).
		Transform().
		TransformMarshal(atlas.MakeMarshalTransformFunc(
			func(x MyType) (string, error) {
				return string(x.X) + ":" + string(x.Y) + ":" + string(x.Z), nil
			})).
		TransformUnmarshal(atlas.MakeUnmarshalTransformFunc(
			func(x string) (MyType, error) {
				ss := strings.Split(x, ":")
				if len(ss) != 3 {
					return MyType{}, fmt.Errorf("parsing MyType: string must have 3 parts, separated by colon")
				}
				return MyType{ss[0], ss[1], ss[2]}, nil
			})).
		Complete()

	atl := atlas.MustBuild(
		MyType_AtlasEntry,
		// this is a vararg... stack more entries here!
	)

	var buf bytes.Buffer
	encoder := refmt.NewMarshallerAtlased(json.EncodeOptions{}, &buf, atl)

	err := encoder.Marshal(MyType{"serializes", "as", "string!"})
	fmt.Println(buf.String())
	fmt.Printf("%v\n", err)

}

Output:

"serializes:as:string!"
<nil>

type Unmarshaller 

type Unmarshaller interface {
	Unmarshal(v interface{}) error
}

func NewUnmarshaller 

func NewUnmarshaller(opts DecodeOptions, r io.Reader) Unmarshaller

func NewUnmarshallerAtlased 

func NewUnmarshallerAtlased(opts DecodeOptions, r io.Reader, atl atlas.Atlas) Unmarshaller

Source Files

View all Source files

Directories

Path Synopsis
Package implementing the CBOR -- Concise Binary Object Notation -- http://cbor.io/ -- spec.
 Package implementing the CBOR -- Concise Binary Object Notation -- http://cbor.io/ -- spec.
cmd
refmt
Package implementing the JSON -- http://json.org/ -- spec.
 Package implementing the JSON -- http://json.org/ -- spec.
microbench
tokconv
obj
The `obj` package defines Marshaller and Unmarshaller types, which can be used to convert in-memory values to token streams, and token streams to unpack in-memory values.
 The `obj` package defines Marshaller and Unmarshaller types, which can be used to convert in-memory values to token streams, and token streams to unpack in-memory values.
atlas
Atlas types are used to define how to map Go values into refmt token streams.
 Atlas types are used to define how to map Go values into refmt token streams.
atlas/common
commonatlases is a package full of `atlas.Entry` definions for common types in the standard library.
 commonatlases is a package full of `atlas.Entry` definions for common types in the standard library.
The `shared` package defines helper types and functions used internally by all the other refmt packages; it is not user-facing.
 The `shared` package defines helper types and functions used internally by all the other refmt packages; it is not user-facing.
Testing helper functions.
 Testing helper functions.
tok
Package containing Token struct and TokenType info.
 Package containing Token struct and TokenType info.
fixtures
Token stream test fixtures.
 Token stream test fixtures.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.