binarystruct

package module
v0.0.11 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 27, 2023 License: BSD-3-Clause Imports: 11 Imported by: 3

README

Go Reference

binarystruct : binary data encoder/decoder for native Go structs

Package binarystruct is an automatic type-converting binary data encoder/decoder(or marshaller/unmarshaller) for go-language structs.

Go's built-in binary encoding package, "encoding/binary" is the preferred method to deal with binary data structures. The binary package is quite easy to use, but some cases require additional type conversions when values are tightly packed. For example, an integer value in raw binary structure may be stored as a word or a byte, but the decoded value would be type-casted to an architecture-dependent integer for easy of use in the Go context.

This package simplifies the typecasting burdens by automatically handling conversion of struct fields using field tags.

A Quick Example

Assume we have a binary data structure with a magic header and three integers, byte, word, dword each, like below. By writing binary data types to field tags in Go struct definition, the values are automatically recognized and converted to proper encoding types.

// source binary data
blob := []byte { 0x61, 0x62, 0x63, 0x64,
	0x01,
	0x00, 0x02,
	0x00, 0x00, 0x00, 0x03 }
// [ "abcd", 0x01, 0x0002, 0x00000003 ]

// Go struct, with field types specified in tags
strc := struct {
	Header       string `binary:"[4]byte"` // maps to 4 bytes
	ValueInt8    int    `binary:"int8"`    // maps to single signed byte
	ValueUint16  int    `binary:"uint16"`  // maps to two bytes
	ValueDword32 int    `binary:"dword"`   // maps to four bytes
}{}

// Unmarshal binary data into the struct
readsz, err := binarystruct.Unmarshal(blob, binarystruct.BigEndian, &strc)

// the structure have proper values now
fmt.Println(strc)
// {abcd 1 2 3}

// Marshal a struct to []byte
output, err := binarystruct.Marshal(&strc, binarystruct.BigEndian)
// output == blob

Features

  • Automatic type conversion and range check based on field tag descriptions
  • Variable-length Array and String handling with references to other struct member variables
  • Text encoding support with "golang.org/x/text/encoding" interface

See also

See go reference doc for details.

Documentation

Overview

Package binarystruct is an automatic type-converting binary data marshaller/unmarshaller for go structs.

Binary data formats are usually tightly packed to save spaces. Such data often require type conversions to be used in the Go language context. This package handles type conversions between Go data types and binary types of struct fields according to their tags.

See the struct below for an example. Each field in this struct is tagged with "binary" tags. The three integer fields are tagged as 1-byte, 2-bytes, and 4-bytes long, and the Header string is tagged as a 4-byte sequence. Marshall and Unmarshal function reads the tag and converts each field to the specified binary format.

// a quick example
strc := struct {
	Header       string `binary:"[4]byte"` // marshaled to 4 bytes
	ValueInt8    int    `binary:"int8"`    // marshaled to single byte
	ValueUint16  int    `binary:"uint16"`  // marshaled to two bytes
	ValueDword32 int    `binary:"dword"`   // marshaled to four bytes
}{"abcd", 1, 2, 3}

blob, err := binarystruct.Marshal(strc, binarystruct.BigEndian)
// marshaled blob will be:
// { 0x61, 0x62, 0x63, 0x64,
//   0x01,
//   0x00, 0x02,
//   0x00, 0x00, 0x00, 0x03 }
Example

A complex example

// An example of marshalling a complex struct to []byte
package main

import (
	"fmt"
	"time"

	"github.com/mixcode/binarystruct"
)

type SomeType int

const (
	SomeType0 SomeType = iota
	SomeType1
)

// a complex data structure
type Header struct {
	// Be sure to set double-quotations in tags!
	Magic string `binary:"[4]byte"` // string converted to 4-byte magic header

	// simple values
	Serial   int      `binary:"int32"`  // integer converted to signed dword
	UnixTime int64    `binary:"uint32"` // int64 converted to unsigned dword
	Type     SomeType `binary:"word"`   // 2-byte word
	Flags    uint32   // if no tag is given, then the size will be its natural size (in this case 4-bytes)

	// both From and To will be converted to int16
	From, To int `binary:"int16"`

	// floating point values
	F1 float32 `binary:"float64"`
	F2 float64 `binary:"int16"` // float types could be converted to integer

	// fixed size data
	Name   *string `binary:"string(0x10)"` // fixed size (16-bytes long) string
	Values []int   `binary:"[0x08]byte"`   // fixed size byte array

	// variable length data
	KeySize int    `binary:"uint16"`
	Key     []byte `binary:"[KeySize]byte"`
	// array size and string length may be sums or subs of integer values and other struct fields

	StrBufLen int      `binary:"uint16"`
	Strings   []string `binary:"[4]string(StrBufLen+1)"` // array of fixed sized strings
}

// A complex example
func main() {
	// build a struct
	name := "max 16 chars"
	timestamp, _ := time.Parse("2006-01-02", "1980-01-02")
	header := Header{
		// simple values
		Magic:    "HEAD",
		Serial:   0x01000002,
		UnixTime: timestamp.Unix(),
		Type:     SomeType1,
		Flags:    0xfffefdfc,
		From:     3, To: 4,

		// note that float type conversions may inaccurate
		F1: 5.0,
		F2: 6.0,

		// fixed length data
		Name:   &name, // pointers and interfaces are dereferced when marshaled
		Values: []int{7, 8, 9, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e},

		// variable length data
		KeySize: 4,
		Key:     []byte{0x0f, 0x10, 0x11, 0x12},

		StrBufLen: 7,
		Strings:   []string{"aa", "bb", "cc", "dd"},
	}

	// marshalling a struct to []byte
	data, err := binarystruct.Marshal(&header, binarystruct.BigEndian)
	if err != nil {
		panic(err)
	}

	// marshaled data will be as follows
	// ---------
	// 00000000  48 45 41 44 01 00 00 02  12 cf f7 80 00 01 ff fe  |HEAD............|
	// 00000010  fd fc 00 03 00 04 40 14  00 00 00 00 00 00 00 06  |......@.........|
	// 00000020  6d 61 78 20 31 36 20 63  68 61 72 73 00 00 00 00  |max 16 chars....|
	// 00000030  07 08 09 0a 0b 0c 0d 0e  00 04 0f 10 11 12 00 07  |................|
	// 00000040  61 61 00 00 00 00 00 00  62 62 00 00 00 00 00 00  |aa......bb......|
	// 00000050  63 63 00 00 00 00 00 00  64 64 00 00 00 00 00 00  |cc......dd......|
	// ---------

	// unmarshalling []byte to struct
	restored := Header{}
	readsz, err := binarystruct.Unmarshal(data, binarystruct.BigEndian, &restored)
	if err != nil {
		panic(err)
	}

	if readsz != len(data) {
		panic(fmt.Errorf("read and write size does not match: read %d, write %d", readsz, len(data)))
	}

}
Output:

Example (Arrays)

An example of slices and arrays.

package main

import (
	"fmt"

	"github.com/mixcode/binarystruct"
)

func main() {

	// slices, arrays and strings
	type exampleStruct struct {

		// fixed size array
		I1 []int `binary:"[4]int16"`

		// variable sized array
		N1 int `binary:"uint16"` // a field value
		N2 int `binary:"int8"`   // another field value
		// array size can contain refererences to other fields and simple integer add-subs
		I2 []int `binary:"[N1+N2+2-1]int8"`

		// string
		S1 string `binary:"string(8)"`    // string of buffer size 8
		S2 string `binary:"string(N2+1)"` // string size also can have field reference

		// length-prefixed strings
		BS  string `binary:"bstring"`  // a byte + []byte. the byte contains the string length
		WS  string `binary:"wstring"`  // a word + []byte. the word contains the string length
		DWS string `binary:"dwstring"` // a dword + []byte. the dword contains the string length

		// array of strings
		SA [4]string `binary:"[]string(4)"` // if a fixed size array is given, then the array size may be omitted

		// special case
		SB string `binary:"[4]string(4)"` // only the first string is used. other 3 strings will be ignored
	}

	src := exampleStruct{
		I1: []int{1, 2, 3, 4},

		N1: 1, N2: 2,
		I2: []int{0, 1, 2, 3}, // N1+N2+2-1 elements long

		S1: "abcd",
		S2: "def", // N2+1 byte long

		BS:  "byte_str",
		WS:  "word_str",
		DWS: "dword_str",

		SA: [4]string{"aa", "bb", "cc", "dd"},
		SB: "zz",
	}

	// marshalling a struct to []byte
	data, err := binarystruct.Marshal(&src, binarystruct.LittleEndian)
	if err != nil {
		panic(err)
	}

	// marshalled result:
	// ---
	// +0000  01 00 02 00 03 00 04 00  01 00 02 00 01 02 03 61  |...............a|
	// +0010  62 63 64 00 00 00 00 64  65 66 08 62 79 74 65 5f  |bcd....def.byte_|
	// +0020  73 74 72 08 00 77 6f 72  64 5f 73 74 72 09 00 00  |str..word_str...|
	// +0030  00 64 77 6f 72 64 5f 73  74 72 61 61 00 00 62 62  |.dword_straa..bb|
	// +0040  00 00 63 63 00 00 64 64  00 00 7a 7a 00 00 00 00  |..cc..dd..zz....|
	// +0050  00 00 00 00 00 00 00 00  00 00                    |..........|
	// ---

	// unmarshalling []byte to a struct
	restored := exampleStruct{}
	_, err = binarystruct.Unmarshal(data, binarystruct.LittleEndian, &restored)
	if err != nil {
		panic(err)
	}
	fmt.Println(restored)

}
Output:

{[1 2 3 4] 1 2 [0 1 2 3] abcd def byte_str word_str dword_str [aa bb cc dd] zz}
Example (Pointers)

Pointers and interfaces are automatically allocated if possible.

package main

import (
	"fmt"

	"github.com/mixcode/binarystruct"
)

// struct with pointers
type subStruct struct {
	N int16
	M *int16
}

type exampleStruct struct {
	P1 *int16
	P2 **int16
	P3 *subStruct  // a pointer to another struct
	I  interface{} // an interface
}

// Pointers and interfaces are automatically allocated if possible.
func main() {

	// build a struct with pointers
	n1, n2, m1, m2 := int16(1), int16(2), int16(4), int16(6)
	p1, p2, p3, p4 := &n1, &n2, &m1, &m2

	src := exampleStruct{
		P1: p1, // the final value of pointers and interfaces will be written
		P2: &p2,
		P3: &subStruct{3, p3},
		I:  &subStruct{5, p4},
	}

	// marshalling a struct to []byte
	// pointers and interfaces are traversed when marshaled
	data, err := binarystruct.Marshal(&src, binarystruct.LittleEndian)
	if err != nil {
		panic(err)
	}

	// marshaled result:
	// ---
	// +0000  01 00 00 00 02 00 00 00  03 00 00 00 04 00 00 00
	// +0010  05 00 00 00 06 00 00 00
	// ---

	// unmarshalling the data
	r1 := int16(0)
	pr1 := &r1
	restored := exampleStruct{
		P1: nil,          // intermediate variables may be alloced for nil pointers
		P2: &pr1,         // if a value is set to the pointer, then the value will be overwritten
		P3: nil,          // structs are also allocated
		I:  &subStruct{}, // Interfaces must be pre-set to be unmarshaled
	}
	readsz, err := binarystruct.Unmarshal(data, binarystruct.LittleEndian, &restored)
	if err != nil {
		panic(err)
	}

	if readsz != len(data) {
		panic(fmt.Errorf("read and write size does not match: read %d, write %d", readsz, len(data)))
	}

	// print unmarshaled data
	fmt.Printf("%d %d %d %d %d %d\n",
		*restored.P1, **restored.P2,
		restored.P3.N, *restored.P3.M,
		restored.I.(*subStruct).N, *(restored.I.(*subStruct).M),
	)
	fmt.Println(*pr1)

}
Output:

1 2 3 4 5 6
2

Index

Examples

Constants

View Source
const (

	//
	// Encoding types for struct field tags.
	// Note that type names in tags are case insensitive.
	//
	//
	// Signed values. Vaules must respect its valid range.
	// e.g.) Int8 must be in [-128, 127].
	Int8  eType // `binary:"int8"`
	Int16       // `binary:"int16"`
	Int32       // `binary:"int32"`
	Int64       // `binary:"int64"`
	//
	// Unsigned values. Must respect its valid range.
	// e.g.) Uint8 must be in [0, 255].
	Uint8  // `binary:"uint8"`
	Uint16 // `binary:"uint16"`
	Uint32 // `binary:"uint32"`
	Uint64 // `binary:"uint64"`
	//
	// Sign-agnostic bitmaps.
	// Be careful that these types are ambiguous in type conversion.
	// i.e) int `binary:"byte"` could map both 255 and -1 to 0xff.
	Byte  // 8-bit byte. `binary: "byte"`
	Word  // 16-bit word. `binary: "word"`
	Dword // 32-bit double word. `binary: "dword"`
	Qword // 64-bit quad word. `binary: "qword"`
	//
	// Floating point values.
	Float32 // `binary:"float32"`
	Float64 // `binary:"float64"`
	//
	// String types.
	// When string types are postfixed by '(size)'
	// then the encoded size will be exactly size bytes long.
	String   // []byte. `binary:"string"` `binary:"string(size)"`
	Bstring  // {size Uint8, string [size]byte}  `binary:"bstring"`
	Wstring  // {size Uint16, string [size]byte} `binary:"wstring"`
	Dwstring // {size Uint32, string [size]byte} `binary:"dwstring"`
	// zero-terminated string types.
	Zstring   // zero-terminated byte string, or C-style string. `binary:"zstring"`
	Z16string // zero-word-terminated word string. `binary:"z16string"`

	// Pad is padding zero bytes. Original value is ignored.
	// May be postfixed by '(size)' to set number of bytes.
	// e.g.) `binary:"pad(0x8)"`
	Pad

	// Values with Ignore tag are ignored. `binary:"ignore"`
	Ignore

	// If a field is tagged with Any, or no tag is set,
	// then the the value's default encoding will be used.
	Any
)

Variables

View Source
var (
	// byte orders
	BigEndian    binary.ByteOrder = binary.BigEndian
	LittleEndian binary.ByteOrder = binary.LittleEndian
)
View Source
var (
	// supplied value must be a pointer or an interface
	ErrCannotSet = errors.New("the value cannot be set")

	// the field is tagged as array but underlying value is not
	ErrNotAnArray = errors.New("must be an array or slice type")

	// cannot determine the length of an array or slice
	ErrUnknownLength = errors.New("unknown array, slice or string size")
)
View Source
var (
	// possibly an invalid encoding type appears in a tag
	ErrInvalidType = errors.New("invalid binary type")
)

Functions

func Marshal

func Marshal(govalue interface{}, order ByteOrder) (encoded []byte, err error)

Marshal encodes a go value into binary data and return it as []byte.

Example
package main

import (
	"fmt"

	bst "github.com/mixcode/binarystruct"
)

func main() {
	strc := struct {
		Header       string `binary:"[4]byte"` // marshaled to 4 bytes
		ValueInt8    int    `binary:"int8"`    // marshaled to single byte
		ValueUint16  int    `binary:"uint16"`  // marshaled to two bytes
		ValueDword32 int    `binary:"dword"`   // marshaled to four bytes
	}{"abcd", 1, 2, 3}
	blob, err := bst.Marshal(strc, bst.BigEndian)

	if err != nil {
		panic(err)
	}
	for _, b := range blob {
		fmt.Printf(" %02x", b)
	}
	fmt.Println()

}
Output:

61 62 63 64 01 00 02 00 00 00 03

func Read

func Read(r io.Reader, order ByteOrder, data interface{}) (n int, err error)

Read reads binary data from r and decode it into a Go value. The Go value must be a writable type such as a slice, a pointer or an interface.

func Unmarshal

func Unmarshal(input []byte, order ByteOrder, govalue interface{}) (n int, err error)

Unmarshal decodes binary images into a Go value. The Go value must be a writable type such as a slice, a pointer or an interface.

Example
package main

import (
	"fmt"

	bst "github.com/mixcode/binarystruct"
)

func main() {
	blob := []byte{0x61, 0x62, 0x63, 0x64,
		0x01,
		0x00, 0x02,
		0x00, 0x00, 0x00, 0x03}
	// [ "abcd", 0x01, 0x0002, 0x00000003 ]

	// A quick example
	strc := struct {
		Header       string `binary:"[4]byte"` // marshaled to 4 bytes
		ValueInt8    int    `binary:"int8"`    // marshaled to single byte
		ValueUint16  int    `binary:"uint16"`  // marshaled to two bytes
		ValueDword32 int    `binary:"dword"`   // marshaled to four bytes
	}{}
	readsz, err := bst.Unmarshal(blob, bst.BigEndian, &strc)

	if err != nil {
		panic(err)
	}
	fmt.Println(readsz, strc)

}
Output:

11 {abcd 1 2 3}

func Write

func Write(w io.Writer, order ByteOrder, govalue interface{}) (n int, err error)

Write encodes a go value into binary stream and writes to w.

Types

type ByteOrder

type ByteOrder = binary.ByteOrder

type ByteOrder is an alias of encoding/binary.ByteOrder

type Marshaller

type Marshaller struct {
	TextEncoding map[string]encoding.Encoding // map[encodingName]Encoding
	// contains filtered or unexported fields
}

Marshaller is go-type to binary-type encoder with environmental values

func (*Marshaller) AddTextEncoding added in v0.0.2

func (ms *Marshaller) AddTextEncoding(encodingName string, enc encoding.Encoding)

AddTextEncoding set a new text encoder to a Marshaller. Provided encodingName could be used in string tag's 'encoding' property, like `binary:"string,encoding=encodingName"`

Example
package main

import (
	"fmt"

	"github.com/mixcode/binarystruct"

	"golang.org/x/text/encoding/japanese"
	"golang.org/x/text/encoding/unicode"
)

func main() {

	// make a explicit marshaller
	var marshaller = new(binarystruct.Marshaller)

	// add Japanese Shift-JIS text encoding
	// see "golang.org/x/text/encoding/japanese"
	marshaller.AddTextEncoding("sjis", japanese.ShiftJIS)

	// add UTF-16(little endian with BOM) text encoding
	// see "golang.org/x/text/encoding/unicode"
	marshaller.AddTextEncoding("utf16", unicode.UTF16(unicode.LittleEndian, unicode.UseBOM))

	type st struct {
		// wstring is []byte prefixed by a word for length
		S string `binary:"wstring,encoding=sjis"`
		T string `binary:"wstring,encoding=utf16"`
	}

	in := st{
		S: "こんにちは", // will be encoded to Shift-JIS
		T: "峠丼",    // will be encoded to UTF-16
	}

	// marshalling
	data, err := marshaller.Marshal(&in, binarystruct.LittleEndian)
	if err != nil {
		panic(err)
	}

	fmt.Printf("Marshaled:")
	for _, b := range data {
		fmt.Printf(" %02x", b)
	}
	fmt.Println()

	// unmarshalling
	out := st{}
	_, err = marshaller.Unmarshal(data, binarystruct.LittleEndian, &out)
	if err != nil {
		panic(err)
	}

	fmt.Printf("%v\n", out)

}
Output:

Marshaled: 0a 00 82 b1 82 f1 82 c9 82 bf 82 cd 06 00 ff fe e0 5c 3c 4e
{こんにちは 峠丼}

func (*Marshaller) Marshal

func (ms *Marshaller) Marshal(govalue interface{}, order ByteOrder) (encoded []byte, err error)

Marshaller.Marshal() is binary image encoder with environment in a Marshaller.

func (*Marshaller) Read

func (ms *Marshaller) Read(r io.Reader, order binary.ByteOrder, data interface{}) (n int, err error)

Unmarshaller.Read() is binary stream decoder with environment in a Marshaller.

func (*Marshaller) RemoveTextEncoding added in v0.0.4

func (ms *Marshaller) RemoveTextEncoding(encodingName string)

RemoveTextEncoding removes an encoding from a Marshaller.

func (*Marshaller) Unmarshal

func (ms *Marshaller) Unmarshal(input []byte, order binary.ByteOrder, govalue interface{}) (n int, err error)

Unmarshaller.Unmarshal() is binary image decoder with environment in a Marshaller.

func (*Marshaller) Write

func (ms *Marshaller) Write(w io.Writer, order ByteOrder, data interface{}) (n int, err error)

Marshaller.Write() is binary stream encoder with environment in a Marshaller.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL