fanunmarshal

package module
v0.1.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 31, 2022 License: MIT Imports: 4 Imported by: 0

README

go report card CircleCI GoDoc

README IS WIP

Introduction

fanunmarshal is a concurrent unmarhaller

use with slices of byte slices [][]byte, for example for data coming from Redis using MGet

Basic usage with slice response

import github.com/thisisdevelopment/fanunmarshal


type SomeData struct {
	Data  string `json:"data"`
	Other int    `json:"other"`
}

func initTestData() (testdata [][]byte) {
	testdata = append(testdata, []byte(`{ "data": "1", "other": 42}`))
	testdata = append(testdata, []byte(`{ "data": "2", "other": 42}`))
	testdata = append(testdata, []byte(`{ "data": "3", "other": 42}`))
	testdata = append(testdata, []byte(`{ "data": "4", "other": 42}`))
	testdata = append(testdata, []byte(`{ "data": "5", "other": 42}`))
	testdata = append(testdata, []byte(`{ "data": "6", "other": 42}`))
	return testdata
}


func withSlice() {
    testdata := initTestData() 
	var expected = SomeData{}
	data := fanunmarshal.New().
		WithWorkers(10).
		WithUseJsonIter().
		UnMarshalSlice(testdata, &expected)

	res := map[string]bool{}
	for _, d := range data {
		somedata := d.(*SomeData)
	}
}

func withChannel() {
    var testdata = initTestData()
	var expected = SomeData{}

	fm := New().
		WithWorkers(10).
		WithUseJsonIter().
		DisableAutoScaleDown()

	pipe := fm.MakeChan(testdata)
	outputChan := fm.UnMarshalChan(pipe, &expected, nil)

    /** OR
    	fm := fanunmarshal.New().
	    	WithWorkers(10).
		    WithUseJsonIter()

        dataLength := len(testData)
    	outputChan := fm.UnMarshalChan(pipe, &expected, &dataLength)

    **/ 



	res := map[string]bool{}
	for d := range outputChan {
		somedata := d.(*SomeData)
	}

}

Benchmarks

Total json size 1.4mb. 1000 lines

Standard sequencial unmarshalling

fanunmarshal on  master [!?] via 🐹 v1.18 took 34s 
❯ go test -benchtime=1s -bench=. -count=10 -cpu=4
goos: darwin
goarch: arm64
pkg: github.com/thisisdevelopment/fanunmarshal

BenchmarkPlainUnMarshal-4                    141           8334686 ns/op
BenchmarkPlainUnMarshal-4                    141           8339650 ns/op
BenchmarkPlainUnMarshal-4                    141           8338255 ns/op
BenchmarkPlainUnMarshal-4                    142           8351871 ns/op
BenchmarkPlainUnMarshal-4                    142           8369517 ns/op
BenchmarkPlainUnMarshal-4                    142           8347529 ns/op
BenchmarkPlainUnMarshal-4                    142           8333782 ns/op
BenchmarkPlainUnMarshal-4                    142           8403951 ns/op
BenchmarkPlainUnMarshal-4                    142           8371589 ns/op
BenchmarkPlainUnMarshal-4                    142           8389560 ns/op

Using fanunmarshall 10 workers stdlib json

BenchmarkWithLibSlice_stdlib_10-4            336           3573922 ns/op
BenchmarkWithLibSlice_stdlib_10-4            337           3572657 ns/op
BenchmarkWithLibSlice_stdlib_10-4            338           3607899 ns/op
BenchmarkWithLibSlice_stdlib_10-4            336           3579343 ns/op
BenchmarkWithLibSlice_stdlib_10-4            336           3576072 ns/op
BenchmarkWithLibSlice_stdlib_10-4            336           3568089 ns/op
BenchmarkWithLibSlice_stdlib_10-4            333           3555157 ns/op
BenchmarkWithLibSlice_stdlib_10-4            333           3539808 ns/op
BenchmarkWithLibSlice_stdlib_10-4            336           3645658 ns/op
BenchmarkWithLibSlice_stdlib_10-4            333           3553782 ns/op

Using fanunmarshall 10 workers jsoniter lib

BenchmarkWithLibSlice_jsoniter_10-4          501           2469929 ns/op
BenchmarkWithLibSlice_jsoniter_10-4          474           2497836 ns/op
BenchmarkWithLibSlice_jsoniter_10-4          469           2650896 ns/op
BenchmarkWithLibSlice_jsoniter_10-4          502           2641072 ns/op
BenchmarkWithLibSlice_jsoniter_10-4          512           2712920 ns/op
BenchmarkWithLibSlice_jsoniter_10-4          430           2592961 ns/op
BenchmarkWithLibSlice_jsoniter_10-4          433           2679012 ns/op
BenchmarkWithLibSlice_jsoniter_10-4          436           2694356 ns/op
BenchmarkWithLibSlice_jsoniter_10-4          423           2571725 ns/op
BenchmarkWithLibSlice_jsoniter_10-4          480           2772881 ns/op

PASS

Contributing

You can help to deliver a better fanunmarshaller, check out how you can do things CONTRIBUTING.md

License

© This is Development BV, 2022~time.Now() Released under the MIT License

Documentation

Overview

Package fanunmarshal is a concurrent unmarshaller

use with slices of byte slices [][]byte, for example for data coming from Redis using MGet if we need to unmarshal huge structs, this is where the package shines

Index

Examples

Constants

View Source 
const (
	// default amount of workers
	DefaultWorkers = 2
	// default scaledown the amount of workers set based on the list size
	AutoScaleDown = true
	// default use the standard json encoding lib
	UseStdLib = true
	// use jsoniter (faster)
	UseJsoniter = false
)

Variables

This section is empty.

Functions

This section is empty.

Types

type FanUnMarshal added in v0.1.3 

type FanUnMarshal struct {
	// contains filtered or unexported fields
}

func (*FanUnMarshal) DisableAutoScaleDown added in v0.1.3 

func (f *FanUnMarshal) DisableAutoScaleDown() IFanUnMarshal

DisableAutoScaleDown, disable scaling down the max amount of workers based on your list amount

func (*FanUnMarshal) MakeChan added in v0.1.3 

func (f *FanUnMarshal) MakeChan(data [][]byte) <-chan []byte

MakeChan creates a channel from input byte slice

func (*FanUnMarshal) UnMarshalChan added in v0.1.3 

func (f *FanUnMarshal) UnMarshalChan(pipe <-chan []byte, expected interface{}, dataLength *int) <-chan interface{}

UnMarshalChan unmarshal a channel, returning a channel

Example 
package main

import (
	"fmt"

	fum "github.com/thisisdevelopment/fanunmarshal"
)

func main() {

	// small tiny example
	testdata := [][]byte{
		[]byte(`{ "data": "1", "other": 42}`),
		[]byte(`{ "data": "2", "other": 1337}`),
		[]byte(`{ "data": "3", "other": 161803398875}`),
	}

	// setup receiver struct
	type SomeData struct {
		Data  string `json:"data"`
		Other int    `json:"other"`
	}

	// we expect slice of SomeData
	// setup expected what we receive in the output channel
	var expected = SomeData{}

	// setup instance
	var fm = fum.New().
		WithWorkers(3).        // length of data
		WithUseJsonIter().     // use jsoniter lib
		DisableAutoScaleDown() // because we're not sending the length of the data into UnMarshalChan we need to disable the scaledown

	// setup our input channel
	var pipe = fm.MakeChan(testdata)

	// here we send in a nil as the length of the data, otherwise a pointer to int of length
	var outputChan = fm.UnMarshalChan(pipe, &expected, nil)

	/**
		take notice that order of the channel is not in order you may expect it to be
	**/
	var summed = 0
	// 42 + 1337 + 161803398875 = 161803400254
	for d := range outputChan {
		if data, ok := d.(*SomeData); ok {
			summed += data.Other
		}
	}
	fmt.Printf("%d\n", summed)

}

Output:

161803400254

func (*FanUnMarshal) UnMarshalSlice added in v0.1.3 

func (f *FanUnMarshal) UnMarshalSlice(data [][]byte, expected interface{}) []interface{}

UnMarshalSlice unmarshal a slice of []byte slices, returning the a slice

Example 
package main

import (
	"fmt"
	"sort"

	fum "github.com/thisisdevelopment/fanunmarshal"
)

func main() {
	// small tiny example
	testdata := [][]byte{
		[]byte(`{ "data": "1", "other": 42}`),
		[]byte(`{ "data": "2", "other": 1337}`),
		[]byte(`{ "data": "3", "other": 161803398875}`),
	}

	// setup receiver struct
	type SomeData struct {
		Data  string `json:"data"`
		Other int    `json:"other"`
	}

	// we expect slice of SomeData
	var expected = SomeData{}

	data := fum.New().
		WithWorkers(10).
		WithUseJsonIter().
		UnMarshalSlice(testdata, &expected)

	/** sort it for comparation, just for the example to pass
		take notice that order of the returned slice is not in order you may expect it to be
	**/
	sort.Slice(data, func(i, j int) bool {
		return data[i].(*SomeData).Other < data[j].(*SomeData).Other
	})

	// data is the slice returned from the UnMarshalSlice method
	for _, d := range data {
		fmt.Printf("%+v\n", d)
	}

}

Output:

&{Data:1 Other:42}
&{Data:2 Other:1337}
&{Data:3 Other:161803398875}

func (*FanUnMarshal) WithUseJsonIter added in v0.1.3 

func (f *FanUnMarshal) WithUseJsonIter() IFanUnMarshal

WithUseJsonIter use jsoniter lib instead of default std lib json package

func (*FanUnMarshal) WithWorkers added in v0.1.3 

func (f *FanUnMarshal) WithWorkers(workers uint) IFanUnMarshal

WithWorkers set the amount of workers to work on your list

type IFanUnMarshal added in v0.0.2 

type IFanUnMarshal interface {
	// WithWorkers set the amount of workers to work on your list
	WithWorkers(workers uint) IFanUnMarshal
	// DisableAutoScaleDown, disable scaling down the max amount of workers based on your list amount
	DisableAutoScaleDown() IFanUnMarshal
	// WithUseJsonIter use jsoniter lib instead of default std lib json package
	WithUseJsonIter() IFanUnMarshal
	// UnMarshalSlice unmarshal a slice, returning the slice
	UnMarshalSlice(data [][]byte, expected interface{}) []interface{}
	// MakeChan make a channel based of a slice of byteslices
	MakeChan(data [][]byte) <-chan []byte
	// UnMarshalChan unmarshal a channel, returning a channel
	UnMarshalChan(pipe <-chan []byte, expected interface{}, dataLength *int) <-chan interface{}
}

func New 

func New() IFanUnMarshal

New instance

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.