simdjson-go: github.com/minio/simdjson-go Index | Files | Directories

package simdjson

import "github.com/minio/simdjson-go"

Index

Package Files

find_subroutines_amd64.go parse_json_amd64.go parse_number_amd64.go parse_string_amd64.go parsed_array.go parsed_json.go parsed_object.go parsed_serialize.go simdjson_amd64.go stage1_find_marks_amd64.go stage2_build_tape_amd64.go

Constants

const (
    TagString      = Tag('"')
    TagInteger     = Tag('l')
    TagUint        = Tag('u')
    TagFloat       = Tag('d')
    TagNull        = Tag('n')
    TagBoolTrue    = Tag('t')
    TagBoolFalse   = Tag('f')
    TagObjectStart = Tag('{')
    TagObjectEnd   = Tag('}')
    TagArrayStart  = Tag('[')
    TagArrayEnd    = Tag(']')
    TagRoot        = Tag('r')
    TagEnd         = Tag(0)
)
const ALWAYS_COPY_STRINGS = true

For enhanced performance, simdjson-go can point back into the original JSON buffer for strings, however this can lead to issues in streaming use cases scenarios, or scenarios in which the underlying JSON buffer is reused. So the default behaviour is to create copies of all strings (not just those transformed anyway for unicode escape charactes) into the separate Strings buffer (at the expense of using more memory and less performance).

const DEFAULTDEPTH = 128
const GOLANG_NUMBER_PARSING = true

The current parse_number() code has some precision issues (see PR "Exact float parsing", https://github.com/lemire/simdjson/pull/333)

An example of this (from canada.json):

"-65.619720000000029" --> -65.61972000000004

instead of

"-65.619720000000029" --> -65.61972000000003

There is a slower code path that uses Golang's Atoi() and ParseFloat()

For **benchmarking** both GOLANG_NUMBER_PARSING and ALWAYS_COPY_STRINGS are set to false

const INDEX_SIZE = 1536 // Seems to be a good size for the index buffering
const INDEX_SIZE_WITH_SAFETY_BUFFER = INDEX_SIZE - 128 // Make sure we never write beyond buffer
const INDEX_SLOTS = 16
const JSONTAGMASK = 0xff << 56
const JSONVALUEMASK = 0xffffffffffffff
const RET_ADDRESS_ARRAY_CONST = 3
const RET_ADDRESS_OBJECT_CONST = 2
const RET_ADDRESS_SHIFT = 2

Constants for "return address" modes

const RET_ADDRESS_START_CONST = 1
const STRINGBUFBIT = 0x80000000000000
const STRINGBUFMASK = 0x7fffffffffffff

Variables

var TagToType = [256]Type{
    TagString:      TypeString,
    TagInteger:     TypeInt,
    TagUint:        TypeUint,
    TagFloat:       TypeFloat,
    TagNull:        TypeNull,
    TagBoolTrue:    TypeBool,
    TagBoolFalse:   TypeBool,
    TagObjectStart: TypeObject,
    TagArrayStart:  TypeArray,
    TagRoot:        TypeRoot,
}

TagToType converts a tag to type. For arrays and objects only the start tag will return types. All non-existing tags returns TypeNone.

func ParseNDStream Uses

func ParseNDStream(r io.Reader, res chan<- Stream, reuse <-chan *ParsedJson)

ParseNDStream will parse a stream and return parsed JSON to the supplied result channel. The method will return immediately. Each element is contained within a root tag.

<root>Element 1</root><root>Element 2</root>...

Each result will contain an unspecified number of full elements, so it can be assumed that each result starts and ends with a root tag. The parser will keep parsing until writes to the result stream blocks. A stream is finished when a non-nil Error is returned. If the stream was parsed until the end the Error value will be io.EOF The channel will be closed after an error has been returned. An optional channel for returning consumed results can be provided. There is no guarantee that elements will be consumed, so always use non-blocking writes to the reuse channel.

func SupportedCPU Uses

func SupportedCPU() bool

SupportedCPU will return whether the CPU is supported.

type Array Uses

type Array struct {
    // contains filtered or unexported fields
}

Array represents a JSON array. There are methods that allows to get full arrays if the value type is the same. Otherwise an iterator can be retrieved.

func (*Array) AsFloat Uses

func (a *Array) AsFloat() ([]float64, error)

AsFloat returns the array values as float. Integers are automatically converted to float.

func (*Array) AsInteger Uses

func (a *Array) AsInteger() ([]int64, error)

AsInteger returns the array values as float. Integers are automatically converted to float.

func (*Array) AsString Uses

func (a *Array) AsString() ([]string, error)

AsString returns the array values as a slice of strings. No conversion is done.

func (*Array) FirstType Uses

func (a *Array) FirstType() Type

FirstType will return the type of the first element. If there are no elements, TypeNone is returned.

func (*Array) Interface Uses

func (a *Array) Interface() ([]interface{}, error)

Interface returns the array as a slice of interfaces. See Iter.Interface() for a reference on value types.

func (*Array) Iter Uses

func (a *Array) Iter() Iter

Iter returns the array as an iterator. This can be used for parsing mixed content arrays. The first value is ready with a call to Advance. Calling after last element should have TypeNone.

func (*Array) MarshalJSON Uses

func (a *Array) MarshalJSON() ([]byte, error)

MarshalJSON will marshal the entire remaining scope of the iterator.

func (*Array) MarshalJSONBuffer Uses

func (a *Array) MarshalJSONBuffer(dst []byte) ([]byte, error)

MarshalJSONBuffer will marshal all elements. An optional buffer can be provided for fewer allocations. Output will be appended to the destination.

type CompressMode Uses

type CompressMode uint8
const (
    // CompressNone no compression whatsoever.
    CompressNone CompressMode = iota

    // CompressFast will apply light compression,
    // but will not deduplicate strings which may affect deserialization speed.
    CompressFast

    // CompressDefault applies light compression and deduplicates strings.
    CompressDefault

    // CompressBest
    CompressBest
)

type Element Uses

type Element struct {
    // Name of the element
    Name string
    // Type of the element
    Type Type
    // Iter containing the element
    Iter Iter
}

Element represents an element in an object.

type Elements Uses

type Elements struct {
    Elements []Element
    Index    map[string]int
}

Elements contains all elements in an object kept in original order. And index contains lookup for object keys.

func (Elements) Lookup Uses

func (e Elements) Lookup(key string) *Element

Lookup a key in elements and return the element. Returns nil if key doesn't exist. Keys are case sensitive.

func (Elements) MarshalJSON Uses

func (e Elements) MarshalJSON() ([]byte, error)

MarshalJSON will marshal the entire remaining scope of the iterator.

func (Elements) MarshalJSONBuffer Uses

func (e Elements) MarshalJSONBuffer(dst []byte) ([]byte, error)

MarshalJSONBuffer will marshal all elements. An optional buffer can be provided for fewer allocations. Output will be appended to the destination.

type Iter Uses

type Iter struct {
    // contains filtered or unexported fields
}

Iter represents a section of JSON. To start iterating it, use Advance() or AdvanceIter() methods which will queue the first element. If an Iter is copied, the copy will be independent.

func (*Iter) Advance Uses

func (i *Iter) Advance() Type

Advance will read the type of the next element and queues up the value on the same level.

func (*Iter) AdvanceInto Uses

func (i *Iter) AdvanceInto() Tag

AdvanceInto will read the tag of the next element and move into and out of arrays , objects and root elements. This should only be used for strictly manual parsing.

func (*Iter) AdvanceIter Uses

func (i *Iter) AdvanceIter(dst *Iter) (Type, error)

AdvanceIter will read the type of the next element and return an iterator only containing the object. If dst and i are the same, both will contain the value inside.

func (*Iter) Array Uses

func (i *Iter) Array(dst *Array) (*Array, error)

Array will return the next element as an array. An optional destination can be given.

func (*Iter) Bool Uses

func (i *Iter) Bool() (bool, error)

Bool() returns the bool value.

func (*Iter) Float Uses

func (i *Iter) Float() (float64, error)

Float returns the float value of the next element. Integers are automatically converted to float.

func (*Iter) Int Uses

func (i *Iter) Int() (int64, error)

Int returns the integer value of the next element. Integers and floats within range are automatically converted.

func (*Iter) Interface Uses

func (i *Iter) Interface() (interface{}, error)

Interface returns the value as an interface. Objects are returned as map[string]interface{}. Arrays are returned as []interface{}. Float values are returned as float64. Integer values are returned as int64 or uint64. String values are returned as string. Boolean values are returned as bool. Null values are returned as nil. Root objects are returned as []interface{}.

func (*Iter) MarshalJSON Uses

func (i *Iter) MarshalJSON() ([]byte, error)

MarshalJSON will marshal the entire remaining scope of the iterator.

func (*Iter) MarshalJSONBuffer Uses

func (i *Iter) MarshalJSONBuffer(dst []byte) ([]byte, error)

MarshalJSONBuffer will marshal the remaining scope of the iterator including the current value. An optional buffer can be provided for fewer allocations. Output will be appended to the destination.

func (*Iter) Object Uses

func (i *Iter) Object(dst *Object) (*Object, error)

Object will return the next element as an object. An optional destination can be given.

func (*Iter) PeekNext Uses

func (i *Iter) PeekNext() Type

PeekNext will return the next value type. Returns TypeNone if next ends iterator.

func (*Iter) PeekNextTag Uses

func (i *Iter) PeekNextTag() Tag

PeekNextTag will return the tag at the current offset. Will return TagEnd if at end of iterator.

func (*Iter) Root Uses

func (i *Iter) Root(dst *Iter) (Type, *Iter, error)

Root() returns the object embedded in root as an iterator along with the type of the content of the first element of the iterator. An optional destination can be supplied to avoid allocations.

func (*Iter) String Uses

func (i *Iter) String() (string, error)

String() returns a string value.

func (*Iter) StringBytes Uses

func (i *Iter) StringBytes() ([]byte, error)

StringBytes() returns a byte array.

func (*Iter) StringCvt Uses

func (i *Iter) StringCvt() (string, error)

StringCvt() returns a string representation of the value. Root, Object and Arrays are not supported.

func (*Iter) Type Uses

func (i *Iter) Type() Type

Type returns the queued value type from the previous call to Advance.

func (*Iter) Uint Uses

func (i *Iter) Uint() (uint64, error)

Uint returns the unsigned integer value of the next element. Positive integers and floats within range are automatically converted.

type Object Uses

type Object struct {
    // contains filtered or unexported fields
}

Object represents a JSON object.

func (*Object) FindKey Uses

func (o *Object) FindKey(key string, dst *Element) *Element

FindKey will return a single named element. An optional destination can be given. The method will return nil if the element cannot be found. This should only be used to locate a single key where the object is no longer needed. The object will not be advanced.

func (*Object) Map Uses

func (o *Object) Map(dst map[string]interface{}) (map[string]interface{}, error)

Map will unmarshal into a map[string]interface{} See Iter.Interface() for a reference on value types.

func (*Object) NextElement Uses

func (o *Object) NextElement(dst *Iter) (name string, t Type, err error)

NextElement sets dst to the next element and returns the name. TypeNone with nil error will be returned if there are no more elements.

func (*Object) NextElementBytes Uses

func (o *Object) NextElementBytes(dst *Iter) (name []byte, t Type, err error)

NextElementBytes sets dst to the next element and returns the name. TypeNone with nil error will be returned if there are no more elements. Contrary to NextElement this will not cause allocations.

func (*Object) Parse Uses

func (o *Object) Parse(dst *Elements) (*Elements, error)

Parse will return all elements and iterators. An optional destination can be given. The Object will be consumed.

type ParsedJson Uses

type ParsedJson struct {
    Message []byte
    Tape    []uint64
    Strings []byte
    // contains filtered or unexported fields
}

func Parse Uses

func Parse(b []byte, reuse *ParsedJson) (*ParsedJson, error)

Parse a block of data and return the parsed JSON. An optional block of previously parsed json can be supplied to reduce allocations.

func ParseND Uses

func ParseND(b []byte, reuse *ParsedJson) (*ParsedJson, error)

ParseND will parse newline delimited JSON. An optional block of previously parsed json can be supplied to reduce allocations.

func (*ParsedJson) Iter Uses

func (pj *ParsedJson) Iter() Iter

Iter returns a new Iter.

func (*ParsedJson) Reset Uses

func (pj *ParsedJson) Reset()

type Serializer Uses

type Serializer struct {
    // contains filtered or unexported fields
}

Serializer allows to serialize parsed json and read it back. A Serializer can be reused, but not used concurrently.

func NewSerializer Uses

func NewSerializer() *Serializer

NewSerializer will create and initialize a Serializer.

func (*Serializer) CompressMode Uses

func (s *Serializer) CompressMode(c CompressMode)

func (*Serializer) Deserialize Uses

func (s *Serializer) Deserialize(src []byte, dst *ParsedJson) (*ParsedJson, error)

Deserialize the content in src. Only basic sanity checks will be performed. Slight corruption will likely go through unnoticed. And optional destination can be provided.

func (*Serializer) Serialize Uses

func (s *Serializer) Serialize(dst []byte, pj ParsedJson) []byte

Serialize the data in pj and return the data. An optional destination can be provided.

type Stream Uses

type Stream struct {
    Value *ParsedJson
    Error error
}

A Stream is used to stream back results. Either Error or Value will be set on returned results.

type Tag Uses

type Tag uint8

Tag indicates the data type of a tape entry

func (Tag) String Uses

func (t Tag) String() string

func (Tag) Type Uses

func (t Tag) Type() Type

Type converts a tag to a type. Only basic types and array+object start match a type.

type Type Uses

type Type uint8

Type is a JSON value type.

const (
    TypeNone Type = iota
    TypeNull
    TypeString
    TypeInt
    TypeUint
    TypeFloat
    TypeBool
    TypeObject
    TypeArray
    TypeRoot
)

func (Type) String Uses

func (t Type) String() string

String returns the type as a string.

Directories

PathSynopsis
examples

Package simdjson imports 18 packages (graph) and is imported by 4 packages. Updated 2020-07-02. Refresh now. Tools for package owners.