jsonmuncher

JSON Muncher

A highly efficient streaming JSON parser for Go.

But why though?

Do we really need yet another JSON parser? There are a dozen or so other projects that solve this problem, would one of those not suffice?

Different situations call for different approaches. Each of these projects exists because someone found the existing solutions lacking in some way, a gap that needed filling. This might be related to speed, memory footprint, ease of use, some combination of these, or something else entirely. I've found that none of the existing parsers fill my particular gap.

JSON Muncher is designed to be as fast as possible, but its primary focus is memory efficiency. It employs the following design concepts:

• Interactive. Each step of the parse is explicitly triggered by the caller. In some ways this might be seen as detrimental, as it means a little more code is often needed to parse a file than is required by other parsers. However, it also means you have finer control over how the parse progresses, which can allow for drastic efficiency improvements.
• Streaming. Rather than load the entire file into memory at once, JSON Muncher reads only what it needs from the input stream. This heavily reduces the memory footprint, and is especially helpful when parsing very large files.
• No memory allocation (almost). Allocating memory can be costly, in time as well as in space. JSON Muncher avoids allocations as much as possible, without sacrificing usability. However, sometimes allocations are necessary. Allocations are only made in these extremely limited cases:
  • Two or three allocations are made to initialize the input buffer. This only happens once, when the parse begins.
  • An error is allocated if there is a problem parsing the stream. This happens at most once (usually not at all).
  • If a numeric literal in the JSON stream is too long, a temporary buffer is allocated to store it during parsing. This only happens if the literal exceeds 32 characters in length, which is extremely unlikely in practice.

Performance

If you want to see the raw benchmarks or run them yourself, go here.

Speed

Memory

Allocations

API Reference

GoDoc

The API lives in the jsonmuncher package.

Parse()

func Parse(r io.Reader, size int) (JsonValue, error)

The Parse function starts parsing a stream. The stream is passed as the first argument, and can be anything that implements the io.Reader interface. The second argument is the size of the read buffer, in bytes.

This function will allocate a read buffer of the appropriate size, read a chunk of the stream into it, and start parsing. It returns a JsonValue, which can be used to further parse the stream.

The buffer size can be anything, but a power of two is recommended, to avoid potential hardware-related slowness. 4096 (4KiB) or 8192 (8KiB) are generally both good buffer sizes.

JsonValue

This struct represents a value parsed from the stream. It has two exported fields:

type JsonValue struct {
    Type   JsonType
    Status JsonStatus
}

• Type describes what kind of data the JsonValue contains. Its value might be Null, Bool, Number, String, Array, or Object.
• Status describes the read status of the JsonValue. Its value is one of the following:
  • Incomplete: There was a problem parsing the value, or some other error was encountered. As long as you aren't ignoring errors, you shouldn't see this.
  • Working: The value is partially parsed, and can be parsed further by reading more from the stream.
  • Complete: The value was parsed in its entirety, and can no longer read from the stream.

The rest of the API consists of methods on the JsonValue struct.

ValueBool()

func (data *JsonValue) ValueBool() (bool, error)

If this JsonValue is a Bool, return the value (true or false). Otherwise, return an error.

ValueNum()

func (data *JsonValue) ValueNum() (float64, error)

If this JsonValue is a Number, return the value as a double-precision float. Otherwise, return an error.

Read()

func (data *JsonValue) Read(b []byte) (int, error)

An implementation of the io.Reader interface. If this JsonValue is a String, write its contents into the argument slice, and return the number of bytes written. Otherwise, return an error.

NextKey()

func (data *JsonValue) NextKey() (JsonValue, error)

If this JsonValue is an Object, read to the next key and return it. Otherwise, return an error.

NextValue()

func (data *JsonValue) NextValue() (JsonValue, error)

If this JsonValue is an Object or Array, read to the next value and return it. Otherwise, return an error.

In the case of an Array, each time NextValue() is called, the next item in the array will be returned. In the case of an Object, alternate between calling NextKey() and NextValue() to get each key/value pair in the object, or else data will be skipped (if you miss calling a NextKey() or NextValue(), that key or value is discarded).

When NextKey() or NextValue() is called but the object or array has been read to the end, a jsonmuncher.EndOfValue error is returned.

Close()

func (data *JsonValue) Close() error

An implementation of the io.Closer interface. When called, this "closes" the JsonValue by simply discarding the remainder of the value from the stream. Use this if you don't want the rest of the value's data, as it's a good deal faster than parsing. This is effective when used with Strings, Numbers, Objects, or Arrays.

NextKey(), NextValue(), and Close() cannot be called on an Object or Array that has a partially-parsed child; you must fully parse or Close() a child value before continuing to read its parent. Otherwise, an error is returned.

Compare()

func (data *JsonValue) Compare(vals ...string) (string, bool, error)

A helper function, designed to check the value of a String without allocating memory. Consumes the string in the process. Given one or more strings as arguments, compare the value against each argument. Return true with the matched string if there was an exact match, and return false otherwise.

FindKey()

func (data *JsonValue) FindKey(keys ...string) (string, JsonValue, bool, error)

A helper function, designed to quickly search an Object without allocating memory. Given one or more strings as arguments, check each key in the object and compare it to each argument. When an exact match is found, read the corresponding value from the object, and return true with the matched string and the value. If the object is exhausted and no match is found, return false.

To compare against multiple strings while avoiding allocations, Compare() and FindKey() both must sort their arguments. Both functions perform an in-place sort, but this step is faster if the arguments are already ordered correctly. For this reason, it's recommended that when these functions are used, arguments are passed in alphabetical order. Also, if a slice of arguments is passed using something like jsonval.Compare(sliceval ...), the slice will be sorted in-place by the function, so it may not be in the same order after the function runs.