jstream

Overview ¶

Package jstream defines and implements the JSON Stream protocol

Purpose ¶

The purpose of the jstream protocol is to provide a very simple layer on top of an existing JSON implementation to allow for streaming arbitrary numbers of JSON objects and byte blobs of arbitrary size in a standard way, and to allow for embedding streams within each other.

The order of priorities when designing jstream is as follows:

1. Protocol simplicity
2. Implementation simplicity
3. Efficiency, both in parsing speed and bandwidth

The justification for this is that protocol simplicity generally spills into implementation simplicity anyway, and accounts for future languages which have different properties than current ones. Parsing speed isn't much of a concern when reading data off a network (the primary use-case here), as RTT is always going to be the main blocker. Bandwidth can be a concern, but it's one better solved by wrapping the byte stream with a compressor.

jstream protocol

The jstream protocol is carried over a byte stream (in go: an io.Reader). To read the protocol a JSON object is read off the byte stream and inspected to determine what kind of jstream element it is.

Multiple jstream elements are sequentially read off the same byte stream. Each element may be separated from the other by any amount of whitespace, with whitespace being defined as spaces, tabs, carriage returns, and/or newlines.

jstream elements

There are three jstream element types:

• JSON Value: Any JSON value
• Byte Blob: A stream of bytes of unknown, and possibly infinite, size
• Stream: A heterogenous sequence of jstream elements of unknown, and possibly infinite, size

JSON Value elements are defined as being JSON objects with a `val` field. The value of that field is the JSON Value.

{ "val":{"foo":"bar"} }

Byte Blob elements are defined as being a JSON object with a `bytesStart` field with a value of `true`. Immediately following the JSON object are the bytes which are the Byte Blob, encoded using standard base64. Immediately following the encoded bytes is the character `$`, to indicate the bytes have been completely written. Alternatively the character `!` may be written immediately after the bytes to indicate writing was canceled prematurely by the writer.

{ "bytesStart":true }wXnxQHgUO8g=$
{ "bytesStart":true }WGYcTI8=!

The JSON object may also contain a `sizeHint` field, which gives the estimated number of bytes in the Byte Blob (excluding the trailing delimiter). The hint is neither required to exist or be accurate if it does. The trailing delimeter (`$` or `!`) is required to be sent even if the hint is sent.

Stream elements are defined as being a JSON object with a `streamStart` field with a value of `true`. Immediately following the JSON object will be zero more jstream elements of any type, possibly separated by whitespace. Finally the Stream is ended with another JSON object with a `streamEnd` field with a value of `true`.

{ "streamStart":true }
	{ "val":{"foo":"bar"} }
	{ "bytesStart":true }7TdlDQOnA6isxD9C$
{ "streamEnd":true }

A Stream may also be prematurely canceled by the sending of a JSON object with the `streamCancel` field set to `true` (in place of one with `streamEnd` set to `true`).

The Stream's original JSON object (the "head") may also have a `sizeHint` field, which gives the estimated number of jstream elements in the Stream. The hint is neither required to exist or be accurate if it does. The tail JSON object (with the `streamEnd` field) is required even if `sizeHint` is given.

One of the elements in a Stream may itself be a Stream. In this way Streams may be embedded within each other.

Here's an example of a complex Stream, which carries within it two different streams and some other elements:

{ "streamStart":true }
	{ "val":{"foo":"bar" }
	{ "streamStart":true, "sizeHint":2 }
		{ "val":{"foo":"baz"} }
		{ "val":{"foo":"biz"} }
	{ "streamEnd":true }
	{ "bytesStart":true }X7KCpLIjqIBJt9vA$
	{ "streamStart":true }
		{ "bytesStart":true }0jT+kNCuxHywUYy0$
		{ "bytesStart":true }LUqjR6OACB2p1BG4$
	{ "streamEnd":true }
{ "streamEnd":true }

Finally, the byte stream off of which the jstream is based (i.e. the io.Reader) is implicitly treated as a Stream, with the Stream ending when the byte stream is closed.