goavro: github.com/linkedin/goavro Index | Examples | Files

package goavro

import "github.com/linkedin/goavro"

Package goavro is a library that encodes and decodes Avro data.

Goavro provides methods to encode native Go data into both binary and textual JSON Avro data, and methods to decode both binary and textual JSON Avro data to native Go data.

Goavro also provides methods to read and write Object Container File (OCF) formatted files, and the library contains example programs to read and write OCF files.

Usage Example:

    package main

    import (
        "fmt"

        "github.com/linkedin/goavro"
    )

    func main() {
        codec, err := goavro.NewCodec(`
            {
              "type": "record",
              "name": "LongList",
              "fields" : [
	      {"name": "next", "type": ["null", "LongList", {"type": "long", "logicalType": "timestamp-millis"}], "default": null}
              ]
            }`)
        if err != nil {
            fmt.Println(err)
        }

        // NOTE: May omit fields when using default value
        textual := []byte(`{"next":{"LongList":{}}}`)

        // Convert textual Avro data (in Avro JSON format) to native Go form
        native, _, err := codec.NativeFromTextual(textual)
        if err != nil {
            fmt.Println(err)
        }

        // Convert native Go form to binary Avro data
        binary, err := codec.BinaryFromNative(nil, native)
        if err != nil {
            fmt.Println(err)
        }

        // Convert binary Avro data back to native Go form
        native, _, err = codec.NativeFromBinary(binary)
        if err != nil {
            fmt.Println(err)
        }

        // Convert native Go form to textual Avro data
        textual, err = codec.TextualFromNative(nil, native)
        if err != nil {
            fmt.Println(err)
        }

        // NOTE: Textual encoding will show all fields, even those with values that
        // match their default values
        fmt.Println(string(textual))
        // Output: {"next":{"LongList":{"next":null}}}
    }

Index

Examples

Package Files

array.go binaryReader.go boolean.go bytes.go canonical.go codec.go doc.go enum.go fixed.go floatingPoint.go integer.go logical_type.go map.go name.go null.go ocf.go ocf_reader.go ocf_writer.go rabin.go ravin.go record.go text.go union.go

Constants

const (
    // CompressionNullLabel is used when OCF blocks are not compressed.
    CompressionNullLabel = "null"

    // CompressionDeflateLabel is used when OCF blocks are compressed using the
    // deflate algorithm.
    CompressionDeflateLabel = "deflate"

    // CompressionSnappyLabel is used when OCF blocks are compressed using the
    // snappy algorithm.
    CompressionSnappyLabel = "snappy"
)

Variables

var (
    // MaxBlockCount is the maximum number of data items allowed in a single
    // block that will be decoded from a binary stream, whether when reading
    // blocks to decode an array or a map, or when reading blocks from an OCF
    // stream. This check is to ensure decoding binary data will not cause the
    // library to over allocate RAM, potentially creating a denial of service on
    // the system.
    //
    // If a particular application needs to decode binary Avro data that
    // potentially has more data items in a single block, then this variable may
    // be modified at your discretion.
    MaxBlockCount = int64(math.MaxInt32)

    // MaxBlockSize is the maximum number of bytes that will be allocated for a
    // single block of data items when decoding from a binary stream. This check
    // is to ensure decoding binary data will not cause the library to over
    // allocate RAM, potentially creating a denial of service on the system.
    //
    // If a particular application needs to decode binary Avro data that
    // potentially has more bytes in a single block, then this variable may be
    // modified at your discretion.
    MaxBlockSize = int64(math.MaxInt32)
)
var (
    // RelaxedNameValidation causes name validation to allow the first component
    // of an Avro namespace to be the empty string.
    RelaxedNameValidation bool
)

func FingerprintFromSOE Uses

func FingerprintFromSOE(buf []byte) (uint64, []byte, error)

FingerprintFromSOE returns the unsigned 64-bit Rabin fingerprint from the header of a buffer that encodes a Single-Object Encoded datum. This function is designed to be used to lookup a Codec that can decode the contents of the buffer. Once a Codec is found that has the matching Rabin fingerprint, its NativeFromBinary method may be used to decode the remaining bytes returned as the second return value. On failure this function returns an ErrNotSingleObjectEncoded error.

func decode(codex map[uint64]*goavro.Codec, buf []byte) error {
    // Perform a sanity check on the buffer, then return the Rabin fingerprint
    // of the schema used to encode the data.
    fingerprint, newBuf, err := goavro.FingerprintFromSOE(buf)
    if err != nil {
        return err
    }

    // Get a previously stored Codec from the codex map.
    codec, ok := codex[fingerprint]
    if !ok {
        return fmt.Errorf("unknown codec: %#x", fingerprint)
    }

    // Use the fetched Codec to decode the buffer as a SOE.
    //
    // Faster because SOE magic prefix and schema fingerprint already
    // checked and used to fetch the Codec.  Just need to decode the binary
    // bytes remaining after the prefix were removed.
    datum, _, err := codec.NativeFromBinary(newBuf)
    if err != nil {
        return err
    }

    _, err = fmt.Println(datum)
    return err
}

func Union Uses

func Union(name string, datum interface{}) interface{}

Union wraps a datum value in a map for encoding as a Union, as required by Union encoder.

When providing a value for an Avro union, the encoder will accept `nil` for a `null` value. If the value is non-`nil`, it must be a `map[string]interface{}` with a single key-value pair, where the key is the Avro type name and the value is the datum's value. As a convenience, the `Union` function wraps any datum value in a map as specified above.

func ExampleUnion() {
   codec, err := goavro.NewCodec(`["null","string","int"]`)
   if err != nil {
       fmt.Println(err)
   }
   buf, err := codec.TextualFromNative(nil, goavro.Union("string", "some string"))
   if err != nil {
       fmt.Println(err)
   }
   fmt.Println(string(buf))
   // Output: {"string":"some string"}
}

Code:

codec, err := NewCodec(`["null","string","int"]`)
if err != nil {
    fmt.Println(err)
}
buf, err := codec.TextualFromNative(nil, Union("string", "some string"))
if err != nil {
    fmt.Println(err)
}
fmt.Println(string(buf))

Output:

{"string":"some string"}

Code:

// Supported logical types and their native go types:
// * timestamp-millis - time.Time
// * timestamp-micros - time.Time
// * time-millis      - time.Duration
// * time-micros      - time.Duration
// * date             - int
// * decimal          - big.Rat
codec, err := NewCodec(`["null", {"type": "long", "logicalType": "timestamp-millis"}]`)
if err != nil {
    fmt.Println(err)
}

// Note the usage of type.logicalType i.e. `long.timestamp-millis` to denote the type in a union. This is due to the single string naming format
// used by goavro. Decimal can be both bytes.decimal or fixed.decimal
bytes, err := codec.BinaryFromNative(nil, map[string]interface{}{"long.timestamp-millis": time.Date(2006, 1, 2, 15, 4, 5, 0, time.UTC)})
if err != nil {
    fmt.Println(err)
}

decoded, _, err := codec.NativeFromBinary(bytes)
if err != nil {
    fmt.Println(err)
}
out := decoded.(map[string]interface{})
fmt.Printf("%#v\n", out["long.timestamp-millis"].(time.Time).String())

Output:

"2006-01-02 15:04:05 +0000 UTC"

type Codec Uses

type Codec struct {
    Rabin uint64
    // contains filtered or unexported fields
}

Codec supports decoding binary and text Avro data to Go native data types, and conversely encoding Go native data types to binary or text Avro data. A Codec is created as a stateless structure that can be safely used in multiple go routines simultaneously.

func NewCodec Uses

func NewCodec(schemaSpecification string) (*Codec, error)

NewCodec returns a Codec used to translate between a byte slice of either binary or textual Avro data and native Go data.

Creating a `Codec` is fast, but ought to be performed exactly once per Avro schema to process. Once a `Codec` is created, it may be used multiple times to convert data between native form and binary Avro representation, or between native form and textual Avro representation.

A particular `Codec` can work with only one Avro schema. However, there is no practical limit to how many `Codec`s may be created and used in a program. Internally a `Codec` is merely a named tuple of four function pointers, and maintains no runtime state that is mutated after instantiation. In other words, `Codec`s may be safely used by many go routines simultaneously, as your program requires.

codec, err := goavro.NewCodec(`
    {
      "type": "record",
      "name": "LongList",
      "fields" : [
        {"name": "next", "type": ["null", "LongList"], "default": null}
      ]
    }`)
if err != nil {
        fmt.Println(err)
}

func (*Codec) BinaryFromNative Uses

func (c *Codec) BinaryFromNative(buf []byte, datum interface{}) ([]byte, error)

BinaryFromNative appends the binary encoded byte slice representation of the provided native datum value to the provided byte slice in accordance with the Avro schema supplied when creating the Codec. It is supplied a byte slice to which to append the binary encoded data along with the actual data to encode. On success, it returns a new byte slice with the encoded bytes appended, and a nil error value. On error, it returns the original byte slice, and the error message.

func ExampleBinaryFromNative() {
    codec, err := goavro.NewCodec(`
        {
          "type": "record",
          "name": "LongList",
          "fields" : [
            {"name": "next", "type": ["null", "LongList"], "default": null}
          ]
        }`)
    if err != nil {
        fmt.Println(err)
    }

    // Convert native Go form to binary Avro data
    binary, err := codec.BinaryFromNative(nil, map[string]interface{}{
        "next": map[string]interface{}{
            "LongList": map[string]interface{}{
                "next": map[string]interface{}{
                    "LongList": map[string]interface{}{
                    // NOTE: May omit fields when using default value
                    },
                },
            },
        },
    })
    if err != nil {
        fmt.Println(err)
    }

    fmt.Printf("%#v", binary)
    // Output: []byte{0x2, 0x2, 0x0}
}

func (*Codec) CanonicalSchema Uses

func (c *Codec) CanonicalSchema() string

CanonicalSchema returns the Parsing Canonical Form of the schema according to the Avro specification.

func (*Codec) NativeFromBinary Uses

func (c *Codec) NativeFromBinary(buf []byte) (interface{}, []byte, error)

NativeFromBinary returns a native datum value from the binary encoded byte slice in accordance with the Avro schema supplied when creating the Codec. On success, it returns the decoded datum, a byte slice containing the remaining undecoded bytes, and a nil error value. On error, it returns nil for the datum value, the original byte slice, and the error message.

func ExampleNativeFromBinary() {
    codec, err := goavro.NewCodec(`
        {
          "type": "record",
          "name": "LongList",
          "fields" : [
            {"name": "next", "type": ["null", "LongList"], "default": null}
          ]
        }`)
    if err != nil {
        fmt.Println(err)
    }

    // Convert native Go form to binary Avro data
    binary := []byte{0x2, 0x2, 0x0}

    native, _, err := codec.NativeFromBinary(binary)
    if err != nil {
        fmt.Println(err)
    }

    fmt.Printf("%v", native)
    // Output: map[next:map[LongList:map[next:map[LongList:map[next:<nil>]]]]]
}

func (*Codec) NativeFromSingle Uses

func (c *Codec) NativeFromSingle(buf []byte) (interface{}, []byte, error)

NativeFromSingle converts Avro data from Single-Object-Encoded format from the provided byte slice to Go native data types in accordance with the Avro schema supplied when creating the Codec. On success, it returns the decoded datum, along with a new byte slice with the decoded bytes consumed, and a nil error value. On error, it returns nil for the datum value, the original byte slice, and the error message.

func decode(codec *goavro.Codec, buf []byte) error {
    datum, _, err := codec.NativeFromSingle(buf)
    if err != nil {
        return err
    }
    _, err = fmt.Println(datum)
    return err
}

func (*Codec) NativeFromTextual Uses

func (c *Codec) NativeFromTextual(buf []byte) (interface{}, []byte, error)

NativeFromTextual converts Avro data in JSON text format from the provided byte slice to Go native data types in accordance with the Avro schema supplied when creating the Codec. On success, it returns the decoded datum, along with a new byte slice with the decoded bytes consumed, and a nil error value. On error, it returns nil for the datum value, the original byte slice, and the error message.

func ExampleNativeFromTextual() {
    codec, err := goavro.NewCodec(`
        {
          "type": "record",
          "name": "LongList",
          "fields" : [
            {"name": "next", "type": ["null", "LongList"], "default": null}
          ]
        }`)
    if err != nil {
        fmt.Println(err)
    }

    // Convert native Go form to text Avro data
    text := []byte(`{"next":{"LongList":{"next":{"LongList":{"next":null}}}}}`)

    native, _, err := codec.NativeFromTextual(text)
    if err != nil {
        fmt.Println(err)
    }

    fmt.Printf("%v", native)
    // Output: map[next:map[LongList:map[next:map[LongList:map[next:<nil>]]]]]
}

func (*Codec) Schema Uses

func (c *Codec) Schema() string

Schema returns the original schema used to create the Codec.

func (*Codec) SchemaCRC64Avro Uses

func (c *Codec) SchemaCRC64Avro() int64

SchemaCRC64Avro returns a signed 64-bit integer Rabin fingerprint for the canonical schema. This method returns the signed 64-bit cast of the unsigned 64-bit schema Rabin fingerprint.

DEPRECATED: This method has been replaced by the Rabin structure Codec field and is provided for backward compatibility only.

func (*Codec) SingleFromNative Uses

func (c *Codec) SingleFromNative(buf []byte, datum interface{}) ([]byte, error)

SingleFromNative appends the single-object-encoding byte slice representation of the provided native datum value to the provided byte slice in accordance with the Avro schema supplied when creating the Codec. It is supplied a byte slice to which to append the header and binary encoded data, along with the actual data to encode. On success, it returns a new byte slice with the encoded bytes appended, and a nil error value. On error, it returns the original byte slice, and the error message.

func ExampleSingleItemEncoding() {
    codec, err := goavro.NewCodec(`"int"`)
    if err != nil {
        fmt.Fprintf(os.Stderr, "%s\n", err)
        return
    }

    buf, err := codec.SingleFromNative(nil, 3)
    if err != nil {
        fmt.Fprintf(os.Stderr, "%s\n", err)
        return
    }

    fmt.Println(buf)
    // Output: [195 1 143 92 57 63 26 213 117 114 6]
}

func (*Codec) TextualFromNative Uses

func (c *Codec) TextualFromNative(buf []byte, datum interface{}) ([]byte, error)

TextualFromNative converts Go native data types to Avro data in JSON text format in accordance with the Avro schema supplied when creating the Codec. It is supplied a byte slice to which to append the encoded data and the actual data to encode. On success, it returns a new byte slice with the encoded bytes appended, and a nil error value. On error, it returns the original byte slice, and the error message.

func ExampleTextualFromNative() {
    codec, err := goavro.NewCodec(`
        {
          "type": "record",
          "name": "LongList",
          "fields" : [
            {"name": "next", "type": ["null", "LongList"], "default": null}
          ]
        }`)
    if err != nil {
        fmt.Println(err)
    }

    // Convert native Go form to text Avro data
    text, err := codec.TextualFromNative(nil, map[string]interface{}{
        "next": map[string]interface{}{
            "LongList": map[string]interface{}{
                "next": map[string]interface{}{
                    "LongList": map[string]interface{}{
                    // NOTE: May omit fields when using default value
                    },
                },
            },
        },
    })
    if err != nil {
        fmt.Println(err)
    }

    fmt.Printf("%s", text)
    // Output: {"next":{"LongList":{"next":{"LongList":{"next":null}}}}}
}

type ErrInvalidName Uses

type ErrInvalidName struct {
    Message string
}

ErrInvalidName is the error returned when one or more parts of an Avro name is invalid.

func (ErrInvalidName) Error Uses

func (e ErrInvalidName) Error() string

type ErrNotSingleObjectEncoded Uses

type ErrNotSingleObjectEncoded string

ErrNotSingleObjectEncoded is returned when an attempt is made to decode a single-object encoded value from a buffer that does not have the correct magic prefix.

func (ErrNotSingleObjectEncoded) Error Uses

func (e ErrNotSingleObjectEncoded) Error() string

type ErrWrongCodec Uses

type ErrWrongCodec uint64

ErrWrongCodec is returned when an attempt is made to decode a single-object encoded value using the wrong codec.

func (ErrWrongCodec) Error Uses

func (e ErrWrongCodec) Error() string

type OCFConfig Uses

type OCFConfig struct {
    // W specifies the `io.Writer` to which to send the encoded data,
    // (required). If W is `*os.File`, then creating an OCF for writing will
    // attempt to read any existing OCF header and use the schema and
    // compression codec specified by the existing header, then advance the file
    // position to the tail end of the file for appending.
    W   io.Writer

    // Codec specifies the Codec to use for the new OCFWriter, (optional). If
    // the W parameter above is an `*os.File` which contains a Codec, the Codec
    // in the existing file will be used instead. Otherwise if this Codec
    // parameter is specified, it will be used. If neither the W parameter above
    // is an `*os.File` with an existing Codec, nor this Codec parameter is
    // specified, the OCFWriter will create a new Codec from the schema string
    // specified by the Schema parameter below.
    Codec *Codec

    // Schema specifies the Avro schema for the data to be encoded, (optional).
    // If neither the W parameter above is an `*os.File` with an existing Codec,
    // nor the Codec parameter above is specified, the OCFWriter will create a
    // new Codec from the schema string specified by this Schema parameter.
    Schema string

    // CompressionName specifies the compression codec used, (optional). If
    // omitted, defaults to "null" codec. When appending to an existing OCF,
    // this field is ignored.
    CompressionName string

    //MetaData specifies application specific meta data to be added to
    //the OCF file.  When appending to an existing OCF, this field
    //is ignored
    MetaData map[string][]byte
}

OCFConfig is used to specify creation parameters for OCFWriter.

type OCFReader Uses

type OCFReader struct {
    // contains filtered or unexported fields
}

OCFReader structure is used to read Object Container Files (OCF).

func NewOCFReader Uses

func NewOCFReader(ior io.Reader) (*OCFReader, error)

NewOCFReader initializes and returns a new structure used to read an Avro Object Container File (OCF).

func example(ior io.Reader) error {
    // NOTE: Wrap provided io.Reader in a buffered reader, which improves the
    // performance of streaming file data.
    br := bufio.NewReader(ior)
    ocfr, err := goavro.NewOCFReader(br)
    if err != nil {
        return err
    }
    for ocfr.Scan() {
        datum, err := ocfr.Read()
        if err != nil {
            return err
        }
        fmt.Println(datum)
    }
    return ocfr.Err()
}

func (*OCFReader) Codec Uses

func (ocfr *OCFReader) Codec() *Codec

Codec returns the codec found within the OCF file.

func (*OCFReader) CompressionName Uses

func (ocfr *OCFReader) CompressionName() string

CompressionName returns the name of the compression algorithm found within the OCF file.

func (*OCFReader) Err Uses

func (ocfr *OCFReader) Err() error

Err returns the last error encountered while reading the OCF file. See `NewOCFReader` documentation for an example.

func (*OCFReader) MetaData Uses

func (ocfr *OCFReader) MetaData() map[string][]byte

MetaData returns the file metadata map found within the OCF file

func (*OCFReader) Read Uses

func (ocfr *OCFReader) Read() (interface{}, error)

Read consumes one datum value from the Avro OCF stream and returns it. Read is designed to be called only once after each invocation of the Scan method. See `NewOCFReader` documentation for an example.

func (*OCFReader) RemainingBlockItems Uses

func (ocfr *OCFReader) RemainingBlockItems() int64

RemainingBlockItems returns the number of items remaining in the block being processed.

func (*OCFReader) Scan Uses

func (ocfr *OCFReader) Scan() bool

Scan returns true when there is at least one more data item to be read from the Avro OCF. Scan ought to be called prior to calling the Read method each time the Read method is invoked. See `NewOCFReader` documentation for an example.

func (*OCFReader) SkipThisBlockAndReset Uses

func (ocfr *OCFReader) SkipThisBlockAndReset()

SkipThisBlockAndReset can be called after an error occurs while reading or decoding datum values from an OCF stream. OCF specifies each OCF stream contain one or more blocks of data. Each block consists of a block count, the number of bytes for the block, followed be the possibly compressed block. Inside each decompressed block is all of the binary encoded datum values concatenated together. In other words, OCF framing is at a block level rather than a datum level. If there is an error while reading or decoding a datum, the reader is not able to skip to the next datum value, because OCF does not have any markers for where each datum ends and the next one begins. Therefore, the reader is only able to skip this datum value and all subsequent datum values in the current block, move to the next block and start decoding datum values there.

type OCFWriter Uses

type OCFWriter struct {
    // contains filtered or unexported fields
}

OCFWriter is used to create a new or append to an existing Avro Object Container File (OCF).

func NewOCFWriter Uses

func NewOCFWriter(config OCFConfig) (*OCFWriter, error)

NewOCFWriter returns a new OCFWriter instance that may be used for appending binary Avro data, either by appending to an existing OCF file or creating a new OCF file.

func (*OCFWriter) Append Uses

func (ocfw *OCFWriter) Append(data interface{}) error

Append appends one or more data items to an OCF file in a block. If there are more data items in the slice than MaxBlockCount allows, the data slice will be chunked into multiple blocks, each not having more than MaxBlockCount items.

func (*OCFWriter) Codec Uses

func (ocfw *OCFWriter) Codec() *Codec

Codec returns the codec used by OCFWriter. This function provided because upstream may be appending to existing OCF which uses a different schema than requested during instantiation.

func (*OCFWriter) CompressionName Uses

func (ocfw *OCFWriter) CompressionName() string

CompressionName returns the name of the compression algorithm used by OCFWriter. This function provided because upstream may be appending to existing OCF which uses a different compression algorithm than requested during instantiation. the OCF file.

Package goavro imports 23 packages (graph) and is imported by 27 packages. Updated 2019-06-13. Refresh now. Tools for package owners.