avro

package module
v0.0.0-...-0ff56f7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 30, 2018 License: Apache-2.0 Imports: 15 Imported by: 0

README

Apache Avro for Golang

Build Status GoDoc

Support for decoding/encoding avro using both map-style access (GenericRecord) and to/from arbitrary Go structs (SpecificRecord).

This library started as a fork of elodina/go-avro but has now proceeded to become a maintained library.

Installation

Installation via go get:

go get gopkg.in/avro.v0

Documentation

Some usage examples are located in examples folder:

About This fork

This fork separated from elodina/go-avro in December 2016 because of the project not responding to PR's since around May 2016. Had tried to contact them to get maintainer access but the original maintainer no longer is able to make those changes.

Originally, we were waiting in hope the elodina maintainer would return, but it hasn't happened, so the plan now is to proceed with this as its own library and take PRs, push for feature additions and version bumps.

Documentation

Overview

Package avro encodes/decodes avro schemas to your struct or a map.

Overview

Go-avro parses .avsc schemas from files and then lets you work with them.

schema, err := avro.ParseSchemaFile("person.avsc")
// important: handle err!

Struct Mapping

When using SpecificDecoder, the implementation uses struct tags to map avro messages into your struct. This helps because it makes schema evolution easier and avoids having a ton of casting and dictionary checks in your user code.

Say you had the schema:

{
  "type": "record",
  "name": "Person",
  "fields" : [
    {"name": "id", "type": "int"},
    {"name": "name", "type": "string"}
    {"name": "location", "type": {
      "type": "record",
      "name": "Location",
      "fields": [
        {"name": "latitude", "type": "double"},
        {"name": "longitude", "type": "double"}
      ]
    }}
  ]
}

This could be mapped to a SpecificRecord by using structs:

type Person struct {
	Id       int32     `avro:"id"`
	Name     string    `avro:"name"`
	Location *Location `avro:"location"`
}

type Location struct {
	Latitude  float64
	Longitude float64
}

If the `avro:` struct tag is omitted, the default mapping lower-cases the first letter only. It's better to just explicitly define where possible.

Mapped types:

  • avro 'int' is always 32-bit, so maps to golang 'int32'
  • avro 'long' is always mapped to 'int64'
  • avro 'float' -> float32
  • avro 'double' -> 'float64'
  • most other ones are obvious

Type unions are a bit more tricky. For a complex type union, the only valid mapping is interface{}. However, for a type union with only "null" and one other type (very typical) you can map it as a pointer type and keep type safety.

Index

Examples

Constants

View Source
const (
	// Record schema type constant
	Record int = iota

	// Enum schema type constant
	Enum

	// Array schema type constant
	Array

	// Map schema type constant
	Map

	// Union schema type constant
	Union

	// Fixed schema type constant
	Fixed

	// String schema type constant
	String

	// Bytes schema type constant
	Bytes

	// Int schema type constant
	Int

	// Long schema type constant
	Long

	// Float schema type constant
	Float

	// Double schema type constant
	Double

	// Boolean schema type constant
	Boolean

	// Null schema type constant
	Null

	// Recursive schema type constant. Recursive is an artificial type that means a Record schema without its definition
	// that should be looked up in some registry.
	Recursive
)

Variables

View Source
var ErrBlockNotFinished = errors.New("Block read is unfinished")

Happens when trying to read next block without finishing the previous one.

View Source
var ErrIntOverflow = errors.New("Overflowed an int value")

Happens when the given value to decode overflows maximum int32 value.

View Source
var ErrInvalidBool = errors.New("Invalid bool value")

Happens when given value to decode as bool is neither 0x00 nor 0x01.

View Source
var ErrInvalidFixedSize = errors.New("Invalid Fixed type size")

Happens when avro schema contains invalid value for fixed size.

View Source
var ErrInvalidInt = errors.New("Invalid int value")

Happens when given value to decode as a int is invalid

View Source
var ErrInvalidLong = errors.New("Invalid long value")

Happens when given value to decode as a long is invalid

View Source
var ErrInvalidSchema = errors.New("Invalid schema")

Happens when avro schema is unparsable or is invalid in any other way.

View Source
var ErrInvalidStringLength = errors.New("Invalid string length")

Happens when given value to decode as string has either negative or undecodable length.

View Source
var ErrLongOverflow = errors.New("Overflowed a long value")

Happens when the given value to decode overflows maximum int64 value.

View Source
var ErrNegativeBytesLength = errors.New("Negative bytes length")

Happens when given value to decode as bytes has negative length.

View Source
var ErrNotAvroFile = errors.New("Not an Avro data file")

Indicates the given file to decode does not correspond to Avro data file format.

View Source
var ErrSchemaNotSet = errors.New("Schema not set")

Happens when a datum reader has no set schema.

View Source
var ErrUnexpectedEOF = io.ErrUnexpectedEOF

Signals that an end of file or stream has been reached unexpectedly.

View Source
var ErrUnionTypeOverflow = errors.New("Union type overflow")

UnionTypeOverflow happens when the numeric index of the union type is invalid.

Functions

func GetFullName

func GetFullName(schema Schema) string

GetFullName returns a fully-qualified name for a schema if possible. The format is namespace.name.

func LoadSchemas

func LoadSchemas(path string) map[string]Schema

LoadSchemas loads and parses a schema file or directory. Directory names MUST end with "/"

func NewFieldDoesNotExistError

func NewFieldDoesNotExistError(field string) error

Specify a custom error message for indicating which necessary field in the struct is missing.

Types

type ArraySchema

type ArraySchema struct {
	Items      Schema
	Properties map[string]interface{}
}

ArraySchema implements Schema and represents Avro array type.

func (*ArraySchema) GetName

func (*ArraySchema) GetName() string

GetName returns a type name for this ArraySchema.

func (*ArraySchema) MarshalJSON

func (s *ArraySchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*ArraySchema) Prop

func (s *ArraySchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*ArraySchema) String

func (s *ArraySchema) String() string

String returns a JSON representation of ArraySchema.

func (*ArraySchema) Type

func (*ArraySchema) Type() int

Type returns a type constant for this ArraySchema.

func (*ArraySchema) Validate

func (s *ArraySchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type AvroRecord

type AvroRecord interface {
	// Schema returns an Avro schema for this AvroRecord.
	Schema() Schema
}

AvroRecord is an interface for anything that has an Avro schema and can be serialized/deserialized by this library.

type BooleanSchema

type BooleanSchema struct{}

BooleanSchema implements Schema and represents Avro boolean type.

func (*BooleanSchema) GetName

func (*BooleanSchema) GetName() string

GetName returns a type name for this BooleanSchema.

func (*BooleanSchema) MarshalJSON

func (*BooleanSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*BooleanSchema) Prop

func (*BooleanSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for BooleanSchema.

func (*BooleanSchema) String

func (*BooleanSchema) String() string

String returns a JSON representation of BooleanSchema.

func (*BooleanSchema) Type

func (*BooleanSchema) Type() int

Type returns a type constant for this BooleanSchema.

func (*BooleanSchema) Validate

func (*BooleanSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type BytesSchema

type BytesSchema struct{}

BytesSchema implements Schema and represents Avro bytes type.

func (*BytesSchema) GetName

func (*BytesSchema) GetName() string

GetName returns a type name for this BytesSchema.

func (*BytesSchema) MarshalJSON

func (*BytesSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*BytesSchema) Prop

func (*BytesSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for BytesSchema.

func (*BytesSchema) String

func (*BytesSchema) String() string

String returns a JSON representation of BytesSchema.

func (*BytesSchema) Type

func (*BytesSchema) Type() int

Type returns a type constant for this BytesSchema.

func (*BytesSchema) Validate

func (*BytesSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type CodeGenerator

type CodeGenerator struct {
	// contains filtered or unexported fields
}

CodeGenerator is a code generation tool for structs from given Avro schemas.

func NewCodeGenerator

func NewCodeGenerator(schemas []string) *CodeGenerator

NewCodeGenerator creates a new CodeGenerator for given Avro schemas.

func (*CodeGenerator) Generate

func (codegen *CodeGenerator) Generate() (string, error)

Generate generates source code for Avro schemas specified on creation. The ouput is Go formatted source code that contains struct definitions for all given schemas. May return an error if code generation fails, e.g. due to unparsable schema.

type DataBlock

type DataBlock struct {

	// Number of entries encoded in Data.
	NumEntries int64

	// Size of data buffer in bytes.
	BlockSize int

	// Number of unread entries in this DataBlock.
	BlockRemaining int64
	// contains filtered or unexported fields
}

DataBlock is a structure that holds a certain amount of entries and the actual buffer to read from.

type DataFileReader

type DataFileReader struct {
	// contains filtered or unexported fields
}

DataFileReader is a reader for Avro Object Container Files. More here: https://avro.apache.org/docs/current/spec.html#Object+Container+Files

Example
package main

import (
	"log"

	avro "gopkg.in/avro.v0"
)

type SomeStruct struct {
	Name string
}

func main() {
	// Create a reader open for reading on a data file.
	reader, err := avro.NewDataFileReader("filename.avro")
	if err != nil {
		log.Fatal(err)
	}
	defer reader.Close()

	for reader.HasNext() {
		var dest SomeStruct // or a *avro.GenericRecord
		if err := reader.Next(&dest); err != nil {
			// Error specific to decoding a single record
		}
		log.Printf("Decoded record %v", dest)
	}

	// If there was any error that stopped the reader loop, this is how we know
	if err := reader.Err(); err != nil {
		log.Fatal(err)
	}
}
Output:

func NewDataFileReader

func NewDataFileReader(filename string, ignoreMe ...DatumReader) (*DataFileReader, error)

NewDataFileReader enables reading an object container file from the filesystem. May return an error if the file contains invalid data or is just missing.

The second DatumReader argument is deprecated, only there for source compatibility. Will be removed in an upcoming compatibility break.

func (*DataFileReader) Close

func (reader *DataFileReader) Close() error

Close the underlying file if necessary.

Needed with filesystem files if you want to not leak filehandles. Returns any error in closing.

func (*DataFileReader) Err

func (reader *DataFileReader) Err() error

Err returns the last encountered error.

Will not return io.EOF if that was the last error.

func (*DataFileReader) HasNext

func (reader *DataFileReader) HasNext() bool

HasNext is used in a for loop to know you can continue on.

If there was an I/O or decoding error in decoding a block, then HasNext will be false, even if there might be more data in the data file.

It might be possible to recover from corrupted data by jumping to the next block by using the NextBlock() but this is not guaranteed.

func (*DataFileReader) Next

func (reader *DataFileReader) Next(v interface{}) error

Next reads the next value from file and fills the given value with data.

v can be anything a DatumReader would accept, including a pointer to a struct that is compatible with this file's schema, an allocated *GenericRecord, or a **GenericRecord (library allocates for you)

Will error with io.EOF if you're past the end, loop HasNext() to prevent.

func (*DataFileReader) NextBlock

func (reader *DataFileReader) NextBlock() error

NextBlock tells this DataFileReader to skip current block and move to next one.

This is not typically needed as the Next() loop will automatically advance to the next block for you.

May return an error if the block is malformed or io.EOF if no more blocks left to read.

type DataFileWriter

type DataFileWriter struct {
	// contains filtered or unexported fields
}

DataFileWriter lets you write object container files.

func NewDataFileWriter

func NewDataFileWriter(output io.Writer, schema Schema, datumWriter DatumWriter) (writer *DataFileWriter, err error)

NewDataFileWriter creates a new DataFileWriter for given output and schema using the given DatumWriter to write the data to that Writer. May return an error if writing fails.

func (*DataFileWriter) Close

func (w *DataFileWriter) Close() error

Close this DataFileWriter. This is required to finish out the data file format. After Close() is called, this DataFileWriter cannot be used anymore.

func (*DataFileWriter) Flush

func (w *DataFileWriter) Flush() error

Flush out any previously written datums to our underlying io.Writer. Does nothing if no datums had previously been written.

It's up to the library user to decide how often to flush; doing it often will spend a lot of time on tiny I/O but save memory.

func (*DataFileWriter) Write

func (w *DataFileWriter) Write(v interface{}) error

Write out a single datum.

Encoded datums are buffered internally and will not be written to the underlying io.Writer until Flush() is called.

type DatumReader

type DatumReader interface {
	// Reads a single structured entry using this DatumReader according to provided Schema.
	// Accepts a value to fill with data and a Decoder to read from. Given value MUST be of pointer type.
	// May return an error indicating a read failure.
	Read(v interface{}, dec Decoder) error
}

DatumReader is an interface that is responsible for reading structured data according to schema from a decoder

func NewDatumReader

func NewDatumReader(schema Schema) DatumReader

NewDatumReader creates a DatumReader that can handle both GenericRecord and also aribtrary structs.

This is the preferred implementation at this point in time.

type DatumWriter

type DatumWriter interface {
	// Write writes a single entry using this DatumWriter according to provided Schema.
	// Accepts a value to write and Encoder to write to.
	// May return an error indicating a write failure.
	Write(obj interface{}, enc Encoder) error
}

DatumWriter is an interface that is responsible for writing structured data according to schema to an encoder.

func NewDatumWriter

func NewDatumWriter(schema Schema) DatumWriter

NewDatumWriter creates a DatumWriter that can handle both GenericRecord and also aribtrary structs.

This is the preferred implementation at this point in time.

type Decoder

type Decoder interface {
	// Reads a null value. Returns a decoded value and an error if it occurs.
	ReadNull() (interface{}, error)

	// Reads a boolean value. Returns a decoded value and an error if it occurs.
	ReadBoolean() (bool, error)

	// Reads an in value. Returns a decoded value and an error if it occurs.
	ReadInt() (int32, error)

	// Reads a long value. Returns a decoded value and an error if it occurs.
	ReadLong() (int64, error)

	// Reads a float value. Returns a decoded value and an error if it occurs.
	ReadFloat() (float32, error)

	// Reads a double value. Returns a decoded value and an error if it occurs.
	ReadDouble() (float64, error)

	// Reads a bytes value. Returns a decoded value and an error if it occurs.
	ReadBytes() ([]byte, error)

	// Reads a string value. Returns a decoded value and an error if it occurs.
	ReadString() (string, error)

	// Reads an enum value (which is an Avro int value). Returns a decoded value and an error if it occurs.
	ReadEnum() (int32, error)

	// Reads and returns the size of the first block of an array. If call to this return non-zero, then the caller
	// should read the indicated number of items and then call ArrayNext() to find out the number of items in the
	// next block. Returns a decoded value and an error if it occurs.
	ReadArrayStart() (int64, error)

	// Processes the next block of an array and returns the number of items in the block.
	// Returns a decoded value and an error if it occurs.
	ArrayNext() (int64, error)

	// Reads and returns the size of the first block of map entries. If call to this return non-zero, then the caller
	// should read the indicated number of items and then call MapNext() to find out the number of items in the
	// next block. Usage is similar to ReadArrayStart(). Returns a decoded value and an error if it occurs.
	ReadMapStart() (int64, error)

	// Processes the next block of map entries and returns the number of items in the block.
	// Returns a decoded value and an error if it occurs.
	MapNext() (int64, error)

	// Reads fixed sized binary object into the provided buffer.
	// Returns an error if it occurs.
	ReadFixed([]byte) error
}

Decoder is an interface that provides low-level support for deserializing Avro values.

func NewBinaryDecoder

func NewBinaryDecoder(buf []byte) Decoder

NewBinaryDecoder creates a new BinaryDecoder to read from a given buffer.

func NewBinaryDecoderReader

func NewBinaryDecoderReader(r io.Reader) Decoder

NewBinaryDecoderReader creates a new BinaryDecoder to read from a given io.Reader.

This decoder makes a lot of very small reads from the underlying io.Reader. If this is some high-latency object like a network socket or file, consider passing some sort of buffered reader like a bufio.Reader.

type DoubleSchema

type DoubleSchema struct{}

DoubleSchema implements Schema and represents Avro double type.

func (*DoubleSchema) GetName

func (*DoubleSchema) GetName() string

GetName returns a type name for this DoubleSchema.

func (*DoubleSchema) MarshalJSON

func (*DoubleSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*DoubleSchema) Prop

func (*DoubleSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for DoubleSchema.

func (*DoubleSchema) String

func (*DoubleSchema) String() string

Returns a JSON representation of DoubleSchema.

func (*DoubleSchema) Type

func (*DoubleSchema) Type() int

Type returns a type constant for this DoubleSchema.

func (*DoubleSchema) Validate

func (*DoubleSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type Encoder

type Encoder interface {
	// Writes a null value. Doesn't actually do anything but may advance the state of Encoder implementation if it
	// is stateful.
	WriteNull(interface{})

	// Writes a boolean value.
	WriteBoolean(bool)

	// Writes an int value.
	WriteInt(int32)

	// Writes a long value.
	WriteLong(int64)

	// Writes a float value.
	WriteFloat(float32)

	// Writes a double value.
	WriteDouble(float64)

	// Writes a bytes value.
	WriteBytes([]byte)

	// Writes a string value.
	WriteString(string)

	// WriteArrayStart should be called when starting to serialize an array providing it with a number of items in
	// array block.
	WriteArrayStart(int64)

	// WriteArrayNext should be called after finishing writing an array block either passing it the number of items in
	// next block or 0 indicating the end of array.
	WriteArrayNext(int64)

	// WriteMapStart should be called when starting to serialize a map providing it with a number of items in
	// map block.
	WriteMapStart(int64)

	// WriteMapNext should be called after finishing writing a map block either passing it the number of items in
	// next block or 0 indicating the end of map.
	WriteMapNext(int64)

	// Writes raw bytes to this Encoder.
	WriteRaw([]byte)
}

Encoder is an interface that provides low-level support for serializing Avro values.

func NewBinaryEncoder

func NewBinaryEncoder(buffer io.Writer) Encoder

NewBinaryEncoder creates a new BinaryEncoder that will write to a given io.Writer.

type EnumSchema

type EnumSchema struct {
	Name       string
	Namespace  string
	Aliases    []string
	Doc        string
	Symbols    []string
	Properties map[string]interface{}
}

EnumSchema implements Schema and represents Avro enum type.

func (*EnumSchema) GetName

func (s *EnumSchema) GetName() string

GetName returns an enum name for this EnumSchema.

func (*EnumSchema) MarshalJSON

func (s *EnumSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*EnumSchema) Prop

func (s *EnumSchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*EnumSchema) String

func (s *EnumSchema) String() string

String returns a JSON representation of EnumSchema.

func (*EnumSchema) Type

func (*EnumSchema) Type() int

Type returns a type constant for this EnumSchema.

func (*EnumSchema) Validate

func (*EnumSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type FixedSchema

type FixedSchema struct {
	Namespace  string
	Name       string
	Size       int
	Properties map[string]interface{}
}

FixedSchema implements Schema and represents Avro fixed type.

func (*FixedSchema) GetName

func (s *FixedSchema) GetName() string

GetName returns a fixed name for this FixedSchema.

func (*FixedSchema) MarshalJSON

func (s *FixedSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*FixedSchema) Prop

func (s *FixedSchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*FixedSchema) String

func (s *FixedSchema) String() string

String returns a JSON representation of FixedSchema.

func (*FixedSchema) Type

func (*FixedSchema) Type() int

Type returns a type constant for this FixedSchema.

func (*FixedSchema) Validate

func (s *FixedSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type FloatSchema

type FloatSchema struct{}

FloatSchema implements Schema and represents Avro float type.

func (*FloatSchema) GetName

func (*FloatSchema) GetName() string

GetName returns a type name for this FloatSchema.

func (*FloatSchema) MarshalJSON

func (*FloatSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*FloatSchema) Prop

func (*FloatSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for FloatSchema.

func (*FloatSchema) String

func (*FloatSchema) String() string

String returns a JSON representation of FloatSchema.

func (*FloatSchema) Type

func (*FloatSchema) Type() int

Type returns a type constant for this FloatSchema.

func (*FloatSchema) Validate

func (*FloatSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type GenericDatumReader

type GenericDatumReader struct {
	// contains filtered or unexported fields
}

GenericDatumReader implements DatumReader and is used for filling GenericRecords or other Avro supported types (full list is: interface{}, bool, int32, int64, float32, float64, string, slices of any type, maps with string keys and any values, GenericEnums) with data. Each value passed to Read is expected to be a pointer.

func NewGenericDatumReader

func NewGenericDatumReader() *GenericDatumReader

NewGenericDatumReader creates a new GenericDatumReader.

func (*GenericDatumReader) Read

func (reader *GenericDatumReader) Read(v interface{}, dec Decoder) error

Read reads a single entry using this GenericDatumReader. Accepts a value to fill with data and a Decoder to read from. Given value MUST be of pointer type. May return an error indicating a read failure.

func (*GenericDatumReader) SetSchema

func (reader *GenericDatumReader) SetSchema(schema Schema) DatumReader

SetSchema sets the schema for this GenericDatumReader to know the data structure. Note that it must be called before calling Read.

type GenericDatumWriter

type GenericDatumWriter struct {
	// contains filtered or unexported fields
}

GenericDatumWriter implements DatumWriter and is used for writing GenericRecords or other Avro supported types (full list is: interface{}, bool, int32, int64, float32, float64, string, slices of any type, maps with string keys and any values, GenericEnums) to a given Encoder.

func NewGenericDatumWriter

func NewGenericDatumWriter() *GenericDatumWriter

NewGenericDatumWriter creates a new GenericDatumWriter.

func (*GenericDatumWriter) SetSchema

func (writer *GenericDatumWriter) SetSchema(schema Schema) DatumWriter

SetSchema sets the provided schema for this GenericDatumWriter to know the data structure. Note that it must be called before calling Write.

func (*GenericDatumWriter) Write

func (writer *GenericDatumWriter) Write(obj interface{}, enc Encoder) error

Write writes a single entry using this GenericDatumWriter according to provided Schema. Accepts a value to write and Encoder to write to. May return an error indicating a write failure.

type GenericEnum

type GenericEnum struct {
	// Avro enum symbols.
	Symbols []string
	// contains filtered or unexported fields
}

GenericEnum is a generic Avro enum representation. This is still subject to change and may be rethought.

func NewGenericEnum

func NewGenericEnum(symbols []string) *GenericEnum

NewGenericEnum returns a new GenericEnum that uses provided enum symbols.

func (*GenericEnum) Get

func (enum *GenericEnum) Get() string

Get gets the string value for this enum (e.g. symbol).

func (*GenericEnum) GetIndex

func (enum *GenericEnum) GetIndex() int32

GetIndex gets the numeric value for this enum.

func (*GenericEnum) Set

func (enum *GenericEnum) Set(symbol string)

Set sets the string value for this enum (e.g. symbol). Panics if the given symbol does not exist in this enum.

func (*GenericEnum) SetIndex

func (enum *GenericEnum) SetIndex(index int32)

SetIndex sets the numeric value for this enum.

type GenericRecord

type GenericRecord struct {
	// contains filtered or unexported fields
}

GenericRecord is a generic instance of a record schema. Fields are accessible by their name.

func NewGenericRecord

func NewGenericRecord(schema Schema) *GenericRecord

NewGenericRecord creates a new GenericRecord.

func (*GenericRecord) Get

func (gr *GenericRecord) Get(name string) interface{}

Get gets a value by its name.

func (*GenericRecord) Map

func (gr *GenericRecord) Map() map[string]interface{}

Map returns a map representation of this GenericRecord.

func (*GenericRecord) Schema

func (gr *GenericRecord) Schema() Schema

Schema returns a schema for this GenericRecord.

func (*GenericRecord) Set

func (gr *GenericRecord) Set(name string, value interface{})

Set sets a value for a given name.

func (*GenericRecord) String

func (gr *GenericRecord) String() string

String returns a JSON representation of this GenericRecord.

type IntSchema

type IntSchema struct{}

IntSchema implements Schema and represents Avro int type.

func (*IntSchema) GetName

func (*IntSchema) GetName() string

GetName returns a type name for this IntSchema.

func (*IntSchema) MarshalJSON

func (*IntSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*IntSchema) Prop

func (*IntSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for IntSchema.

func (*IntSchema) String

func (*IntSchema) String() string

String returns a JSON representation of IntSchema.

func (*IntSchema) Type

func (*IntSchema) Type() int

Type returns a type constant for this IntSchema.

func (*IntSchema) Validate

func (*IntSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type LongSchema

type LongSchema struct{}

LongSchema implements Schema and represents Avro long type.

func (*LongSchema) GetName

func (*LongSchema) GetName() string

GetName returns a type name for this LongSchema.

func (*LongSchema) MarshalJSON

func (*LongSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*LongSchema) Prop

func (*LongSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for LongSchema.

func (*LongSchema) String

func (*LongSchema) String() string

Returns a JSON representation of LongSchema.

func (*LongSchema) Type

func (*LongSchema) Type() int

Type returns a type constant for this LongSchema.

func (*LongSchema) Validate

func (*LongSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type MapSchema

type MapSchema struct {
	Values     Schema
	Properties map[string]interface{}
}

MapSchema implements Schema and represents Avro map type.

func (*MapSchema) GetName

func (*MapSchema) GetName() string

GetName returns a type name for this MapSchema.

func (*MapSchema) MarshalJSON

func (s *MapSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*MapSchema) Prop

func (s *MapSchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*MapSchema) String

func (s *MapSchema) String() string

String returns a JSON representation of MapSchema.

func (*MapSchema) Type

func (*MapSchema) Type() int

Type returns a type constant for this MapSchema.

func (*MapSchema) Validate

func (s *MapSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type Marshaler

type Marshaler interface {
	MarshalAvro(enc Encoder) error
}

Marshaler is an interface that may be implemented to avoid using runtime reflection during serialization. Implementing it is optional and may be used as an optimization. Falls back to using reflection if not implemented.

type NullSchema

type NullSchema struct{}

NullSchema implements Schema and represents Avro null type.

func (*NullSchema) GetName

func (*NullSchema) GetName() string

GetName returns a type name for this NullSchema.

func (*NullSchema) MarshalJSON

func (*NullSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*NullSchema) Prop

func (*NullSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for NullSchema.

func (*NullSchema) String

func (*NullSchema) String() string

String returns a JSON representation of NullSchema.

func (*NullSchema) Type

func (*NullSchema) Type() int

Type returns a type constant for this NullSchema.

func (*NullSchema) Validate

func (*NullSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type RecordSchema

type RecordSchema struct {
	Name       string   `json:"name,omitempty"`
	Namespace  string   `json:"namespace,omitempty"`
	Doc        string   `json:"doc,omitempty"`
	Aliases    []string `json:"aliases,omitempty"`
	Properties map[string]interface{}
	Fields     []*SchemaField `json:"fields"`
}

RecordSchema implements Schema and represents Avro record type.

func (*RecordSchema) GetName

func (s *RecordSchema) GetName() string

GetName returns a record name for this RecordSchema.

func (*RecordSchema) MarshalJSON

func (s *RecordSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*RecordSchema) Prop

func (s *RecordSchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*RecordSchema) String

func (s *RecordSchema) String() string

String returns a JSON representation of RecordSchema.

func (*RecordSchema) Type

func (*RecordSchema) Type() int

Type returns a type constant for this RecordSchema.

func (*RecordSchema) Validate

func (s *RecordSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type RecursiveSchema

type RecursiveSchema struct {
	Actual *RecordSchema
}

RecursiveSchema implements Schema and represents Avro record type without a definition (e.g. that should be looked up).

func (*RecursiveSchema) GetName

func (s *RecursiveSchema) GetName() string

GetName returns a record name for enclosed RecordSchema.

func (*RecursiveSchema) MarshalJSON

func (s *RecursiveSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*RecursiveSchema) Prop

func (*RecursiveSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for RecursiveSchema.

func (*RecursiveSchema) String

func (s *RecursiveSchema) String() string

String returns a JSON representation of RecursiveSchema.

func (*RecursiveSchema) Type

func (*RecursiveSchema) Type() int

Type returns a type constant for this RecursiveSchema.

func (*RecursiveSchema) Validate

func (s *RecursiveSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type Schema

type Schema interface {
	// Returns an integer constant representing this schema type.
	Type() int

	// If this is a record, enum or fixed, returns its name, otherwise the name of the primitive type.
	GetName() string

	// Gets a custom non-reserved property from this schema and a bool representing if it exists.
	Prop(key string) (interface{}, bool)

	// Converts this schema to its JSON representation.
	String() string

	// Checks whether the given value is writeable to this schema.
	Validate(v reflect.Value) bool
}

Schema is an interface representing a single Avro schema (both primitive and complex).

func MustParseSchema

func MustParseSchema(rawSchema string) Schema

MustParseSchema is like ParseSchema, but panics if the given schema cannot be parsed.

func ParseSchema

func ParseSchema(rawSchema string) (Schema, error)

ParseSchema parses a given schema without provided schemas to reuse. Equivalent to call ParseSchemaWithResistry(rawSchema, make(map[string]Schema)) May return an error if schema is not parsable or has insufficient information about any type.

func ParseSchemaFile

func ParseSchemaFile(file string) (Schema, error)

ParseSchemaFile parses a given file. May return an error if schema is not parsable or file does not exist.

func ParseSchemaWithRegistry

func ParseSchemaWithRegistry(rawSchema string, schemas map[string]Schema) (Schema, error)

ParseSchemaWithRegistry parses a given schema using the provided registry for type lookup. Registry will be filled up during parsing. May return an error if schema is not parsable or has insufficient information about any type.

func Prepare

func Prepare(schema Schema) Schema

Prepare optimizes a schema for decoding/encoding.

It makes a recursive copy of the schema given and returns an immutable wrapper of the schema with some optimizations applied.

type SchemaField

type SchemaField struct {
	Name       string      `json:"name,omitempty"`
	Doc        string      `json:"doc,omitempty"`
	Default    interface{} `json:"default"`
	Type       Schema      `json:"type,omitempty"`
	Properties map[string]interface{}
}

SchemaField represents a schema field for Avro record.

func (*SchemaField) MarshalJSON

func (s *SchemaField) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema field as JSON.

func (*SchemaField) Prop

func (this *SchemaField) Prop(key string) (interface{}, bool)

Gets a custom non-reserved property from this schemafield and a bool representing if it exists.

func (*SchemaField) String

func (s *SchemaField) String() string

String returns a JSON representation of SchemaField.

type SpecificDatumReader

type SpecificDatumReader struct {
	// contains filtered or unexported fields
}

SpecificDatumReader implements DatumReader and is used for filling Go structs with data. Each value passed to Read is expected to be a pointer.

func NewSpecificDatumReader

func NewSpecificDatumReader() *SpecificDatumReader

NewSpecificDatumReader creates a new SpecificDatumReader.

func (*SpecificDatumReader) Read

func (reader *SpecificDatumReader) Read(v interface{}, dec Decoder) error

Read reads a single structured entry using this SpecificDatumReader. Accepts a Go struct with exported fields to fill with data and a Decoder to read from. Given value MUST be of pointer type. Field names should match field names in Avro schema but be exported (e.g. "some_value" in Avro schema is expected to be Some_value in struct) or you may provide Go struct tags to explicitly show how to map fields (e.g. if you want to map "some_value" field of type int to SomeValue in Go struct you should define your struct field as follows: SomeValue int32 `avro:"some_field"`). May return an error indicating a read failure.

func (*SpecificDatumReader) SetSchema

func (reader *SpecificDatumReader) SetSchema(schema Schema) DatumReader

SetSchema sets the schema for this SpecificDatumReader to know the data structure. Note that it must be called before calling Read.

type SpecificDatumWriter

type SpecificDatumWriter struct {
	// contains filtered or unexported fields
}

SpecificDatumWriter implements DatumWriter and is used for writing Go structs in Avro format.

Example (Basic)
package main

import (
	"bytes"
	"log"

	avro "gopkg.in/avro.v0"
)

var someSchema avro.Schema

type SomeStruct struct {
	Name string
}

func main() {
	writer := avro.NewSpecificDatumWriter()
	writer.SetSchema(someSchema)

	var buf bytes.Buffer
	v := &SomeStruct{Name: "abc"}

	if err := writer.Write(v, avro.NewBinaryEncoder(&buf)); err != nil {
		log.Fatal(err) // i/o errors OR encoding errors
	}

	// buf.Bytes() now contains the encoded value of 'v'
}
Output:

Example (Full)
package main

import (
	"bytes"
	"fmt"
	"log"

	avro "gopkg.in/avro.v0"
)

func main() {
	// Parse a schema from JSON to get the schema object.
	schema, err := avro.ParseSchema(`{
		"type": "record",
		"name": "Person",
		"fields": [
			{"name": "first_name", "type": "string"},
			{"name": "age", "type": "int"}
		]
	}`)
	if err != nil {
		log.Fatal(err)
	}

	// This is a struct which supports the above schema.
	type Person struct {
		FirstName string `avro:"first_name"` // the avro: tag specifies which avro field matches this field.
		Age       int32  `avro:"age"`
	}

	// Create a SpecificDatumWriter, which you can re-use multiple times.
	writer := avro.NewSpecificDatumWriter()
	writer.SetSchema(schema)

	// Write a person to a byte buffer as avro.
	person := &Person{
		FirstName: "Bob",
		Age:       48,
	}

	var buf bytes.Buffer
	encoder := avro.NewBinaryEncoder(&buf)
	err = writer.Write(person, encoder)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(buf.Bytes())
}
Output:

[6 66 111 98 96]

func NewSpecificDatumWriter

func NewSpecificDatumWriter() *SpecificDatumWriter

NewSpecificDatumWriter creates a new SpecificDatumWriter.

func (*SpecificDatumWriter) SetSchema

func (writer *SpecificDatumWriter) SetSchema(schema Schema) DatumWriter

SetSchema sets the provided schema for this SpecificDatumWriter to know the data structure. Note that it must be called before calling Write.

func (*SpecificDatumWriter) Write

func (writer *SpecificDatumWriter) Write(obj interface{}, enc Encoder) error

Write writes a single Go struct using this SpecificDatumWriter according to provided Schema. Accepts a value to write and Encoder to write to. Field names should match field names in Avro schema but be exported (e.g. "some_value" in Avro schema is expected to be Some_value in struct) or you may provide Go struct tags to explicitly show how to map fields (e.g. if you want to map "some_value" field of type int to SomeValue in Go struct you should define your struct field as follows: SomeValue int32 `avro:"some_field"`). May return an error indicating a write failure.

type StringSchema

type StringSchema struct{}

StringSchema implements Schema and represents Avro string type.

func (*StringSchema) GetName

func (*StringSchema) GetName() string

GetName returns a type name for this StringSchema.

func (*StringSchema) MarshalJSON

func (*StringSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*StringSchema) Prop

func (*StringSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for StringSchema.

func (*StringSchema) String

func (*StringSchema) String() string

Returns a JSON representation of StringSchema.

func (*StringSchema) Type

func (*StringSchema) Type() int

Type returns a type constant for this StringSchema.

func (*StringSchema) Validate

func (*StringSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type UnionSchema

type UnionSchema struct {
	Types []Schema
}

UnionSchema implements Schema and represents Avro union type.

func (*UnionSchema) GetName

func (*UnionSchema) GetName() string

GetName returns a type name for this UnionSchema.

func (*UnionSchema) GetType

func (s *UnionSchema) GetType(v reflect.Value) int

GetType gets the index of actual union type for a given value.

func (*UnionSchema) MarshalJSON

func (s *UnionSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*UnionSchema) Prop

func (*UnionSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for UnionSchema.

func (*UnionSchema) String

func (s *UnionSchema) String() string

String returns a JSON representation of UnionSchema.

func (*UnionSchema) Type

func (*UnionSchema) Type() int

Type returns a type constant for this UnionSchema.

func (*UnionSchema) Validate

func (s *UnionSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type Unmarshaler

type Unmarshaler interface {
	UnmarshalAvro(dec Decoder) error
}

Unmarshaler is an interface that may be implemented to avoid using runtime reflection during deserialization. Implementing it is optional and may be used as an optimization. Falls back to using reflection if not implemented.

Directories

Path Synopsis
examples

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL