origins

package module
v0.0.0-...-f714a01 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 18, 2016 License: BSD-2-Clause Imports: 18 Imported by: 0

README

Origins

Build Status Coverage Status GoDoc

Origins is an open source bi-temporal database for storing and retrieving facts about the state of things. It supports "time-travel" queries, aggregate views, and change detection.

  • This project is in an alpha stage of development.
  • Interested in working on a temporal database written in Go? Get in touch!
  • For more information consult the documentation.
  • Learn how you can contribute.

Docker

Run in-memory Origins HTTP service.

docker run -p 49110:49110 dbhi/origins http

Use the BoltDB storage engine with a volume.

docker run \
    -p 49110:49110 \
    -v <host-dir>:/data \
    dbhi/origins http \
    --storage=boltdb \
    --path=/data/origins.boltdb

Development

Environment

Install Go

brew on OS X.

brew install go

Otherwise follow these instructions.

Install Go dependencies

make install
Testing

Ensure the test dependencies are installed:

make test-install

Then run:

make test
Building

To the build the origins command in ./cmd/origins locally, simply run:

make build

To build binaries for each platform, ensure the build dependencies are installed:

make build-install

Then run:

make build

Documentation

Overview

Origins is an open source bi-temporal database for storing and retrieving facts about the state of things. It supports "time-travel" queries, aggregate views, and change detection.

The primary interface is the CLI which can be installed by running:

go get -u github.com/chop-dbhi/origins/cmd/origins

This package defines some of the primitive structures and algorithms for manipulating, reading, and writing facts and is used to build higher level client APIs.

Fact sorting is done using the Timsort algorithm which is hybrid algorithm of merge sort and insertion sort. This is chosen because facts are generally partially sorted by entity since facts are derived from higher level objects.

For comparison, comparators for the default Quicksort algorithm are implemented for benchmarking purposes.

Wikipedia: https://en.wikipedia.org/wiki/Timsort Comparison to quicksort: http://stackoverflow.com/a/19587279/407954

Index

Constants

View Source
const (
	AttrsDomain         = "origins.attrs"
	TypesDomain         = "origins.types"
	MacrosDomain        = "origins.macros"
	DomainsDomain       = "origins.domains"
	TransactionsDomain  = "origins.transactions"
	CardinalitiesDomain = "origins.cardinalities"
)
View Source
const Version = "0.1"

Variables

View Source
var (
	ErrHeaderRequired = errors.New("A header is required with field names.")
	ErrRequiredFields = errors.New("The entity, attribute, and value fields are required.")
)
View Source
var StorageEngines = map[string]storage.Initializer{
	"bolt":   boltdb.Init,
	"boltdb": boltdb.Init,
	"mem":    memory.Init,
	"memory": memory.Init,
}

Registered storage engines with their initializers. Aliases are supported as separate entries.

Functions

func AEVTComparator

func AEVTComparator(f1, f2 *Fact) bool

AEVTComparator compares two facts using an attribute-entity-value-time sort.

func AVETComparator

func AVETComparator(f1, f2 *Fact) bool

AVETComparator compares two facts using an attribute-value-entity-time sort.

func AttributeComparator

func AttributeComparator(f1, f2 *Fact) bool

AttributeComparator compares two attribute identities.

func Copy

func Copy(it Iterator, w Writer) (int, error)

Copy reads all facts from the reader and copies them to the writer. The number of facts written is returned and an error if present.

func Decompressor

func Decompressor(r io.Reader, f string) (io.Reader, error)

Decompressor returns a reader that wraps the input reader for decompression.

func DetectFileCompression

func DetectFileCompression(n string) string

DetectFileCompression detects the file compression type based on the filename.

func DetectFileFormat

func DetectFileFormat(n string) string

DetectFileFormat detects the file format based on the filename.

func EAVTComparator

func EAVTComparator(f1, f2 *Fact) bool

EAVTComparator compares two facts using an entity-attribute-value-time sort.

func EntityComparator

func EntityComparator(f1, f2 *Fact) bool

EntityComparator compares two entity identities.

func Exists

func Exists(iter Iterator, predicate func(*Fact) bool) bool

Exists takes an iterator and predicate function and returns true if the predicate matches.

func IdentComparator

func IdentComparator(id1, id2 *Ident) bool

IdentComparator compares to identities.

func Init

func Init(name string, opts *storage.Options) (storage.Engine, error)

Init initializes a store with the specified storage and options.

func Map

func Map(iter Iterator, proc func(*Fact) error) error

Map takes an iterator and passes each fact into the map function.

func MapFacts

func MapFacts(iter FactsIterator, proc func(facts Facts) error) error

func Read

func Read(it Iterator, buf Facts) (int, error)

Read consumes up to len(buf) facts of the iterator and puts them into a buffer.

func Sort

func Sort(facts Facts, comp Comparator)

Sort performs an in-place sort of the facts using the built-in sort method.

func TimeComparator

func TimeComparator(f1, f2 *Fact) bool

TimeComparator compares two times.

func Timsort

func Timsort(facts Facts, comp Comparator)

Timsort performs an in-place sort of the facts using the Timsort algorithm.

func TransactionComparator

func TransactionComparator(f1, f2 *Fact) bool

TransactionComparator compares two value identities.

func Transactions

func Transactions(iter Iterator) ([]uint64, error)

Transactions extract a unique set of transaction IDs from the iterator.

func VAETComparator

func VAETComparator(f1, f2 *Fact) bool

VAETComparator compares two facts using an value-attribute-entity-time sort.

func ValueComparator

func ValueComparator(f1, f2 *Fact) bool

ValueComparator compares two value identities.

Types

type Buffer

type Buffer struct {
	// contains filtered or unexported fields
}

Buffer holds a slice of facts. The buffer dynamically grows as facts are written to it. The position is maintained across reads.

func NewBuffer

func NewBuffer(buf Facts) *Buffer

NewBuffer initializes a buffer of facts.

func (*Buffer) Append

func (b *Buffer) Append(buf ...*Fact) (int, error)

Write takes a slice of facts and writes them to the buffer.

func (*Buffer) Err

func (b *Buffer) Err() error

func (*Buffer) Facts

func (b *Buffer) Facts() Facts

Facts returns a copy of the unread portion of facts.

func (*Buffer) Grow

func (b *Buffer) Grow(n int)

Grow increase the buffer size by a minimum of n.

func (*Buffer) Len

func (b *Buffer) Len() int

Len returns the length of the unread portion of the buffer.

func (*Buffer) Next

func (b *Buffer) Next() *Fact

Next returns the next unread fact in the buffer.

func (*Buffer) Reset

func (b *Buffer) Reset()

Reset resets the buffer so it has not content. This is equivalent to calling b.Truncate(0).

func (*Buffer) Truncate

func (b *Buffer) Truncate(n int)

Truncate resets the read and write position to the specified index.

func (*Buffer) Write

func (b *Buffer) Write(f *Fact) error

Write writes a fact to the buffer.

type CSVReader

type CSVReader struct {
	// contains filtered or unexported fields
}

func NewCSVReader

func NewCSVReader(r io.Reader) *CSVReader

func (*CSVReader) Err

func (r *CSVReader) Err() error

Err returns the error produced while reading.

func (*CSVReader) Next

func (r *CSVReader) Next() *Fact

Next returns the next fact in the stream.

type CSVWriter

type CSVWriter struct {
	// contains filtered or unexported fields
}

func NewCSVWriter

func NewCSVWriter(w io.Writer) *CSVWriter

func (*CSVWriter) Flush

func (w *CSVWriter) Flush() error

func (*CSVWriter) Write

func (w *CSVWriter) Write(f *Fact) error

type Comparator

type Comparator func(f1, f2 *Fact) bool

Comparator is a function type that compares two facts for the purposes of sorting. It returns true if the first fact should come before the second fact.

type Fact

type Fact struct {
	// Denotes whether the fact is asserted or retracted.
	Operation Operation

	// The domain the fact applies to. The motivation is to enable asserting
	// facts about entities in different domains. This model decouples content
	// from context to support auxillary information.
	Domain string

	// The entity, attribute, and value of the fact. Also referred to as a
	// "triple", these make up the content of the fact. The entity and attribute
	// must have a domain associated with it. If not specified, they default to
	// the fact domain. If a value has a domain specified, it will be interpreted
	// as a reference to an entity, otherwise it will be treated as a literal.
	Entity    *Ident
	Attribute *Ident
	Value     *Ident

	// The time the fact is true in the world. Also known as the "valid time",
	// this can be set if the fact is true at an earlier or later time than
	// when it was added.
	Time time.Time

	// Transaction that processed this fact. The transaction serves as the
	// temporal component of the fact. Time-travel queries rely on the
	// transaction time to filter out facts not applicable in the specified
	// time range.
	Transaction uint64
}

Fact is the fundamental unit of the information model. It is an immutable declaration of something that is true.

func First

func First(iter Iterator, predicate func(*Fact) bool) *Fact

First takes an iterator and predicate function and returns the first fact that matches the predicate.

func (*Fact) MarshalJSON

func (f *Fact) MarshalJSON() ([]byte, error)

func (*Fact) String

func (f *Fact) String() string

String returns a string representation of the fact.

type Facts

type Facts []*Fact

Facts is a slice of facts.

func ReadAll

func ReadAll(it Iterator) (Facts, error)

ReadAll reads all facts from the reader.

func Reflect

func Reflect(v interface{}) (Facts, error)

Reflect takes a value and returns a set of partially defined facts containing attribute and value components. Currently, struct values or pointers to struct values are supported. Struct fields with a primitive type are included (support for pointers coming soon). The `origins` tag can be used to specify an alternate identity name, an attribute domain name or omit the field all together.

func (Facts) String

func (fs Facts) String() string

type FactsIterator

type FactsIterator interface {
	Next() Facts
	Err() error
}

func Groupby

func Groupby(iter Iterator, cmp func(f1, f2 *Fact) bool) FactsIterator

type Flusher

type Flusher interface {
	Flush() error
}

Flusher is an interface that defines the Flush method. Types that are buffered and require being *flushed* at the end of processing implement this method.

type Ident

type Ident struct {
	Domain string
	Name   string
}

Ident models an identity of something. For entities and attributes a domain is required. For values, if the domain is ommitted, the identity is considered a literal value.

func NewIdent

func NewIdent(domain, name string) (*Ident, error)

NewIdent validates and initializes a new identity value.

func ParseIdent

func ParseIdent(s string) (*Ident, error)

ParseIdent parses a fully qualified identity string, validates the contents and returns an identity value.

func (*Ident) Is

func (id *Ident) Is(b *Ident) bool

Is returns true if the passed ident is the same as the current one.

func (*Ident) String

func (id *Ident) String() string

String returns the fully qualified identity string.

type Identifier

type Identifier interface {
	Ident() *Ident
}

Identifier defines the Ident method that returns and Ident value. Types that implement will used during reflection to be properly represented as facts.

type Idents

type Idents []*Ident

Idents is a slice of idents.

func Attributes

func Attributes(iter Iterator) (Idents, error)

Attributes extract a unique set of attribute identities from the iterator.

func Entities

func Entities(iter Iterator) (Idents, error)

Entities extract a unique set of entity identities from the iterator.

func Values

func Values(iter Iterator) (Idents, error)

Values extract a unique set of values identities from the iterator.

func (Idents) Len

func (ids Idents) Len() int

func (Idents) Less

func (ids Idents) Less(i, j int) bool

func (Idents) Swap

func (ids Idents) Swap(i, j int)

type Iterator

type Iterator interface {
	// Next returns the next available fact in the stream.
	Next() *Fact

	// Err returns an error if one occurred while iterating facts.
	Err() error
}

Iterator is an interface that reads from an underlying stream of facts and makes facts available through the Next() method. The usage of the this interface is as follows:

// The fact will be nil if the underlying stream is exhaused
// or an error occurred.
for f := it.Next(); f != nil {
	// Do something with the fact
}

if err := it.Err(); err != nil {
	// Handle the error.
}

func Entity

func Entity(iter Iterator, id *Ident) Iterator

Entity takes an iterator and returns the all facts about an entity.

func Filter

func Filter(iter Iterator, predicate func(*Fact) bool) Iterator

Filter filters facts consumed from the iterator and returns an iterator.

func Slice

func Slice(iter Iterator, offset, limit int) Iterator

Slice applies an offset and limit to the iterator.

type Operation

type Operation int8

Operation denotes the validity of a fact. A fact can be asserted in which case it is deemed valid and true or retracted which denotes the fact is no longer valid or true. A fact may be retracted without being asserted first. This denotes some fact is known not to be true without necessarily knowing what is true.

const (
	Assertion Operation = iota
	Retraction
)

func ParseOperation

func ParseOperation(s string) (Operation, error)

ParseOperation normalizes the string operation denoting an assertion or retraction of a fact.

func (Operation) MarshalJSON

func (o Operation) MarshalJSON() ([]byte, error)

func (Operation) String

func (o Operation) String() string

type UniversalReader

type UniversalReader struct {
	// contains filtered or unexported fields
}

UniversalReader wraps an io.Reader and replaces carriage returns with newlines.

func NewUniversalReader

func NewUniversalReader(r io.Reader) *UniversalReader

NewUniversalReader returns a UniversalReader that wraps the passed io.Reader.

func (*UniversalReader) Read

func (c *UniversalReader) Read(buf []byte) (int, error)

type Writer

type Writer interface {
	// Write writes a fact to the underlying stream.
	Write(*Fact) error
}

Writer is an interface that defines the Write method. It takes a fact and writes it to the underlying value.

Directories

Path Synopsis
cmd
Package dal is a generated protocol buffer package.
Package dal is a generated protocol buffer package.
The schema package defines types for representing the optional schema support in Origins.
The schema package defines types for representing the optional schema support in Origins.
The storage package defines a key-value based storage engine interface.
The storage package defines a key-value based storage engine interface.
boltdb
The boltdb package implements a storage engine for BoltDB.
The boltdb package implements a storage engine for BoltDB.
memory
The memory package implements an in-memory storage storage.
The memory package implements an in-memory storage storage.
The transactor package contains the APIs for writing facts to storage.
The transactor package contains the APIs for writing facts to storage.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL