goethe

Goethe

Goethe is a lightweight append-only event log server. It's meant to be used in event driven systems to enable patterns like event sourcing.

TODOs

Architecture

Concepts

Goethe is meant to be the event store in event driven systems. This section shall guide you through the top level concepts.

Events

Event driven systems use events as their main communication channels. Events are basically facts about things that happend in the system. Naturally, events are immutable, they are facts. This is why having an append-only event log in an event driven system seems to make sense.

Goethe does not put any constraints on the events you want to be flowing through your system, and you will find a lot of good advice like in the CloudEvents Spec. We consider events as plain old byte arrays, for you to have the complete freedom about what you are doing.

Cursors

We want to deliver maximum ease to Goethe clients. The default usecase for a topic consumer is to start we she left off before. Of course there are times when replays are necessary and a consumer wants to read a complete topic from its beginnig.

This is why we think about cursors owned by a certain service pointing to a certain event in a certain topic.

A consumer is able to tell Goethe a specific event to start consuming from. This might be due to the consumer stored her last known position or to replay events since a certain occurence. For consumers' convenience we also enable reading from where a consumer left off by leaving the cursor's current_event field blank and Goethe starts from server's the last known cursor position of the consumer.

Future Features:

If a consumer wants to read events from a topic at a specific point in time, she can just provide a cursor pointing to that time. Goethe will select the next event which was commited after that time.

In cases where consumers start to become slow, you might start scaling the consumer service horizontally to increase performance and share the load among the instances. This is also something which can be addressed by the use of proper cursors. If multiple instances use the same consumer field in the cursors, the consumer's cursor is moved for all instances.

Storage

Each Goethe instance uses a local LevelDB (the Go implementation) as its storage backend. LevelDB is incredibly fast in sequential reading and writing and therefore performs very well in an append-only log. LevelDB's default key sorting algorithm uses bytewise comparison.

To achieve maximum performance, our event keys need to be poperly sorted by their occurence. This is why we store them behind keys with the following pattern: {TOPIC_ID}:{EVENT_ID} where EVENT_ID is the UNIX nano timestamp when the event was originally received. Nano timestamps are defacto unique in a single topic, hence, collisions are very unlikely. Having event keys with topic prefixex, all events of a certain topics are arranged in a sequence.

That results in reading sequentially from a topic and therefore getting maximum speed.

Build

To build the server simply run: make build, to build the CLI use the make cli command.

Generate Spec sources

This service provides a grpc API. To generate the new stubs after changing it:

• make sure to have protoc installed
• make sure to have protoc-gen-go installed; run install_proto.sh
• run make spec

License

Unless otherwise noted, the Goethe source files are distributed under the MIT license found in the LICENSE file.