inceptiondb

InceptionDB

A durable in-memory database to store JSON documents.

Write operations are stored into an append-only journal file. The state (documents and indexes) is maintained in memory.

Operations are strongly consistent (atomic) at document level and eventually consistent if multiple documents are involved in the same operation. Replica nodes will be eventually consistent.

It can be used stand-alone or embedded into your golang project.

Interact with an HTTP API (see examples here).

Motivation

InceptionDB was born to be the persistence layer of another project called Bitumen. Bitumen is a distributed NAS that should store family memories, pictures and videos that should last for 50 or 100 years.

One of the biggest motivations is the code should be simple and easy to maintain (yes, even the database itself) and also 100% free of license restrictions.

The second biggest motivation is to have fun and learn some things :D.

Performance

InceptionDB has been designed to be small and easy to read and modify. It is not blazing fast by design but it can burn your wires. Here some numbers from my laptop:

• +200K inserts per second (with 2 unique indexes + concurrency)
• +100K queries per second (on btree index + concurrency)
• +8M documents per second fetching documents from the same query

Documentation

Find API documentation autogenerated examples: ./doc/examples

Technical overview

InceptionDB stores all the data in memory and also a copy on disk in the form of a journal.

When the service starts, the journal is read and applied to recreate the last valid state in memory. From that point on, it is ready to continue operation. One lateral effect is that you can recover the state of the whole database in any point in the past.

Supported indexes:

• Map index, options:
  • field key to be indexed
  • sparse value can be undefined (so, document is not indexed and reachable by the index)
  • multivaluated multiple values will point to the same record (if the value is an array of strings)
• Btree index, options:
  • fields compound keys
  • sparse if indexed fields are undefined, document is not indexed
  • unique only unique tuples are indexed

It does not implement a scheduler, so the index must be explicitly indicated by the user, otherwise a fullscan traversal will be performed.

Features

• API oriented - HTTP is the only interface so that it can be used by any language with any technology.
• Based on journal
• Fast writes

Future work

There are some features planned for the future: replication, trigger http events, atomic patch defined by javascript, historical data,...

Getting started

Just clone the repo, execute make run (golang is required) and open (http://127.0.0.1:8080/)[http://127.0.0.1:8080/].

Choose a good name:

And insert your first JSON: