zapp

package module
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 19, 2023 License: MIT Imports: 13 Imported by: 0

README

Zapp - a high performant Key-Value Database

Zapp is a pure golang implemented kv database. It's very simple, but it's high efficient and allows building amazing systems on top of it.

And also this is a parody on all these modern pathos databases, which appear and die every day.

Docs

Why this name

Zapp (database) is called after one of the main Futurama's characters, Zapp Branigan. Just like the prototype, the database is extremely efficient, fast, extremely sexy, smart, mature, space efficient, hardworking... aaaaaand... fast... aaaaand sexy... I'm not lying, I swear! You can watch the show yourself, if you don't believe me!

How cool Zapp is?

First, it's very fast! Super fast! Here are the real performance testing numbers:

  • Set - 1 338 284 QPS
  • Get - 4 418 378 QPS
  • Delete - 1 874 350 QPS

This is 50x times more than a single PostgreSQL can do!

Zapp is also fully ACID compatible! Check this out:

  • Atomicity - we have transactions, which are executed and committed at once. We can not rollback them. No rollbacks - no problems!
  • Consistency - we almost do not have any constraints! So your data model will always be consistent!
  • Isolation - parallel requests are fully isolated from each other. In fact, Zapp provides serializable isolation level!
  • Durability - you can enable Write Ahead Logging, and your data will be 100% safe!

So, in fact, Zapp is much cooler, than any other database. It's more fast and has the same ACID features!

I'm joking :)

Although this project is a joke, Zapp can be really used in production, if you need a durable performant key value storage. If you are not 100% sure, you are ready for this, then better use Redis or PostgreSQL.

Of course, in most cases you will not have millions QPS. Take a look at real performance testing on an isolated VPS in Cloud. If you want to learn more, how to tune performance, read the Performance Tips Docs.

Inspiration

Worth mentioning, that Zapp was inspired by another golang kv database sniper, which was really used in production. But there are differences: Zapp uses another collision resolving algorithm, Zapp provides data safety guarantees with the help of Write Ahead logging and etc.

Mac OS tip

How to Change Open Files Limit on OS X and macOS

Plans and Features

  • Creating segment's data file and filling it data
  • Selecting segment based on key hash value
  • Get, Set, Delete operations
  • Expiration time support. Expired keys are collected in background
  • Optional Write Ahead Logging. Each write operation generate a new WAL entry with a unique LSN.
  • Restoring Segment's data from WAL, if last applied LSN is lower, then the most recent LSN from WAL
  • Unit Test Coverage for most of the current Segment's logic. Perform a lot of testing for correctness
  • Write a performance testing code and make real performance testing on a VPS
  • Write Docs and release to public
  • Implement a mutable Min-Heap data structure inside Zapp to track items, which are about to expire. This is a replacement for an O(N) algorithm of checking each item in current collect-expired-items process
  • Implement metrics reporting: performance, keys, dataset size, segments etc.
  • Implement Zapp as a standalone daemon server with some standard Key-Value protocol. For example, Memcached protocol
  • Implement a CLI application to manage Zapp daemon: restart, start, stop, metrics view, data access etc.
  • Implement synchronous and asynchronous replication protocol for creating a cluster of Zapp with master-slave replication. Choose between logic and physic replication. The master pushes changes to sync replicas, async replicas pull changes from the master
  • Implement the mechanism of promoting a slave instance into a master
  • Implement a coordinator application with consensus algorithm inside for managing the cluster for automatic master failover

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrNotFound = errors.New("key not found")

	ErrSegmentMagicNumbersDoNotMatch = errors.New("file magic numbers do not match")
	ErrSegmentUnknownVersionNumber   = errors.New("unknown version number of segment file")
	ErrUnknownBlobStatus             = errors.New("unkown blob status")

	ErrInvalidPath        = errors.New("invalid path for storing data")
	ErrInvalidSegmentsNum = errors.New("invalid number of segments")

	ErrClosed = errors.New("segment is closed")
)

Functions

This section is empty.

Types

type DB 

type DB struct {
	// contains filtered or unexported fields
}

func New 

func New(params Params) (*DB, error)

func (*DB) Close 

func (db *DB) Close()

func (*DB) Delete 

func (db *DB) Delete(key string) error

func (*DB) Get 

func (db *DB) Get(key string) ([]byte, error)

func (*DB) Set 

func (db *DB) Set(key string, data []byte, ttl time.Duration) error

type Params 

type Params struct {
	// contains filtered or unexported fields
}

type ParamsBuilder 

type ParamsBuilder struct {
	// contains filtered or unexported fields
}

func NewParamsBuilder 

func NewParamsBuilder(path string) *ParamsBuilder

func (*ParamsBuilder) Params 

func (pb *ParamsBuilder) Params() Params

func (*ParamsBuilder) RemoveExpiredDeltaMax 

func (pb *ParamsBuilder) RemoveExpiredDeltaMax(delta time.Duration) *ParamsBuilder

RemoveExpiredDeltaMax sets maximum random delta added to ExpiredPeriod to generate expired check period for each segment. Only works if ExpiredPeriod > 0

func (*ParamsBuilder) RemoveExpiredPeriod 

func (pb *ParamsBuilder) RemoveExpiredPeriod(period time.Duration) *ParamsBuilder

RemoveExpiredPeriod sets the period time for scheduling a background process of periodic marking expired items as deleted in inmemory state. 0 value disables the background process running

func (*ParamsBuilder) SegmentsNum 

func (pb *ParamsBuilder) SegmentsNum(number int) *ParamsBuilder

func (*ParamsBuilder) SyncPeriod 

func (pb *ParamsBuilder) SyncPeriod(period time.Duration) *ParamsBuilder

SyncPeriod sets the period time for scheduling a background process of periodic calling fsync on data file and creating checkpoint in WAL. 0 value disables the background process running

func (*ParamsBuilder) SyncPeriodDeltaMax 

func (pb *ParamsBuilder) SyncPeriodDeltaMax(delta time.Duration) *ParamsBuilder

SyncPeriodDeltaMax sets maximum random delta added to SyncPeriod to generate sync period for each segment. Only works if SyncPeriod > 0

func (*ParamsBuilder) UseWAL 

func (pb *ParamsBuilder) UseWAL(use bool) *ParamsBuilder

Source Files

View all Source files

Directories

Path Synopsis
docs
examples/with-wal
examples/without-wal

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.