mydis

Mydis

Introduction

Distributed, reliable database and cache library, server, and client. Inspired by Redis, Mydis is written entirely in Go and can be used as a library, embedded into an existing application, or as a standalone client/server.

Basics

Mydis can store multiple types of data: strings, bytes, integers, floats, lists, and hashes (objects that hold key/value pairs). Each item is referenced with a key, a string of any length. The Mydis library, server, and client are thread/goroutine-safe. Client and server communication is handled with gRPC. All data types can have an expiration value set. Backwards compatibility with HTTP/1.1 is handled by gRPC-Gateway and must be run on a separate port. Both client and peer connections using gRPC are encrypted by default.

Details

Under the hood, the production-ready Etcd system is used. The creators describe it as such:

Etcd is a distributed, reliable key-value store for the most critical data of a distributed system.

In an effort to keep the focus on reliability and consistency, the authors of Etcd decided to allow only a single data type for both keys and values: byte arrays. Mydis builds upon the solid Etcd framework to provide more data types and features such as atomic list operations and distributed locks.

Load Balancing

The client supports Round Robin load balancing by calling mydis.NewClientConfigAddresses(addresses []string) or by changing the Addresses value in an existing ClientConfig instance. The client will connect to all servers specified and round robin each request across the pool of connections. Since 1.2.0, Mydis supports graceful reconnects in the event of a server or network outage.

Versioning

Versioning is done using SemVer. All bug fixes and new features should happen in separate branches and a pull request made when ready to merge. The master branch is the development branch and stable releases are tagged with the version number.

Configuration

Without any configuration, Mydis starts up as a single node cluster on port 8383 with a storage limit of 2GB. This can be changed by creating a YAML configuration file named mydis.conf in either /etc/mydis or in the same directory as the executable. To change the storage limit from the 2GB default, set quota-backend-bytes to the number of bytes the new limit should be. Clustering configuration is explained in another section. It is recommended to leave listen-client-urls and advertise-client-urls as an empty list unless you want clients to be able to bypass Mydis and connect directly to Etcd.

See Etcd's documentation for more configuration options.

The gRPC listening port can be changed from the default of 8383 by specifying the environment variable MYDIS_ADDRESS. The default value is 0.0.0.0:8383. Likewise, the gRPC-Gateway port can be changed by specifying the environment variable PORT. The default value is 8000.

Clustering

Clustering is handled entirely by Etcd. Its documentation explains the configuration required to create a cluster. To summarize, clusters should always have an odd number of nodes.

Example cluster configuration:

Nodes

• node1=10.0.0.1
• node2=10.0.0.2
• node3=10.0.0.3

Config for Node1

initial-advertise-peer-urls:
- https://10.0.0.1:2380

listen-peer-urls:
- https://10.0.0.1:2380

ClientAutoTLS: true
PeerAutoTLS: true

name: node1
initial-cluster: node1=https://10.0.0.1:2380,node2=https://10.0.0.2:2380,node3=https://10.0.0.3:2380
initial-cluster-token: name-of-cluster

Config for Node2

initial-advertise-peer-urls:
- https://10.0.0.2:2380

listen-peer-urls:
- https://10.0.0.2:2380

ClientAutoTLS: true
PeerAutoTLS: true

name: node2
initial-cluster: node1=https://10.0.0.1:2380,node2=https://10.0.0.2:2380,node3=https://10.0.0.3:2380
initial-cluster-token: name-of-cluster

Some notes about this configuration:

• Nodes will communicate over an encrypted HTTPS socket using auto generated keys.
• Mydis will use the value of ClientAutoTLS or the client-specific TLS options to enable client encryption.
• It is highly recommended to always use ClientAutoTLS: true and PeerAutoTLS: true unless you specify your own TLS configuration. The client tries to use AutoTLS by default.
• The IP address 127.0.0.1 or 0.0.0.0 should never be used in any configuration option. The node's actual IP address should be used.
• The name is used in initial-cluster to determine which address in the list belongs to the current node, so it must match.
• The only fields that should change between nodes are: initial-advertise-peer-urls, listen-peer-urls, and name.
• While setting initial-cluster-token is optional, it's always a good idea to give each cluster a name.

See Etcd's clustering guide for more information.

Server API

When used as a library, the server API uses protocol buffer messages with context, as required by gRPC.

Client API

Some getter functions return a helper object, called Value that allows you to specify what data type you would like the response in. The data type functions available include:

• Bytes()
• String()
• Bool()
• Proto(proto.Message)
• Int()
• Float()
• Time()
• Duration()
• List()
• Map()

All of these functions return the desired data type and an error if there was a problem getting or converting the value.

Keys

Keys are strings of any length.

Functions

• Keys() []string: Get list of keys available in the database.
• KeysWithPrefix() []string: Gets a list of keys with the given prefix.
• Has(key) bool: Determine if a key exists.
• SetExpire(key, exp): Reset the expiration of a key to the number of seconds from now.
• Delete(key): Delete a key.
• Clear(): Clear the database.

Strings/Bytes

Strings can be text or byte arrays. Conversion between string and bytes is done without any additional memory allocations.

Functions

• Get(key) Value: Get a value, returns ErrKeyNotFound if key doesn't exist.
• GetMany(keyList) map[string]Value: Get multiple values.
• GetWithPrefix(prefix) map[string]Value: Gets the keys with the given prefix.
• Set(key, value): Set a value.
• SetNX(key, value) bool: Set a value only if the key doesn't exist, returns true if changed.
• SetMany(values) map[string]string: Set many values, returning a map[key]errorText for any errors.
• Length(key) int64: Get the number of bytes stored at the given key.

Numbers

Numbers can be 64-bit integers or floating-point values.

Functions

• IncrementInt(key, by) int64: Increment an integer and return new value, starts at zero if key doesn't exist.
• IncrementFloat(key, by) float64: Increment a float and return new value, starts at zero if key doesn't exist.
• DecrementInt(key, by) int64: Decrement an integer and return new value, starts at zero if key doesn't exist.
• DecrementFloat(key, by) float64: Decrement a float and return new value, starts at zero if key doesn't exist.

Lists

Lists are lists of values.

Functions

• GetListItem(key, index) Value: Get a single item from a list by index, returns ErrKeyNotFound if key doesn't exist, or ErrorListIndexOutOfRange if index is out of range.
• SetListItem(key, index, value): Set a single item in a list by index.
• ListHas(key, value) int64: Determines if a list has a value, returns index or -1 if not found.
• ListLimit(key, limit): Sets the maximum length of a list, removing items from the top once limit is reached. Evaluated on insert and append only.
• ListInsert(key, index, value): Insert an item in a list at the given index, creates new list and inserts item at index zero if key doesn't exist.
• ListAppend(key, value): Append an item to the end of a list, creates new list if key doesn't exist.
• ListPopLeft(key) Value: Remove and return the first item in a list, returns ErrListEmpty if list is empty.
• ListPopLeftBlock(key, seconds) Value: Remove and return the first item in a list, or wait the given seconds for a new value, setting to zero waits forever.
• ListPopRight(key) Value: Remove and return the last item in a list, returns ErrListEmpty if list is empty.
• ListPopRightBlock(key, seconds) Value: Remove and return the last item in a list, or wait the given seconds for a new value, setting to zero waits forever.
• ListDelete(key, index): Remove an item from a list by index, returns an error if key or index doesn't exist.
• ListDeleteItem(key, value) int64: Search for and remove the first occurrence of value from the list, returns index of item or -1 for not found.
• ListLength(key) int64: Get the number of items in a list.

Hashes

Hashes are objects with multiple string fields.

Functions

• GetHashField(key, field) Value: Get a single field from a hash, returns ErrHashFieldNotFound if field doesn't exist.
• GetHashFields(key, fields) map[string]Value: Get multiple fields from a hash, non-existent keys will not be added to the resulting map.
• HashHas(key) bool: Determines if a hash has a field, returns true or false.
• HashFields(key) []string: Get a list of hash fields.
• HashValues(key) []Value: Get a list of hash values.
• HashLength(key) int64: Get the number of fields in a hash.
• SetHashField(key, field, value): Set a single field in a hash, creates new hash if key doesn't exist.
• SetHashFields(key, values): Set multiple fields in a hash, creates new hash if key doesn't exist.
• DelHashField(key, field): Delete a single field from a hash.

Locks

Keys can be locked from modification.

Functions

• Lock(key): Lock a key, waiting a default of 5 seconds if a lock already exists on the key before returning ErrKeyLocked.
• LockWithTimeout(key, seconds): Lock a key, waiting for the given number of seconds if already locked before returning ErrKeyLocked.
• Unlock(key): Unlock a key.
• UnlockThenSet(key, value): Unlock a key, then immediately set its value.
• SetLockTimeout(seconds): Sets the default timeout in seconds if key is already locked.

Events

Using the event handling feature, you can be notified when a key changes.

Functions

• Watch(key, prefix): Get a notification event when a key changes. When calling one of the set functions, subscribed clients will be notified, including the sender if subscribed. If prefix is true, watches all keys with the given prefix.
• UnWatch(key, prefix): Stop getting notifications when a key changes.
• NewEventChannel(): Returns a new Event channel.
• CloseEventChannel(id): Closes an Event channel.

Authentication

Authentication is handled entirely by Etcd. All auth commands are proxied to Etcd for processing. Once authentication is enabled, the functions below (with the exepction of Authenticate and LogOut) will require a user with the root role. Etcd requires a root user and role to be created before authentication can be enabled. Any user can be assigned the root role afterwards, but the root user must remain for recovery purposes.

Functions

• AuthEnable(): Enable authentication (must have root user and role created).
• AuthDisable(): Disable authentication.
• Authenticate(username, password): Authenticate as the given user.
• LogOut(): Client-only method, clears the authentication token set by the call to Authenticate.
• UserAdd(username, password): Creates a new user.
• UserGet(username): Get list of roles for the given user.
• UserList(): Get list of all usernames.
• UserDelete(username): Delete a user.
• UserChangePassword(username, password): Change a user's password.
• UserGrantRole(username, role): Grant the user a role.
• UserRevokeRole(username, role): Revoke a role from the user.
• RoleAdd(role): Create a new role.
• RoleGet(role): Get list of permissions for the given role.
• RoleList(): Get list of all roles.
• RoleDelete(role): Delete a role.
• RoleGrantPermission(role, perm): Grant the role a permission.
• RoleRevokePermission(role, perm): Revoke a permission from a role.

See Etcd's API documentation or the Authentication Guide for version 2. Even though the guide is for an older version of Etcd, the concepts and configuration haven't changed in version 3.

Usage

Server Library

server := mydis.NewServer(mydis.NewServerConfig())
if err := server.Start(":8000", ":8383"); err != nil {
	log.Fatalln(err)
}

Client Library

import "github.com/deejross/mydis/client"

myClient := client.NewClient(myClient.NewClientConfig("localhost:8383"))
s, err := myClient.Get("key").String()
if err != nil {
	return err
}

Limitations

Etcd limits message sizes to 1.5MB, so values cannot be larger than this. The maximum storage size is configurable up to 8GB.

Help Wanted

If you encounter a bug or have an idea for a feature, please open a GitHub issue.

Development

If you are interested in setting up an environment to modify Mydis and submit a pull request, you can follow the instructions in the SETUP document.

License

Mydis is licensed under the Apache 2.0 license. See the LICENSE file for details.