README
¶
huffleraft
Replicated fault-tolerant key-value store driven by hashicorp/raft and dgraph-io/badger
Please read the
example
to see the system in action, especially for CRUD operations on a 5 node cluster!
For full documentation and more thorough descriptions: godoc
import "github.com/s4ayub/huffleraft"
Features and Purpose:
A RaftStore instance encapsulates a raft node (from hashicorp/raft) for consensus, a storage system (from dgraph-io/badger) for key-value pairs and an HTTP server to accept accept and redirect requests to the leader node.
The purpose of this package is to explore the raft consensus algorithm, specifically hashicorp's implementation. This package is effective for experimenting with raft because the API is such that the user doesn't even have to build any HTTP requests, rather, the package does that in the background through Get, Set, Delete, and Join. This makes it very easy and quick to play around with raft in conjunction with other code.
- Perform CRUD operations on draph-io's Badger storage in a distributed manner
- Fault tolerant as per the raft consensus algorithm
- Commands can be performed on any node in a cluster and they'll be redirected to the leader of the cluster
- Logs are truncated using the Snapshot, Restore and Persist mechanism provided by hashicorp/raft (checkout fsm.go)
Example logs:
Once a couple RaftStore instances are running, and a cluster is formed, the logs may look something like this after a Set command has been performed on them:
2017/09/04 04:02:07 [DEBUG] raft: Node 127.0.0.1:8740 updated peer set (2): [127.0.0.1:8730 127.0.0.1:8750 127.0.0.1:8740]
2017/09/04 04:02:07 [DEBUG] raft-net: 127.0.0.1:8730 accepted connection from: 127.0.0.1:62560
2017/09/04 04:02:07 [DEBUG] raft-net: 127.0.0.1:8740 accepted connection from: 127.0.0.1:62561
[raftStore | 127.0.0.1:8750]2017/09/04 04:02:11 key: yes, value: no, set
Motivation and References:
I wanted to dive deeper into distributed systems by building one of my own. The following repository was helpful in the development of huffleraft:
- hraftd
- This is a simple reference use of hashicorp/raft also made to study the implementation of the consensus algorithm. However, it required the user to interact with the system as a client and use curl to submit requests to the server. My package builds on top of this by being embeddable inside code, implementing command redirection to the leader node and by using dragph-io's badger as a faster storage compared to a native Go map. The people at hashicorp were very quick to answer any questions I had as well.
Design Decisions:
-
Leader re-direction:
- The
RaftStorestruct encases an http listener to handle requests and aRaftServerto handle the consensus. There are two important addresses associated with each raft store. The httpAddr listens for http requests, and the raftAddr which the raft server uses for its transport layer between raft nodes. As per the consensus algorithm, commands such as setting, deleting and joining must be sent to the leader node only. The leader's raftAddr can easily be accessed in a cluster byraftServer.Leader()(raftServer being an instance of Raft from hashicorp/raft). However, the httpAddr associated cannot be easily retrieved, hence, a link must be made between the raftAddr and the httpAddr, since the httpAddr is the one that the user would be sending their commands to. I decided that the httpAddr would always be 1 port away from the raftAddr. The httpAddr for a RaftStore instance can always be retrieved now by adding 1 to its raftAddr.
- The
-
Using dgraph-io/badger:
- Bagder was proposed as a performant Go alternative to RocksDB which made it an attractive storage back-end for the key-value store.
Caveats and Suggested Improvements:
- Cannot use side by side ports (8866 and 8867 for example) for two raft servers because the port next to the one on which a raft server is initiated on, is used for http requests. This is done to establish a link between the raftAddr and the httpAddr as discussed above. This routing system can likely be improved upon using a custom multiplexer setup where requests can be sent to the same port that the raft node exists on. Something like this in rqlite: https://github.com/rqlite/rqlite/tree/master/tcp
- More robust error handling
Documentation
¶
Overview ¶
huffleraft provides a package to run a distributed key value store driven by dgraph-io/badger for storage and hashicorp/raft for consensus. It can be integrated wtihin code much like embedded storage systems like leveldb. The package can be imported and does NOT require the user to make HTTP requests through curl and treat the system as an external application. Rather, using the API, this system can be embedded within applications. Firstly, the user makes a new badger store using NewBadgerKV. This is the storage for the key-value pairs for each raft node. A RaftStore using an appropriate port and directory. Start() is called on the RaftStore which starts the HTTP server to accept requests. This RaftStore can be interacted with by using Get, Set, Delete, and Join (to add nodes to a cluster). Check out the fsm.go file to see how logs are truncated using raft's Snapshot, Restore, and Persist mechanism. For more details check out the github repo: https://github.com/s4ayub/huffleraft and also the examples/ folder!!!
Index ¶
- type BadgerKV
- type RaftStore
- func (rs *RaftStore) Close()
- func (rs *RaftStore) Delete(key string) error
- func (rs *RaftStore) Get(key string) ([]byte, error)
- func (rs *RaftStore) Join(addr string) error
- func (rs *RaftStore) ServeHTTP(w http.ResponseWriter, r *http.Request)
- func (rs *RaftStore) Set(key, value string) error
- func (rs *RaftStore) Start() error
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BadgerKV ¶
type BadgerKV struct {
// contains filtered or unexported fields
}
func NewBadgerKV ¶
NewBadgerKV returns a BadgerKV. If the directory in which the data will be stored does not exist, then that directory will be made for the user.
type RaftStore ¶
A RaftStore encapsulates the http server (httpAddr, ln), a raft node (raftDir string, raftAddr string, RaftServer *raft.Raft) and a key-value store (kvs *BadgerKV).
func NewRaftKV ¶
NewRaftKV returns a RaftStore. nableSingle is used to determine whether a node should be allowed to elect itself as a leader. Set this option to true if you plan to send commands to a single node.
func (*RaftStore) Close ¶
func (rs *RaftStore) Close()
Close closes the HTTP server of the RaftStore.
func (*RaftStore) Delete ¶
Sends a request to delete a key to the http server of the leader node in the raft cluster. This key is deleted inside the badger stores of the RaftStore instances throughout cluster. This command REQUIRES a leader in a cluster.
func (*RaftStore) Join ¶
The addr provided is that of the node that would like to join a cluster, and the method Join is run on a node within that cluster. It sends a request to join a cluster to the http server of the leader node in the cluster. If there is no leader, then the http address of the node this method was run on will be used. This command does not require a leader in a cluster.
func (*RaftStore) ServeHTTP ¶
func (rs *RaftStore) ServeHTTP(w http.ResponseWriter, r *http.Request)
ServeHTTP allows the RaftStore to serve HTTP requests.
func (*RaftStore) Set ¶
Sends a request to set a key-value to the http server of the leader node in the raft cluster. This pair is then stored inside the badger stores of the RaftStore instances throughout cluster. This command REQUIRES a leader in a cluster.
func (*RaftStore) Start ¶
Start starts the http service using the listener within a RaftStore. The HTTP server is used to redirect commands like Set, Delete and Join to the leader RaftStore in a cluster. The HTTP address is always one away from the Raft address which the raft node uses for communication with other raft nodes.
Source Files
Directories