README
¶
ePaxos
Actor based implementation of the ePaxos consensus algorithm.
Quick Start
To install this implementation of ePaxos on a system:
$ go get github.com/bbengfort/epaxos/...
This should install the epaxos command on your system. Create a configuration file that defines the peers for the network and other parameters as follows:
{
"timeout": "100ms",
"aggregate": true,
"log_level": 5,
"peers": [
{
"pid": 1,
"name": "alpha",
"ip_address": "127.0.0.1",
"domain": "localhost",
"port": 3264
},
{
"pid": 2,
"name": "bravo",
"ip_address": "127.0.0.1",
"domain": "localhost",
"port": 3265
},
{
"pid": 3,
"name": "charlie",
"ip_address": "127.0.0.1",
"domain": "localhost",
"port": 3266
}
]
}
The configuration file can be stored as a .toml, .json, .yml, .yaml file in the following locations:
/etc/epaxos.json~/.epaxos.json$(pwd)/epaxos.json
Or the path to the configuration file can be passed to the command at runtime. To run an ePaxos replica process:
$ epaxos serve -n alpha
The -n command specifies which peer configures the local replica, by default if no name is specified, then the hostname of the machine is used. To commit a command to the ePaxos log:
$ epaxos commit -k "key" -v "value"
This commits the command named "key" with the specified "value" to the log. Note that the client is automatically redirected to a leader in a round-robin fashion and requires the same configuration to connect.
Documentation
¶
Overview ¶
Package epaxos implements the ePaxos consensus algorithm.
Index ¶
- Constants
- Variables
- func LogLevel() string
- func SetLogLevel(level uint8)
- func SetLogger(l *log.Logger)
- type Actor
- type Callback
- type Client
- type ComplexValidator
- type Config
- func (c *Config) GetName() (name string, err error)
- func (c *Config) GetPath() (string, error)
- func (c *Config) GetPeer() (peers.Peer, error)
- func (c *Config) GetQuorum() uint32
- func (c *Config) GetRemotes() (remotes []peers.Peer, err error)
- func (c *Config) GetThrifty() []uint32
- func (c *Config) GetTimeout() (time.Duration, error)
- func (c *Config) GetUptime() (time.Duration, error)
- func (c *Config) Load() error
- func (c *Config) Update(o *Config) error
- func (c *Config) Validate() error
- type Event
- type EventType
- type Logs
- type Remote
- type Remotes
- type Replica
- func (r *Replica) Broadcast(req *pb.PeerRequest, toall bool)
- func (r *Replica) Close() error
- func (r *Replica) Commit(inst *pb.Instance)
- func (r *Replica) Connect() error
- func (r *Replica) Consensus(stream pb.Epaxos_ConsensusServer) (err error)
- func (r *Replica) Dispatch(e Event) error
- func (r *Replica) Handle(e Event) error
- func (r *Replica) Listen() error
- func (r *Replica) Propose(ctx context.Context, in *pb.ProposeRequest) (*pb.ProposeReply, error)
Constants ¶
const ( LogTrace uint8 = iota LogDebug LogInfo LogCaution LogStatus LogWarn LogSilent )
Levels for implementing the debug and trace message functionality.
const CautionThreshold = 80
CautionThreshold for issuing caution logs after accumulating cautions.
const DefaultRetries = 3
DefaultRetries specifies the number of times to attempt a commit.
const MessageBufferSize = 1024
MessageBufferSize represents the number of messages that can be queued to send to the remote server before the process starts to block to prevent back pressure.
const PackageVersion = "0.1"
PackageVersion of the current ePaxos implementation
Variables ¶
var ( ErrEventTypeError = errors.New("captured event with wrong value type") ErrEventSourceError = errors.New("captured event with wrong source type") ErrUnknownState = errors.New("epaxos in an unknown state") ErrNotListening = errors.New("replica is not listening for events") ErrRetries = errors.New("could not connect after several attempts") ErrNoNetwork = errors.New("no network specified in the configuration") ErrBenchmarkMode = errors.New("specify either fixed duration or maximum operations benchmark mode") ErrBenchmarkRun = errors.New("benchmark has already been run") )
Standard errors for primary operations.
Functions ¶
func LogLevel ¶
func LogLevel() string
LogLevel returns a string representation of the current level
func SetLogLevel ¶
func SetLogLevel(level uint8)
SetLogLevel modifies the log level for messages at runtime. Ensures that the highest level that can be set is the trace level.
Types ¶
type Actor ¶
type Actor interface {
Listen() error // Run the actor model listen for events and handle them
Close() error // Stop the actor from receiving new events (handles remaining pending events)
Dispatch(Event) error // Outside callers can dispatch events to the actor
Handle(Event) error // Handler method for each event in sequence
}
Actor objects listen for events (messages) and then can create more actors, send more messages or make local decisions that modify their own private state. Actors implement lockless concurrent operation (indeed, none of the structs in this package implement mutexes and are not thread safe independently). Concurrency here is based on the fact that only a single actor is initialized and reads event objects one at a time off of a buffered channel. All actor methods should be private as a result so they are not called from other threads.
type Client ¶
Client maintains network information embedded in the configuration to connect to an ePaxos consensus quorum and make propose requests.
func (*Client) Propose ¶
func (c *Client) Propose(access pb.AccessType, key string, value []byte) (rep *pb.ProposeReply, err error)
Propose an operation to be applied to the state store.
type ComplexValidator ¶
type ComplexValidator struct {
TagName string
}
ComplexValidator validates complex types that multiconfig doesn't understand
func (*ComplexValidator) Validate ¶
func (v *ComplexValidator) Validate(s interface{}) error
Validate implements the multiconfig.Validator interface.
type Config ¶
type Config struct {
Name string `required:"false" json:"name,omitempty"` // unique name of the local replica, hostname by default
Seed int64 `required:"false" json:"seed,omitempty"` // random seed to initialize random generator
Timeout string `default:"500ms" validate:"duration" json:"timeout"` // timeout to wait for responses (parseable duration)
Aggregate bool `default:"false" json:"aggregate"` // aggregate operations from multiple concurrent clients
Thrifty bool `default:"false" json:"thrifty"` // whether or not to send thrifty quorum messages
LogLevel int `default:"3" validate:"uint" json:"log_level"` // verbosity of logging, lower is more verbose
Peers []peers.Peer `json:"peers"` // definition of all hosts on the network
// Experimental configuration
// TODO: remove after benchmarks
Uptime string `required:"false" validate:"duration" json:"uptime"` // run for a time limit and then shutdown
Metrics string `requred:"false" json:"metrics"` // location to write benchmarks to disk
}
Config uses the multiconfig loader and validators to store configuration values required to run ePaxos. Configuration can be stored as a JSON, TOML, or YAML file in the current working directory as epaxos.json, in the user's home directory as .epaxos.json or in /etc/epaxos.json (with the extension of the file format of choice). Configuration can also be added from the environment using environment variables prefixed with $EPAXOS_ and the all caps version of the configuration name.
func (*Config) GetName ¶
GetName returns the name of the local host defined by the configuration or using the hostname by default.
func (*Config) GetPath ¶
GetPath searches possible configuration paths returning the first path it finds; this path is used when loading the configuration from disk. An error is returned if no configuration file exists.
func (*Config) GetPeer ¶
GetPeer returns the local peer configuration or an error if no peer is found in the configuration. If the name is not set on the configuration, the hostname is used.
func (*Config) GetQuorum ¶
GetQuorum returns the number of replicas required for a quourm based on the peers defined in the configuration.
func (*Config) GetRemotes ¶
GetRemotes returns all peer configurations for remote hosts on the network, e.g. by excluding the local peer configuration.
func (*Config) GetThrifty ¶
GetThrifty returns the peers to send broadcast messages to. If not thrifty, it returns nil, otherwise it returns the next n peers by PID where n is one less than the majority of replicas.
func (*Config) GetTimeout ¶
GetTimeout parses the timeout duration and returns it.
func (*Config) Load ¶
Load the configuration from default values, then from a configuration file, and finally from the environment. Validate the configuration when loaded.
type Event ¶
type Event interface {
Type() EventType
Source() interface{}
Value() interface{}
}
Event represents actions that occur during consensus. Listeners can register callbacks with event handlers for specific event types.
type EventType ¶
type EventType uint16
EventType is an enumeration of the kind of events that can occur.
type Logs ¶
type Logs struct {
// contains filtered or unexported fields
}
Logs is a 2D array that maintains the state of all replicas by keeping track of Instances of operations that can be applied to the cluster state. This type exposes a number of helpful methods to interact with the log and sequence numbers.
An instance in a log is identified by (pid, slot) where the PID is the id of the replica who is the quorum leader for the instance and the slot is the index in the log for that replica where the instance resides. Instances are further defined by sequence numbers and dependencies. Sequence numbers expose the order that the instance was applied to this particular log, and dependencies order all instnaces with respect to all internal logs.
func (*Logs) Create ¶
Create an instance based on the current log state and append it. This is a helper method for quickly adding a set of operations to the log in a consistent manner.
type Remote ¶
Remote maintains a connection to a peer on the network.
func (*Remote) Close ¶
Close the messenger process gracefully by closing the messages channel, wait for the messenger to finish sending the last messages, then clean up the connection to the remote peer. Note that any sends after Close() will cause a panic.
func (*Remote) Connect ¶
Connect to the remote peer and establish birdirectional streams. This is the external version of connect that is used to establish the go routine that will send and receive messages from the remote host and dispatch them to the replic'as event chan. It is the messenger routine's responsibility to keep the channel open.
func (*Remote) Send ¶
func (c *Remote) Send(req *pb.PeerRequest)
Send a message to the remote. This places the message on a buffered channel, which will be sent in the order they are received. The response is dispatched to the actor event listener to be handled in the order responses are received.
func (*Remote) SendBeacon ¶
func (c *Remote) SendBeacon()
SendBeacon sends a beacon message that establishes a conenction to the remote.
type Replica ¶
Replica represents the local consensus replica and is the primary object implemented in a running system. There should only be one replica per process.
func (*Replica) Broadcast ¶
func (r *Replica) Broadcast(req *pb.PeerRequest, toall bool)
Broadcast a request to all members in the quorum using thrifty communications if so configured. The toall flag forces the request to be broadcast even if thrifty.
func (*Replica) Close ¶
Close the event handler and stop listening for events. TODO: gracefully shutdown the grpc server as well.
func (*Replica) Commit ¶
Commit an instance and broadcast the commit to all members in the quroum and reply to the client(s) that initiated the proposal.
func (*Replica) Connect ¶
Connect the replica to its remote peers. If in thrifty mode, only connects to its thrifty neighbors rather than establishing connections to all peers.
func (*Replica) Consensus ¶
func (r *Replica) Consensus(stream pb.Epaxos_ConsensusServer) (err error)
Consensus receives PeerRequest messages from remote peers and dispatches them to the primary replica process. This method waits for the handler to create a reply before receiving the next message.
func (*Replica) Propose ¶
func (r *Replica) Propose(ctx context.Context, in *pb.ProposeRequest) (*pb.ProposeReply, error)
Propose is the primary entry point for client requests. This method is the gRPC handler that essentially dispatches the propose event to the replica and listens for the replica to send back a response so the client can be replied to.
Source Files
Directories