memlist

package module
v0.0.0-...-2da433a Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 28, 2023 License: MIT Imports: 14 Imported by: 1

README

Memlist

Go Report Card GoDoc

Memlist is a peer-to-peer cluster membership package that uses the SWIM protocol, to detect member failures and communicate them through a gossip-style method. This package can be used as a platform to manage membership within a distributed system. Enabling things such as service discovery and data consistency among the entire cluster.

Memlist is a weakly-consistent protocol, meaning data will eventually become consistent among the entire cluster. The speed at which it becomes consistent largely depends on the chosen configuration. Node failures are detected though a periodic "ping-ack" communication and then disseminated across the cluster, in the case of a new gossip event.

Usage

Memlist is very simple to use and only takes a few lines to get up and running.

// You can use the default local config or create your own configuration.
conf := memlist.DefaultLocalConfig()
mem, err := memlist.Create(conf)
if err != nil {
	log.Fatal("Failed to create member: ", err)
}

// Join the cluster with the address of one of the members of that cluster.
if err := mem.Join(":7990"); err != nil {
	log.Fatal("Failed to join another cluster: ", err)
}

// Get information about other nodes in the cluster.
for node := range mem.AllNodes() {
	log.Println(node)
}

// Now the member will keep track of the cluster in the background, enabling 
// you to go ahead and do other needed operations.
// 
// A Listener can be used to receive events when a change in membership occurs. 
// This must be implemented by the client and injected into the Config before 
// creating a member. 

In Action

Watch as memlist sucessfully identifies and forms a cluster with three nodes and automatically detects a failure in the THIRD node and successfully spreading this information accross the rest of the cluster.

https://user-images.githubusercontent.com/56979977/138744132-60343718-998c-401c-b279-8030e3efb247.mov

Connect & Contact

Email - mathewestafanous13@gmail.com

Website - https://mathewestafanous.com

GitHub - https://github.com/Mathew-Estafanous

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config

type Config struct {
	// A Name for the Node that is unique to the entire cluster.
	Name string

	// Configuration related to which address and Port the Node will
	// bind to and listen on.
	BindAddr string
	BindPort uint16

	// PingInterval is the time between attempts at random pings towards
	// another Node. Decreasing the interval will result in more frequent
	// checks at the cost of increased bandwidth usage.
	PingInterval time.Duration
	// PingTimeout is the timeout a Node will wait for an ACK response from
	// any member before determining that the Node is potentially unhealthy.
	PingTimeout time.Duration

	// PiggyBackLimit is used as the maximum number of gossip events that are added
	// (piggyback) off of a Node's ping/ack messages.
	//
	// Consider a slider when determining the limit. The lower the limit the smaller
	// the messages and latency will decrease; however, gossip events will take
	// longer to disseminate. Opposite, when increasing the limit, messages will
	// be larger and latency increases while gossip will spread exponentially faster.
	//
	// DefaultLocalConfig chooses 4 as an arbitrary limit.
	PiggyBackLimit int

	// IndirectChecks is the number of nodes that will be contacted in the case
	// that an indirect sendPing is required. Increasing the number of checks will
	// also increase the chances of an indirect sendPing succeeding. This is at the
	// expense of bandwidth usage.
	IndirectChecks int

	// Transport is an optional field for the client to define custom
	// communication among other nodes. If this field is left nil, Create will
	// by default use a NetTransport in the Member.
	Transport Transport

	// TCPTimeout is the time in which a TCP connection will be attempted. If no
	// connection is made before reaching the timeout, then the attempt will fail.
	TCPTimeout time.Duration

	// EventListener can be used to inject the client's implementation of the Listener.
	// If nothing is injected, then Create will use a fake listener in its place.
	EventListener Listener
}

Config provides different options that can be adjusted according to what is needed of the member. Some fields are required and should not be left empty when creating a new member.

func DefaultLocalConfig

func DefaultLocalConfig() *Config

DefaultLocalConfig returns a configuration that is set up for a local environment.

type Gossip

type Gossip struct {
	Gt gossipType

	// Node is the peer that the Gossip refers to.
	Node Node
}

Gossip represents a message that is sent to peers regarding changes in the state of the cluster and other nodes.

type Listener

type Listener interface {
	// OnMembershipChange is called when there is a change in the state of a
	// peer or if a peer has joined/left the cluster.
	OnMembershipChange(peer Node)
}

Listener is implemented by the client as a way to hook into the membership and gossip layer of the Member. Providing a way to listen for important events. This listener must be thread-safe.

type Member

type Member struct {
	// contains filtered or unexported fields
}

func Create

func Create(conf *Config) (*Member, error)

Create a new member using the given configuration file. The member will start listening for packets that are sent from the Transport.

The Member has not joined any cluster at this point. To join a cluster of nodes, look into using Join.

Upon creating a member with the Config, the data must NOT be altered and is assumed to remain unchanged throughout the life of the Member.

func (*Member) AllNodes

func (m *Member) AllNodes() []Node

AllNodes will return every known alive node at the time.

func (*Member) Join

func (m *Member) Join(addr string, data any) error

Join will attempt to join the member into a cluster of nodes by connecting to the Node at the given address. An error will be return if the member failed to join a cluster.

func (*Member) Leave

func (m *Member) Leave(timeout time.Duration) error

Leave will safely stop all running processes and will notify other nodes that it will be leaving the cluster. This is a blocking operation until the member has successfully left the cluster or the timeout has been reached.

func (*Member) Shutdown

func (m *Member) Shutdown() error

Shutdown will stop all background processes such as responding to received packets. No message will be sent regarding leaving the cluster and as a result the member will eventually be considered 'dead'.

This method should only be called once. If called more than once, a non-nil error will be returned.

If you want your member to notify other members about leaving the cluster look into using Leave instead.

func (*Member) TotalNodes

func (m *Member) TotalNodes() int

TotalNodes returns the total number of known peers that are alive.

type NetTransport

type NetTransport struct {
	// contains filtered or unexported fields
}

NetTransport is the standard implementation of the Transport and should be enough for most use cases.

func NewNetTransport

func NewNetTransport(addr string, port uint16) (*NetTransport, error)

NewNetTransport will create and return a NetTransport that is properly setup with udp and tcp listeners.

func (*NetTransport) DialAndConnect

func (n *NetTransport) DialAndConnect(addr string, timeout time.Duration) (net.Conn, error)

func (*NetTransport) Packets

func (n *NetTransport) Packets() <-chan *Packet

func (*NetTransport) SendTo

func (n *NetTransport) SendTo(b []byte, addr string) error

func (*NetTransport) Shutdown

func (n *NetTransport) Shutdown() error

func (*NetTransport) Stream

func (n *NetTransport) Stream() <-chan net.Conn

type Node

type Node struct {
	Name  string
	Addr  string
	Port  uint16
	State StateType
	Data  any
}

Node represents a single node within the cluster and their state within the cluster.

func (*Node) String

func (n *Node) String() string

type Packet

type Packet struct {
	// Buf is the raw content of the payload.
	Buf []byte

	// From exposes the peer's (sender) address.
	From net.Addr
}

Packet represents the incoming packetCh and the peer's associated data including the message payload.

type StateType

type StateType int
const (
	Alive StateType = iota
	Dead
	Left
)

type Transport

type Transport interface {
	// SendTo will forward the provided byte payload to the given address.
	// This message is expected to be done in a connectionless manner, meaning
	// a response is not guaranteed when the method returns.
	SendTo(b []byte, addr string) error

	// DialAndConnect will create a connection to another peer allowing for a
	// direct two-way connection between both peers.
	DialAndConnect(addr string, timeout time.Duration) (net.Conn, error)

	// Packets returns a channel that is used to receive incoming packets
	// from other peers.
	Packets() <-chan *Packet

	// Stream returns a read only channel that is used to receive incoming
	// streamCh connections from other peers. A streamCh is usually sent during
	// attempts at syncing state between two peers.
	Stream() <-chan net.Conn

	// Shutdown allows for the transport to clean up all listeners safely.
	Shutdown() error
}

Transport is an interface designed to abstract away the communication details among the member nodes.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL