ringpop

package module
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 22, 2022 License: MIT Imports: 24 Imported by: 70

README

ringpop-go Build Status Coverage Status

(This project is no longer under active development.)

Ringpop is a library that brings cooperation and coordination to distributed applications. It maintains a consistent hash ring on top of a membership protocol and provides request forwarding as a routing convenience. It can be used to shard your application in a way that's scalable and fault tolerant.

Getting started

To install ringpop-go:

go get github.com/uber/ringpop-go

Developing

First make certain that thrift (OSX: brew install thrift) and glide are in your path (above). Then,

make setup

to install remaining golang dependencies and install the pre-commit hook.

Finally, run the tests by doing:

make test

Documentation

Interested in where to go from here? Read the docs at ringpop.readthedocs.org

Documentation

Overview

Package ringpop is a library that maintains a consistent hash ring atop a gossip-based membership protocol. It can be used by applications to arbitrarily shard data in a scalable and fault-tolerant manner.

Index

Constants

View Source 
const StatPeriodDefault = time.Duration(5 * time.Second)

StatPeriodDefault defines the default emission period for a periodic stat.

View Source 
const StatPeriodNever = time.Duration(-1)

StatPeriodNever defines a "period" which disables a periodic stat emission.

Variables

View Source 
var (
	// ErrNotBootstrapped is returned by public methods which require the ring to
	// be bootstrapped before they can operate correctly.
	ErrNotBootstrapped = errors.New("ringpop is not bootstrapped")

	// ErrEphemeralAddress is returned by the address resolver if TChannel is
	// using port 0 and is not listening (and thus has not been assigned a port by
	// the OS).
	ErrEphemeralAddress = errors.New("unable to resolve this node's address from channel that is not yet listening")

	// ErrChannelNotListening is returned on bootstrap if TChannel is not
	// listening.
	ErrChannelNotListening = errors.New("tchannel is not listening")

	// ErrInvalidIdentity is returned when the identity value is invalid.
	ErrInvalidIdentity = errors.New("a hostport is not valid as an identity")
)

Functions

func DeserializeThrift 

func DeserializeThrift(b []byte, s athrift.TStruct) error

DeserializeThrift takes a byte slice and attempts to write it into the given thrift struct using the thrift binary protocol. This is a temporary measure before frames can forwarded directly past the endpoint to the proper destinaiton.

func SerializeThrift 

func SerializeThrift(s athrift.TStruct) ([]byte, error)

SerializeThrift takes a thrift struct and returns the serialized bytes of that struct using the thrift binary protocol. This is a temporary measure before frames can forwarded directly past the endpoint to the proper destinaiton.

Types

type AddressResolver added in v0.8.0 

type AddressResolver func() (string, error)

AddressResolver is a function that returns the listen interface/port that Ringpop should use as its address.

type Arg 

type Arg struct{}

Arg is a blank arg

type Interface 

type Interface interface {
	Destroy()
	App() string
	WhoAmI() (string, error)
	Uptime() (time.Duration, error)
	Bootstrap(opts *swim.BootstrapOptions) ([]string, error)
	Checksum() (uint32, error)
	Lookup(key string) (string, error)
	LookupN(key string, n int) ([]string, error)
	GetReachableMembers(predicates ...swim.MemberPredicate) ([]string, error)
	CountReachableMembers(predicates ...swim.MemberPredicate) (int, error)

	HandleOrForward(key string, request []byte, response *[]byte, service, endpoint string, format tchannel.Format, opts *forward.Options) (bool, error)
	Forward(dest string, keys []string, request []byte, service, endpoint string, format tchannel.Format, opts *forward.Options) ([]byte, error)

	Labels() (*swim.NodeLabels, error)

	// events.EventRegistar
	// mockery has troubles generating a working mock when the interface is
	// embedded therefore the definitions are copied here.
	AddListener(events.EventListener) bool
	RemoveListener(events.EventListener) bool

	// DEPRECATED, use AddListener (!) kept around for backwards compatibility
	// but will start logging warnings
	RegisterListener(events.EventListener)

	// swim.SelfEvict
	// mockery has troubles generating a working mock when the interface is
	// embedded therefore the definitions are copied here.
	RegisterSelfEvictHook(hooks swim.SelfEvictHook) error
	SelfEvict() error
}

Interface specifies the public facing methods a user of ringpop is able to use.

type Option 

type Option func(*Ringpop) error

An Option is a modifier functions that configure/modify a real Ringpop object.

There are typically two types of runtime options you can provide: flags (functions that modify the object) and value options (functions the accept user-specific arguments and then return a function that modifies the object).

For more information, see: http://commandcenter.blogspot.com/2014/01/self-referential-functions-and-design.html

func Address added in v0.8.0 

func Address(address string) Option

Address is used to specify a static hostport string as this Ringpop instance's address.

Example: 

ringpop.New("my-app",
    ringpop.Channel(myChannel),
    ringpop.Address("10.32.12.2:21130"),
)

You should make sure the address matches the listening address of the TChannel object.

By default, you do not need to provide an address. If you do not provide one, the address will be resolved automatically by the default resolver.

func AddressResolverFunc added in v0.8.0 

func AddressResolverFunc(resolver AddressResolver) Option

AddressResolverFunc is used to specify a function that will be called when the Ringpop instance needs to resolve its address (typically, on bootstrap).

func Channel 

func Channel(ch shared.TChannel) Option

Channel is used to provide a TChannel instance that Ringpop should use for all communication.

Example: 

rp, err := ringpop.New("my-app", ringpop.Channel(myChannel))

Channel is a required option. The constructor will throw an error if this option is not present.

func Clock added in v0.3.0 

func Clock(c clock.Clock) Option

Clock is used to set the Clock mechanism. Testing harnesses will typically replace this with a mocked clock.

func FaultyPeriod added in v0.4.0 

func FaultyPeriod(period time.Duration) Option

FaultyPeriod configures the period Ringpop keeps a faulty node in its memberlist. Even though the node will not receive any traffic it is still present in the list in case it will come back online later. After this timeout ringpop will remove the node from its membership list permanently. If a node happens to come back after it has been removed from the membership Ringpop still allows it to join and take its old position in the hashring. To remove the node from the distributed membership it will mark it as a tombstone which can be removed from every members membership list independently.

func HashRingConfig 

func HashRingConfig(c *hashring.Configuration) Option

HashRingConfig takes a `HashRingConfiguration` struct that can be used to configure the hash ring.

Example: 

rp, err := ringpop.New("my-app",
    ringpop.Channel(myChannel),
    ringpop.HashRingConfig(&HashRingConfiguration{
        ReplicaPoints: 100,
    }),
)

See documentation on the `HashRingConfiguration` struct for more information about what options are available.

func Identity 

func Identity(identity string) Option

Identity can be used to specify a custom string as the unique identifier for this node. The identity should be unique amongst other Ringpop instances; it is used in the hashring.

By default, the hostport/address of the node is used as the identity in the hashring. An error is thrown if a hostport is manually specified using this option, as this would lead to unexpected behaviour. If you want to override the node's listening address, use the `Address` option.

func LabelLimitCount added in v0.7.0 

func LabelLimitCount(count int) Option

LabelLimitCount limits the number of labels an application can set on this node.

func LabelLimitKeySize added in v0.7.0 

func LabelLimitKeySize(size int) Option

LabelLimitKeySize limits the size that a key of a label can be.

func LabelLimitValueSize added in v0.7.0 

func LabelLimitValueSize(size int) Option

LabelLimitValueSize limits the size that a value of a label can be.

func LogLevels added in v0.3.0 

func LogLevels(levels map[string]logging.Level) Option

LogLevels is used to set the severity log level for all Ringpop named loggers.

func Logger 

func Logger(l log.Logger) Option

Logger is used to specify a bark-compatible logger that will be used for all Ringpop logging. If a logger is not provided, one will be created automatically.

func MembershipChecksumStatPeriod added in v0.3.0 

func MembershipChecksumStatPeriod(period time.Duration) Option

MembershipChecksumStatPeriod configures the period between emissions of the stat 'membership.checksum-periodic'. Using a value <=0 (or StatPeriodNever) will disable emission of this stat. Using a value in (0, 10ms) will return an error, as that value is unrealistically small. Normal values must therefore be >=10ms. StatPeriodDefault defines the default.

func NodeOverride added in v0.9.0 

func NodeOverride(node swim.NodeInterface) Option

NodeOverride is used to optionally override the default SWIM-based gossip-style node membership protocol used for membership management.

func RequiresAppInPing added in v0.8.4 

func RequiresAppInPing(requiresAppInPing bool) Option

RequiresAppInPing configures if ringpop node should reject pings that don't contain app name

func RingChecksumStatPeriod added in v0.3.0 

func RingChecksumStatPeriod(period time.Duration) Option

RingChecksumStatPeriod configures the period between emissions of the stat 'ring.checksum-periodic'. Using a value <=0 (or StatPeriodNever) will disable emission of this stat. Using a value in (0, 10ms) will return an error, as that value is unrealistically small. Normal values must therefore be >=10ms. StatPeriodDefault defines the default.

func SelfEvictPingRatio added in v0.8.0 

func SelfEvictPingRatio(ratio float64) Option

SelfEvictPingRatio configures the maximum percentage/ratio of the members to actively ping while self evicting. A bigger ratio would allow for bigger batch sizes during restarts without the self eviction being lost due to all nodes having the knowledge being shutdown at the same time.

A smaller ratio will cause less network traffic and therefore slightly faster shutdown times. A ratio that exceeds 1 will be capped to one when the self eviction is executed as it does not make sense to send the gossip to the same node twice. A negative value will cause no pings to be sent out during self eviction.

In no case will there be more pings sent then makes sense by the limit of the current piggyback count

func Statter 

func Statter(s log.StatsReporter) Option

Statter is used to specify a bark-compatible (bark.StatsReporter) stats reporter that will be used to record ringpop stats. If a statter is not provided, stats will be emitted to a null stats-reporter.

func SuspectPeriod added in v0.4.0 

func SuspectPeriod(period time.Duration) Option

SuspectPeriod configures the period it takes ringpop to declare a node faulty after ringpop has first detected the node to be unresponsive to a healthcheck. When a node is declared faulty it is removed from the consistent hashring and stops forwarding traffic to that node. All keys previously routed to that node will then be routed to the new owner of the key

func TombstonePeriod added in v0.4.0 

func TombstonePeriod(period time.Duration) Option

TombstonePeriod configures the period of the last time of the lifecycle in of a node in the membership list. This period should give the gossip protocol the time it needs to disseminate this change. If configured too short the node in question might show up again in faulty state in the distributed memberlist of Ringpop.

type Ringpop 

type Ringpop struct {
	// make ringpop an event Emitter
	events.AsyncEventEmitter
	// contains filtered or unexported fields
}

Ringpop is a consistent hashring that uses a gossip protocol to disseminate changes around the ring.

func New 

func New(app string, opts ...Option) (*Ringpop, error)

New returns a new Ringpop instance.

func (*Ringpop) App 

func (rp *Ringpop) App() string

App returns the name of the application this Ringpop instance belongs to. The application name is set in the constructor when the Ringpop instance is created.

func (*Ringpop) Bootstrap 

func (rp *Ringpop) Bootstrap(bootstrapOpts *swim.BootstrapOptions) ([]string, error)

Bootstrap starts communication for this Ringpop instance.

When Bootstrap is called, this Ringpop instance will attempt to contact other instances from the DiscoverProvider.

If no seed hosts are provided, a single-node cluster will be created.

func (*Ringpop) Checksum 

func (rp *Ringpop) Checksum() (uint32, error)

Checksum returns the current checksum of this Ringpop instance's hashring.

func (*Ringpop) CountReachableMembers 

func (rp *Ringpop) CountReachableMembers(predicates ...swim.MemberPredicate) (int, error)

CountReachableMembers returns the number of members currently in this instance's active membership list that match all provided predicates.

func (*Ringpop) Destroy 

func (rp *Ringpop) Destroy()

Destroy stops all communication. Note that this does not close the TChannel instance that was passed to Ringpop in the constructor. Once an instance is destroyed, it cannot be restarted.

func (*Ringpop) Forward 

func (rp *Ringpop) Forward(dest string, keys []string, request []byte, service, endpoint string,
	format tchannel.Format, opts *forward.Options) ([]byte, error)

Forward forwards the request to given destination host and returns the response.

func (*Ringpop) GetReachableMembers 

func (rp *Ringpop) GetReachableMembers(predicates ...swim.MemberPredicate) ([]string, error)

GetReachableMembers returns a slice of members currently in this instance's active membership list that match all provided predicates.

func (*Ringpop) HandleEvent 

func (rp *Ringpop) HandleEvent(event events.Event)

HandleEvent is used to satisfy the swim.EventListener interface. No touchy.

func (*Ringpop) HandleOrForward 

func (rp *Ringpop) HandleOrForward(key string, request []byte, response *[]byte, service, endpoint string,
	format tchannel.Format, opts *forward.Options) (bool, error)

HandleOrForward returns true if the request should be handled locally, or false if it should be forwarded to a different node. If false is returned, forwarding is taken care of internally by the method, and, if no error has occured, the response is written in the provided response field.

func (*Ringpop) Labels added in v0.7.0 

func (rp *Ringpop) Labels() (*swim.NodeLabels, error)

Labels provides access to a mutator of ringpop Labels that will be shared on the membership. Changes made on the mutator are synchronized accross the cluster for other members to make local decisions on.

func (*Ringpop) Lookup 

func (rp *Ringpop) Lookup(key string) (string, error)

Lookup returns the address of the server in the ring that is responsible for the specified key. It returns an error if the Ringpop instance is not yet initialized/bootstrapped.

func (*Ringpop) LookupN 

func (rp *Ringpop) LookupN(key string, n int) ([]string, error)

LookupN returns the addresses of all the servers in the ring that are responsible for the specified key. It returns an error if the Ringpop instance is not yet initialized/bootstrapped.

func (*Ringpop) Ready 

func (rp *Ringpop) Ready() bool

Ready returns whether or not ringpop is bootstrapped and ready to receive requests.

func (*Ringpop) RegisterListener 

func (rp *Ringpop) RegisterListener(l events.EventListener)

RegisterListener is DEPRECATED, use AddListener. This function is kept around for the time being to make sure that ringpop is a drop in replacement for now. It should not be used by new projects, to accomplish this it will log a warning message that the developer can understand. A release in the future will remove this function completely which will cause a breaking change to the ringpop public interface.

func (*Ringpop) RegisterSelfEvictHook added in v0.8.0 

func (rp *Ringpop) RegisterSelfEvictHook(hooks swim.SelfEvictHook) error

RegisterSelfEvictHook registers the eviction hooks that need to be executed before and after self eviction from the membership. An error will be returned if ringpop was unable to register the hooks. This could happen in the following cases: - Ringpop has not been bootstrapped yet - SelfEvict has already been called - The hook is already registered

func (*Ringpop) SelfEvict added in v0.8.0 

func (rp *Ringpop) SelfEvict() error

SelfEvict should be called before shutting down the application. When calling this function ringpop will gracefully evict itself from the network. Utilities that hook into ringpop will have the opportunity to hook into this system to gracefully handle the shutdown of ringpop.

func (*Ringpop) Uptime 

func (rp *Ringpop) Uptime() (time.Duration, error)

Uptime returns the amount of time that this Ringpop instance has been bootstrapped for.

func (*Ringpop) WhoAmI 

func (rp *Ringpop) WhoAmI() (string, error)

WhoAmI returns the address of the current/local Ringpop node. It returns an error if Ringpop is not yet initialized/bootstrapped.

Source Files

View all Source files

Directories

Path Synopsis
jsonfile
statichosts
test/mocks
examples
keyvalue
keyvalue/gen-go/keyvalue
Package keyvalue is generated code used to make or handle TChannel calls using Thrift.
 Package keyvalue is generated code used to make or handle TChannel calls using Thrift.
ping-json
ping-thrift
ping-thrift-gen
ping-thrift-gen/gen-go/ping
Package ping is generated code used to make or handle TChannel calls using Thrift.
 Package ping is generated code used to make or handle TChannel calls using Thrift.
ping-thrift/gen-go/ping
Package ping is generated code used to make or handle TChannel calls using Thrift.
 Package ping is generated code used to make or handle TChannel calls using Thrift.
role-labels
role-labels/gen-go/role
Package role is generated code used to make or handle TChannel calls using Thrift.
 Package role is generated code used to make or handle TChannel calls using Thrift.
Package forward provides a mechanism to forward TChannel requests.
 Package forward provides a mechanism to forward TChannel requests.
Package hashring provides a hashring implementation that uses a red-black Tree.
 Package hashring provides a hashring implementation that uses a red-black Tree.
Package replica extends Ringpop functionality by providing a mechanism to replicate a request to multiple nodes in the ring.
 Package replica extends Ringpop functionality by providing a mechanism to replicate a request to multiple nodes in the ring.
scripts
testpop
test
mocks
mocks/logger
remoteservice/mocks
thrift/pingpong
Package pingpong is generated code used to make or handle TChannel calls using Thrift.
 Package pingpong is generated code used to make or handle TChannel calls using Thrift.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.