README
¶
Cirrus
Cirrus lets you manage your Snowflake node IDs.
Etymology
High-level clouds, called cirrus clouds, can reach heights of 20,000 feet (6,000 meters) and are typically thin. They do not produce rain and often indicate fair weather. They are usually made up of ice.
Not Production Ready
See TODO list below.
What's Snowflake?
Snowflake IDs are special 8-byte IDs generated by Twitter's Snowflake algorithm.
Each Snowflake ID is a 63 bit int.
- 41 bits for a timestamp (milliseconds since a custom epoch)
- 10 bits for a node ID (ranging from 0 to 1023)
- 12 bits for a sequence number (ranging from 0 to 4095)
Why Cirrus?
If you're planning on horizontally scaling (i.e., replicating) your app, you'll need a way of doling out node IDs.
In the era of Docker and Kubernetes, containers and pods are expected to be ephemeral. They can go down at any moment. If that happens, they have no way of relinquishing their node IDs, and so that node ID is essentially lost.
Fortunately, checking for "stale" IDs is super easy, so we can ensure we have a dynamic and reliable way of re-requisitioning and reallocating node IDs.
What does Cirrus do?
Cirrus maps node IDs to app IDs.
An app ID is a string that can be anything (e.g., an int, a UUID, a hash, etc).
A node ID is an int ranging from 0 to 1023.
Apps must periodically send heartbeats to Cirrus, otherwise their node IDs get requisitioned.
TODO
Questions
- How do we know if a node has actually died? What if it's just busy and stays alive? There might be a moment where 2 apps generate PKs with the same node ID? That would be disastrous.
Single point of failure
Cirrus needs to be able scale out so it's not a single point of failure.
We need a couple things:
- Replace
AvailableNodeIdschannel with a distributed queue system (e.g., RabbitMQ, Cherami, etc) - Replace
Heartbeatsmap with a distributed store (e.g., etcd)
Usability
- Usage section in README
- Configurable durations
- Dockerfile
- Helm chart
Documentation
¶
Index ¶
Constants ¶
const ( // NumNodes is the number of nodes we offer (max 1024) NumNodes = 10 // GrpcPort is the port our gRPC server runs on GrpcPort = 50051 )
Variables ¶
var ( // HeartbeatPeriodicity is how often apps should heartbeat before we requisition their node IDs. HeartbeatPeriodicity = getHeartbeatPeriodicity() // StaleCheckPeriodicity is how often we check for stale node IDs StaleCheckPeriodicity = time.Second * 5 )
var AvailableNodeIds = make(chan int, NumNodes)
AvailableNodeIds is a channel of available node IDs. As nodes die, their node IDs will be requisitioned and sent into the channel.
var Heartbeats map[int]time.Time
Heartbeats stores the last time nodes sent a heartbeat.
Functions ¶
func PruneStaleEntries ¶
PruneStaleEntries checks for stale node IDs.
Types ¶
type GrpcServer ¶
type GrpcServer struct {
// contains filtered or unexported fields
}
GrpcServer is a gRPC server that provides an endpoint which allows apps to send "heartbeats" to request node IDs.
func (*GrpcServer) Heartbeat ¶
func (service *GrpcServer) Heartbeat(ctx context.Context, in *pb.HeartbeatRequest) (*pb.HeartbeatResponse, error)
Heartbeat is a gRPC endpoint to allow apps to request a node ID.
Source Files
Directories