leaderelection

GoLea

GoLea provides the capability for a set of distributed processes to compete for leadership for a shared resource. It is implemented using Zookeeper for the underlying support. For a general description of leader election, see the wikipedia page describing leader election.

Terms

How to use

This section provides an overview of the various phases of an election. The file Election_test.go is a good example of how a client is expected to use the leader election package. The integration tests (see the integrationtest directory) may also provide some insight into how to use the package although their intent is primarily to test the package vs. provide examples.

Leader Election (LE) clients will have to import the following packages:

import (
	"github.com/samuel/go-zookeeper/zk"
	"github.com/Comcast/go-leaderelection"
)

Leader election clients must provide a ZK connection when creating an election instance. The rationale behind this is to prevent applications that participate in multiple elections from letting the library create multiple ZK connections behind the scenes. This allows the application to optimize the use of ZK connections.

Create a Resource

A resource is represented as a ZK node. During the design and implementation of the library an explicit decision was made to not provide the capability in the library to create election resources. This decision may be revisited in the future.

// Create the ZK connection
zkConn, _, err := zk.Connect([]string{zkURL}, heartBeat)
if err != nil {
    log.Printf("Error in zk.Connect (%s): %v",
        zkURL, err)
    return nil, err
}

// Create the election node in ZooKeeper
_, err = zkConn.Create(electionNode,
    []byte(""), 0, zk.WorldACL(zk.PermAll))
if err != nil {
    log.Printf("Error creating the election node (%s): %v",
        electionNode, err)
    return nil, err
}

candidate, err := leaderelection.NewElection(zkConn, electionNode)

Delete an Election

err := leaderelection.DeleteElection(zkConn, electionNode)

or

leaderelection.EndElection()

Request Leadership for an Election

Leadership election is inherently an asynchronous process. For candidates that win an election the inital response is fairly rapid. Even so, leaders can still receive errors from the election that will impact their leadership of a resource. The most common example is when the underlying election instance gets disconnected from Zookeeper. In the event of errors, leaders expected to resign their leadership. This is likely a futile process if the ZooKeeper connection is lost, but it's good practice to do this for any error.

Followers of an election can wait an indeterminate amount of time to become the leader of an election. If a follower becomes a leader they will be notified. In some cases the election may end before a follower can become a leader. Like leaders, followers can receive errors from the election. Like leaders, they should resign from the election in this case.

Finally, an election can be unilaterally ended by any actor in the application. As with errors, this results in a status notification to all candidates, including the leader. Candidates are expected to resign from the election in this case as well.

Given that leadership election is an asynchronous process clients should start the election as a goroutine as shown below. All events pertaining to the status of an election are communicated via channels. So the typical pattern is for a client to monitor the election status in a for/select loop as shown below.

// Each candidation should register interest in participating for an election
candidate, err := NewElection(zkConn, "/election", "myhostname")
...
// Each candidate should begin their participation in the
// by starting a goroutine.
go candidate.ElectLeader()
...

Monitor election and candidate's role in the election

var status Status
var ok bool

for {
    select {
    case status, ok = <-leaderElector.Status():
        if !ok {
            fmt.Println("\t\t\tChannel closed, election is terminated!!!")
            respCh <- ElectionResponse{false, status.CandidateID}
            leaderElector.Resign()
            wg.Done()
            return
        }
        if status.Err != nil {
            fmt.Println("Received election status error <<", status.Err, ">> for candidate <", leaderElector.candidateID, ">.")
            leaderElector.Resign()
            wg.Done()
            return
        }

        fmt.Println("Candidate received status message: <", status, ">.")
        if status.Role == Leader {
            doLeaderStuff(leaderElector, status, respCh, connFailCh, waitFor)
            wg.Done()
            return
        }
    case <-connFailCh:
        fmt.Println("\t\t\tZK connection failed for candidate <", status.CandidateID, ">, exiting")
        respCh <- ElectionResponse{false, status.CandidateID}
        leaderElector.Resign()
        wg.Done()
        return
    case <-time.After(time.Second * 100):
        fmt.Println("\t\t\tERROR!!! Timer expired, stop waiting to become leader for", status.CandidateID)
        leaderElector.Resign()
        respCh <- ElectionResponse{false, status.CandidateID}
        wg.Done()
        return
}

}

Resign from an Election

candidate.Resign()

Leadership change

See the Monitor election and candidate's role in the election above.

Query status

Candidates are always notified when an election's status changes. It is up to the client to cache this status if they need to reference it between status changes.

Prerequisites

1. go-leaderelection uses github.com/samuel/go-zookeeper/zk.
2. All tests require the availablility of a Zookeeper installation. zkServer.sh must be in the path. Election_test.go requires that Zookeeper be running. The integration tests control Zookeeper so Zookeeper should not be running when executing the integration tests.

Testing the package has additional prerequisites:

1. The integration tests leverage github.com/Comcast/goint. This package must be installed prior to executing integration tests.