esnctl

esnctl

Elasticsearch Node Controller with AWS Auto Scaling Group

Why

Graceful Elasticsearch node addition/removal requires several steps.

Add node

1. Disable shard reallocation
   • see https://www.elastic.co/guide/en/elasticsearch/reference/1.4/cluster-nodes-shutdown.html
2. Add node
3. Enable shard reallocation

Remove node

1. Remove node from load balancer
2. Wait for connection draining
3. Remove node from shard allocation targets
   • see http://stackoverflow.com/a/23905040
4. Wait for that shards on target node escape to other nodes
5. (Es 1.x only) Shut down node

So far we have conducted this by hand. However, it sometimes causes operation errors. We realize that these operations should be automated and conducted by ONE action.

Required environment

• Elasticsearch 1.x / 2.x / 5.x
• Elasticsearch cluster is running on AWS EC2 instances
  • Using EC2 Discovery
• EC2 instances are managed by AWS Auto Scaling Groups
  • Instances (= Nodes) can be added/removed by modifying DesiredCapacity
• EC2 instances and Auto Scaling Group are attached to Target Group
  • Cluster can be accessed through Application Load Balancer

(TODO: architecture image here)

Installation

Precompiled binary

Precompiled binaries for Windows, OS X, Linux are available at Releases.

From source

$ go get -d github.com/dtan4/esnctl
$ cd $GOPATH/src/github.com/dtan4/esnctl
$ make deps
$ make install

Usage

To run esnctl add or esnctl remove, you need to set valid AWS credentials beforehand.

export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
export AWS_REGION=xx-yyyy-0

esnctl list

List nodes

$ esnctl add \
  --cluster-url http://elasticsearch.example.com
ip-10-0-1-21.ap-northeast-1.compute.internal
ip-10-0-1-22.ap-northeast-1.compute.internal
ip-10-0-1-23.ap-northeast-1.compute.internal

esnctl add

Add nodes

$ esnctl add \
  --cluster-url http://elasticsearch.example.com \
  --group elasticsearch \
  -n 2
===> Disabling shard reallocation...
===> Launching 2 instances on elasticsearch...
===> Waiting for nodes join to Elasticsearch cluster...
........................
===> Enabling shard reallocation...
===> Finished!

esnctl remove

Remove a node

Only 1 node can be removed at the same time.

$ esnctl remove \
  --cluster-url http://elasticsearch.example.com \
  --group elasticsearch \
  --node-name ip-10-0-1-21.ap-northeast-1.compute.internal
===> Retrieving target instance ID...
===> Retrieving target group...
===> Detaching instance from target group...
............................................................
===> Excluding target node from shard allocation group...
===> Waiting for shards escape from target node...
..................
===> Shutting down target node...
===> Detaching target instance...
===> Finished!

Author

Daisuke Fujita (@dtan4)

License