deathnode

Deathnode

Purpose

Gracefully Mesos agent destroy system for AWS Autoscaling.

Deathnode ensures that agents from an Autoscaling group in AWS are destroyed only after they have being drained, allowing to implement scale-in or red/black deployments with no impact to the customers.

It's implementation is based using:

• Mesos Maintenance Primitives (http://mesos.apache.org/documentation/latest/maintenance/)
• AWS Autoscaling ProtectFromScaleIn (http://docs.aws.amazon.com/autoscaling/latest/userguide/as-instance-termination.html)
• AWS Autoscaling Lifecycle Hooks (http://docs.aws.amazon.com/autoscaling/latest/userguide/lifecycle-hooks.html)

How does it do it?

Deathnode monitors the autoscaling groups from the Mesos Agents. Whenever it detects that one instance from an autoscaling group should be removed, it will:

• Find the best agent to be killed
• Tag the instance as being deleted
• Remove instance protection from the instance
• Set the instance in maintenance mode

Then deathnode will keep monitoring this agent, completing destroy lifecycle once it's drained.

Usage

Here you can find an example of usage:

./deathnode -autoscalingGroupName ${ASG_NAME} -delayDelete 300 -mesosUrl ${MESOS_URL} -polling 60 -protectedFrameworks Eremetic -debug

Constraints

When removing an instance, contraints are used by deathnode to filter which instances are not able to be picked up as candidates (best efford). Multiple contraints can be specified.

• noContraint: Applies no constraints
• protectedConstraint: Do not pick instances that has tasks from protected frameworks
• filterFrameworkConstraint: Do not pick instances that has tasks from the specified framework
• taskNameRegexpConstraint: Do not pick instances that has tasks that it's name match a certain regexp

Build

To execute the test, run:

make test

To build the app, run:

make build

To build a docker image, run:

DOCKERTAG="version" make docker

Limitations

• Most of the Mesos frameworks doesn't implement Maintenance primitives.
• Maintenance primitives are not available through Mesos native integration