k8s-eniconfig-controller

ENIConfig Controller

This controller is still in an alpha state, please file issues and pull requests as you run into issues. Thanks 🎉

This repository will implement auto annotating your Kubernetes data plane nodes with a desired ENIConfig name. This was originally implemented in the amazon-vpc-cni-k8s project in this pull request - https://github.com/aws/amazon-vpc-cni-k8s/pull/165

Prerequisites

Before patching the node with the annotation, you will need to change the AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG environment variable in the AWS CNI daemonset to true. By default, pods share the same subnet and security groups as the worker node's primary interface. When you set this variable to true it causes ipamD to use the security groups and VPC subnet in a worker node's ENIConfig for elastic network interface allocation. The subnet in the ENIConfig must belong to the same Availability Zone that the worker node resides in.

This operator can be configured to use an arbitrary EC2 tag for annotation. If no tag is set, the operator will use a tag called k8s.amazonaws.com/eniConfig. While the operator will automatically apply the annotation to your worker nodes, it does not verify that subnet in the eniconfig is in the same AZ as the worker node.

The worker nodes also have to be assigned an IAM role with a policy that allows the DescribeTags action to the EC2 instance. This is configured by default when you use eksctl to provision a cluster.

Prerequisites

Before patching the node with the annotation, you will need to change the AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG environment variable in the AWS CNI daemonset to true. By default, pods share the same subnet and security groups as the worker node's primary interface. When you set this variable to true it causes ipamD to use the security groups and VPC subnet in a worker node's ENIConfig for elastic network interface allocation. The subnet in the ENIConfig must belong to the same Availability Zone that the worker node resides in.

This operator can be configured to use an arbitrary EC2 tag for annotation. If no tag is set, the operator will use a tag called k8s.amazonaws.com/eniConfig. Before the operator will automatically apply the annotation to your worker nodes, it will verify that subnet in the eniconfig is in the same AZ as the worker node.

The worker nodes also have to be assigned an IAM role with a policy that allows the DescribeTags action to the EC2 instance. This is configured by default when you use eksctl to provision a cluster.

Deploying

Before you deploy the controller, you will need to make sure your instances have the proper policies assigned.

POLICY_ARN=$(aws iam create-policy \
                 --policy-name eniconfig-controller-policy \
                 --policy-document  file://configs/eniconfig-controller-policy.json | jq -r ".Policy.Arn")

Now that you have this defined you can add this to the worker node role, alternatively you could use a pod identity project to allow the pod to assume your role.

aws iam attach-role-policy \
    --role-name {WORKER NODE ROLE NAME} \
    --policy-arn $POLICY_ARN

kubectl apply -f https://raw.githubusercontent.com/christopherhein/eniconfig-controller/master/configs/eniconfig-controller.yaml

Notes about helm

Because this Deployment has to be running on before other pods are scheduled you cannot use helm. You will notice in the configs/eniconfig-controller.yaml it runs on hostNetwork: true to allow the pod to take the Instance Tag and look up the proper subnets to apply.

Running in Dev

# assumes you have a working kubeconfig, not required if operating in-cluster
$ go build -o eniconfig-controller .
$ ./eniconfig-controller -kubeconfig=$KUBECONFIG

Running in Dev

# assumes you have a working kubeconfig, not required if operating in-cluster
$ go build -o eniconfig-controller .
$ ./eniconfig-controller -kubeconfig=$HOME/.kube/config -eniconfig-name=name-of-eni

## Releasing

To release this project all you have to do is run `make release`.

## License

This library is licensed under the Apache 2.0 License.