k8s-aws-operator

k8s-aws-operator

Manage AWS Elastic IPs (EIPs) and Elastic Network Interfaces (ENIs) as Custom Resources in your Kubernetes cluster and assign them your pods.

Requirements

• Your pod IPs must be allocated from your VPC subnets. This is the default setup on AWS EKS by using the AWS VPC CNI plugin.
• If you wish egress traffic to be sourced from assigned EIPs: In AWS VPC CNI plugin, AWS_VPC_K8S_CNI_EXTERNALSNAT must be set to true or AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS must include the destination CIDR's.
• Your worker nodes must reside in a public subnet with an internet gateway attached.

Installation

Create an IAM role

Create an IAM role with the policy here.

Install the operator

Run:

$ helm install --namespace kube-system --set aws.region=us-east-1 oci://ghcr.io/goto-opensource/k8s-aws-operator --version v1.0.0 # adjust version

If you want to use IAM roles for service accounts, add the required trust relationship with your cluster to the IAM role and add the corresponding annotation on the service account (e.g. by setting the Helm value serviceAccount.annotations."eks.amazonaws.com/role-arn" accordingly).

Usage

EIPs

Basic usage

Create a new file example.yaml:

apiVersion: aws.k8s.logmein.com/v1alpha1
kind: EIP
metadata:
  name: my-eip
spec:
  tags:
    owner: My team

Apply it:

$ kubectl apply -f example.yaml
eip.aws.k8s.logmein.com/my-eip created

Describe it:

$ kubectl get eip my-eip
NAME     STATE      PUBLIC IP       POD
my-eip   allocated  34.228.250.93

Request a random address from a BYOIP address pool:

apiVersion: aws.k8s.logmein.com/v1alpha1
kind: EIP
# ...
spec:
  publicIPv4Pool: <your pool ID here>
  # ...

Request a specific address from a BYOIP address pool:

apiVersion: aws.k8s.logmein.com/v1alpha1
kind: EIP
# ...
spec:
  publicIPv4Address: 12.34.56.78
  # ...

Adjust example.yaml to include an assignment section:

apiVersion: aws.k8s.logmein.com/v1alpha1
kind: EIP
metadata:
  name: my-eip
spec:
  tags:
    owner: My team
  assignment:
    podName: some-pod

Apply it:

$ kubectl apply -f example.yaml
eip.aws.k8s.logmein.com/my-eip configured

Describe it:

$ kubectl get eip my-eip
NAME     STATE      PUBLIC IP       POD
my-eip   assigned   34.228.250.93   my-pod

Allocating and assigning can also be done in one step.

Remove the assignment section again and reapply the manifest.

$ kubectl delete eip my-eip
eip.aws.k8s.logmein.com/my-eip deleted

Unassigning and releasing can also be done in one step.

One EIP per pod in a deployment / statefulset

You can use an initContainer as part of your pod definition to create the EIP or EIPAssociation custom resource. This requires that your pod has RBAC permissions to create EIP/ EIPAssociation resources.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: eip-user
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: eip-user-role
rules:
- apiGroups:
  - aws.k8s.logmein.com
  resources:
  - eips
  - eipassociations
  verbs:
  - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: eip-user-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: eip-user-role
subjects:
- kind: ServiceAccount
  name: eip-user
---
apiVersion: apps/v1
kind: Deployment
# ...
spec:
  # ...
  template:
    spec:
      # ...
      serviceAccountName: eip-user
      initContainers:
      - name: init-eip
        image: <some image that has kubectl>
        env:
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        command:
        - /bin/sh
        - -c
        - |
            # allocate and assign EIP
            cat <<EOS | kubectl apply -f-
            apiVersion: aws.k8s.logmein.com/v1alpha1
            kind: EIP
            metadata:
              name: $(MY_POD_NAME)
              namespace: $(MY_POD_NAMESPACE)
            spec:
              tags:
                owner: My team
                pod: $(MY_POD_NAME)
                namespace: $(MY_POD_NAMESPACE)
              assignment:
                podName: $(MY_POD_NAME)
            EOS

            # wait for EIP to be assigned
            while [ "$(kubectl get eip $(MY_POD_NAME) -o jsonpath='{.status.state}')" != "assigned" ]
            do
              sleep 1
            done

You can ensure that an EIP is released when your pod is terminated by including ownerReferences in your EIP resource. Setting blockOwnerDeletion: true prevents the pod from vanishing until the EIP is unassigned and released.

apiVersion: aws.k8s.logmein.com/v1alpha1
kind: EIP
metadata:
  name: my-eip
  ownerReferences:
  - apiVersion: v1
    kind: Pod
    name: some-pod
    uid: ... # put the UID of the pod here
    blockOwnerDeletion: true
spec:
  tags:
    owner: My team
  assignment:
    podName: some-pod

Here is an example of how to create an EIPAssociation to have a static EIP assigned/unassigned to a pod.

apiVersion: aws.k8s.logmein.com/v1alpha1
kind: EIPAssociation
metadata:
  name: my-eip-association
  ownerReferences:
  - apiVersion: v1
    kind: Pod
    name: some-pod
    uid: ... # put the UID of the pod here
    blockOwnerDeletion: true
spec:
  eipName: eip-name
  assignment:
    podName: some-pod

ENIs

To be documented