azure

Kubernetes Virtual Kubelet with ACI

Azure Container Instances (ACI) provide a hosted environment for running containers in Azure. When using ACI, there is no need to manage the underlying compute infrastructure, Azure handles this management for you. When running containers in ACI, you are charged by the second for each running container.

The Azure Container Instances provider for the Virtual Kubelet configures an ACI instance as a node in any Kubernetes cluster. When using the Virtual Kubelet ACI provider, pods can be scheduled on an ACI instance as if the ACI instance is a standard Kubernetes node. This configuration allows you to take advantage of both the capabilities of Kubernetes and the management value and cost benefit of ACI.

This document details configuring the Virtual Kubelet ACI provider.

Table of Contents

• Prerequiste
• Quick set-up with the ACI Connector
• Manual set-up
• Validate the Virtual Kubelet ACI provider
• Schedule a pod in ACI
• Work arounds
• Upgrade the ACI Connector
• Remove the Virtual Kubelet

Prerequisite

This guide assumes that you have a Kubernetes cluster up and running (can be minikube) and that kubectl is already configured to talk to it.

Other pre-requesites are:

• A Microsoft Azure account.
• Install the Azure CLI.
• Install the Kubernetes CLI.
• Install the Helm CLI.

Install the Azure CLI

Install az by following the instructions for your operating system. See the full installation instructions if yours isn't listed below.

MacOS

brew install azure-cli

Windows

Download and run the Azure CLI Installer (MSI).

Ubuntu 64-bit

1. Add the azure-cli repo to your sources:

   echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ wheezy main" | \
        sudo tee /etc/apt/sources.list.d/azure-cli.list
   

2. Run the following commands to install the Azure CLI and its dependencies:

   sudo apt-key adv --keyserver packages.microsoft.com --recv-keys 52E16F86FEE04B979B07E28DB02C46DF417A0893
   sudo apt-get install apt-transport-https
   sudo apt-get update && sudo apt-get install azure-cli
   

Install the Kubernetes CLI

Install kubectl by running the following command:

az aks install-cli

Install the Helm CLI

Helm is a tool for installing pre-configured applications on Kubernetes. Install helm by running the following command:

MacOS

brew install kubernetes-helm

Windows

1. Download the latest Helm release.
2. Decompress the tar file.
3. Copy helm.exe to a directory on your PATH.

Linux

curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash

Cluster and Azure Account Setup

Now that we have all the tools, we will set up your Azure account to work with ACI.

Configure your Azure account

First let's identify your Azure subscription and save it for use later on in the quickstart.

1. Run az login and follow the instructions in the command output to authorize az to use your account

2. List your Azure subscriptions:

   az account list -o table
   

3. Copy your subscription ID and save it in an environment variable:

   Bash

   export AZURE_SUBSCRIPTION_ID="<SubscriptionId>"
   

   PowerShell

   $env:AZURE_SUBSCRIPTION_ID = "<SubscriptionId>"
   

4. Enable ACI in your subscription:

   az provider register -n Microsoft.ContainerInstance
   

Quick set-up with the ACI Connector

The Azure cli can be used to install the ACI provider. We like to say Azure's provider or implementation for Virtual Kubelet is the ACI Connector. For this section Virtual Kubelet's specific ACI provider will be referenced as the the ACI Connector. If you continue with this section you can skip sections below up to "Schedule a pod in ACI", as we use Azure Container Service (AKS) to easily deploy and install the connector, thus it is assumed that you've created an AKS cluster.

To install the ACI Connector use the az cli and the aks namespace. Make sure to use the resource group of the aks cluster you've created and the name of the aks cluster you've created. You can choose the connector name to be anything. Choose any command below to install the Linux, Windows, or both the Windows and Linux Connector.

Note: You might need to specify the --aci-resource-group, due to a bug in the az cli. The resource group is then auto-generated. To find the name navigate to the Azure Portal resource groups, scroll down and find the name that matches MC_aks cluster name_aks rg_location.

1. Install the Linux ACI Connector

   Bash

   az aks install-connector --resource-group <aks cluster rg> --name <aks cluster name> 
   

2. Install the Windows ACI Connector

   Bash

   az aks install-connector --resource-group <aks cluster rg> --name <aks cluster name> --os-type windows 
   

3. Install both the Windows and Linux ACI Connectors

   Bash

   az aks install-connector --resource-group <aks cluster rg> --name <aks cluster name> --os-type both 
   

Now you are ready to deploy a pod to the connector so skip to the "Schedule a pod in ACI" section.

Manual set-up

Create a Resource Group for ACI

To use Azure Container Instances, you must provide a resource group. Create one with the az cli using the following command.

export ACI_REGION=eastus
az group create --name aci-group --location "$ACI_REGION"
export AZURE_RG=aci-group

Create a service principal

This creates an identity for the Virtual Kubelet ACI provider to use when provisioning resources on your account on behalf of Kubernetes.

1. Create a service principal with RBAC enabled for the quickstart:

   az ad sp create-for-rbac --name virtual-kubelet-quickstart -o table
   

2. Save the values from the command output in environment variables:

   Bash

   export AZURE_TENANT_ID=<Tenant>
   export AZURE_CLIENT_ID=<AppId>
   export AZURE_CLIENT_SECRET=<Password>
   

   PowerShell

   $env:AZURE_TENANT_ID = "<Tenant>"
   $env:AZURE_CLIENT_ID = "<AppId>"
   $env:AZURE_CLIENT_SECRET = "<Password>"
   

Deployment of the ACI provider in your cluster

Run these commands to deploy the virtual kubelet which connects your Kubernetes cluster to Azure Container Instances.

export VK_RELEASE=virtual-kubelet-0.2.0

If your cluster is an AKS cluster:

RELEASE_NAME=virtual-kubelet
NODE_NAME=virtual-kubelet
CHART_URL=https://github.com/virtual-kubelet/virtual-kubelet/raw/master/charts/$VK_RELEASE.tgz

curl https://raw.githubusercontent.com/virtual-kubelet/virtual-kubelet/master/scripts/createCertAndKey.sh > createCertAndKey.sh
chmod +x createCertAndKey.sh
. ./createCertAndKey.sh

helm install "$CHART_URL" --name "$RELEASE_NAME" \
  --set provider=azure \
  --set providers.azure.targetAKS=true \
  --set providers.azure.tenantId=$AZURE_TENANT_ID \
  --set providers.azure.subscriptionId=$AZURE_SUBSCRIPTION_ID \
  --set providers.azure.clientId=$AZURE_CLIENT_ID \
  --set apiserverCert=$cert \
  --set apiserverKey=$key

For any other type of Kubernetes cluster:

RELEASE_NAME=virtual-kubelet
NODE_NAME=virtual-kubelet
CHART_URL=https://github.com/virtual-kubelet/virtual-kubelet/raw/master/charts/$VK_RELEASE.tgz

curl https://raw.githubusercontent.com/virtual-kubelet/virtual-kubelet/master/scripts/createCertAndKey.sh > createCertAndKey.sh
chmod +x createCertAndKey.sh
. ./createCertAndKey.sh

helm install "$CHART_URL" --name "$RELEASE_NAME" \
  --set provider=azure \
  --set rbac.install=true \
  --set providers.azure.targetAKS=false \
  --set providers.azure.tenantId=$AZURE_TENANT_ID \
  --set providers.azure.subscriptionId=$AZURE_SUBSCRIPTION_ID \
  --set providers.azure.clientId=$AZURE_CLIENT_ID \
  --set providers.azure.clientKey=$AZURE_CLIENT_SECRET \
  --set providers.azure.aciResourceGroup=$AZURE_RG \
  --set providers.azure.aciRegion=$ACI_REGION \
  --set apiserverCert=$cert \
  --set apiserverKey=$key

If your cluster has RBAC enabled set rbac.install=true

Output:

NAME:   virtual-kubelet
LAST DEPLOYED: Thu Feb 15 13:17:01 2018
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Secret
NAME                             TYPE    DATA  AGE
virtual-kubelet-virtual-kubelet  Opaque  3     1s

==> v1beta1/Deployment
NAME                             DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
virtual-kubelet-virtual-kubelet  1        1        1           0          1s

==> v1/Pod(related)
NAME                                              READY  STATUS             RESTARTS  AGE
virtual-kubelet-virtual-kubelet-7bcf5dc749-6mvgp  0/1    ContainerCreating  0         1s


NOTES:
The virtual kubelet is getting deployed on your cluster.

To verify that virtual kubelet has started, run:

```cli
  kubectl --namespace=default get pods -l "app=virtual-kubelet-virtual-kubelet"

Validate the Virtual Kubelet ACI provider

To validate that the Virtual Kubelet has been installed, return a list of Kubernetes nodes using the kubectl get nodes command. You should see a node that matches the name given to the ACI connector.

kubectl get nodes

Output:

NAME                                        STATUS    ROLES     AGE       VERSION
virtual-kubelet-virtual-kubelet             Ready     <none>    2m        v1.8.3
aks-nodepool1-39289454-0                    Ready     agent     22h       v1.7.7
aks-nodepool1-39289454-1                    Ready     agent     22h       v1.7.7
aks-nodepool1-39289454-2                    Ready     agent     22h       v1.7.7

Schedule a pod in ACI

Create a file named virtual-kubelet-test.yaml and copy in the following YAML. Replace the nodeName value with the name given to the virtual kubelet node.

apiVersion: v1
kind: Pod
metadata:
  name: helloworld
spec:
  containers:
  - image: microsoft/aci-helloworld
    imagePullPolicy: Always
    name: helloworld
    resources:
      requests:
        memory: 1G
        cpu: 1
    ports:
    - containerPort: 80
      name: http
      protocol: TCP
    - containerPort: 443
      name: https
  dnsPolicy: ClusterFirst
  nodeSelector:
    kubernetes.io/role: agent
    beta.kubernetes.io/os: linux
    type: virtual-kubelet
  tolerations:
  - key: virtual-kubelet.io/provider
    operator: Exists
  - key: azure.com/aci
    effect: NoSchedule

Notice that Virtual-Kubelet nodes are tainted by default to avoid unexpected pods running on them, i.e. kube-proxy, other virtual-kubelet pods, etc. To schedule a pod to them, you need to add the tolerations to your pod spec:

  tolerations:
  - key: virtual-kubelet.io/provider
    operator: Exists
  - key: azure.com/aci
    effect: NoSchedule

Also, specify a nodeSelector so the pods will be forced onto the Virtual-Kubelet node.

  nodeSelector:
    kubernetes.io/role: agent
    beta.kubernetes.io/os: linux
    type: virtual-kubelet

Run the application with the kubectl create command.

kubectl create -f virtual-kubelet-test.yaml

Use the kubectl get pods command with the -o wide argument to output a list of pods with the scheduled node.

kubectl get pods -o wide

Notice that the helloworld pod is running on the virtual-kubelet node.

NAME                                            READY     STATUS    RESTARTS   AGE       IP             NODE
aci-helloworld-2559879000-8vmjw                 1/1       Running   0          39s       52.179.3.180   virtual-kubelet

To validate that the container is running in an Azure Container Instance, use the az container list Azure CLI command.

az container list -o table

Output:

Name                             ResourceGroup    ProvisioningState    Image                     IP:ports         CPU/Memory       OsType    Location
-------------------------------  ---------------  -------------------  ------------------------  ---------------  ---------------  --------  ----------
helloworld-2559879000-8vmjw  myResourceGroup    Succeeded            microsoft/aci-helloworld  52.179.3.180:80  1.0 core/1.5 gb  Linux     eastus

Work arounds for the ACI Connector

If your pod that's scheduled onto the Virtual Kubelet node is in a pending state please add these workarounds to your Virtual Kubelet pod spec.

First, grab the logs from your ACI Connector pod, with the following command.

kubectl logs virtual-kubelet-virtual-kubelet-7bcf5dc749-6mvgp 

Stream or pod watcher errors

If you see the following errors in the logs:

ERROR: logging before flag.Parse: E0914 00:02:01.546132       1 streamwatcher.go:109] Unable to decode an event from the watch stream: stream error: stream ID 181; INTERNAL_ERROR
time="2018-09-14T00:02:01Z" level=error msg="Pod watcher connection is closed unexpectedly" namespace= node=virtual-kubelet-myconnector-linux operatingSystem=Linux provider=azure

Then copy the master URL with cluster-info.

kubectl cluster-info

Output:

Kubernetes master is running at https://aksxxxx-xxxxx-xxxx-xxxxxxx.hcp.uksouth.azmk8s.io:443

Edit your aci-connector deployment by first getting the deployment name.

kubectl get deploy 

Output:

NAME                           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
virtual-kubelet-virtual-kubelet 1         1         1            1           5d
aci-helloworld                  1         1         1            0           12m

Edit the deployment.

kubectl edit deploy virtual-kubelet-virtual-kubelet 

Add the following name and value to the deployment in the enviorment section. Use your copied master URL.

--name: MASTER_URI
  value: https://aksxxxx-xxxxx-xxxx-xxxxxxx.hcp.uksouth.azmk8s.io:443

Taint deprecated errors

If you see the following errors in the logs:

Flag --taint has been deprecated, Taint key should now be configured using the VK_TAINT_KEY environment variable

Then edit your aci-connector deployment by first grabbing the deployment name.

kubectl get deploy 

Output:

NAME                           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
virtual-kubelet-virtual-kubelet 1         1         1            1           5d
aci-helloworld                  1         1         1            0           12m

Edit the connector deployment.

kubectl edit deploy virtual-kubelet-virtual-kubelet 

Add the following as an enviorment variable within the deployment.

--name: VK_TAINT_KEY
  value: azure.com/aci

Also, delete the following argument in your pod spec:

- --taint
  - azure.com/aci

Upgrade the ACI Connector

If you've installed Virtual Kubelet with the Azure cli so you're using the ACI Connector implementation, then you are also able to upgrade the connector to the latest release. Run the following command to upgrade your ACI Connector.

az aks upgrade-connector --resource-group <aks cluster rg> --name <aks cluster name> --connector-name myconnector --os-type linux

Remove the Virtual Kubelet

You can remove your Virtual Kubelet node by deleting the Helm deployment. Run the following command:

helm delete virtual-kubelet --purge

If you used the ACI Connector installation then use the following command to remove the the ACI Connector from your cluster.

az aks remove-connector --resource-group <aks cluster rg> --name <aks cluster name> --connector-name myconnector --os-type linux