k8s-sysdig-adapter

Kubernetes Custom Metrics Adapter for Sysdig

Table of contents:

• Introduction
• Prerequisites
• Installation
• Playground
• Troubleshooting
• Contributing
• Relevant links
• Credits and License

Introduction

k8s-sysdig-adapter is an implementation of the Custom Metrics API using Sysdig Monitor.

If you have a Kubernetes cluster and you are a Sysdig user, this adapter enables you to create horizontal pod autoscalers based on metrics provided by Sysdig's monitoring solution.

In the following example, we're creating an autoscaler based on the net.http.request.count metric. The autoscaler will adjust the number of pods deployed for our service as the metric fluctuates over or below the threshold (targetValue).

---
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
  name: kuard
spec:
  scaleTargetRef:
    kind: Deployment
    name: kuard
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Object
    object:
      target:
        kind: Service
        name: kuard
      metricName: net.http.request.count
      targetValue: 100

Prerequisites

You're going to need:

• Kubernetes 1.6+
• Sysdig Monitor - see the installation instructions.
• Sysdig Monitor Access Key - which you've used during the installation of Sysdig Monitor.
• Sysdig Monitor API Token - see where to find it in these instructions. Do not confuse the API token with the agent access key, they're not the same! This is the API token that our metrics server is going to use when accessing the API.

Installation

The configuration files that you can find under the deploy directory are just for reference. Every deployment is unique so tweak them as needed. At the very least, you need to use your own access key and API token as follows:

• 01-sysdig-daemon-set.yml installs the Sysdig agent - edit it so it uses your own access key.
• 03-sysdig-metrics-server.yml installs a secret in Kubernetes containing the Sysdig Monitor API token - edit it to use your own token.

Now we're ready to start! 🎉

1. For the purpose of this example we're going to deploy kuard, a demo application found in the "Kubernetes Up and Running" book. This application is deployed with three replicas by default.

   $ kubectl apply -f deploy/00-kuard.yml
   

   Let's check that it's running:

    $ kubectl get pods -l app=kuard -o wide
    NAME                    READY     STATUS    RESTARTS   AGE       IP          NODE
    kuard-bcc7bf7df-clv2f   1/1       Running   0          1m        10.46.0.2   worker-node-2
    kuard-bcc7bf7df-d9svn   1/1       Running   0          1m        10.40.0.2   worker-node-1
    kuard-bcc7bf7df-zg8nc   1/1       Running   0          1m        10.46.0.3   worker-node-2
   

2. Install Sysdig Monitor if you haven't done it yet - they have great docs that you can use. In this example, we're going to install it using a DaemonSet object as follows:

   $ kubectl apply -f deploy/01-sysdig-daemon-set.yml
   

   Don't forget to add your own access key to the file!

3. The following command is going to deploy a number of required objects like a custom namespace custom-metrics, required RBAC roles, permissions, bindings and the service object for our metrics server:

   $ kubectl apply -f deploy/02-sysdig-metrics-rbac.yml
   

4. Deploy the metrics server with:

   $ kubectl apply -f deploy/03-sysdig-metrics-server.yml
   

   Don't forget to add your own API token to the file!

   It should be possible to retrieve the full list of metrics available using the following command:

   $ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq -r ".resources[].name"
   

5. Deploy our custom autoscaler that scales our service based on the net.http.request.count metric.

   $ kubectl apply -f deploy/04-kuard-hpa.yml
   

At this point you should be able to see the autoscaler in action. In the example, we set a threshold of 100 requests per minute. Let's generate some traffic with hey:

$ hey -c 5 -q 85 -z 24h http://10.103.86.213

Finally, use the following command to watch the autoscaler:

$ kubectl get hpa kuard-autoscaler -w
NAME               REFERENCE          TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
kuard-autoscaler   Deployment/kuard   105763m/100   3         10        8          2d

Playground

You can find a playground based on Vagrant virtual machines under the playground directory in this repository. You can use it to demo this project or for development purposes.

minikube has not been tried yet! See issue #3 for more details.

Troubleshooting

If you encounter any problems that the documentation does not address, file an issue.

Contributing

Thanks for taking the time to join our community and start contributing!

• Please familiarize yourself with the Code of Conduct before contributing.
• See CONTRIBUTING.md for information about setting up your environment, the workflow that we expect, and instructions on the developer certificate of origin that we require.
• Check out the issues and our roadmap.

Relevant links

From the Kubernetes project:

• Types in Custom Metrics API
• Library for writing a Custom Metrics API server
• Library for writing a Kubernetes-style API Server
• Documentation: Horizontal Pod Autoscaling

Other links:

• Sysdig's blog post about HPA
• Kubernetes Prometheus Adapter
• kubeadm workshop
• Basic Custom Metrics API server using minikube

Credits and license

This project wouldn't be possible without the great job done by others. See the links above for some of the things I've been using in different ways.

Thanks to Dave Cheney and Joe Beda for being so welcoming with my contributions to the Contour project - that was my first experience as a developer in the Kubernetes project. The tagging policy and the contributing guide in this project is based on Contour's.

This project is open source and it uses a permissive license.