splunk-service

• datadog-service
  • Quickstart
  • If you already have a Keptn cluster running
  • Compatibility Matrix
  • Installation
    • Up- or Downgrading
    • Uninstall
  • Running tests on your local machine
    • Run tests
  • Development
    • Where to start
    • Common tasks
    • Testing Cloud Events
  • Automation
    • GitHub Actions: Automated Pull Request Review
    • GitHub Actions: Unit Tests
  • How to release a new version of this service
  • Known problems
  • License

datadog-service

This implements the datadog-service that integrates the Datadog observability platform with Keptn. This enables you to use Datadog as the source for the Service Level Indicators (SLIs) that are used for Keptn Quality Gates. If you want to learn more about Keptn visit us on keptn.sh

Check the issue on the main repo for more info: https://github.com/keptn/keptn/issues/2652

Quickstart

If you are on Mac or Linux, you can use examples/kup.sh to set up a local Keptn installation that uses Datadog. This script creates a local minikube cluster, installs Keptn, Istio, Datadog and the Datadog integration for Keptn (check the script for pre-requisites).

To use the script,

export DD_API_KEY="<your-datadog-api-key>" DD_APP_KEY="<your-datadog-app-key>" DD_SITE="datadoghq.com" 
examples/kup.sh

Check the official docs for how to create the Datadog API key and Application key

Note: Application keys get the same permissions as you. You might want to narrow down the permissions (datadog-service only reads metrics from the API. Check the official docs linked above for more information).

If you already have a Keptn cluster running

1. Install datadog

Add datadog helm repo:

helm repo add datadog https://helm.datadoghq.com

Install datadog helm chart:

export DD_API_KEY="<your-datadog-api-key>" DD_APP_KEY="<your-datadog-app-key>" DD_SITE="datadoghq.com" 
helm install datadog --set datadog.apiKey=${DD_API_KEY} datadog/datadog --set datadog.appKey=${DD_APP_KEY} --set datadog.site=${DD_SITE} --set clusterAgent.enabled=true --set clusterAgent.metricsProvider.enabled=true --set clusterAgent.createPodDisruptionBudget=true --set clusterAgent.replicas=2

1. Install Keptn datadog-service to integrate Datadog with Keptn

export DD_API_KEY="<your-datadog-api-key>" DD_APP_KEY="<your-datadog-app-key>" DD_SITE="datadoghq.com" 
# cd datadog-service
helm install datadog-service ./helm --set datadogservice.ddApikey=${DD_API_KEY} --set datadogservice.ddAppKey=${DD_APP_KEY} --set datadogservice.ddSite=${DD_SITE}

1. Add SLI and SLO

keptn add-resource --project="<your-project>" --stage="<stage-name>" --service="<service-name>" --resource=/path-to/your/sli-file.yaml --resourceUri=datadog/sli.yaml
keptn add-resource --project="<your-project>"  --stage="<stage-name>" --service="<service-name>" --resource=/path-to/your/slo-file.yaml --resourceUri=slo.yaml

Example:

keptn add-resource --project="podtatohead" --stage="hardening" --service="helloservice" --resource=./quickstart/sli.yaml --resourceUri=datadog/sli.yaml
keptn add-resource --project="podtatohead" --stage="hardening" --service="helloservice" --resource=./quickstart/slo.yaml --resourceUri=slo.yaml

Check ./quickstart/sli.yaml and ./quickstart/slo.yaml for example SLI and SLO.

1. Configure Keptn to use datadog SLI provider Use keptn CLI version 0.15.0 or later.

keptn configure monitoring datadog --project <project-name>  --service <service-name>

1. Trigger delivery

keptn trigger delivery --project=<project-name> --service=<service-name> --image=<image> --tag=<tag>

Example:

keptn trigger delivery --project=podtatohead --service=helloservice --image=docker.io/jetzlstorfer/helloserver --tag=0.1.1

Observe the results in the Keptn Bridge

Compatibility Matrix

Please fill in your versions accordingly

datadog-service version will match Keptn version starting from 0.15.0 version of Keptn e.g., datadog-service 0.15.x is compatible with Keptn 0.15.x

Installation

export DD_API_KEY="<your-datadog-api-key>" DD_APP_KEY="<your-datadog-app-key>" DD_SITE="datadoghq.com" 
# cd datadog-service
helm install datadog-service ./helm --set datadogservice.ddApikey=${DD_API_KEY} --set datadogservice.ddAppKey=${DD_APP_KEY} --set datadogservice.ddSite=${DD_SITE}

Tell Keptn to use datadog as SLI provider for your project/service

keptn configure monitoring datadog --project <project-name>  --service <service-name>

This should install the datadog-service together with a Keptn distributor into the keptn namespace, which you can verify using

kubectl -n keptn get deployment datadog-service -o wide
kubectl -n keptn get pods -l run=datadog-service

Up- or Downgrading

Adapt and use the following command in case you want to up- or downgrade your installed version (specified by the $VERSION placeholder):

helm upgrade datadog-service ./helm --set datadogservice.ddApikey=${DD_API_KEY} --set datadogservice.ddAppKey=${DD_APP_KEY} --set datadogservice.ddSite=${DD_SITE}

Uninstall

To delete a deployed datadog-service helm chart:

helm uninstall datadog-service

Running tests on your local machine

port-forward Keptn API so that our tests can create/delete Keptn resources

kubectl port-forward svc/api-gateway-nginx 5000:80 -nkeptn # in a separate terminal window

from datadog-service repo

export ENABLE_E2E_TEST=true

export KEPTN_ENDPOINT=http://localhost:5000/api

export KEPTN_API_TOKEN=$(kubectl get secret keptn-api-token -n keptn -ojsonpath='{.data.keptn-api-token}' | base64 -d)

Run tests

gotestsum --format standard-verbose -- -timeout=120m  ./test/e2e/...

Development

Development can be conducted using any GoLang compatible IDE/editor (e.g., Jetbrains GoLand, VSCode with Go plugins).

It is recommended to make use of branches as follows:

• master contains the latest potentially unstable version
• release-* contains a stable version of the service (e.g., release-0.1.0 contains version 0.1.0)
• create a new branch for any changes that you are working on, e.g., feature/my-cool-stuff or bug/overflow
• once ready, create a pull request from that branch back to the master branch

When writing code, it is recommended to follow the coding style suggested by the Golang community.

Where to start

If you don't care about the details, your first entrypoint is eventhandlers.go. Within this file you can add implementation for pre-defined Keptn Cloud events.

To better understand all variants of Keptn CloudEvents, please look at the Keptn Spec.

If you want to get more insights into processing those CloudEvents or even defining your own CloudEvents in code, please look into main.go (specifically processKeptnCloudEvent), helm/templates, consult the Keptn docs as well as existing Keptn Core and Keptn Contrib services.

Common tasks

• Build the binary: go build -ldflags '-linkmode=external' -v -o datadog-service
• Run tests: go test -race -v ./...
• Build the docker image: docker build . -t ghcr.io/keptn-sandbox/datadog-service:latest
• Run the docker image locally: docker run --rm -it -p 8080:8080 ghcr.io/keptn-sandbox/datadog-service:latest
• Push the docker image to DockerHub: docker push ghcr.io/keptn-sandbox/datadog-service:latest
• Watch the deployment using kubectl: kubectl -n keptn get deployment datadog-service -o wide
• Get logs using kubectl: kubectl -n keptn logs deployment/datadog-service -f
• Watch the deployed pods using kubectl: kubectl -n keptn get pods -l run=datadog-service

Testing Cloud Events

We have dummy cloud-events in the form of RFC 2616 requests in the test-events/ directory. These can be easily executed using third party plugins such as the Huachao Mao REST Client in VS Code.

Automation

GitHub Actions: Automated Pull Request Review

This repo uses reviewdog for automated reviews of Pull Requests.

You can find the details in .github/workflows/reviewdog.yml.

GitHub Actions: Unit Tests

This repo has automated unit tests for pull requests.

You can find the details in .github/workflows/tests.yml.

How to release a new version of this service

It is assumed that the current development takes place in the master branch (either via Pull Requests or directly).

To make use of the built-in automation using GH Actions for releasing a new version of this service, you should

• branch away from master to a branch called release-x.y.z (where x.y.z is your version),
• check the output of GH Actions builds for the release branch,
• verify that your image was built and pushed to GHCR with the right tags,
• update the image tags in [deploy/service.yaml], and
• test your service against a working Keptn installation.

If any problems occur, fix them in the release branch and test them again.

Once you have confirmed that everything works and your version is ready to go, you should

• create a new release on the release branch using the GitHub releases page, and
• merge any changes from the release branch back to the master branch.

Known problems

1. If the evaluation window of the query is too short, the api might return an empty result which datadog-service treats as 0 and fails the evaluation. Issue
2. There is an on-purpose 60s delay before the datadog metrics API is called. This is because, calling the metrics API earlier leads to incorrect data. Issue
3. Does not support default queries for throughput, error rate, request latency etc., i.e., you have to enter the entire query. Issue

License

Please find more information in the LICENSE file.