watchdog

Coinbase Watchdog

A Datadog codification framework.

The goal of Coinbase Watchdog is to make users lives easier by monitoring Datadog components and ensuring their consistency with data stored in git repository.

Coinbase Watchdog operates in two different ways:

• Code driven: Users can make a change to component configuration and submit a pull request. Once merged, Coinbase Watchdog will automatically call the Datadog API to update with latest change.
• UI driven: Coinbase Watchdog will monitor Datadog components for changes. If a change is detected (between actual component and data stored in git) a new pull request will be created. If the pull request was closed, Coinbase Watchdog will restore the component from git, discarding the change.

Quick guide

• Create config files in watchdog-resources repository under config folder. This folder may contain an arbitrary number of sub folders. This gives users ability to organize the configuration hierarchy for their best needs. Config must be in yaml format and have a .yaml file extension.

Example config file

# cat ./config/infra/sre/components.yaml
meta:
    team: infra/sre

dashboards:
    - 954604

monitors:
    - 6065878

How to setup Coinbase Watchdog from scratch

• Create a github app, follow the guide on github.com
  • Give permission to read/write to Pull Requests
  • Generate a new RSA private key in github UI.
• Generate an rsa-ssh key from github app private RSA key: ssh-keygen -y -f <private.key>
  • Add the generated public rsa-ssh key to github repo, where config and data will be stored under Settings -> Deploy Keys. Make sure to check Allow write access. This will grant Coinbase Watchdog permissions to push changes to remote branch and open PRs.

Coinbase Watchdog parameters

All parameters must be set via environment variables.

There are 2 different categories of parameters: system and user parameters.

System parameters:

• DD_API_KEY, required - Datadog API key.
• DD_APP_KEY, required - Datadog APP key.
• DATADOG_POLLING_SCHEDULER, optional, default set to "simple" - Datadog polling scheduler method.
• DATADOG_POLLING_INTERVAL, optional, default set to "20s" - Datadog poling interval.
• GITHUB_ASSETS_STORE_PATH, optional, default set to "data" - Base directory in watchdog-resources repo to store components data to.
• GITHUB_BASE_URL, optional, default set to github.com - Set the default github URL. Useful for github EE.
• GITHUB_APP_PRIVATE_KEY, required - Private key generated by github app.
• GITHUB_PROJECT_OWNER, required - Organization name which contains the watchdog-resources repo.
• GITHUB_REPO, required - Name of repo to save datadog components, usually watchdog-resources.
• GITHUB_APP_INTEGRATION_ID, required - Github app integration ID.
• GITHUB_APP_INSTALLATION_ID, required - Github installation ID.
• GITHUB_WEBHOOK_SECRET", optional, unset - Github webhook secret.
• LOGGING_LEVEL, optional, unset - Set the logging level (info/debug/warning).
• LOGGING_JSON, optional, default set to false - Output JSON logs.
• HTTP_SECRET, optional, unset - Secret used to access HTTP endpoints. (Refer to design doc for more details)
• HTTP_PORT, optional, default set to "3000" - Port to listen on
• SLACK_TOKEN, optional, - Use slack token for notifications.

User parameters:

• USER_CONFIG_PATH, optional, default set to "/config" - Prefix to base path with user configs.
• USER_CONFIG_GIT_URL, required - URL to github repo with user configs.
• USER_CONFIG_UPDATE_INTERVAL, optional, default set to "10m" - Coinbase Watchdog will automatically reload user configs every 10 minutes.
• USER_CONFIG_GIT_PRIVATE_KEY, required - Private key to clone the repo (the public key must be in Deploy keys).