telemetry-manager

Telemetry Manager

Overview

To implement Kyma's strategy of moving from in-cluster observability backends to a Telemetry component that integrates with external backends, Telemetry Manager is a Kubernetes operator that provides APIs for configurable logging, tracing, and monitoring.

Telemetry Manager has been bootstrapped with Kubebuilder 3.6.0. Additional APIs can also be added by Kubebuilder.

Configurable Logging

The logging controllers generate a Fluent Bit DaemonSet and configuration from one or more LogPipeline and LogParser custom resources. The controllers ensure that all Fluent Bit Pods run the current configuration by restarting Pods after the configuration has changed. See all CRD attributes and some examples.

Further design decisions and test results are documented in Dynamic Logging Backend Configuration.

Configurable Tracing

The trace controller creates an OpenTelemetry Collector deployment and related Kubernetes objects from a TracePipeline custom resource. The collector is configured to receive traces using the OTLP and OpenCensus protocols, and forwards the received traces to a configurable OTLP backend.

See Dynamic Trace Backend Configuration for further information.

Configurable Monitoring

The metric controller creates an OpenTelemetry Collector and related Kubernetes objects from a MetricPipeline custom resource. The collector is deployed as a Gateway. The controller is configured to receive metrics in the OTLP protocol and forward them to a configurable OTLP backend.

See Dynamic Monitoring Backend Configuration for further information.

Development

Prerequisites

• Install kubebuilder 3.6.0, which is the base framework for this controller. Required to add new APIs.
• Install Golang 1.19 or newer (for development and local execution).
• Install Docker.
• Install golangci-lint.

Other dependencies will be downloaded by the make targets to the bin sub-folder.

Available Commands

For development, you can use the following commands:

• Run unit tests

make test

• Create a k3d cluster on Docker, deploy Telemetry Manager, and run integration tests

make e2e-test

• Run golangci-lint and lint manifests

make lint

• Autofix all automatically-fixable linter complaints

make lint-autofix

• Regenerate YAML manifests (CRDs and RBAC)

make manifests

• Install CRDs to cluster in current kubeconfig context

make install

• Uninstall CRDs to cluster in current kubeconfig context

make uninstall

• Run the operator locally (uses current kubeconfig context)

make run

• Build container image and deploy to cluster in current kubeconfig context

export IMG=<my container repo>
make docker-build
make docker-push
make deploy

• Clean up everything

make undeploy

Deploying ModuleTemplate with the Lifecycle Manager

Check the documentation here.

Troubleshooting

Enable pausing reconciliations

You must pause reconciliations to be able to debug the pipelines and, for example, try out a different pipeline configuration or a different OTel configuration. To pause reconciliations, create a telemetry-override-config in the operators Namespace. Here is an example of such a ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: telemetry-override-config
data:
  override-config: |
    global:
      logLevel: debug
    tracing:
      paused: true
    logging:
      paused: true
    metrics:
      paused: true

The global, tracing, logging and metrics fields are optional.

Debugging steps

1. Create an overriding telemetry-override-config ConfigMap.

2. Perform debugging operations.

3. Remove the created ConfigMap.

4. To reset the debug actions, perform a restart of Telemetry Manager.

   kubectl rollout restart deployment telemetry-controller-manager
   

Caveats If you change the pipeline CR when the reconciliation is paused, these changes will not be applied immediately but in a periodic reconciliation cycle of one hour. To reconcile earlier, restart Telemetry Manager.

Profiling

Telemetry Manager has pprof-based profiling activated and exposed on port 6060. Use port-forwarding to access the pprof endpoint. You can find additional information in the Go pprof package documentation.