ecsobserver

Amazon Elastic Container Service Observer

Status
Stability	beta
Distributions	contrib
Issues
Code Owners	@dmitryax, @rmfitzpatrick

The ecsobserver uses the ECS/EC2 API to discover prometheus scrape targets from all running tasks and filter them based on service names, task definitions and container labels.

NOTE: If you run collector as a sidecar, you should consider use ECS resource detector instead. However, it does not have service, EC2 instances etc. because it only queries local API.

Config

The configuration is based on existing cloudwatch agent ECS discovery . A full collector config looks like the following:

extensions:
  ecs_observer:
    refresh_interval: 60s # format is https://golang.org/pkg/time/#ParseDuration
    cluster_name: 'Cluster-1' # cluster name need manual config
    cluster_region: 'us-west-2' # region can be configured directly or use AWS_REGION env var
    result_file: '/etc/ecs_sd_targets.yaml' # the directory for file must already exists
    services:
      - name_pattern: '^retail-.*$'
    docker_labels:
      - port_label: 'ECS_PROMETHEUS_EXPORTER_PORT'
    task_definitions:
      - job_name: 'task_def_1'
        metrics_path: '/metrics'
        metrics_ports:
          - 9113
          - 9090
        arn_pattern: '.*:task-definition/nginx:[0-9]+'

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "ecs-task"
          file_sd_configs:
            - files:
                - '/etc/ecs_sd_targets.yaml' # MUST match the file name in ecs_observer.result_file
          relabel_configs: # Relabel here because label with __ prefix will be dropped by receiver.
            - source_labels: [ __meta_ecs_cluster_name ] # ClusterName
              action: replace
              target_label: ClusterName
            - source_labels: [ __meta_ecs_service_name ] # ServiceName
              action: replace
              target_label: ServiceName
            - action: labelmap # Convert docker labels on container to metric labels
              regex: ^__meta_ecs_container_labels_(.+)$ # Capture the key using regex, e.g. __meta_ecs_container_labels_Java_EMF_Metrics -> Java_EMF_Metrics
              replacement: '$$1'

processors:
  batch:

# Use awsemf for CloudWatch Container Insights Prometheus. The extension does not have requirement on exporter.
exporters:
  awsemf:

service:
  pipelines:
    metrics:
      receivers: [ prometheus ]
      processors: [ batch ]
      exporters: [ awsemf ]
  extensions: [ ecs_observer ]

Output configuration

result_file specifies where to write the discovered targets. It MUST match the files defined in file_sd_configs for prometheus receiver. See output format for the detailed format.

Filters configuration

There are three type of filters, and they share the following common optional properties.

• job_name
• metrics_path
• metrics_ports an array of port number

Example

ecs_observer:
  job_name: 'ecs-sd-job'
  services:
    - name_pattern: ^retail-.*$
      container_name_pattern: ^java-api-v[12]$
    - name_pattern: game
      metrics_path: /v3/343
      job_name: guilty-spark
  task_definitions:
    - arn_pattern: '*memcached.*'
    - arn_pattern: '^proxy-.*$'
      metrics_ports:
        - 9113
        - 9090
      metrics_path: /internal/metrics
  docker_labels:
    - port_label: ECS_PROMETHEUS_EXPORTER_PORT
    - port_label: ECS_PROMETHEUS_EXPORTER_PORT_V2
      metrics_path_label: ECS_PROMETHEUS_EXPORTER_METRICS_PATH

ECS Service Name based filter Configuration

ECS Task Definition based filter Configuration

Docker Label based filter Configuration

Specify label keys to look up value

Authentication

It uses the default credential chain, on ECS it is advised to use ECS task role. You need to deploy the collector as an ECS task/service with the following permissions .

EC2 access is required for getting private IP for ECS EC2. However, EC2 permission can be removed if you are only using Fargate because task ip comes from awsvpc instead of host.

ec2:DescribeInstances
ecs:ListTasks
ecs:ListServices
ecs:DescribeContainerInstances
ecs:DescribeServices
ecs:DescribeTasks
ecs:DescribeTaskDefinition

Design

• Discovery
• Notify receiver
• Output format

Discovery mechanism

The extension polls ECS API periodically to get all running tasks and filter out scrape targets. There are 3 types of filters for discovering targets, targets match the filter are kept. Targets from different filters are merged base on address/metrics_path before updating/creating receiver.

ECS Service Name based filter

ECS Service is a deployment that manages multiple tasks with same definition (like Deployment and DaemonSet in k8s).

The service configuration matches both service name and container name (if not empty).

NOTE: name of the service is added as label value with key ServiceName.

# Example 1: Matches all containers that are started by retail-* service
name_pattern: ^retail-.*$
---
# Example 2: Matches all container with name java-api in cash-app service 
name_pattnern: ^cash-app$
container_name_pattern: ^java-api$
---
# Example 3: Override default metrics_path (i.e. /metrics)
name_pattern: ^log-replay-worker$
metrics_path: /v3/metrics

ECS Task Definition based filter

ECS task definition contains one or more containers (like Pod in k8s). Long running applications normally uses service. while short running (batch) jobs can be created from task definitions directly .

The task definition matches both task definition name and container name (of not empty). Optional config like metrics_path, metrics_ports, job_name can override default value.

# Example 1: Matches all the tasks created from task definition that contains memcached in its arn
arn_pattern: "*memcached.*"

Docker Label based filter

Docker label can be specified in task definition. Only port_label is used when checking if the container should be included. Optional config like metrics_path_label, job_name_label can override default value.

# Example 1: Matches all the container that has label ECS_PROMETHEUS_EXPORTER_PORT_NGINX
port_label: 'ECS_PROMETHEUS_EXPORTER_PORT_NGINX'
---
# Example 2: Override job name based on label MY_APP_JOB_NAME
port_label: 'ECS_PROMETHEUS_EXPORTER_PORT_MY_APP'
job_name_label: 'MY_APP_JOB_NAME'

Notify Prometheus Receiver of discovered targets

There are three ways to notify a receiver

• Use file based service discovery in prometheus config and updates the file.
• Use receiver creator framework to create a new receiver for new endpoints.
• Register as a prometheus discovery plugin.

Generate target file for file based discovery

• Status: implemented

This is current approach used by cloudwatch-agent and also recommended by prometheus . It's easier to debug and the main drawback is it only works for prometheus. Another minor issue is fsnotify may not work properly occasionally and delay the update.

Receiver creator framework

• Status: pending

This is a generic approach that creates a new receiver at runtime based on discovered endpoints. The main problem is performance issue as described in this issue.

Register as prometheus discovery plugin

• Status: pending

Because both the collector and prometheus is written in Go, we can call discover.RegisterConfig to make it a valid config for prometheus (like other in tree plugins like kubernetes). The drawback is the configuration is now under prometheus instead of extension and can cause confusion.

Output Format

Example in unit test.

The format is based on cloudwatch agent , ec2 sd and kubernetes sd. Task and labels from task definition are always included. EC2 info is only included when task is running on ECS EC2 ( i.e. not on Fargate).

Unlike cloudwatch agent, all the additional labels starts with __meta_ecs_ prefix. If they are not renamed during relabel, they will all get dropped in prometheus receiver and won't pass down along the pipeline.

The number of dimensions supported by AWS EMF exporter is limited by its backend. The labels can be modified/filtered at different stages, prometheus receiver relabel, Metrics Transform Processor and EMF exporter Metric Declaration

Essential Labels

Required for prometheus to scrape the target.

Additional Labels

Additional information from ECS and EC2.

Serialization

• Labels, all the label value are encoded as string. (e.g. strconv.Itoa(123)).
• Go struct, all the non string types are converted. labels and tags are passed as map[string]string instead of []KeyValue
• Prometheus target, each taget

// PrometheusECSTarget contains address and labels extracted from a running ECS task 
// and its underlying EC2 instance (if available).
// 
// For serialization
// - FromLabels and ToLabels converts it between map[string]string.
// - FromTargetYAML and ToTargetYAML converts it between prometheus file discovery format in YAML. 
// - FromTargetJSON and ToTargetJSON converts it between prometheus file discovery format in JSON. 
type PrometheusECSTarget struct {
	Address                string            `json:"address"`
	MetricsPath            string            `json:"metrics_path"`
	Job                    string            `json:"job"`
	TaskDefinitionFamily   string            `json:"task_definition_family"`
	TaskDefinitionRevision int               `json:"task_definition_revision"`
	TaskLaunchType         string            `json:"task_launch_type"`
	TaskGroup              string            `json:"task_group"`
	TaskTags               map[string]string `json:"task_tags"`
	ContainerName          string            `json:"container_name"`
	ContainerLabels        map[string]string `json:"container_labels"`
	HealthStatus           string            `json:"health_status"`
	EC2InstanceId          string            `json:"ec2_instance_id"`
	EC2InstanceType        string            `json:"ec2_instance_type"`
	EC2Tags                map[string]string `json:"ec2_tags"`
	EC2VPCId               string            `json:"ec2_vpc_id"`
	EC2PrivateIP           string            `json:"ec2_private_ip"`
	EC2PublicIP            string            `json:"ec2_public_ip"`
}

Delta

Delta is not supported because there is no watch API in ECS (unlike k8s, see known issues). The output always contains all the targets. Caller/Consumer need to implement their own logic to calculate the targets diff if they only want to process new targets.

Known issues

• There is no list watch API in ECS (unlike k8s), and we fetch ALL the tasks and filter it locally. If the poll interval is too short or there are multiple instances doing discovery, you may hit the (undocumented) API rate limit. In memory caching is implemented to reduce calls for task definition and ec2.
• A single collector may not be able to handle a large cluster, you can use hashmod in relabel_config to do static sharding. However, too many collectors may trigger the rate limit on AWS API as each shard is fetching ALL the tasks during discovery regardless of number of shards.

Implementation

The implementation has two parts, core ecs service discovery logic and adapter for notifying discovery results.

Packages

• extension/observer/ecsobserver main logic
• internal/ecsmock mock ECS cluster
• internal/errctx structured error wrapping

Flow

The pseudocode showing the overall flow.

NewECSSD() {
  session := awsconfig.NewSssion()
  ecsClient := awsecs.NewClient(session)
  filters := config.NewFileters()
  decorator := awsec2.NewClient(session)
  for {
    select {
    case <- timer:
      // Fetch ALL
      tasks := ecsClient.FaetchAll()
      // Filter
      filteredTasks := fileters.Apply(tasks)
      // Add EC2 info
      decorator.Apply(filteredTask)
      // Generate output
      if writeResultFile {
         writeFile(fileteredTasks, /etc/ecs_sd.yaml)
      } else {
          notifyObserver()
      }
    }
  }
}

Metrics

Following metrics are logged at debug level. TODO(pingleig): Is there a way for otel plugins to export custom metrics to otel's own /metrics.

Error Handling

• Auth and cluster not found error will cause the extension to stop (calling ReportStatus). Although IAM role can be updated at runtime without restarting the collector, it's better to fail to make the problem obvious. Same applies to cluster not found. In the future we can add config to downgrade those errors if user want to monitor an ECS cluster with collector running outside the cluster, the collector can run anywhere as long as it can reach scrape targets and AWS API.
• If we have non-critical error, we overwrite existing file with whatever targets we have, we might not have all the targets due to throttle etc.

Unit Test

A mock ECS and EC2 server is in internal/ecsmock, see fetcher_test for its usage.

Integration Test

Will be implemented in AOT Testing Framework to run against actual ECS service on both EC2 and Fargate.

Changelog

• 2021-06-02 first version that actually works on ECS by @pingleig, thanks @anuraaga @Aneurysm9 @jrcamp @mxiamxia for reviewing (all the PRs ...)
• 2021-02-24 Updated doc by @pingleig
• 2020-12-29 Initial implementation by Raphael in #1920