ecsservicediscovery

package
v1.3000030.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 2, 2023 License: MIT Imports: 16 Imported by: 0

README

ECS Prometheus Exporter Auto Discovery

Overview

This module provides the Prometheus exporter auto discovery functionality based on the ECS task metadata.

There are two modes to discover the Prometheus targets based on the customer config:

  • Mode 1: Docker label Based: Customers add the docker labels to the containers to indicate the port and metric path of the Prometheus metrics. Customers configure the CWAgent to discover the Prometheus targets with the matching docker label and container port.
  • Mode 2: ECS Task Definition ARN based: Customers configure the CWAgent to discover the Prometheus targets when its ECS task definition arn matches the configured regex and it matches the configured container ports.

Two modes can be enabled together and CWAgent will de-dup the discovered targets based on: {private_ip}:{port}/{metrics_path}

Service Discovery Workflow
  1. List the current running ECS task ARNs for the specific ECS cluster by ECS:ListTasks paginated call
  2. Describe the ECS tasks based on the ListTasks response ECS:DescribeTasks batch call
  3. Get the ECS Task Definition from LRU cache, if there is none in cache, call ECS:DescribeTaskDefinition and cache. LRU cache size (2000) based on ECS service quota
  4. Check the Container Docker Label if there is Docker Label based Service Discovery config
  5. Check the Task Definition ARN if there is Task Definition ARN Regex config
  6. Filter the ECS tasks that match the above two checking for further processing
  7. Get the containerInstance/ec2 instance info from LRU cache if the tasks is running on EC2 launch type. LRU cache size (2000) based on ECS service quota
  8. Call ECS: DescribeContainerInstances and EC2:DescribeInstances for the instances have not been cached. Batching Call with batch size = 100.
  9. Export the ECS Prometheus targets into file configured by sd_result_file

Configuration Options

Overall Configuration Options
Configuration Field Description
sd_frequency Mandatory frequency to discover the prometheus exporters
sd_target_cluster Mandatory target ECS cluster name for service discovery
sd_cluster_region Mandatory the target ECS clusters' AWS region name
sd_result_file Mandatory path of the yaml file for the Prometheus target results
docker_label Optional docker label based service discovery configurations. If this structure is nil, docker label based service discovery is disabled
task_definition_list Optional ECS task definition based service discovery configurations slice. If this slice is empty, task definition based service discovery is disabled
Service Endpoint Based Auto Discovery
Configuration Field Description
sd_service_name_pattern Mandatory ECS service name regex pattern
sd_metrics_ports Mandatory semicolon separated containerPort for Prometheus metrics.
sd_container_name_pattern Optional ECS task container name regex pattern
sd_metrics_path Optional Prometheus metric path. If not specified, the default path /metrics is assumed
sd_job_name Optional Prometheus scrape job name. If not specified, the job name in prometheus.yaml is used
Docker Label Based Auto Discovery Configuration
Configuration Field Description
sd_port_label Mandatory Container's docker label name that specify the containerPort for Prometheus metrics. Only one value is allowed
sd_metrics_path_label Optional Container's docker label name that specify the Prometheus metric path. If not specified, the default path /metrics is assumed.
sd_job_name_label Optional Container's docker label name that specify the Prometheus scrape job name. If not specified, the job name in prometheus.yaml is used.
Task Definition Based Auto Discovery
Configuration Field Description
sd_task_definition_arn_pattern Mandatory ECS task definition arn regex pattern
sd_metrics_ports Mandatory semicolon separated containerPort for Prometheus metrics.
sd_container_name_pattern Optional ECS task container name regex pattern
sd_metrics_path Optional Prometheus metric path. If not specified, the default path /metrics is assumed
sd_job_name Optional Prometheus scrape job name. If not specified, the job name in prometheus.yaml is used
Configuration Example

Sample Configuration in TOML format:

    [inputs.prometheus.ecs_service_discovery]
      sd_cluster_region = "us-east-2"
      sd_frequency = "15s"
      sd_result_file = "/opt/aws/amazon-cloudwatch-agent/etc/ecs_sd_targets.yaml"
      sd_target_clusters = "EC2-Justin-Testing; Fargate-Justin-Testing"
      [inputs.prometheus.ecs_service_discovery.docker_label]
        sd_job_name_label = "ECS_PROMETHEUS_JOB_NAME"
        sd_metrics_path_label = "ECS_PROMETHEUS_METRICS_PATH"
        sd_port_label = "ECS_PROMETHEUS_EXPORTER_PORT_SUBSET_A"

      [[inputs.prometheus.ecs_service_discovery.task_definition_list]]
        sd_job_name = "task_def_1"
        sd_metrics_path = "/stats/metrics"
        sd_metrics_ports = "9901;9404;9406"
        sd_task_definition_arn_pattern = ".*:task-definition/bugbash-java-fargate-awsvpc-task-def-only:[0-9]+"

      [[inputs.prometheus.ecs_service_discovery.task_definition_list]]
        sd_container_name_pattern = "^bugbash-jar.*$"
        sd_metrics_ports = "9902"
        sd_task_definition_arn_pattern = ".*:task-definition/nginx:[0-9]+"

Permission

ECS Task Role needs to be granted the following permission so CWAgent can query the ECS/EC2 frontend to get the task meteData.

  • ECS Policy
ECS:ListTasks,
ECS:DescribeContainerInstances,
ECS:DescribeTasks,
ECS:DescribeTaskDefinition
EC2:DescribeInstances

Example Result

- targets:
  - 10.6.1.95:32785
  labels:
    __metrics_path__: /metrics
    ECS_PROMETHEUS_EXPORTER_PORT_SUBSET_B: "9406"
    ECS_PROMETHEUS_JOB_NAME: demo-jar-ec2-bridge-subset-b-dynamic
    ECS_PROMETHEUS_METRICS_PATH: /metrics
    InstanceType: t3.medium
    LaunchType: EC2
    SubnetId: subnet-0347624eeea6c5969
    TaskDefinitionFamily: demo-jar-ec2-bridge-dynamic-port-subset-b
    TaskGroup: family:demo-jar-ec2-bridge-dynamic-port-subset-b
    TaskRevision: "7"
    VpcId: vpc-033b021cd7ecbcedb
    container_name: demo-jar-ec2-bridge-dynamic-port-subset-b
    job: task_def_2
- targets:
  - 10.6.1.95:32783
  labels:
    __metrics_path__: /metrics
    ECS_PROMETHEUS_EXPORTER_PORT_SUBSET_B: "9406"
    ECS_PROMETHEUS_JOB_NAME: demo-jar-ec2-bridge-subset-b-dynamic
    ECS_PROMETHEUS_METRICS_PATH: /metrics
    InstanceType: t3.medium
    LaunchType: EC2
    SubnetId: subnet-0347624eeea6c5969
    TaskDefinitionFamily: demo-jar-ec2-bridge-dynamic-port-subset-b
    TaskGroup: family:demo-jar-ec2-bridge-dynamic-port-subset-b
    TaskRevision: "7"
    VpcId: vpc-033b021cd7ecbcedb
    container_name: demo-jar-ec2-bridge-dynamic-port-subset-b
    job: task_def_2

Documentation

Index

Constants

View Source 
const (
	AWSAPIDescribeContainerInstances = "AWSCLI_DescribeContainerInstances"
	AWSCLIDescribeInstancesRequest   = "AWSCLI_DescribeInstancesRequest"
	AWSCLIDescribeTaskDefinition     = "AWSCLI_DescribeTaskDefinition"
	AWSCLIListServices               = "AWSCLI_ListServices"
	AWSCLIListTasks                  = "AWSCLI_ListTasks"
	AWSCLIDescribeTasks              = "AWSCLI_DescribeTasks"
	LRUCacheGetEC2MetaData           = "LRUCache_Get_EC2MetaData"
	LRUCacheGetTaskDefinition        = "LRUCache_Get_TaskDefinition"
	LRUCacheSizeContainerInstance    = "LRUCache_Size_ContainerInstance"
	LRUCacheSizeTaskDefinition       = "LRUCache_Size_TaskDefinition"
	ExporterDiscoveredTargetCount    = "Exporter_DiscoveredTargetCount"
)
View Source 
const (
	AwsSdkLevelRetryCount = 3
)

Variables

This section is empty.

Functions

func StartECSServiceDiscovery 

func StartECSServiceDiscovery(sd *ServiceDiscovery, shutDownChan chan interface{}, wg *sync.WaitGroup)

Types

type ContainerInstanceProcessor 

type ContainerInstanceProcessor struct {
	// contains filtered or unexported fields
}

Add the Container instance metadata for ECS Clusters on Linux EC2 Instances

func NewContainerInstanceProcessor 

func NewContainerInstanceProcessor(ecs *ecs.ECS, ec2 *ec2.EC2, s *ProcessorStats) *ContainerInstanceProcessor

func (*ContainerInstanceProcessor) Process 

func (p *ContainerInstanceProcessor) Process(cluster string, taskList []*DecoratedTask) ([]*DecoratedTask, error)

func (*ContainerInstanceProcessor) ProcessorName 

func (p *ContainerInstanceProcessor) ProcessorName() string

type DecoratedTask 

type DecoratedTask struct {
	Task           *ecs.Task
	TaskDefinition *ecs.TaskDefinition
	EC2Info        *EC2MetaData
	ServiceName    string

	DockerLabelBased    bool
	TaskDefinitionBased bool
}

func (*DecoratedTask) ExporterInformation 

func (t *DecoratedTask) ExporterInformation(config *ServiceDiscoveryConfig, dockerLabelRegex *regexp.Regexp, targets map[string]*PrometheusTarget)

func (*DecoratedTask) String 

func (t *DecoratedTask) String() string

type DockerLabelConfig 

type DockerLabelConfig struct {
	JobNameLabel     string `toml:"sd_job_name_label"`
	PortLabel        string `toml:"sd_port_label"`
	MetricsPathLabel string `toml:"sd_metrics_path_label"`
}

type DockerLabelDiscoveryProcessor 

type DockerLabelDiscoveryProcessor struct {
	// contains filtered or unexported fields
}

Tag the Tasks that matched the Docker Label based SD Discovery

func NewDockerLabelDiscoveryProcessor 

func NewDockerLabelDiscoveryProcessor(d *DockerLabelConfig) *DockerLabelDiscoveryProcessor

func (*DockerLabelDiscoveryProcessor) Process 

func (p *DockerLabelDiscoveryProcessor) Process(cluster string, taskList []*DecoratedTask) ([]*DecoratedTask, error)

func (*DockerLabelDiscoveryProcessor) ProcessorName 

func (p *DockerLabelDiscoveryProcessor) ProcessorName() string

type EC2MetaData 

type EC2MetaData struct {
	ContainerInstanceId string
	ECInstanceId        string
	PrivateIP           string
	InstanceType        string
	VpcId               string
	SubnetId            string
}

type Processor 

type Processor interface {
	Process(cluster string, taskList []*DecoratedTask) ([]*DecoratedTask, error)
	ProcessorName() string
}

type ProcessorStats 

type ProcessorStats struct {
	// contains filtered or unexported fields
}

func (*ProcessorStats) AddStats 

func (sd *ProcessorStats) AddStats(name string)

func (*ProcessorStats) AddStatsCount 

func (sd *ProcessorStats) AddStatsCount(name string, count int)

func (*ProcessorStats) GetStats 

func (sd *ProcessorStats) GetStats(name string) int

func (*ProcessorStats) ResetStats 

func (sd *ProcessorStats) ResetStats()

func (*ProcessorStats) ShowStats 

func (sd *ProcessorStats) ShowStats()

type PrometheusTarget 

type PrometheusTarget struct {
	Targets []string          `yaml:"targets"`
	Labels  map[string]string `yaml:"labels"`
}

type ServiceDiscovery 

type ServiceDiscovery struct {
	Config *ServiceDiscoveryConfig
	// contains filtered or unexported fields
}

type ServiceDiscoveryConfig 

type ServiceDiscoveryConfig struct {
	Frequency            string                       `toml:"sd_frequency"`
	ResultFile           string                       `toml:"sd_result_file"`
	TargetCluster        string                       `toml:"sd_target_cluster"`
	TargetClusterRegion  string                       `toml:"sd_cluster_region"`
	ServiceNamesForTasks []*ServiceNameForTasksConfig `toml:"service_name_list_for_tasks"`
	DockerLabel          *DockerLabelConfig           `toml:"docker_label"`
	TaskDefinitions      []*TaskDefinitionConfig      `toml:"task_definition_list"`
}

type ServiceDiscoveryError 

type ServiceDiscoveryError struct {
	// contains filtered or unexported fields
}

func (ServiceDiscoveryError) Error 

func (p ServiceDiscoveryError) Error() string

type ServiceEndpointDiscoveryProcessor added in v1.247347.1 

type ServiceEndpointDiscoveryProcessor struct {
	// contains filtered or unexported fields
}

Tag the Tasks that match the Service Name based Service Discovery

func NewServiceEndpointDiscoveryProcessor added in v1.247347.1 

func NewServiceEndpointDiscoveryProcessor(ecs *ecs.ECS, serviceNamesForTasks []*ServiceNameForTasksConfig, s *ProcessorStats) *ServiceEndpointDiscoveryProcessor

func (*ServiceEndpointDiscoveryProcessor) Process added in v1.247347.1 

func (p *ServiceEndpointDiscoveryProcessor) Process(cluster string, taskList []*DecoratedTask) ([]*DecoratedTask, error)

func (*ServiceEndpointDiscoveryProcessor) ProcessorName added in v1.247347.1 

func (p *ServiceEndpointDiscoveryProcessor) ProcessorName() string

type ServiceNameForTasksConfig added in v1.247347.1 

type ServiceNameForTasksConfig struct {
	ContainerNamePattern string `toml:"sd_container_name_pattern"`
	JobName              string `toml:"sd_job_name"`
	MetricsPath          string `toml:"sd_metrics_path"`
	MetricsPorts         string `toml:"sd_metrics_ports"`
	ServiceNamePattern   string `toml:"sd_service_name_pattern"`
	// contains filtered or unexported fields
}

func (*ServiceNameForTasksConfig) String added in v1.247347.1 

func (s *ServiceNameForTasksConfig) String() string

type TargetsExportProcessor 

type TargetsExportProcessor struct {
	// contains filtered or unexported fields
}

func NewTargetsExportProcessor 

func NewTargetsExportProcessor(sdConfig *ServiceDiscoveryConfig, s *ProcessorStats) *TargetsExportProcessor

func (*TargetsExportProcessor) Process 

func (p *TargetsExportProcessor) Process(cluster string, taskList []*DecoratedTask) ([]*DecoratedTask, error)

func (*TargetsExportProcessor) ProcessorName 

func (p *TargetsExportProcessor) ProcessorName() string

type TaskDefinitionConfig 

type TaskDefinitionConfig struct {
	ContainerNamePattern string `toml:"sd_container_name_pattern"`
	JobName              string `toml:"sd_job_name"`
	MetricsPath          string `toml:"sd_metrics_path"`
	MetricsPorts         string `toml:"sd_metrics_ports"`
	TaskDefArnPattern    string `toml:"sd_task_definition_arn_pattern"`
	// contains filtered or unexported fields
}

func (*TaskDefinitionConfig) String 

func (t *TaskDefinitionConfig) String() string

type TaskDefinitionDiscoveryProcessor 

type TaskDefinitionDiscoveryProcessor struct {
	// contains filtered or unexported fields
}

Tag the Tasks that match the Task Definition ARN based Service Discovery

func NewTaskDefinitionDiscoveryProcessor 

func NewTaskDefinitionDiscoveryProcessor(taskDefinitions []*TaskDefinitionConfig) *TaskDefinitionDiscoveryProcessor

func (*TaskDefinitionDiscoveryProcessor) Process 

func (p *TaskDefinitionDiscoveryProcessor) Process(cluster string, taskList []*DecoratedTask) ([]*DecoratedTask, error)

func (*TaskDefinitionDiscoveryProcessor) ProcessorName 

func (p *TaskDefinitionDiscoveryProcessor) ProcessorName() string

type TaskDefinitionProcessor 

type TaskDefinitionProcessor struct {
	// contains filtered or unexported fields
}

Decorate the tasks with the ECS task definition

func NewTaskDefinitionProcessor 

func NewTaskDefinitionProcessor(ecs *ecs.ECS, s *ProcessorStats) *TaskDefinitionProcessor

func (*TaskDefinitionProcessor) Process 

func (p *TaskDefinitionProcessor) Process(cluster string, taskList []*DecoratedTask) ([]*DecoratedTask, error)

func (*TaskDefinitionProcessor) ProcessorName 

func (p *TaskDefinitionProcessor) ProcessorName() string

type TaskFilterProcessor 

type TaskFilterProcessor struct {
}

Filter out the tasks not matching the discovery configs Filter out the tasks with nil task definition

func NewTaskFilterProcessor 

func NewTaskFilterProcessor() *TaskFilterProcessor

func (*TaskFilterProcessor) Process 

func (p *TaskFilterProcessor) Process(cluster string, taskList []*DecoratedTask) ([]*DecoratedTask, error)

func (*TaskFilterProcessor) ProcessorName 

func (p *TaskFilterProcessor) ProcessorName() string

type TaskProcessor 

type TaskProcessor struct {
	// contains filtered or unexported fields
}

Get all running tasks for the target cluster

func NewTaskProcessor 

func NewTaskProcessor(ecs *ecs.ECS, s *ProcessorStats) *TaskProcessor

func (*TaskProcessor) Process 

func (p *TaskProcessor) Process(cluster string, taskList []*DecoratedTask) ([]*DecoratedTask, error)

func (*TaskProcessor) ProcessorName 

func (p *TaskProcessor) ProcessorName() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.