awsemfexporter

package module
v0.0.0-...-b02dcfa Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 14, 2022 License: Apache-2.0 Imports: 28 Imported by: 0

README

AWS CloudWatch EMF Exporter for OpenTelemetry Collector

Status
Stability beta
Supported pipeline types metrics
Distributions contrib

This exporter converts OpenTelemetry metrics to AWS CloudWatch Embedded Metric Format(EMF) and then sends them directly to CloudWatch Logs using the PutLogEvents API.

Data Conversion

Convert OpenTelemetry Int64DataPoints, DoubleDataPoints, SummaryDataPoints metrics datapoints into CloudWatch EMF structured log formats and send it to CloudWatch. Logs and Metrics will be displayed in CloudWatch console.

Exporter Configuration

The following exporter configuration parameters are supported.

Name Description Default
log_group_name Customized log group name which supports {ClusterName} and {TaskId} placeholders. One valid example is /aws/metrics/{ClusterName}. It will search for ClusterName (or aws.ecs.cluster.name) resource attribute in the metrics data and replace with the actual cluster name. If none of them are found in the resource attribute map, {ClusterName} will be replaced by undefined. Similar way, for the {TaskId}, it searches for TaskId (or aws.ecs.task.id) key in the resource attribute map. For {NodeName}, it searches for NodeName (or k8s.node.name) "/metrics/default"
log_stream_name Customized log stream name which supports {TaskId}, {ClusterName}, {NodeName}, {ContainerInstanceId}, and {TaskDefinitionFamily} placeholders. One valid example is {TaskId}. It will search for TaskId (or aws.ecs.task.id) resource attribute in the metrics data and replace with the actual task id. If none of them are found in the resource attribute map, {TaskId} will be replaced by undefined. Similarly, for the {TaskDefinitionFamily}, it searches for TaskDefinitionFamily (or aws.ecs.task.family). For the {ClusterName}, it searches for ClusterName (or aws.ecs.cluster.name). For {NodeName}, it searches for NodeName (or k8s.node.name). For {ContainerInstanceId}, it searches for ContainerInstanceId (or aws.ecs.container.instance.id). (Note: ContainerInstanceId (or aws.ecs.container.instance.id) only works for AWS ECS EC2 launch type. "otel-stream"
log_retention LogRetention is the option to set the log retention policy for only newly created CloudWatch Log Groups. Defaults to Never Expire if not specified or set to 0. Possible values for retention in days are 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 2192, 2557, 2922, 3288, or 3653. "Never Expire"
namespace Customized CloudWatch metrics namespace "default"
endpoint Optionally override the default CloudWatch service endpoint.
no_verify_ssl Enable or disable TLS certificate verification. false
proxy_address Upload Structured Logs to AWS CloudWatch through a proxy.
region Send Structured Logs to AWS CloudWatch in a specific region. If this field is not present in config, environment variable "AWS_REGION" can then be used to set region. determined by metadata
role_arn IAM role to upload segments to a different account.
max_retries Maximum number of retries before abandoning an attempt to post data. 1
dimension_rollup_option DimensionRollupOption is the option for metrics dimension rollup. Three options are available: NoDimensionRollup, SingleDimensionRollupOnly and ZeroAndSingleDimensionRollup "ZeroAndSingleDimensionRollup" (Enable both zero dimension rollup and single dimension rollup)
resource_to_telemetry_conversion "resource_to_telemetry_conversion" is the option for converting resource attributes to telemetry attributes. It has only one config onption- enabled. For metrics, if enabled=true, all the resource attributes will be converted to metric labels by default. See Resource Attributes to Metric Labels section below for examples. enabled=false
output_destination "output_destination" is an option to specify the EMFExporter output. Currently, two options are available. "cloudwatch" or "stdout" cloudwatch
parse_json_encoded_attr_values List of attribute keys whose corresponding values are JSON-encoded strings and will be converted to JSON structures in emf logs. For example, the attribute string value "{\"x\":5,\"y\":6}" will be converted to a json object: {"x": 5, "y": 6} [ ]
metric_declarations List of rules for filtering exported metrics and their dimensions. [ ]
metric_descriptors List of rules for inserting or updating metric descriptors. [ ]
metric_declaration

A metric_declaration section characterizes a rule to be used to set dimensions for exported metrics, filtered by the incoming metrics' labels and metric names.

Name Description Default
dimensions List of dimension sets to be exported. Dimension sets that include dimensions that are not labels are ignored. Use empty dimension set [] for metrics without labels. [[ ]]
metric_name_selectors List of regex strings to filter metric names by.
label_matchers (Optional) list of label matching rules to filter metrics by their labels. This rule is applied to any metric that matches any of the label matchers. [ ]
label_matcher

A label_matcher section defines a matching rule against the labels of the incoming metric. Only metrics that match the rules will be used by the surrounding metric_declaration.

Name Description Default
label_names List of label names to filter by. Their corresponding values are concatenated using the separator and matched against the configured regular expression.
separator (Optional) separator placed between concatenated label values. ";"
regex Regex string to be matched against concatenated label values.
metric_descriptor

A metric descriptor section allows the schema of a metric to be overwritten before sending out to the CloudWatch backend service. Currently, we only support unit override.

Name Description Default
metric_name The name of the metric to be overwritten.
unit The overwritten value of unit. The MetricDatum contains a ful list of supported unit values.
overwrite true if the schema should be overwritten with the given specification, otherwise it will only be configured if empty. false

AWS Credential Configuration

This exporter follows default credential resolution for the aws-sdk-go.

Follow the guidelines for the credential configuration.

Configuration Examples

Resource Attributes to Metric Labels

resource_to_telemetry_conversion option can be enabled to convert all the resource attributes to metric labels. By default, this option is disabled. Users need to set enabled=true to opt-in. See the config example below.

exporters:
    awsemf:
        region: 'us-west-2'
        resource_to_telemetry_conversion:
            enabled: true
Metric Declaration

The following is an example of how to use metric_declaration to select what metrics should be exported.

exporters:
  awsemf:
    region: 'us-west-2'
    output_destination: stdout
    dimension_rollup_option: "NoDimensionRollup"
    metric_declarations:
      - dimensions: [[]]
        metric_name_selectors:
          # Metric without label
          - "^node_load15$"
      - dimensions: [[device, fstype], []]
        metric_name_selectors:
          - "^node_filesystem_readonly$"

Documentation

Overview

Package awsemfexporter implements an OpenTelemetry Collector exporter that sends EmbeddedMetricFormat to AWS CloudWatch Logs in the region the collector is running in using the PutLogEvents API.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewFactory

func NewFactory() exporter.Factory

NewFactory creates a factory for AWS EMF exporter.

Types

type Config

type Config struct {
	config.ExporterSettings `mapstructure:",squash"`
	// AWSSessionSettings contains the common configuration options
	// for creating AWS session to communicate with backend
	awsutil.AWSSessionSettings `mapstructure:",squash"`
	// LogGroupName is the name of CloudWatch log group which defines group of log streams
	// that share the same retention, monitoring, and access control settings.
	LogGroupName string `mapstructure:"log_group_name"`
	// LogStreamName is the name of CloudWatch log stream which is a sequence of log events
	// that share the same source.
	LogStreamName string `mapstructure:"log_stream_name"`
	// Namespace is a container for CloudWatch metrics.
	// Metrics in different namespaces are isolated from each other.
	Namespace string `mapstructure:"namespace"`
	// DimensionRollupOption is the option for metrics dimension rollup. Three options are available, default option is "ZeroAndSingleDimensionRollup".
	// "ZeroAndSingleDimensionRollup" - Enable both zero dimension rollup and single dimension rollup
	// "SingleDimensionRollupOnly" - Enable single dimension rollup
	// "NoDimensionRollup" - No dimension rollup (only keep original metrics which contain all dimensions)
	DimensionRollupOption string `mapstructure:"dimension_rollup_option"`

	// LogRetention is the option to set the log retention policy for the CloudWatch Log Group. Defaults to Never Expire if not specified or set to 0
	// Possible values are 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 2192, 2557, 2922, 3288, or 3653
	LogRetention int64 `mapstructure:"log_retention"`

	// ParseJSONEncodedAttributeValues is an array of attribute keys whose corresponding values are JSON-encoded as strings.
	// Those strings will be decoded to its original json structure.
	ParseJSONEncodedAttributeValues []string `mapstructure:"parse_json_encoded_attr_values"`

	// MetricDeclarations is the list of rules to be used to set dimensions for exported metrics.
	MetricDeclarations []*MetricDeclaration `mapstructure:"metric_declarations"`

	// MetricDescriptors is the list of override metric descriptors that are sent to the CloudWatch
	MetricDescriptors []MetricDescriptor `mapstructure:"metric_descriptors"`

	// OutputDestination is an option to specify the EMFExporter output. Default option is "cloudwatch"
	// "cloudwatch" - direct the exporter output to CloudWatch backend
	// "stdout" - direct the exporter output to stdout
	// TODO: we can support directing output to a file (in the future) while customer specifies a file path here.
	OutputDestination string `mapstructure:"output_destination"`

	// EKSFargateContainerInsightsEnabled is an option to reformat certin metric labels so that they take the form of a high level object
	// The end result will make the labels look like those coming out of ECS and be more easily injected into cloudwatch
	// Note that at the moment in order to use this feature the value "kubernetes" must also be added to the ParseJSONEncodedAttributeValues array in order to be used
	EKSFargateContainerInsightsEnabled bool `mapstructure:"eks_fargate_container_insights_enabled"`

	// ResourceToTelemetrySettings is the option for converting resource attrihutes to telemetry attributes.
	// "Enabled" - A boolean field to enable/disable this option. Default is `false`.
	// If enabled, all the resource attributes will be converted to metric labels by default.
	ResourceToTelemetrySettings resourcetotelemetry.Settings `mapstructure:"resource_to_telemetry_conversion"`
	// contains filtered or unexported fields
}

Config defines configuration for AWS EMF exporter.

func (*Config) Validate

func (config *Config) Validate() error

Validate filters out invalid metricDeclarations and metricDescriptors

type LabelMatcher

type LabelMatcher struct {
	// List of label names to filter by. Their corresponding values are concatenated using
	// the separator and matched against the specified regular expression.
	LabelNames []string `mapstructure:"label_names"`
	// (Optional) Separator placed between concatenated source label values. (Default: ';')
	Separator string `mapstructure:"separator"`
	// Regex string to be used to match against values of the concatenated labels.
	Regex string `mapstructure:"regex"`
	// contains filtered or unexported fields
}

LabelMatcher defines a label filtering rule against the labels of incoming metrics. Only metrics that match the rules will be used by the surrounding MetricDeclaration.

func (*LabelMatcher) Matches

func (lm *LabelMatcher) Matches(labels map[string]string) bool

Matches returns true if given set of labels matches the LabelMatcher's rules.

type MetricDeclaration

type MetricDeclaration struct {
	// Dimensions is a list of dimension sets (which are lists of dimension names) to be
	// included in exported metrics. If the metric does not contain any of the specified
	// dimensions, the metric would be dropped (will only show up in logs).
	Dimensions [][]string `mapstructure:"dimensions"`
	// MetricNameSelectors is a list of regex strings to be matched against metric names
	// to determine which metrics should be included with this metric declaration rule.
	MetricNameSelectors []string `mapstructure:"metric_name_selectors"`
	// (Optional) List of label matchers that define matching rules to filter against
	// the labels of incoming metrics.
	LabelMatchers []*LabelMatcher `mapstructure:"label_matchers"`
	// contains filtered or unexported fields
}

MetricDeclaration characterizes a rule to be used to set dimensions for certain incoming metrics, filtered by their metric names.

func (*MetricDeclaration) ExtractDimensions

func (m *MetricDeclaration) ExtractDimensions(labels map[string]string) (dimensions [][]string)

ExtractDimensions filters through the dimensions defined in the given metric declaration and returns dimensions that only contains labels from in the given label set.

func (*MetricDeclaration) MatchesLabels

func (m *MetricDeclaration) MatchesLabels(labels map[string]string) bool

MatchesLabels returns true if the given OTLP Metric's name matches any of the Metric Declaration's label matchers.

func (*MetricDeclaration) MatchesName

func (m *MetricDeclaration) MatchesName(metricName string) bool

MatchesName returns true if the given OTLP Metric's name matches any of the Metric Declaration's metric name selectors.

type MetricDescriptor

type MetricDescriptor struct {
	// MetricName is the name of the metric
	MetricName string `mapstructure:"metric_name"`
	// Unit defines the override value of metric descriptor `unit`
	Unit string `mapstructure:"unit"`
	// Overwrite set to true means the existing metric descriptor will be overwritten or a new metric descriptor will be created; false means
	// the descriptor will only be configured if empty.
	Overwrite bool `mapstructure:"overwrite"`
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL