prometheusreceiver

Prometheus Receiver

Status
Stability	beta: metrics
Distributions	core, contrib
Issues
Code Owners	@Aneurysm9, @dashpole

Receives metric data in Prometheus format. See the Design for additional information on this receiver.

⚠️ Warning

Note: This component is currently work in progress. It has several limitations and please don't use it if the following limitations is a concern:

• Collector cannot auto-scale the scraping yet when multiple replicas of the collector is run.
• When running multiple replicas of the collector with the same config, it will scrape the targets multiple times.
• Users need to configure each replica with different scraping configuration if they want to manually shard the scraping.
• The Prometheus receiver is a stateful component.

Unsupported features

The Prometheus receiver is meant to minimally be a drop-in replacement for Prometheus. However, there are advanced features of Prometheus that we don't support and thus explicitly will return an error for if the receiver's configuration YAML/code contains any of the following

• alert_config.alertmanagers
• alert_config.relabel_configs
• remote_read
• remote_write
• rule_files

Getting Started

This receiver is a drop-in replacement for getting Prometheus to scrape your services. It supports the full set of Prometheus configuration in scrape_config, including service discovery. Just like you would write in a YAML configuration file before starting Prometheus, such as with:

Note: Since the collector configuration supports env variable substitution $ characters in your prometheus configuration are interpreted as environment variables. If you want to use $ characters in your prometheus configuration, you must escape them using $$.

prometheus --config.file=prom.yaml

Feature gates:

• receiver.prometheusreceiver.UseCreatedMetric: Start time for Summary, Histogram and Sum metrics can be retrieved from _created metrics. Currently, this behaviour is disabled by default. To enable it, use the following feature gate option:

"--feature-gates=receiver.prometheusreceiver.UseCreatedMetric"

• receiver.prometheusreceiver.EnableNativeHistograms: process and turn native histogram metrics into OpenTelemetry exponential histograms. For more details consult the Prometheus native histograms section.

"--feature-gates=receiver.prometheusreceiver.EnableNativeHistograms"

• report_extra_scrape_metrics: Extra Prometheus scrape metrics can be reported by setting this parameter to true

You can copy and paste that same configuration under:

receivers:
  prometheus:
    config:

For example:

receivers:
    prometheus:
      config:
        scrape_configs:
          - job_name: 'otel-collector'
            scrape_interval: 5s
            static_configs:
              - targets: ['0.0.0.0:8888']
          - job_name: k8s
            kubernetes_sd_configs:
            - role: pod
            relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              regex: "true"
              action: keep
            metric_relabel_configs:
            - source_labels: [__name__]
              regex: "(request_duration_seconds.*|response_duration_seconds.*)"
              action: keep

The prometheus receiver also supports additional top-level options:

• trim_metric_suffixes: [Experimental] When set to true, this enables trimming unit and some counter type suffixes from metric names. For example, it would cause singing_duration_seconds_total to be trimmed to singing_duration. This can be useful when trying to restore the original metric names used in OpenTelemetry instrumentation. Defaults to false.
• use_start_time_metric: When set to true, this enables retrieving the start time of all counter metrics from the process_start_time_seconds metric. This is only correct if all counters on that endpoint started after the process start time, and the process is the only actor exporting the metric after the process started. It should not be used in "exporters" which export counters that may have started before the process itself. Use only if you know what you are doing, as this may result in incorrect rate calculations. Defaults to false.
• start_time_metric_regex: The regular expression for the start time metric, and is only applied when use_start_time_metric is enabled. Defaults to process_start_time_seconds.

For example,

receivers:
    prometheus:
      trim_metric_suffixes: true
      use_start_time_metric: true
      start_time_metric_regex: foo_bar_.*
      config:
        scrape_configs:
          - job_name: 'otel-collector'
            scrape_interval: 5s
            static_configs:
              - targets: ['0.0.0.0:8888']

Prometheus native histograms

Native histograms are an experimental feature of Prometheus.

To start scraping native histograms, set config.global.scrape_protocols to [ PrometheusProto, OpenMetricsText1.0.0, OpenMetricsText0.0.1, PrometheusText0.0.4 ] in the receiver configuration. This requirement will be lifted once Prometheus can scrape native histograms over text formats.

To enable converting native histograms to OpenTelemetry exponential histograms, enable the feature gate receiver.prometheusreceiver.EnableNativeHistograms. The feature is considered experimental.

This feature applies to the most common integer counter histograms, gauge histograms are dropped. In case a metric has both the conventional (aka classic) buckets and also native histogram buckets, only the native histogram buckets will be taken into account to create the corresponding exponential histogram. To scrape the classic buckets instead use the scrape option scrape_classic_histograms.

OpenTelemetry Operator

Additional to this static job definitions this receiver allows to query a list of jobs from the OpenTelemetryOperators TargetAllocator or a compatible endpoint.

receivers:
  prometheus:
    target_allocator:
      endpoint: http://my-targetallocator-service
      interval: 30s
      collector_id: collector-1

The target_allocator section embeds the full confighttp client configuration.

Exemplars

This receiver accepts exemplars coming in Prometheus format and converts it to OTLP format.

1. Value is expected to be received in float64 format
2. Timestamp is expected to be received in ms
3. Labels with key span_id in prometheus exemplars are set as OTLP span id and labels with key trace_id are set as trace id
4. Rest of the labels are copied as it is to OTLP format

Resource and Scope

This receiver drops the target_info prometheus metric, if present, and uses attributes on that metric to populate the OpenTelemetry Resource.

It drops otel_scope_name and otel_scope_version labels, if present, from metrics, and uses them to populate the OpenTelemetry Instrumentation Scope name and version. It drops the otel_scope_info metric, and uses attributes (other than otel_scope_name and otel_scope_version) to populate Scope Attributes.