otel2influx

package module
v0.5.8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 22, 2023 License: MIT Imports: 16 Imported by: 2

README

OpenTelemetry to InfluxDB Line Protocol Converter

Go Reference

This package converts OpenTelemetry traces, metrics, and logs to InfluxDB line protocol. The schema is optimized for InfluxDB/IOx, a timeseries database engine that is currently under development.

Docker Image: WIP OpenTelemetry Collector Contrib

Docker Image: WIP Telegraf

Definitions

"InfluxDB"

InfluxDB "IOx"

InfluxDB "Line Protocol"

"OpenTelemetry"

OpenTelemetry "Signal"

OpenTelemetry Signal "Attribute" and Attribute naming guidelines.

OpenTelemetry Signal "Label": similar to Signal Attribute; values are type string only

OpenTelemetry Signal "Resource"

OpenTelemetry Signal "Instrumentation Library"

Signals have properties that are not free-form Attributes or Labels. These properties distinguish the OpenTelemetry Signal types. For example:

  • trace ID, span ID, parent span ID
  • metric name
  • log severity, log span ID

Conversion

Spans are assigned measurement spans. Metric points are assigned the measurement named in the Metric.name protocol buffer field. Logs are assigned measurement logs.

This exporter converts all Attributes to Line Protocol fields, without modification. In the case that application Attribute keys conflict with Resource or Instrumentation Library Attribute keys, the application loses.

The exporter converts other Signal properties to fields with key names borrowed from OTLP protocol buffer messages. For example:

  • Span.start_time_unix_nano (type fixed64) -> InfluxDB line protocol timestamp
  • Span.trace_id (type bytes) -> trace_id as hexadecimal string
  • Span.name (type string) -> name
  • Span.end_time_unix_nano (type fixed64) -> end_time_unix_nano as uint64
  • Metric.time_unix_nano (type fixed64) -> InfluxDB line protocol timestamp
  • LogRecord.time_unix_nano (type fixed64) -> InfluxDB line protocol timestamp
    • This is an optional field in the OpenTelemetry data model, but required in InfluxDB.
    • This exporter drops the LogRecord if this field value is not set.
  • LogRecord.severity_number (type enum) -> severity_number as int32
  • LogRecord.body (type opentelemetry.proto.common.v1.AnyValue) -> body as string

Some exceptions to the above exist. For example:

  • Span.events (type repeated message) -> measurement logs
  • Span.links (type repeated message) -> measurment span-links
  • Metric.description is ignored
  • Metric.unit is ignored
  • Metric values are assigned field keys gauge, counter, count, sum, inf
    • Metric conversion follows Prometheus conventions for compatibility
  • LogRecord.flags is ignored
    • This is an enum with no values defines yet

Example Line Protocol

TODO(jacobmarble): update this section

Tracing Spans
spans end_time_unix_nano="2021-02-19 20:50:25.6893952 +0000 UTC",instrumentation_library_name="tracegen",kind="SPAN_KIND_INTERNAL",name="okey-dokey",net.peer.ip="1.2.3.4",parent_span_id="d5270e78d85f570f",peer.service="tracegen-client",service.name="tracegen",span.kind="server",span_id="4c28227be6a010e1",status_code="STATUS_CODE_OK",trace_id="7d4854815225332c9834e6dbf85b9380" 1613767825689169000
spans end_time_unix_nano="2021-02-19 20:50:25.6893952 +0000 UTC",instrumentation_library_name="tracegen",kind="SPAN_KIND_INTERNAL",name="lets-go",net.peer.ip="1.2.3.4",peer.service="tracegen-server",service.name="tracegen",span.kind="client",span_id="d5270e78d85f570f",status_code="STATUS_CODE_OK",trace_id="7d4854815225332c9834e6dbf85b9380" 1613767825689135000
spans end_time_unix_nano="2021-02-19 20:50:25.6895667 +0000 UTC",instrumentation_library_name="tracegen",kind="SPAN_KIND_INTERNAL",name="okey-dokey",net.peer.ip="1.2.3.4",parent_span_id="b57e98af78c3399b",peer.service="tracegen-client",service.name="tracegen",span.kind="server",span_id="a0643a156d7f9f7f",status_code="STATUS_CODE_OK",trace_id="fd6b8bb5965e726c94978c644962cdc8" 1613767825689388000
spans end_time_unix_nano="2021-02-19 20:50:25.6895667 +0000 UTC",instrumentation_library_name="tracegen",kind="SPAN_KIND_INTERNAL",name="lets-go",net.peer.ip="1.2.3.4",peer.service="tracegen-server",service.name="tracegen",span.kind="client",span_id="b57e98af78c3399b",status_code="STATUS_CODE_OK",trace_id="fd6b8bb5965e726c94978c644962cdc8" 1613767825689303300
spans end_time_unix_nano="2021-02-19 20:50:25.6896741 +0000 UTC",instrumentation_library_name="tracegen",kind="SPAN_KIND_INTERNAL",name="okey-dokey",net.peer.ip="1.2.3.4",parent_span_id="6a8e6a0edcc1c966",peer.service="tracegen-client",service.name="tracegen",span.kind="server",span_id="d68f7f3b41eb8075",status_code="STATUS_CODE_OK",trace_id="651dadde186b7834c52b13a28fc27bea" 1613767825689480300
Metrics
avalanche_metric_mmmmm_0_71 cycle_id="0",gauge=29,host.name="generate-metrics-avalanche",label_key_kkkkk_0="label_val_vvvvv_0",label_key_kkkkk_1="label_val_vvvvv_1",label_key_kkkkk_2="label_val_vvvvv_2",label_key_kkkkk_3="label_val_vvvvv_3",label_key_kkkkk_4="label_val_vvvvv_4",label_key_kkkkk_5="label_val_vvvvv_5",label_key_kkkkk_6="label_val_vvvvv_6",label_key_kkkkk_7="label_val_vvvvv_7",label_key_kkkkk_8="label_val_vvvvv_8",label_key_kkkkk_9="label_val_vvvvv_9",port="9090",scheme="http",series_id="3",service.name="otel-collector" 1613772311130000000
avalanche_metric_mmmmm_0_71 cycle_id="0",gauge=16,host.name="generate-metrics-avalanche",label_key_kkkkk_0="label_val_vvvvv_0",label_key_kkkkk_1="label_val_vvvvv_1",label_key_kkkkk_2="label_val_vvvvv_2",label_key_kkkkk_3="label_val_vvvvv_3",label_key_kkkkk_4="label_val_vvvvv_4",label_key_kkkkk_5="label_val_vvvvv_5",label_key_kkkkk_6="label_val_vvvvv_6",label_key_kkkkk_7="label_val_vvvvv_7",label_key_kkkkk_8="label_val_vvvvv_8",label_key_kkkkk_9="label_val_vvvvv_9",port="9090",scheme="http",series_id="4",service.name="otel-collector" 1613772311130000000
avalanche_metric_mmmmm_0_71 cycle_id="0",gauge=22,host.name="generate-metrics-avalanche",label_key_kkkkk_0="label_val_vvvvv_0",label_key_kkkkk_1="label_val_vvvvv_1",label_key_kkkkk_2="label_val_vvvvv_2",label_key_kkkkk_3="label_val_vvvvv_3",label_key_kkkkk_4="label_val_vvvvv_4",label_key_kkkkk_5="label_val_vvvvv_5",label_key_kkkkk_6="label_val_vvvvv_6",label_key_kkkkk_7="label_val_vvvvv_7",label_key_kkkkk_8="label_val_vvvvv_8",label_key_kkkkk_9="label_val_vvvvv_9",port="9090",scheme="http",series_id="5",service.name="otel-collector" 1613772311130000000
avalanche_metric_mmmmm_0_71 cycle_id="0",gauge=90,host.name="generate-metrics-avalanche",label_key_kkkkk_0="label_val_vvvvv_0",label_key_kkkkk_1="label_val_vvvvv_1",label_key_kkkkk_2="label_val_vvvvv_2",label_key_kkkkk_3="label_val_vvvvv_3",label_key_kkkkk_4="label_val_vvvvv_4",label_key_kkkkk_5="label_val_vvvvv_5",label_key_kkkkk_6="label_val_vvvvv_6",label_key_kkkkk_7="label_val_vvvvv_7",label_key_kkkkk_8="label_val_vvvvv_8",label_key_kkkkk_9="label_val_vvvvv_9",port="9090",scheme="http",series_id="6",service.name="otel-collector" 1613772311130000000
avalanche_metric_mmmmm_0_71 cycle_id="0",gauge=51,host.name="generate-metrics-avalanche",label_key_kkkkk_0="label_val_vvvvv_0",label_key_kkkkk_1="label_val_vvvvv_1",label_key_kkkkk_2="label_val_vvvvv_2",label_key_kkkkk_3="label_val_vvvvv_3",label_key_kkkkk_4="label_val_vvvvv_4",label_key_kkkkk_5="label_val_vvvvv_5",label_key_kkkkk_6="label_val_vvvvv_6",label_key_kkkkk_7="label_val_vvvvv_7",label_key_kkkkk_8="label_val_vvvvv_8",label_key_kkkkk_9="label_val_vvvvv_9",port="9090",scheme="http",series_id="7",service.name="otel-collector" 1613772311130000000
Logs
logs fluent.tag="fluent.info",pid=18i,ppid=9i,worker=0i 1613769568895331700
logs fluent.tag="fluent.debug",instance=1720i,queue_size=0i,stage_size=0i 1613769568895697200
logs fluent.tag="fluent.info",worker=0i 1613769568896515100

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func InstrumentationScopeToTags added in v0.3.0 

func InstrumentationScopeToTags(instrumentationScope pcommon.InstrumentationScope, tags map[string]string) map[string]string

func ResourceToTags added in v0.2.5 

func ResourceToTags(resource pcommon.Resource, tags map[string]string) map[string]string

Types

type InfluxWriter 

type InfluxWriter interface {
	NewBatch() InfluxWriterBatch
}

type InfluxWriterBatch added in v0.3.0 

type InfluxWriterBatch interface {
	EnqueuePoint(ctx context.Context, measurement string, tags map[string]string, fields map[string]interface{}, ts time.Time, vType common.InfluxMetricValueType) error
	WriteBatch(ctx context.Context) error
}

type NoopInfluxWriter added in v0.5.0 

type NoopInfluxWriter struct{}

func (*NoopInfluxWriter) EnqueuePoint added in v0.5.0 

func (w *NoopInfluxWriter) EnqueuePoint(ctx context.Context, measurement string, tags map[string]string, fields map[string]interface{}, ts time.Time, vType common.InfluxMetricValueType) error

func (*NoopInfluxWriter) NewBatch added in v0.5.0 

func (w *NoopInfluxWriter) NewBatch() InfluxWriterBatch

func (*NoopInfluxWriter) WriteBatch added in v0.5.0 

func (w *NoopInfluxWriter) WriteBatch(ctx context.Context) error

type OtelLogsToLineProtocol 

type OtelLogsToLineProtocol struct {
	// contains filtered or unexported fields
}

func NewOtelLogsToLineProtocol 

func NewOtelLogsToLineProtocol(config *OtelLogsToLineProtocolConfig) (*OtelLogsToLineProtocol, error)

func (*OtelLogsToLineProtocol) WriteLogs 

func (c *OtelLogsToLineProtocol) WriteLogs(ctx context.Context, ld plog.Logs) error

type OtelLogsToLineProtocolConfig added in v0.5.0 

type OtelLogsToLineProtocolConfig struct {
	Logger common.Logger
	Writer InfluxWriter
	// LogRecordDimensions are log record attributes to be used as line protocol tags.
	// These are always included as tags, if available:
	// - trace ID
	// - span ID
	// The default values:
	// - service.name
	// Other common attributes can be found here:
	// - https://github.com/open-telemetry/opentelemetry-collector/tree/main/semconv
	// When using InfluxDB for both logs and traces, be certain that LogRecordDimensions
	// matches the tracing SpanDimensions value.
	LogRecordDimensions []string
}

func DefaultOtelLogsToLineProtocolConfig added in v0.5.0 

func DefaultOtelLogsToLineProtocolConfig() *OtelLogsToLineProtocolConfig

type OtelMetricsToLineProtocol 

type OtelMetricsToLineProtocol struct {
	// contains filtered or unexported fields
}

func NewOtelMetricsToLineProtocol 

func NewOtelMetricsToLineProtocol(config *OtelMetricsToLineProtocolConfig) (*OtelMetricsToLineProtocol, error)

func (*OtelMetricsToLineProtocol) WriteMetrics 

func (c *OtelMetricsToLineProtocol) WriteMetrics(ctx context.Context, md pmetric.Metrics) error

type OtelMetricsToLineProtocolConfig added in v0.5.0 

type OtelMetricsToLineProtocolConfig struct {
	Logger common.Logger
	Writer InfluxWriter
	Schema common.MetricsSchema
}

func DefaultOtelMetricsToLineProtocolConfig added in v0.5.0 

func DefaultOtelMetricsToLineProtocolConfig() *OtelMetricsToLineProtocolConfig

type OtelTracesToLineProtocol 

type OtelTracesToLineProtocol struct {
	// contains filtered or unexported fields
}

func NewOtelTracesToLineProtocol 

func NewOtelTracesToLineProtocol(config *OtelTracesToLineProtocolConfig) (*OtelTracesToLineProtocol, error)

func (*OtelTracesToLineProtocol) WriteTraces 

func (c *OtelTracesToLineProtocol) WriteTraces(ctx context.Context, td ptrace.Traces) error

type OtelTracesToLineProtocolConfig added in v0.5.0 

type OtelTracesToLineProtocolConfig struct {
	Logger common.Logger
	Writer InfluxWriter
	// SpanDimensions are span attributes to be used as line protocol tags.
	// These are always included as tags:
	// - trace ID
	// - span ID
	// The default values are strongly recommended for use with Jaeger:
	// - service.name
	// - span.name
	// Other common attributes can be found here:
	// - https://github.com/open-telemetry/opentelemetry-collector/tree/main/semconv
	SpanDimensions []string
}

func DefaultOtelTracesToLineProtocolConfig added in v0.5.0 

func DefaultOtelTracesToLineProtocolConfig() *OtelTracesToLineProtocolConfig

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.