README
¶
** Important Note: This repository has been donated to and is now part of the OpenTelemetry project, specifically under otel-arrow and otel-arrow-collector sub-projects.**
OTel Arrow Protocol implementation
This package is a reference implementation of the OTel Arrow protocol specified in this OTEP. All OTLP entities are covered (metrics, logs, and traces) as well as all sub-elements such as events, links, gauge, sum, summary, histograms, ... The overall goal is to optimize the compression ratio for telemetry data transmission as well as the end-to-end performance between telemetry data producers and receivers.
This package is still experimental and subject to change. It is currently used by an experimental OTLP/Arrow gRPC exporter and receiver.
Important links:
- OTEP - protocol specification.
- Donation - approved by the Technical Committee (repo not yet transferred in OTel org).
- Arrow Data Model - Mapping OTLP entities to Arrow Schemas.
- Benchmark results - Based on synthetic and production data.
- Validation process - Encoding/Decoding validation process.
- Articles describing some of the Arrow techniques used behind the scenes to optimize compression ratio and memory usage:
Benchmark summary
The following chart shows the compressed message size (in bytes) as a function of the batch size for metrics (univariate), logs, and traces. The bottom of the chart shows the reduction factor for both the standard OTLP protocol (with ZSTD compression) and the OTel Arrow protocol (ZSTD) in comparison with an uncompressed OTLP protocol.
The next chart follows the same logic but shows the results for multivariate metrics (see left column).
For more details, see the following benchmark results page.
Phase 1 (current implementation)
This first step is intended to address the specific use cases of traffic reduction. Based on community feedback, many companies want to reduce the cost of transferring telemetry data over the Internet. By adding a collector that acts as a point of integration and traffic conversion at the edge of a client environment, we can take advantage of the columnar format to eliminate redundant data and optimize the compression rate. This is illustrated in the following diagram.
Note 1: A fallback mechanism can be used to handle the case where the new protocol is not supported by the target. More on this mechanism in this section of the OTEP.
The experimental collector implements on top of this library a new Arrow Receiver and Exporter able to fallback on standard OTLP when needed. The following diagram is an overview of this integration. The internal representation of the data has not been updated and this collector is still fundamentally row-oriented internally.
Note 2: A future phase 2 of this project will focus on implementing end-to-end OTel Arrow to improve the overall performance.
Developers
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change. For more information, please read CONTRIBUTING.
How to change the protobuf specification
To (re)generate the ArrowStreamService gRPC service, you need to install the protoc compiler and the protoc-gen-grpc plugin.
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2
export PATH="$PATH:$(go env GOPATH)/bin"
cd ./proto
./generate.sh
Once the *.pb.go files are generated, you need to replace the content of the api/collector/arrow/v1 directory by the
generated files present in the ./proto/api/collector/arrow/v1 directory.
Integration with the OpenTelemetry Collector
The integration of this package with the OpenTelemetry Collector is done in the following experimental repository:
This above repository houses a fork of the entire core OpenTelemetry Collector, where the complete branch history is kept, including "mainline" Collector commits as well as Arrow-component development commits.
Because that repository contains portions that are not part of the OTel-Arrow project, the components are being maintained in this repository until they can be merged into the OpenTemetry-Collector-Contrib repository.
Collector components copied from that repository are currently
available in the
./collector
sub-package of this repository.
Examples demonstrating how to configure and test an OpenTelemetry
Collector with OTel-Arrow exporter and receiver components are located
in ./collector/examples, including:
examples/bridge: A compression bridge between "edge" and "saas" collectors.examples/metadata-bridge: A compression bridge between "edge" and "saas" collectors with metadata support, allowing request headers to transit via OTel-Arrow.examples/loopback: A collector that writes Arrow to and from itself.examples/recorder: A collector with support for recording data files for diagnostic and benchmark purposes.examples/synthesize: A collector with support for synthesizing telemetry data using a telemetry-generator component.
License
OTel Arrow Protocol Adapter is licensed under Apache 2.0.
Directories
¶
| Path | Synopsis |
|---|---|
|
api
|
|
|
experimental/arrow/v1/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
collector
module
|
|
|
cmd/otelarrowcol
Program otelarrowcol is an OpenTelemetry Collector binary.
|
Program otelarrowcol is an OpenTelemetry Collector binary. |
|
gen/exporter/fileexporter
Package fileexporter exports data to files.
|
Package fileexporter exports data to files. |
|
gen/exporter/otlpexporter
Package otlpexporter exports data by using the OTLP format to a gPRC endpoint.
|
Package otlpexporter exports data by using the OTLP format to a gPRC endpoint. |
|
gen/exporter/otlpexporter/internal/arrow/grpcmock
Package grpcmock is a generated GoMock package.
|
Package grpcmock is a generated GoMock package. |
|
gen/internal/sharedcomponent
Package sharedcomponent exposes util functionality for receivers and exporters that need to share state between different signal types instances such as net.Listener or os.File.
|
Package sharedcomponent exposes util functionality for receivers and exporters that need to share state between different signal types instances such as net.Listener or os.File. |
|
gen/receiver/otlpreceiver
Package otlpreceiver receives data in OTLP format.
|
Package otlpreceiver receives data in OTLP format. |
|
gen/receiver/otlpreceiver/internal/arrow/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
pkg
|
|
|
arrow
Package arrow provides a set of utility functions to access Arrow data structures.
|
Package arrow provides a set of utility functions to access Arrow data structures. |
|
benchmark
Package benchmark is a framework for benchmarking the performance of the OTLP protocol vs the OTLP Arrow protocol.
|
Package benchmark is a framework for benchmarking the performance of the OTLP protocol vs the OTLP Arrow protocol. |
|
benchmark/dataset
Package dataset defines the concept of dataset used in this benchmarking framework.
|
Package dataset defines the concept of dataset used in this benchmarking framework. |
|
benchmark/profileable
Package profileable defines the different protocols that can be profiled.
|
Package profileable defines the different protocols that can be profiled. |
|
benchmark/profileable/arrow
Package arrow implements the Profile interface for the OTLP Arrow protocol.
|
Package arrow implements the Profile interface for the OTLP Arrow protocol. |
|
benchmark/profileable/otlp
Package otlp implements the Profile interface for the OTLP protocol.
|
Package otlp implements the Profile interface for the OTLP protocol. |
|
datagen
Package datagen is a basic framework for generating fake telemetry data for benchmarking.
|
Package datagen is a basic framework for generating fake telemetry data for benchmarking. |
|
otel
Package otel provides a set of functions to convert OTLP entities to OTLP Arrow entities and vice versa.
|
Package otel provides a set of functions to convert OTLP entities to OTLP Arrow entities and vice versa. |
|
otel/arrow_record
Package arrow_record contains the consumer and producer for OTLP Arrow protocol.
|
Package arrow_record contains the consumer and producer for OTLP Arrow protocol. |
|
otel/arrow_record/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
otel/assert
Package assert provides a set of helper functions to assert conditions in OTLP Arrow tests.
|
Package assert provides a set of helper functions to assert conditions in OTLP Arrow tests. |
|
otel/common
Package common defines the common types and functions used by the packages logs, metrics and traces.
|
Package common defines the common types and functions used by the packages logs, metrics and traces. |
|
otel/common/arrow
Package arrow contains common types and functions used to convert OTLP entities into their Arrow representation.
|
Package arrow contains common types and functions used to convert OTLP entities into their Arrow representation. |
|
otel/common/otlp
Package otlp contains common types and functions used to convert OTLP Arrow entities into their OTLP representation.
|
Package otlp contains common types and functions used to convert OTLP Arrow entities into their OTLP representation. |
|
otel/constants
Package constants defines the constants used in the sibling packages.
|
Package constants defines the constants used in the sibling packages. |
|
otel/logs
Package logs provides functions to convert OTLP logs to OTLP Arrow logs and vice versa.
|
Package logs provides functions to convert OTLP logs to OTLP Arrow logs and vice versa. |
|
otel/logs/arrow
Package arrow contains types and functions used to convert OTLP logs into their Arrow representation.
|
Package arrow contains types and functions used to convert OTLP logs into their Arrow representation. |
|
otel/logs/otlp
Package otlp contains types and functions used to convert OTLP Arrow logs into their OTLP representation.
|
Package otlp contains types and functions used to convert OTLP Arrow logs into their OTLP representation. |
|
otel/metrics
Package metrics provides functions to convert OTLP metrics to OTLP Arrow metrics and vice versa.
|
Package metrics provides functions to convert OTLP metrics to OTLP Arrow metrics and vice versa. |
|
otel/metrics/arrow
Package arrow contains types and functions used to convert OTLP metrics into their Arrow representation.
|
Package arrow contains types and functions used to convert OTLP metrics into their Arrow representation. |
|
otel/metrics/otlp
Package otlp contains types and functions used to convert OTLP Arrow metrics into their OTLP representation.
|
Package otlp contains types and functions used to convert OTLP Arrow metrics into their OTLP representation. |
|
otel/traces
Package traces provides functions to convert OTLP traces to OTLP Arrow traces and vice versa.
|
Package traces provides functions to convert OTLP traces to OTLP Arrow traces and vice versa. |
|
otel/traces/arrow
Package arrow contains types and functions used to convert OTLP traces into their Arrow representation.
|
Package arrow contains types and functions used to convert OTLP traces into their Arrow representation. |
|
otel/traces/otlp
Package otlp contains types and functions used to convert OTLP Arrow traces into their OTLP representation.
|
Package otlp contains types and functions used to convert OTLP Arrow traces into their OTLP representation. |
|
Package tools contains tools used to benchmark OTLP, OTLP Arrow protocols, and to generate fake data.
|
Package tools contains tools used to benchmark OTLP, OTLP Arrow protocols, and to generate fake data. |
|
logs_benchmark
Package main contains a CLI tool for benchmarking the logs between OTLP and OTLP Arrow protocols.
|
Package main contains a CLI tool for benchmarking the logs between OTLP and OTLP Arrow protocols. |
|
logs_gen
Package main contains a CLI tool for generating fake OTLP logs.
|
Package main contains a CLI tool for generating fake OTLP logs. |
|
metrics_benchmark
Package main contains a CLI tool for benchmarking the metrics between OTLP and OTLP Arrow protocols.
|
Package main contains a CLI tool for benchmarking the metrics between OTLP and OTLP Arrow protocols. |
|
metrics_gen
Package main contains a CLI tool for generating fake OTLP metrics.
|
Package main contains a CLI tool for generating fake OTLP metrics. |
|
trace_benchmark
Package main contains a CLI tool for benchmarking the traces between OTLP and OTLP Arrow protocols.
|
Package main contains a CLI tool for benchmarking the traces between OTLP and OTLP Arrow protocols. |
|
trace_gen
Package main contains a CLI tool for generating fake OTLP traces.
|
Package main contains a CLI tool for generating fake OTLP traces. |
|
trace_head
Package main contains a CLI tool used to extract the first n spans from a trace file (protobuf file).
|
Package main contains a CLI tool used to extract the first n spans from a trace file (protobuf file). |
|
trace_verify
* Copyright The OpenTelemetry Authors * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
|
* Copyright The OpenTelemetry Authors * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. |