otel-file-importer

OTEL File Importer

This tool was created to import files generated by the OTEL collector file exporter and send these events to Honeycomb.

It supports environments and services as well as honeycomb classic.

Usage

You can use the start flag to determine when to start populating the data in Honeycomb. It will manipulate the timestamps on the events to start the specified time ago from now. For example passing 3h will start populating the first event from 3 hours ago and continue on from there. Otherwise it will use the timestamps from the file.

Populating an environments and services setup:

cd $HOUND_ROOT
go run tools/otel-file-importer --path=output.json --key={API_KEY} --start=3h

Populating a honeycomb classic setup:

cd $HOUND_ROOT
go run tools/otel-file-importer --path=output.json --key={API_KEY} --start=3h --dataset={DATASET}

Producing the JSON file

This can be produced from an OpenTelemetry collector instance, using the file exporter as configured below.

exporters:
  file:
    path: /cache/output.json

To have this output the file when deployed to a local kubernetes installation with Docker for Mac you will need to add a volume to the container spec in the opentelemetry collector deployment as so:

volumes:
- hostPath:
    path: /Users/martin/output
    type: Directory
  name: cache

File format

The file should be in Protobuf JSON encoding using OpenTelemetry protocol.

Example

{
    "resourceSpans": [
        {
            "resource": {
                "attributes": [
                    {
                        "key": "ip",
                        "value": {
                            "stringValue": "10.1.0.30"
                        }
                    },
                    {
                        "key": "service.name",
                        "value": {
                            "stringValue": "checkout"
                        }
                    }
                ]
            },
            "scopeSpans": [
                {
                    "scope": {
                        "name": "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",
                        "version": "semver:0.29.0"
                    },
                    "spans": [
                        {
                            "traceId": "7c36457c3ffc495a72cedce096a85fc7",
                            "spanId": "07eaf42a1f4c1c42",
                            "parentSpanId": "6f1f8ff2a0f8f964",
                            "name": "msdemo.CheckoutService/GetCacheSize",
                            "kind": "SPAN_KIND_SERVER",
                            "startTimeUnixNano": "1655433591053554727",
                            "endTimeUnixNano": "1655433591054202575",
                            "attributes": [
                                {
                                    "key": "rpc.system",
                                    "value": {
                                        "stringValue": "grpc"
                                    }
                                },
                                {
                                    "key": "rpc.service",
                                    "value": {
                                        "stringValue": "msdemo.CheckoutService"
                                    }
                                },
                                {
                                    "key": "rpc.method",
                                    "value": {
                                        "stringValue": "GetCacheSize"
                                    }
                                },
                                {
                                    "key": "net.peer.ip",
                                    "value": {
                                        "stringValue": "10.1.0.32"
                                    }
                                },
                                {
                                    "key": "net.peer.port",
                                    "value": {
                                        "stringValue": "40702"
                                    }
                                },
                                {
                                    "key": "rpc.grpc.status_code",
                                    "value": {
                                        "intValue": "0"
                                    }
                                }
                            ],
                            "events": [
                                {
                                    "timeUnixNano": "1655433591053589997",
                                    "name": "message",
                                    "attributes": [
                                        {
                                            "key": "message.type",
                                            "value": {
                                                "stringValue": "RECEIVED"
                                            }
                                        },
                                        {
                                            "key": "message.id",
                                            "value": {
                                                "intValue": "1"
                                            }
                                        },
                                        {
                                            "key": "message.uncompressed_size",
                                            "value": {
                                                "intValue": "0"
                                            }
                                        }
                                    ]
                                },
                                {
                                    "timeUnixNano": "1655433591054198035",
                                    "name": "message",
                                    "attributes": [
                                        {
                                            "key": "message.type",
                                            "value": {
                                                "stringValue": "SENT"
                                            }
                                        },
                                        {
                                            "key": "message.id",
                                            "value": {
                                                "intValue": "1"
                                            }
                                        },
                                        {
                                            "key": "message.uncompressed_size",
                                            "value": {
                                                "intValue": "0"
                                            }
                                        }
                                    ]
                                }
                            ],
                            "status": {}
                        }
                    ]
                }
            ],
            "schemaUrl": "https://opentelemetry.io/schemas/v1.7.0"
        }
    ]
}