tel

= Telemetry.V2 Otel Protocol

Framework which aims to ease logging affair: `Logs`, `Traces` and `Metrics` .

V2 version launch usage of OpenTelemetry specification for all logging directions.
This mean that all logging propagators uses `OTEL` protocol.

Tel use `zap.Logger` as the heart of system.
That why it's pass all zap functions through.

== Motto

Ony context all logs.

Decrease external dependencies as match as possible.

== Features

.All-In-One
Library establish connection via GRPC OTLP protocol with `opentelemetry-collector-contrib` (official OTEL server) and send `logs`, `traces` and `metrics`.
Collector by the way distribute them to `Loki`, `Tempo` and `Prometheus` or any other services which you prefer and which collector support.
Furthermore, we prepared for you working `dashboards` in `./__grafana` folder which created for our `middlewares` for most popular servers and clients.

.Logs
Our goal to support https://grafana.com/docs/loki/latest/logql/log_queries/#logfmt[logfmt] format for `Loki` viewing.
Via simple `zap` library interface.

By the way, you can enrich log attributes which should be written only when whey would really need

[source,go]
----
// create copy of ctx which we enrich with log attributes
cxt := tel.Global().Copy().Ctx()

// pass ctx in controller->store->root layers and enrich information
err := func (ctx context.Context) error{
	tel.FromCtx(ctx).PutAttr(extract...)
	return return fmt.Errorf("some error")
}(ctx)

// and you write log message only when it really needed
// with all putted attribute via ctx from ALL layers earlier
// No need to look previous info/debug messages
// All needed information in one message with all attributes which you already added, but which would be writen only when you really do call `Error()`, `Info()`, `Debug()` and so on
//
// for example: only when you got error
if err != nil{
	tel.FromCtx(ctx).Error("error happened", tel.Error(err))
}

----

.Trace
Library simplify usage with creating `Spans` of trace

Also, you can send not only logs and also encroach `trace events`

[source,go]
----
	span, ctx := tel.StartSpanFromContext(req.Context(), "my trace")
	defer span.End()

	tel.FromCtx(ctx).Info("this message will be saved both in log and trace",
		// and this will come to the trace as attribute
		tel.Int("code", errCode))
----

.Metrics
Simplify working with metrics
[source,go]

----
	m := tel.Global().Meter("github.com/MyRepo/MyLib/myInstrumenation")

	requestLatency, err := m.SyncFloat64().Histogram("demo_client.request_latency",
		instrument.WithDescription("The latency of requests processed"))
	if err != nil {
		t.Fatal("metric load error", tel.Error(err))
	}

    ...
    start := time.Now()
    ....

    ms := float64(time.Now().Sub(start).Microseconds())

    requestLatency.Record(ctx, ms,
        attribute.String("userID", "e64916d9-bfd0-4f79-8ee3-847f2d034d20"),
        attribute.Int("orderID", 1),
    )
----

.Middleware
* Recovery flow
* Instantiate new copy of `tel` for further handler
* Basic metrics with respective dashboard for grafana
* Trace propagation
** client part - send (inject) current trace span to the server
** server part - read (extract) trace and create new trace child one (or absolutly new if no trace info was provided or this info where not properly wrapped via propagator protocol of OTEL specification)

== Logging stack

Logging data exported via `OTEL's` GRPC protocol. `tel` developed to trespass it via https://github.com/open-telemetry/opentelemetry-collector[open-telemetry collector] which should route log data up to any desired log receivers.

Keep in mind that collector has plugin version https://github.com/open-telemetry/opentelemetry-collector-contrib[collector contrib] - this is gateway-adapter to numerous protocols which not yet support `OTEL`, for example grafana loki.

For instance, you can use `opentelemetry-collector-contrib` as `tel` receiver and route logging data to `Grafana Loki`, trace data to `Grafana Tempo` and metric data to `Prometheus + Grafana ;)`

=== Grafana references feature

==== loki to tempo

`tel` approach to put `traceID` field with actual trace ID.
All our middlewares should do that or developer should do it by himself

Just call `UpdateTraceFields` before write some logs
[source,go]

----
tel.UpdateTraceFields(ctx)
----

understood grafana should setup `derivedFields` for Loki data source
[source,yaml]

----
  - name: Loki
    type: loki
    url: http://loki:3100
    uid: loki
    jsonData:
      derivedFields:
        - datasourceUid: tempo
          matcherRegex: "traceID=(\\w+)"
          name: trace
          url: '$${__value.raw}'
----

==== tempo to loki

We match `tempo` with `loki` by `service_name` label.
All logs should contain traceID by any key form and `service_name`.
In grafana tempo datasource should be configured with `tracesToLogs`
==== prometheus to loki
[source,yaml]

----
  - name: Tempo
    type: tempo
    access: proxy
    orgId: 1
    url: http://tempo:3200
    basicAuth: false
    isDefault: false
    version: 1
    editable: false
    apiVersion: 1
    uid: tempo
    jsonData:
      nodeGraph:
        enabled: true
      tracesToLogs:
        datasourceUid: loki
        filterBySpanID: false
        filterByTraceID: true
        mapTagNamesEnabled: false
        tags:
          - service_name
----

== Install

[source,bash]
----
go get github.com/d7561985/tel/v2@latest
----

=== collector

OTEL collector configuration (labels) part of setup, this mean if you not properly setup it - you wouldn't be able to see appropriate result

[source,yaml]
----
include::example/demo/otel-collector-config.yaml[]
----

== Features

* `OTEL` logs implementation

== Env

.OTEL_SERVICE_NAME
service name

`type`: string

.NAMESPACE
project namespace

`type`: string

.DEPLOY_ENVIRONMENT
ENUM: dev, stage, prod

`type`: string

.LOG_LEVEL
info log

.LOG_ENCODE
valid options: `console` and `json` or "none"

none - disable print to console (only OTEL or critical errors)

`type`: string
NOTE:  debug, info, warn, error, dpanic, panic, fatal

.DEBUG
for IsDebug() function

`type`: bool

.SENTRY_DSN
sentry dns

`type`: string

.MONITOR_ENABLE
default: `true`

.MONITOR_ADDR
address where `health`, `prometheus` would be listen

NOTE: address logic represented in net.Listen description

.OTEL_ENABLE
default: `true`

.OTEL_COLLECTOR_GRPC_ADDR
Address to otel collector server via GRPC protocol

.OTEL_EXPORTER_WITH_INSECURE
With insecure ...

.OTEL_RESOURCE_ATTRIBUTES
This optional variable, handled by open-telemetry SDK.
Separator is semicolon.
Put additional resources variables, very suitable!

== Modules, Plugins and ect

Some plugins require external packages, we don'ty like unnecessary increasing dependencies.
Thus offer sub-modules which should be added separately

=== Middlewares

==== http

Already present in tel package as it uses native golang code

[source,go]
----
package main

import mw "github.com/d7561985/tel/v2/middleware/http"

...
func (s *Server) createServeMux() http.Handler {
	mux := mw.NewServeMux(mw.WithTel(s.tel))
	mux.Handle("/customer", http.HandlerFunc(s.customer))
	return mux
}
...
----

include::middleware/gin/README.adoc[]

==== grpc

[source,bash]
----
go get -v github.com/d7561985/tel/middleware/grpc/v2@latest
----

server:

[source,go]
----
import(
    mw "github.com/d7561985/tel/middleware/grpc/v2"
)
func main(){
	server := grpc.NewServer(
		grpc.UnaryInterceptor(mw.UnaryServerInterceptorAll(mw.WithTel(&tele))),
		grpc.StreamInterceptor(mw.StreamServerInterceptor()),
	)
}

----

client:

[source,go]
----
import(
    mw "github.com/d7561985/tel/middleware/grpc/v2"
)
func main(){
    conn, err := grpc.Dial(hostPort, grpc.WithTransportCredentials(insecure.NewCredentials()),
            grpc.WithUnaryInterceptor(mw.UnaryClientInterceptorAll(mw.WithTel(&tele))),
            grpc.WithStreamInterceptor(mw.StreamClientInterceptor()),
        )
}

----

==== NATS

[source,bash]
----
go get -v github.com/d7561985/tel/middleware/natsmw/v2@latest
----

==== chi

[source,bash]
----
go get -v github.com/d7561985/tel/middleware/chi/v2@latest
----

==== echo

[source,bash]
----
go get -v github.com/d7561985/tel/middleware/echo/v2@latest
----

==== Propagators

https://opentelemetry.io/docs/reference/specification/context/api-propagators/[specification]

In few words: this is a way how trace propagate between services

.github.com/d7561985/tel/v2/propagators/natsprop
Just helper which uses any TextMapPropagator (by default globally declared or via WithPropagators option).
Suitable propagate traces (`propagation.TraceContext`) or baggage(`propagation.Baggage`).

=== Plugins

==== Logging

github.com/d7561985/tel/plugins/pgx/v2

[source,bash]
----
go get -v github.com/d7561985/tel/plugins/pgx/v2@latest
----

== ToDo

* [ ] Expose health check to specific metric
* [ ] Duplicate trace messages for root - ztrace.New just add to chain tree

== Usage

Tale look in `example/demo` folder.