labkit: Index | Examples | Files | Directories

package tracing

import ""

Package tracing is the primary entrypoint into LabKit's distributed tracing functionality.

(This documentation assumes some minimal knowledge of Distributed Tracing, and uses tracing terminology without providing definitions. Please review for an broad overview of distributed tracing if you are not familiar with the technology)

Internally the `tracing` package relies on Opentracing, but avoids leaking this abstraction. In theory, LabKit could replace Opentracing with another distributed tracing interface, such as Zipkin or OpenCensus, without needing to make changes to the application (other than updating the LabKit dependency, of course).

This design decision is deliberate: the package should not leak the underlying tracing implementation.

The package provides three primary exports:

* `tracing.Initialize()` for initializing the global tracer using the `GITLAB_TRACING` environment variable. * An HTTP Handler middleware, `tracing.Handler()`, for instrumenting incoming HTTP requests. * An HTTP RoundTripper, `tracing.NewRoundTripper()` for instrumenting outbound HTTP requests to other services.

The provided example in `example_test.go` demonstrates usage of both the HTTP Middleware and the HTTP RoundTripper.

*Initializing the global tracer*

Opentracing makes use of a global tracer. Opentracing ships with a default NoOp tracer which does nothing at all. This is always configured, meaning that, without initialization, Opentracing does nothing and has a very low overhead.

LabKit's tracing is configured through an environment variable, `GITLAB_TRACING`. This environment variable contains a "connection string"-like configuration, such as:

* `opentracing://jaeger?udp_endpoint=localhost:6831` * `opentracing://datadog` * `opentracing://lightstep`

The parameters for these connection-strings are implementation specific.

This configuration is identical to the one used to configure GitLab's ruby tracing libraries in the `Gitlab::Tracing` package. Having a consistent configuration makes it easy to configure multiple processes at the same time. For example, in GitLab Development Kit, tracing can be configured with a single environment variable, `GITLAB_TRACING=... gdk run`, since `GITLAB_TRACING` will configure Workhorse (written in Go), Gitaly (written in Go) and GitLab's rails components, using the same configuration.

*Compiling applications with Tracing support*

Go's Opentracing interface does not allow tracing implementations to be loaded dynamically; implementations need to be compiled into the application. With LabKit, this is done conditionally, using build tags. Two build tags need to be specified:

* `tracer_static` - this compiles in the static plugin registry support * `tracer_static_[DRIVER_NAME]` - this compile in support for the given driver.

For example, to compile support for Jaeger, compile your Go app with `tracer_static,tracer_static_jaeger`

Note that multiple (or all) drivers can be compiled in alongside one another: using the tags: `tracer_static,tracer_static_jaeger,tracer_static_lightstep,tracer_static_datadog`

If the `GITLAB_TRACING` environment variable references an unknown or unregistered driver, it will log a message and continue without tracing. This is a deliberate decision: the risk of bringing down a cluster during a rollout with a misconfigured tracer configuration is greater than the risk of an operator loosing some time because their application was not compiled with the correct tracers.

*Using the HTTP Handler middleware to instrument incoming HTTP requests*

When an incoming HTTP request arrives on the server, it may already include Distributed Tracing headers, propagated from an upstream service.

The tracing middleware will attempt to extract the tracing information from the headers (the exact headers used are tracing implementation specific), set up a span and pass the information through the request context.

It is up to the Opentracing implementation to decide whether the span will be sent to the tracing infrastructure. This will be implementation-specific, but generally relies on server load, sampler configuration, whether an error occurred, whether certain spans took an anomalous amount of time, etc.

*Using the HTTP RoundTripper to instrument outgoing HTTP requests*

The RoundTripper should be added to the HTTP client RoundTripper stack (see the example). When an outbound HTTP request is sent from the HTTP client, the RoundTripper will determine whether there is an active span and if so, will inject headers into the outgoing HTTP request identifying the span. The details of these headers is implementation specific.

It is important to ensure that the context is passed into the outgoing request, using `req.WithContext(ctx)` so that the correct span information can be injected into the request headers.

*Propagating tracing information to child processes*

Sometimes we want a trace to continue from a parent process to a spawned child process. For this, the tracing package provides `tracing.NewEnvInjector()` and `tracing.ExtractFromEnv()`, for the parent and child processes respectively.

NewEnvInjector() will configure a []string array of environment variables, ensuring they have the correct tracing configuration and any trace and span identifiers. NewEnvInjector() should be called in the child process and will extract the trace and span information from the environment.

Please review the examples in the godocs for details of how to implement both approaches.

This example shows how to initialize tracing and then wrap all incoming calls.


// Tell the tracer to initialize as service "gitlab-wombat"

tr := &http.Transport{
    MaxIdleConns:       10,
    IdleConnTimeout:    30 * time.Second,
    DisableCompression: true,

client := &http.Client{
    Transport: tracing.NewRoundTripper(tr),

// Listen and propagate traces
    // Add the tracing middleware in
        http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            req, err := http.NewRequest(http.MethodGet, "http://localhost:8080/bar", nil)
            if err != nil {

            req = req.WithContext(r.Context())

            resp, err := client.Do(req)
            if err != nil {
            defer resp.Body.Close()

            _, err = io.Copy(w, resp.Body)
            if err != nil {
        // Use this route identifier with the tracing middleware

// Listen and propagate traces
    // Add the tracing middleware in
        http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            fmt.Fprintf(w, "bar")
        // Use this route identifier with the tracing middleware

log.Fatal(http.ListenAndServe(":0", nil))



Package Files

doc.go env_extractor.go env_extractor_option.go env_injector.go env_injector_option.go errors.go inbound_http.go inbound_http_options.go initialization.go initialization_options.go outbound_http.go outbound_http_options.go


var ErrConfiguration = fmt.Errorf("tracing is not properly configured")

ErrConfiguration is returned when the tracer is not properly configured.

func ExtractFromEnv Uses

func ExtractFromEnv(ctx context.Context, opts ...ExtractFromEnvOption) (context.Context, func())

ExtractFromEnv will extract a span from the environment after it has been passed in from the parent process. Returns a new context, and a defer'able function, which should be called on process termination.


// Tell the tracer to initialize as service "gitlab-child-process"

ctx, finished := tracing.ExtractFromEnv(context.Background())
defer finished()

// Program execution happens here...
func(_ context.Context) {}(ctx)

func Handler Uses

func Handler(h http.Handler, opts ...HandlerOption) http.Handler

Handler will extract tracing from inbound request.

func Initialize Uses

func Initialize(opts ...InitializationOption) io.Closer

Initialize will initialize distributed tracing.

func NewRoundTripper Uses

func NewRoundTripper(delegate http.RoundTripper, opts ...RoundTripperOption) http.RoundTripper

NewRoundTripper acts as a "client-middleware" for outbound http requests adding instrumentation to the outbound request and then delegating to the underlying transport.

type EnvInjector Uses

type EnvInjector func(ctx context.Context, env []string) []string

EnvInjector will inject tracing information into an environment in preparation for spawning a child process. This includes trace and span identifiers, as well as the GITLAB_TRACING configuration. Will gracefully degrade if tracing is not configured, or an active span is not currently available.

func NewEnvInjector Uses

func NewEnvInjector(opts ...EnvInjectorOption) EnvInjector

NewEnvInjector will create a new environment injector.


envInjector := tracing.NewEnvInjector()

cmd := exec.Command("ls")
env := []string{

// envInjector will inject any required values
cmd.Env = envInjector(context.Background(), env)

if err := cmd.Run(); err != nil {
    log.WithError(err).Fatal("Command failed")

type EnvInjectorOption Uses

type EnvInjectorOption func(*envInjectorConfig)

EnvInjectorOption will configure an environment injector.

type ExtractFromEnvOption Uses

type ExtractFromEnvOption func(*extractFromEnvConfig)

ExtractFromEnvOption will configure an environment injector.

type HandlerOption Uses

type HandlerOption func(*handlerConfig)

HandlerOption will configure a correlation handler.

func WithRouteIdentifier Uses

func WithRouteIdentifier(routeIdentifier string) HandlerOption

WithRouteIdentifier allows a RouteIdentifier attribute to be set in the handler. This value will appear in the traces.

type InitializationOption Uses

type InitializationOption func(*initializationConfig)

InitializationOption will configure a correlation handler.

func WithConnectionString Uses

func WithConnectionString(connectionString string) InitializationOption

WithConnectionString allows the opentracing connection string to be overridden. By default this will be retrieved from the GITLAB_TRACING environment variable.

func WithServiceName Uses

func WithServiceName(serviceName string) InitializationOption

WithServiceName allows the service name to be configured for the tracer this will appear in traces.

type OperationNamer Uses

type OperationNamer func(*http.Request) string

OperationNamer will return an operation name given an HTTP request.

type RoundTripperOption Uses

type RoundTripperOption func(*roundTripperConfig)

RoundTripperOption will configure a correlation handler.



Package tracing imports 18 packages (graph) and is imported by 20 packages. Updated 2021-01-12. Refresh now. Tools for package owners.