trc

package module
v0.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 17, 2023 License: Apache-2.0 Imports: 18 Imported by: 0

README

trc Go Reference Latest Release Build Status

trc provides in-process request tracing, an efficient alternative to logging. The package is heavily inspired by https://golang.org/x/net/trace, much gratitude to those authors.

Most users should import package eztrc, which offers an API designed for most common use cases.

Here's a quick-and-dirty usage example for a typical HTTP server.

func main() {
	server := NewServer(...) // your HTTP server
	traced := eztrc.Middleware(categorize)(server)
	go func() { log.Fatal(http.ListenAndServe(":8080", traced)) }() // normal API

	traces := eztrc.Handler()
	go func() { log.Fatal(http.ListenAndServe(":8081", traces)) }() // traces UI

	select {}
}

func categorize(r *http.Request) string {
	return r.Method + " " + r.URL.Path // assuming a fixed and finite set of possible paths
}

func someFunction(ctx context.Context, ...) {
	eztrc.Tracef(ctx, "this is a log statement")
	// ...
}

Traces can be viewed, queried, etc. through a web UI.

See the examples directory for more complete example applications.

The current API is experimental and unstable. Breaking changes are guaranteed. Use at your own risk.

Documentation

Overview

Package trc provides in-process request tracing, an efficient alternative to logging. The package is inspired by https://golang.org/x/net/trace, much gratitude to those authors.

The basic idea is that applications should log not by sending events to a destination like stdout or a file on disk, but instead by adding events to a value retrieved from the context, known as a Trace.

Traces are created for each operation processed by your application, e.g. every incoming HTTP request. Each trace is given a semantically meaningful category, and injected into the downstream context so applications can add events over the course of the operation. The trace is marked as finished when the operation completes.

Collector collects traces into per-category ring buffers. Collected traces can be queried over HTTP via [trcweb.NewServer]. That interface is fairly rich, allowing traces to be selected by category, minimum duration, successful vs. errored, and so on.

There are a few caveats. This approach is only suitable for applications that do their work in the context of a trace-related operation, and which reliably have access to a context value. Only the most recent traces are maintained, so long term historical data is not available. And, becase traces are maintained in memory, if a process crashes or restarts, all previous data is by default lost.

Even with these caveats, in-process request tracing often provides a better user experience than traditional logging. The value of application telemetry tends to be highly correlated to age. A rich interface over just the most recent data can be surprisingly powerful.

Most applications should not import this package directly, and should instead use github.com/peterbourgon/trc/eztrc, which provides an API specifically designed for common use cases.

Index

Constants

View Source
const (
	// SearchLimitMin is the minimum search limit.
	SearchLimitMin = 1

	// SearchLimitDefault is the default search limit.
	SearchLimitDefault = 10

	// SearchLimitMax is the maximum search limit.
	SearchLimitMax = 250
)

Variables

View Source
var DefaultBucketing = []time.Duration{
	0,
	100 * time.Microsecond,
	1 * time.Millisecond,
	5 * time.Millisecond,
	10 * time.Millisecond,
	25 * time.Millisecond,
	50 * time.Millisecond,
	100 * time.Millisecond,
	1000 * time.Millisecond,
}

DefaultBucketing is the default set of time buckets used in search stats.

Functions

func SetTraceMaxEvents added in v0.0.3

func SetTraceMaxEvents(n int)

SetTraceMaxEvents sets the max number of events that will be stored in a core trace. Once a core trace has the maximum number of events, additional events increment a "truncated" counter, which is represented as a single final event. The default is 1000, the minimum is 10, and the maximum is 10000.

Changing this value does not affect traces that have already been created.

func SetTraceStacks added in v0.0.3

func SetTraceStacks(enable bool)

SetTraceStacks sets a boolean that determines whether trace events include stack traces. By default, trace event stacks are enabled, because they're generally very useful. However, computing stack traces can be the single most computationally heavy part of using package trc, so disabling them altogether can be a significant performance optimization.

Changing this value does not affect traces that have already been created.

Types

type Broker added in v0.0.3

type Broker struct {
	// contains filtered or unexported fields
}

Broker allows traces to be published to a set of subscribers.

func NewBroker added in v0.0.3

func NewBroker() *Broker

NewBroker returns a new, empty broker.

func (*Broker) Publish added in v0.0.3

func (b *Broker) Publish(ctx context.Context, tr Trace)

Publish the trace, transformed via NewStreamTrace, to any active and matching subscribers. Sends to subscribers don't block and will drop.

func (*Broker) Stream added in v0.0.3

func (b *Broker) Stream(ctx context.Context, f Filter, ch chan<- Trace) (StreamStats, error)

Stream will forward a copy of every trace created in the collector matching the filter to the provided channel. If the channel is full, traces will be dropped. For reasons of efficiency, streamed trace events don't have stacks. Stream blocks until the context is canceled.

Note that if the filter has IsActive true, the caller will receive not only complete matching traces as they are finished, but also a single-event trace for each individual matching event as they are created. This can be an enormous volume of data, please be careful.

func (*Broker) StreamStats added in v0.0.3

func (b *Broker) StreamStats(ctx context.Context, ch chan<- Trace) (StreamStats, error)

StreamStats returns statistics about a currently active subscription.

type CategoryStats added in v0.0.3

type CategoryStats struct {
	Category     string    `json:"category"`
	EventCount   int       `json:"event_count"`
	ActiveCount  int       `json:"active_count"`
	BucketCounts []int     `json:"bucket_counts"`
	ErroredCount int       `json:"errored_count"`
	Oldest       time.Time `json:"oldest"`
	Newest       time.Time `json:"newest"`
	// contains filtered or unexported fields
}

CategoryStats represents statistics for all traces in a specific category.

func NewCategoryStats added in v0.0.3

func NewCategoryStats(category string, bucketing []time.Duration) *CategoryStats

NewCategoryStats returns an empty category stats for the given category, and with the given bucketing.

func (*CategoryStats) EventRate added in v0.0.3

func (cs *CategoryStats) EventRate() (r float64)

EventRate is an approximate measure of events per second in this category.

func (*CategoryStats) IsZero added in v0.0.3

func (cs *CategoryStats) IsZero() bool

IsZero returns true if the stats are not properly initialized or empty.

func (*CategoryStats) Merge added in v0.0.3

func (cs *CategoryStats) Merge(other *CategoryStats)

Merge the other category stats into this one.

func (*CategoryStats) TotalCount added in v0.0.3

func (cs *CategoryStats) TotalCount() int

TotalCount returns the total number of traces in the category.

func (*CategoryStats) TraceRate added in v0.0.3

func (cs *CategoryStats) TraceRate() (r float64)

TraceRate is an approximate measure of traces per second in this category.

type Collector added in v0.0.3

type Collector struct {
	// contains filtered or unexported fields
}

Collector maintains a set of traces in memory, grouped by category.

func NewCollector added in v0.0.3

func NewCollector(cfg CollectorConfig) *Collector

NewCollector returns a new collector with the provided config.

func NewDefaultCollector added in v0.0.3

func NewDefaultCollector() *Collector

NewDefaultCollector returns a new collector with the source "default" and using New to produce new traces.

func (*Collector) NewTrace added in v0.0.3

func (c *Collector) NewTrace(ctx context.Context, category string) (context.Context, Trace)

NewTrace produces a new trace in the collector with the given category, injects it into the given context, and returns a new derived context containing the trace, as well as the new trace itself.

func (*Collector) Search added in v0.0.3

func (c *Collector) Search(ctx context.Context, req *SearchRequest) (*SearchResponse, error)

Search the collector for traces, according to the provided search request.

func (*Collector) SetCategorySize added in v0.0.3

func (c *Collector) SetCategorySize(cap int) *Collector

SetCategorySize resets the max size of each category in the collector. If any categories are currently larger than the given capacity, they will be reduced by dropping old traces. The default capacity is 1000.

The method returns its receiver to allow for builder-style construction.

func (*Collector) SetDecorators added in v0.0.3

func (c *Collector) SetDecorators(decorators ...DecoratorFunc) *Collector

SetDecorators completely resets the decorators used by the collector.

The method returns its receiver to allow for builder-style construction.

func (*Collector) SetNewTrace added in v0.0.3

func (c *Collector) SetNewTrace(newTrace NewTraceFunc) *Collector

SetNewTrace sets the new trace function used by the collector.

The method returns its receiver to allow for builder-style construction.

func (*Collector) SetSourceName added in v0.0.3

func (c *Collector) SetSourceName(name string) *Collector

SetSourceName sets the source used by the collector.

The method returns its receiver to allow for builder-style construction.

func (*Collector) Stream added in v0.0.3

func (c *Collector) Stream(ctx context.Context, f Filter, ch chan<- Trace) (StreamStats, error)

Stream traces matching the filter to the channel, returning when the context is canceled. See Broker.Stream for more details.

func (*Collector) StreamStats added in v0.0.3

func (c *Collector) StreamStats(ctx context.Context, ch chan<- Trace) (StreamStats, error)

StreamStats returns statistics about a currently active subscription.

type CollectorConfig added in v0.0.3

type CollectorConfig struct {
	// Source is used as the source for all traces created within the collector.
	// If not provided, the "default" source is used.
	Source string

	// NewTrace is used to construct the traces in the collector. If not
	// provided, the [New] function is used.
	NewTrace NewTraceFunc

	// Decorators are applied to every new trace created in the collector.
	Decorators []DecoratorFunc

	// Broker is used for streaming traces and events. If not provided, a new
	// broker will be constructed and used.
	Broker *Broker
}

CollectorConfig captures the configuration parameters for a collector.

type DecoratorFunc added in v0.0.3

type DecoratorFunc func(Trace) Trace

DecoratorFunc is a function that decorates a trace in some way. It's similar to an HTTP middleware. Decorators can be provided to a Collector and will be applied to every trace created in that collector.

func LogDecorator added in v0.0.3

func LogDecorator(dst io.Writer) DecoratorFunc

LogDecorator logs a simple string to the provided destination when the trace is created, on every event, and when the trace is finished. The logged string is a reduced form of the full trace, containing only the trace ID and the single event that triggered the log.

func LoggerDecorator added in v0.0.3

func LoggerDecorator(logger *log.Logger) DecoratorFunc

LoggerDecorator is like LogDecorator, but uses a log.Logger.

type Event

type Event struct {
	When    time.Time `json:"when"`
	What    string    `json:"what"`
	Stack   []Frame   `json:"stack,omitempty"`
	IsError bool      `json:"is_error,omitempty"`
}

Event is a traced event, similar to a log event, which is created in the context of a specific trace, via methods like Tracef.

type Filter added in v0.0.3

type Filter struct {
	Sources     []string       `json:"sources,omitempty"`
	IDs         []string       `json:"ids,omitempty"`
	Category    string         `json:"category,omitempty"`
	IsActive    bool           `json:"is_active,omitempty"`
	IsFinished  bool           `json:"is_finished,omitempty"`
	MinDuration *time.Duration `json:"min_duration,omitempty"`
	IsSuccess   bool           `json:"is_success,omitempty"`
	IsErrored   bool           `json:"is_errored,omitempty"`
	Query       string         `json:"query,omitempty"`
	// contains filtered or unexported fields
}

Filter is a set of rules that can be applied to an individual trace, which will either be allowed (pass) or rejected (fail).

func (*Filter) Allow added in v0.0.3

func (f *Filter) Allow(tr Trace) bool

Allow returns true if the provided trace satisfies all of the conditions in the filter.

func (*Filter) Normalize added in v0.0.3

func (f *Filter) Normalize() []error

Normalize must be called before the filter can be used.

func (Filter) String added in v0.0.3

func (f Filter) String() string

String returns an operator-readable representation of the filter.

type Frame

type Frame struct {
	Function string `json:"function"`
	FileLine string `json:"fileline"`
}

Frame is a single call frame in an event's call stack.

func (Frame) CompactFileLine

func (fr Frame) CompactFileLine() string

CompactFileLine returns a human-readable representation of the file and line, intended to be used in user-facing interfaces.

type MultiSearcher added in v0.0.3

type MultiSearcher []Searcher

MultiSearcher allows multiple searchers to be searched as one.

func (MultiSearcher) Search added in v0.0.3

Search scatters the request over the searchers, gathers responses, and merges them into a single response returned to the caller.

type NewTraceFunc added in v0.0.3

type NewTraceFunc func(ctx context.Context, source string, category string, decorators ...DecoratorFunc) (context.Context, Trace)

NewTraceFunc describes a function that produces a new trace with a specific source and category, and which is decorated by the given decorators. It returns a context containing the new trace, as well as the new trace itself.

type SearchRequest added in v0.0.3

type SearchRequest struct {
	Bucketing  []time.Duration `json:"bucketing,omitempty"`
	Filter     Filter          `json:"filter,omitempty"`
	Limit      int             `json:"limit,omitempty"`
	StackDepth int             `json:"stack_depth,omitempty"` // 0 is default stacks, -1 for no stacks
}

SearchRequest describes a complete search request.

func (*SearchRequest) Normalize added in v0.0.3

func (req *SearchRequest) Normalize() []error

Normalize ensures the search request is valid, modifying it if necessary. It returns any errors encountered in the process.

func (SearchRequest) String added in v0.0.3

func (req SearchRequest) String() string

String implements fmt.Stringer.

type SearchResponse added in v0.0.3

type SearchResponse struct {
	Request    *SearchRequest `json:"request,omitempty"`
	Sources    []string       `json:"sources"`
	TotalCount int            `json:"total_count"`
	MatchCount int            `json:"match_count"`
	Traces     []*StaticTrace `json:"traces"`
	Stats      *SearchStats   `json:"stats,omitempty"`
	Problems   []string       `json:"problems,omitempty"`
	Duration   time.Duration  `json:"duration"`
}

SearchResponse returned by a search request.

type SearchStats added in v0.0.3

type SearchStats struct {
	Bucketing  []time.Duration           `json:"bucketing"`
	Categories map[string]*CategoryStats `json:"categories"`
}

SearchStats are statistics over the complete set of traces that were queried as part of a search request.

func NewSearchStats added in v0.0.3

func NewSearchStats(bucketing []time.Duration) *SearchStats

NewSearchStats creates a new and empty search stats, with the given time buckets for grouping finished traces.

func (*SearchStats) AllCategories added in v0.0.3

func (ss *SearchStats) AllCategories() []*CategoryStats

AllCategories returns category stats for all known categories, as well as the synthetic Overall category.

func (*SearchStats) IsZero added in v0.0.3

func (ss *SearchStats) IsZero() bool

IsZero returns true if the stats are empty.

func (*SearchStats) Merge added in v0.0.3

func (ss *SearchStats) Merge(other *SearchStats)

Merge the other stats into this one.

func (*SearchStats) Observe added in v0.0.3

func (ss *SearchStats) Observe(trs ...Trace)

Observe the given traces into the search stats.

func (*SearchStats) Overall added in v0.0.3

func (ss *SearchStats) Overall() *CategoryStats

Overall returns a synthetic category stats representing all categories.

type Searcher added in v0.0.3

type Searcher interface {
	Search(context.Context, *SearchRequest) (*SearchResponse, error)
}

Searcher models anything that can serve search requests.

type StaticTrace added in v0.0.3

type StaticTrace struct {
	TraceSource      string        `json:"source"`
	TraceID          string        `json:"id"`
	TraceCategory    string        `json:"category"`
	TraceStarted     time.Time     `json:"started"`
	TraceDuration    time.Duration `json:"duration"`
	TraceDurationStr string        `json:"duration_str,omitempty"`
	TraceDurationSec float64       `json:"duration_sec,omitempty"`
	TraceFinished    bool          `json:"finished,omitempty"`
	TraceErrored     bool          `json:"errored,omitempty"`
	TraceEvents      []Event       `json:"events,omitempty"`
}

StaticTrace is a "snapshot" of a trace which can be sent over the wire.

func NewSearchTrace added in v0.0.3

func NewSearchTrace(tr Trace) *StaticTrace

NewSearchTrace produces a static trace intended for a search response.

func NewStreamTrace added in v0.0.3

func NewStreamTrace(tr Trace) *StaticTrace

NewStreamTrace produces a static trace meant for streaming. If the trace is active, only the most recent event is included. Also, stacks are removed from every event.

func (*StaticTrace) Category added in v0.0.3

func (st *StaticTrace) Category() string

Category implements the Trace interface.

func (*StaticTrace) Duration added in v0.0.3

func (st *StaticTrace) Duration() time.Duration

Duration implements the Trace interface.

func (*StaticTrace) Errored added in v0.0.3

func (st *StaticTrace) Errored() bool

Errored implements the Trace interface.

func (*StaticTrace) Errorf added in v0.0.3

func (st *StaticTrace) Errorf(format string, args ...any)

Errorf implements the Trace interface.

func (*StaticTrace) Events added in v0.0.3

func (st *StaticTrace) Events() []Event

Events implements the Trace interface.

func (*StaticTrace) Finish added in v0.0.3

func (st *StaticTrace) Finish()

Finish implements the Trace interface.

func (*StaticTrace) Finished added in v0.0.3

func (st *StaticTrace) Finished() bool

Finished implements the Trace interface.

func (*StaticTrace) ID added in v0.0.3

func (st *StaticTrace) ID() string

ID implements the Trace interface.

func (*StaticTrace) LazyErrorf added in v0.0.3

func (st *StaticTrace) LazyErrorf(format string, args ...any)

LazyErrorf implements the Trace interface.

func (*StaticTrace) LazyTracef added in v0.0.3

func (st *StaticTrace) LazyTracef(format string, args ...any)

LazyTracef implements the Trace interface.

func (*StaticTrace) Source added in v0.0.3

func (st *StaticTrace) Source() string

Source implements the Trace interface.

func (*StaticTrace) Started added in v0.0.3

func (st *StaticTrace) Started() time.Time

Started implements the Trace interface.

func (*StaticTrace) Tracef added in v0.0.3

func (st *StaticTrace) Tracef(format string, args ...any)

Tracef implements the Trace interface.

func (*StaticTrace) TrimStacks added in v0.0.3

func (st *StaticTrace) TrimStacks(depth int) *StaticTrace

TrimStacks reduces the stacks of every event in the trace based on depth. A depth of 0 means "no change" -- to remove stacks, use a depth of -1.

type StreamStats added in v0.0.3

type StreamStats struct {
	// Skips is how many traces were considered but didn't pass the filter.
	Skips int `json:"skips"`

	// Sends is how many traces were successfully sent to the subscriber.
	Sends int `json:"sends"`

	// Drops is how many traces were dropped due to lack of capacity.
	Drops int `json:"drops"`
}

StreamStats is metadata about a currently active subscription.

func (StreamStats) String added in v0.0.3

func (s StreamStats) String() string

String implements fmt.Stringer.

type Trace

type Trace interface {
	// ID returns an identifier for the trace which should be automatically
	// generated during construction, and should be unique within a given
	// instance.
	ID() string

	// Source returns a human-readable string representing the origin of the
	// trace, which is typically the instance of the program where the trace was
	// constructed.
	Source() string

	// Category returns the category of the trace, which should be provided by
	// the caller when the trace is created.
	Category() string

	// Started returns when the trace was created, preferably in UTC.
	Started() time.Time

	// Duration returns how long the trace is (or was) active, which is the time
	// between the started timestamp and when the trace was finished. If the
	// trace is still active, it returns the time since the started timestamp.
	Duration() time.Duration

	// Tracef adds a normal event to the trace, with the given format string and
	// args. Args are evaluated immediately.
	Tracef(format string, args ...any)

	// LazyTracef adds a normal event to the trace, with the given format string
	// and args. Args are stored in their raw form and evaulated lazily, when
	// the event is first read. Callers should be very careful to ensure that
	// args passed to this method will remain valid indefinitely.
	LazyTracef(format string, args ...any)

	// Errorf adds an error event to the trace, with the given format string and
	// args. It marks the trace as errored. Args are evaluated immediately.
	Errorf(format string, args ...any)

	// LazyErrorf adds an error event to the trace, with the given format string
	// and args. It marks the trace as errored. Args are stored in their raw
	// form and evaulated lazily, when the event is first read. Callers should
	// be very careful to ensure that args passed to this method will remain
	// valid indefinitely.
	LazyErrorf(format string, args ...any)

	// Finish marks the trace as finished. Once finished, a trace is "frozen",
	// and any method that would modify the trace becomes a no-op.
	Finish()

	// Finished returns true if Finish has been called.
	Finished() bool

	// Errored returns true if Errorf or LazyErrorf has been called.
	Errored() bool

	// Events returns all of the events collected by the trace, oldest to
	// newest. Events are produced by Tracef, LazyTracef, Errorf, and
	// LazyErrorf.
	Events() []Event
}

Trace is a collection of metadata and events for a single operation, typically a request, in a program. Traces are normally accessed through a context, and maintained in a Collector.

New produces a default implementation of a Trace which is suitable for most use cases. Consumers can extend that implementation via DecoratorFunc, or provide their own implementation entirely. Trace implementations must be safe for concurrent use.

Note that traces are typically created for every operation, but are accessed only upon explicit request, for example when an operator is diagnosing a problem. Consequently, traces are written far more often than they are read. Implementations should keep this access pattern in mind, and optimize for writes rather than reads.

Trace implementations may optionally implement SetMaxEvents(int), to allow callers to modify the maximum number of events that will be stored in the trace. This method, if it exists, is called by SetMaxEvents.

Trace implementations may optionally implement Free(), to release any resources claimed by the trace to an e.g. sync.Pool. This method, if it exists, is called by the Collector when a trace is dropped.

func Get

func Get(ctx context.Context) Trace

Get the trace from the context, if it exists. If not, an "orphan" trace is created and returned (but not injected into the context).

func MaybeGet

func MaybeGet(ctx context.Context) (Trace, bool)

MaybeGet returns the trace in the context, if it exists. If not, MaybeGet returns a nil trace and false.

func New

func New(ctx context.Context, source, category string, decorators ...DecoratorFunc) (context.Context, Trace)

New creates a new core trace with the given source and category, and injects it into the given context. It returns a new context containing that trace, and the trace itself.

func Prefix

func Prefix(ctx context.Context, format string, args ...any) (context.Context, Trace)

Prefix decorates the trace in the context such that every trace event will be prefixed with the string specified by format and args. Those args are not evaluated when Prefix is called, but are instead prefixed to the format and args of trace events made against the returned trace.

func Put

func Put(ctx context.Context, tr Trace) (context.Context, Trace)

Put the given trace into the context, and return a new context containing that trace, as well as the trace itself. If the context already contained a trace, it becomes "shadowed" by the new trace.

func Region

func Region(ctx context.Context, name string) (context.Context, Trace, func())

Region provides more detailed tracing of regions of code, usually functions, which is visible in the trace event "what" text. It decorates the trace in the context by annotating events with the provided name, and also creates a standard library runtime/trace.Region with the same name.

Typical usage is as follows.

func foo(ctx context.Context, id int) {
    ctx, tr, finish := trc.Region(ctx, "foo")
    defer finish()
    ...
}

This produces hierarchical trace events as follows.

→ foo
· trace event in foo
· another event in foo
· → bar
· · something in bar
· ← bar [1.23ms]
· final event in foo
← foo [2.34ms]

Region can significantly impact performance. Use it sparingly.

func SetMaxEvents

func SetMaxEvents(tr Trace, maxEvents int) (Trace, bool)

SetMaxEvents tries to set the max events for a specific trace, by checking if the trace implements the method SetMaxEvents(int), and, if so, calling that method with the given max events value. Returns the given trace, and a boolean representing whether or not the call was successful.

Directories

Path Synopsis
cmd
trc
trc is a CLI tool for interacting with trc web servers.
trc is a CLI tool for interacting with trc web servers.
Package eztrc provides an easy-to-use API for common tracing use cases.
Package eztrc provides an easy-to-use API for common tracing use cases.
internal
trcdebug
Package trcdebug contains debug information for the trc package.
Package trcdebug contains debug information for the trc package.
trcringbuf
Package trcringbuf provides a fixed-size ring buffer for traces.
Package trcringbuf provides a fixed-size ring buffer for traces.
trcutil
Package trcutil contains simple helper functions used throughout the module.
Package trcutil contains simple helper functions used throughout the module.
Package trcweb provides an HTTP interface to traces.
Package trcweb provides an HTTP interface to traces.
assets
Package assets contains assets for the trc web interface.
Package assets contains assets for the trc web interface.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL