appdash: Index | Examples | Files | Directories

package appdash

import ""

Package appdash provides a Go app performance tracing suite.

Appdash allows you to trace the end-to-end performance of hierarchically structured applications. You can, for example, measure the time and see the detailed information of each HTTP request and SQL query made by an entire distributed web application.

Web Front-end

The cmd/appdash tool launches a web front-end which displays a web UI for viewing collected app traces. It is effectively a remote collector which your application can connect and send events to.

Timing and application-specific metadata information can be viewed in a nice timeline view for each span (e.g. HTTP request) and it's children.

The web front-end can also be embedded in your own Go HTTP server by utilizing the traceapp sub-package, which is effectively what cmd/appdash serves internally.

HTTP and SQL tracing

Sub-packages for HTTP and SQL event tracing are provided for use with appdash, which allows it to function equivalently to Google's Dapper and Twitter's Zipkin performance tracing suites.

Appdash Structure

The most high-level structure is a Trace, which represents the performance of an application from start to finish (in an HTTP application, for example, the loading of a web page).

A Trace is a tree structure that is made up of several spans, which are just IDs (in an HTTP application, these ID's are passed through the stack via a few special headers).

Each span ID has a set of Events that directly correspond to it inside a Collector. These events can be any combination of message, log, time-span, or time-stamped events (the cmd/appdash web UI displays these events as appropriate).

Inside your application, a Recorder is used to send events to a Collector, which can be a remote HTTP(S) collector, a local in-memory or persistent collector, etc. Additionally, you can implement the Collector interface yourself and store events however you like.



Package Files

collector.go doc.go event.go id.go memstats.go multi.go recorder.go reflect.go span.go store.go trace.go


const SchemaPrefix = "_schema:"
const (
    // SpanIDDelimiter is the delimiter used to concatenate an
    // SpanID's components.
    SpanIDDelimiter = "/"


var (
    // ErrBadSpanID is returned when the span ID cannot be parsed.
    ErrBadSpanID = errors.New("bad span ID")
var ErrQueueDropped = errors.New("ChunkedCollector queue entirely dropped (trace data will be missing)")

ErrQueueDropped is the error returns by ChunkedCollector.Flush and ChunkedCollector.Collect when the internal queue has grown too large and has been subsequently dropped.

var (
    // ErrTraceNotFound is returned by Store.GetTrace when no trace is
    // found with the given ID.
    ErrTraceNotFound = errors.New("trace not found")

func LogWithTimestamp Uses

func LogWithTimestamp(msg string, timestamp time.Time) logEvent

LogWithTimestamp returns an Event with an explicit timestamp that contains only a human readable message.

func PersistEvery Uses

func PersistEvery(s PersistentStore, interval time.Duration, file string) error

PersistEvery persists s's data to a file periodically.

func RegisterEvent Uses

func RegisterEvent(e Event)

RegisterEvent registers an event type for use with UnmarshalEvents.

Events must be registered with this package in order for unmarshaling to work. Much like the image package, sometimes blank imports will be used for packages that register Appdash events with this package:

    _ ""
    _ ""

func UnmarshalEvent Uses

func UnmarshalEvent(as Annotations, e Event) error

UnmarshalEvent unmarshals annotations into an event.

func UnmarshalEvents Uses

func UnmarshalEvents(anns Annotations, events *[]Event) error

UnmarshalEvents unmarshals all events found in anns into events. Any schemas found in anns that were not registered (using RegisterEvent) are ignored; missing a schema is not an error.

type AggregatedResult Uses

type AggregatedResult struct {
    // RootSpanName is the name of the root span of the traces that were
    // aggregated to form this result.
    RootSpanName string

    // Average, Minimum, Maximum, and standard deviation of the total trace
    // times (earliest span start time, latest span end time) of all traces
    // that were aggregated to produce this result, respectively.
    Average, Min, Max, StdDev time.Duration

    // Samples is the number of traces that were sampled in order to produce
    // this result.
    Samples int64

    // Slowest is the N-slowest trace IDs that were part of this group, such
    // that these are the most valuable/slowest traces for inspection.
    Slowest []ID

AggregatedResult represents a set of traces that were aggregated together by root span name to produce some useful metrics (average trace time, minimum time, a link to the slowest traces, etc).

type Aggregator Uses

type Aggregator interface {
    // Aggregate should return the aggregated data for all traces within the
    // past 72/hr, such that:
    //  Aggregate(-72 * time.Hour, 0)
    // would return all possible results.
    Aggregate(start, end time.Duration) ([]*AggregatedResult, error)

Aggregator is a type of store that can aggregate its trace data and return results about it.

type Annotation Uses

type Annotation struct {
    // Key is the annotation's key.
    Key string

    // Value is the annotation's value, which may be either human or
    // machine readable, depending on the schema of the event that
    // generated it.
    Value []byte

An Annotation is an arbitrary key-value property on a span.

func (Annotation) Important Uses

func (a Annotation) Important() bool

Important determines if this annotation's key is considered important to any of the registered event types.

type Annotations Uses

type Annotations []Annotation

Annotations is a list of annotations (on a span).

func MarshalEvent Uses

func MarshalEvent(e Event) (Annotations, error)

MarshalEvent marshals an event into annotations.

func (Annotations) String Uses

func (as Annotations) String() string

String returns a formatted list of annotations.

func (Annotations) StringMap Uses

func (as Annotations) StringMap() map[string]string

StringMap returns the annotations as a key-value map. Only one annotation for a key appears in the map, and it is chosen arbitrarily among the annotations with the same key.

type ChunkedCollector Uses

type ChunkedCollector struct {
    // Collector is the underlying collector that spans are sent to.

    // MinInterval specifies the minimum interval at which to call Flush
    // automatically (in a separate goroutine, as to not to block the caller who
    // may be recording time-sensitive operations).
    // Default MinInterval = 500 * time.Millisecond (500ms).
    MinInterval time.Duration

    // FlushTimeout, if non-zero, specifies the time after which a flush operation
    // is considered timed out. If timeout occurs, the pending queue is entirely
    // dropped (trace data lost) and ErrQueueDropped is returned by Flush.
    // Default FlushTimeout = 50 * time.Millisecond (50ms).
    FlushTimeout time.Duration

    // MaxQueueSize, if non-zero, is the maximum size in bytes that the pending
    // queue of collections may grow to before being entirely dropped (trace data
    // lost). In the event that the queue is dropped, Collect will return
    // ErrQueueDropped.
    // Default MaxQueueSize = 32 * 1024 * 1024 (32 MB).
    MaxQueueSize uint64

    // Log, if non-nil, is used to log warnings like when the queue is entirely
    // dropped (and hence trace data was lost).
    Log *log.Logger

    // OnFlush, if non-nil, will be directly invoked at the start of each Flush
    // operation that is performed by this collector. queueSize is the number of
    // entries in the queue (i.e. number of underlying collections that will
    // occur).
    // It is primarily used for debugging purposes.
    OnFlush func(queueSize int)
    // contains filtered or unexported fields

ChunkedCollector groups annotations together that have the same span and calls its underlying collector's Collect method with the chunked data periodically, instead of immediately. This is more efficient, especially in the case of the underlying Collector being an RemoteCollector, because whole spans are collected at once (rather than in parts). It also prevents the caller, usually a Recorder, from blocking time-sensitive operations on collection (which, in the case of RemoteCollector, may involve connecting to a remote server).

Inherently, ChunkedCollector queues all collections prior to them being flushed out to the underlying Collector. Because of this it's important to understand the various boundaries that are imposed to avoid any sort of queue backlogging or perceived memory leaks.

The flow of a ChunkedCollector is that:

- It receives a collection.
  - If the queue size exceeds MaxQueueSize in bytes, the pending queue is
    entirely dropped and ErrQueueDropped is returned.
  - Otherwise, if the queue would not exceed that size, the collection is
    added to the queue.
- After MinInterval (or if Flush is called manually), all queued collections
  are passed off to the underlying collector. If the overall Flush time
  measured after each underlying Collect call exceeds FlushTimeout, the
  pending queue is entirely dropped and ErrQueueDropped is returned.
- If the queue has been entirely dropped as a result of one of the above
  cases, entire traces and/or parts of their data will be missing. For this
  reason, you may specify a Log for debugging purposes.

func NewChunkedCollector Uses

func NewChunkedCollector(c Collector) *ChunkedCollector

NewChunkedCollector is shorthand for:

c := &ChunkedCollector{
	Collector:    c,
	MinInterval:  500 * time.Millisecond,
	FlushTimeout: 2 * time.Second,
	MaxQueueSize: 32 * 1024 * 1024, // 32 MB
	Log:          log.New(os.Stderr, "appdash: ", log.LstdFlags),

func (*ChunkedCollector) Collect Uses

func (cc *ChunkedCollector) Collect(span SpanID, anns ...Annotation) error

Collect adds the span and annotations to a local buffer until the next call to Flush (or when MinInterval elapses), at which point they are sent (grouped by span) to the underlying collector.

func (*ChunkedCollector) Flush Uses

func (cc *ChunkedCollector) Flush() error

Flush immediately sends all pending spans to the underlying collector.

func (*ChunkedCollector) Stop Uses

func (cc *ChunkedCollector) Stop()

Stop stops the collector. After stopping, no more data will be sent to the underlying collector and calls to Collect will fail.

type Collector Uses

type Collector interface {
    Collect(SpanID, ...Annotation) error

A Collector collects events that occur in spans.

func NewLocalCollector Uses

func NewLocalCollector(s Store) Collector

NewLocalCollector returns a Collector that writes directly to a Store.

type CollectorServer Uses

type CollectorServer struct {

    // Log is the logger to use for errors and warnings. If nil, a new
    // logger is created.
    Log *log.Logger

    // Debug is whether to log debug messages.
    Debug bool

    // Trace is whether to log all data that is received.
    Trace bool
    // contains filtered or unexported fields

A CollectorServer listens for spans and annotations and adds them to a local collector.

func NewServer Uses

func NewServer(l net.Listener, c Collector) *CollectorServer

NewServer creates and starts a new server that listens for spans and annotations on l and adds them to the collector c.

Call the CollectorServer's Start method to start listening and serving.

func (*CollectorServer) Start Uses

func (cs *CollectorServer) Start()

Start starts the server.

type DeleteStore Uses

type DeleteStore interface {

    // Delete deletes traces given their trace IDs.
    Delete(...ID) error

A DeleteStore is a Store that can delete traces.

type Event Uses

type Event interface {
    // Schema should return the event's schema, a constant string, for example
    // the sqltrace package defines SQLEvent which returns just "SQL".
    Schema() string

An Event is a record of the occurrence of something.

func Log Uses

func Log(msg string) Event

Log returns an Event whose timestamp is the current time that contains only a human-readable message.

func Msg Uses

func Msg(msg string) Event

Msg returns an Event that contains only a human-readable message.

func SpanName Uses

func SpanName(name string) Event

SpanName returns an Event containing a human readable Span name.

type EventMarshaler Uses

type EventMarshaler interface {
    // MarshalEvent should marshal this event itself into a set of annotations, or
    // return an error.
    MarshalEvent() (Annotations, error)

EventMarshaler is the interface implemented by an event that can marshal a representation of itself into annotations.

type EventSchemaUnmarshalError Uses

type EventSchemaUnmarshalError struct {
    Found  []string // schemas found in the annotations
    Target string   // schema of the target event

An EventSchemaUnmarshalError is when annotations are attempted to be unmarshaled into an event object that does not match any of the schemas in the annotations.

func (*EventSchemaUnmarshalError) Error Uses

func (e *EventSchemaUnmarshalError) Error() string

type EventUnmarshaler Uses

type EventUnmarshaler interface {
    // UnmarshalEvent should unmarshal the given annotations into a event of the
    // same type, or return an error.
    UnmarshalEvent(Annotations) (Event, error)

EventUnmarshaler is the interface implemented by an event that can unmarshal an annotation representation of itself.

type ID Uses

type ID uint64

An ID is a unique, uniformly distributed 64-bit ID.

func ParseID Uses

func ParseID(s string) (ID, error)

ParseID parses the given string as a hexadecimal string.

func (ID) MarshalJSON Uses

func (id ID) MarshalJSON() ([]byte, error)

MarshalJSON encodes the ID as a hex string.

func (ID) String Uses

func (id ID) String() string

String returns the ID as a hex string.

func (*ID) UnmarshalJSON Uses

func (id *ID) UnmarshalJSON(data []byte) error

UnmarshalJSON decodes the given data as either a hexadecimal string or JSON integer.

type ImportantEvent Uses

type ImportantEvent interface {
    Important() []string

ImportantEvent is an event that can describe in particular which annotation keys it finds important. Only important annotation keys are displayed in the web UI by default.

type LimitStore Uses

type LimitStore struct {
    // Max is the maximum number of traces that the store should keep.
    Max int

    // DeleteStore is the underlying store that spans are saved to and
    // deleted from.
    // contains filtered or unexported fields

A LimitStore wraps another store and deletes the oldest trace when the number of traces reaches the capacity (Max).

func (*LimitStore) Collect Uses

func (ls *LimitStore) Collect(id SpanID, anns ...Annotation) error

Collect calls the underlying store's Collect, deleting the oldest trace if the capacity has been reached.

type MemoryStore Uses

type MemoryStore struct {
    sync.Mutex // protects trace
    // contains filtered or unexported fields

A MemoryStore is an in-memory Store that also implements the PersistentStore interface.

func NewMemoryStore Uses

func NewMemoryStore() *MemoryStore

NewMemoryStore creates a new in-memory store

func (*MemoryStore) Collect Uses

func (ms *MemoryStore) Collect(id SpanID, as ...Annotation) error

Collect implements the Collector interface by collecting the events that occurred in the span in-memory.

func (*MemoryStore) Delete Uses

func (ms *MemoryStore) Delete(traces ...ID) error

Delete implements the DeleteStore interface by deleting the traces given by their span ID's from this in-memory store.

func (*MemoryStore) ReadFrom Uses

func (ms *MemoryStore) ReadFrom(r io.Reader) (int64, error)

ReadFrom implements the PersistentStore interface by using gob-decoding to load ms's internal data structures from the reader r.

func (*MemoryStore) Trace Uses

func (ms *MemoryStore) Trace(id ID) (*Trace, error)

Trace implements the Store interface by returning the Trace (a tree of spans) for the given trace span ID or, if no such trace exists, by returning ErrTraceNotFound.

func (*MemoryStore) Traces Uses

func (ms *MemoryStore) Traces(opts TracesOpts) ([]*Trace, error)

Traces implements the Queryer interface.

func (*MemoryStore) Write Uses

func (ms *MemoryStore) Write(w io.Writer) error

Write implements the PersistentStore interface by gob-encoding and writing ms's internal data structures out to w.

type PersistentStore Uses

type PersistentStore interface {
    Write(io.Writer) error
    ReadFrom(io.Reader) (int64, error)

PersistentStore is a Store that can persist its data and read it back in.

type Queryer Uses

type Queryer interface {
    // Traces returns an implementation-defined list of traces according to the options.
    Traces(opts TracesOpts) ([]*Trace, error)

A Queryer indexes spans and makes them queryable.

func MultiQueryer Uses

func MultiQueryer(q ...Queryer) Queryer

MultiQueryer returns a Queryer whose Traces method returns a union of all traces across each queryer.

type RecentStore Uses

type RecentStore struct {
    // MinEvictAge is the minimum age of a trace before it is evicted.
    MinEvictAge time.Duration

    // DeleteStore is the underlying store that spans are saved to and
    // deleted from.

    // Debug is whether to log debug messages.
    Debug bool
    // contains filtered or unexported fields

A RecentStore wraps another store and deletes old traces after a specified amount of time.

func (*RecentStore) Collect Uses

func (rs *RecentStore) Collect(id SpanID, anns ...Annotation) error

Collect calls the underlying store's Collect and records the time that this trace was first seen.

type Recorder Uses

type Recorder struct {
    // Logger, if non-nil, causes errors to be written to this logger directly
    // instead of being manually checked via the Error method.
    Logger *log.Logger

    SpanID // the span ID that annotations are about
    // contains filtered or unexported fields

A Recorder is associated with a span and records annotations on the span by sending them to a collector.

func NewRecorder Uses

func NewRecorder(span SpanID, c Collector) *Recorder

NewRecorder creates a new recorder for the given span and collector. If c is nil, NewRecorder panics.

func (*Recorder) Annotation Uses

func (r *Recorder) Annotation(as ...Annotation)

Annotation records raw annotations on the span.

func (*Recorder) Child Uses

func (r *Recorder) Child() *Recorder

Child creates a new Recorder with the same collector and a new child SpanID whose parent is this recorder's SpanID.

func (*Recorder) Errors Uses

func (r *Recorder) Errors() []error

Errors returns all errors encountered by the Recorder since the last call to Errors. After calling Errors, the Recorder's list of errors is emptied.

func (*Recorder) Event Uses

func (r *Recorder) Event(e Event)

Event records any event that implements the Event, TimespanEvent, or TimestampedEvent interfaces.

func (*Recorder) Finish Uses

func (r *Recorder) Finish()

Finish finishes recording and saves the recorded information to the underlying collector. If Finish is not called, then no data will be written to the underlying collector. Finish must be called once, otherwise r.error is called, this constraint ensures that collector is called once per Recorder, in order to avoid for performance reasons extra operations(span look up & span's annotations update) within the collector.

func (*Recorder) Log Uses

func (r *Recorder) Log(msg string)

Log records a Log event (an event with the current timestamp and a human-readable message) on the span.

func (*Recorder) LogWithTimestamp Uses

func (r *Recorder) LogWithTimestamp(msg string, timestamp time.Time)

LogWithTimestamp records a Log event with an explicit timestamp

func (*Recorder) Msg Uses

func (r *Recorder) Msg(msg string)

Msg records a Msg event (an event with a human-readable message) on the span.

func (*Recorder) Name Uses

func (r *Recorder) Name(name string)

Name sets the name of this span.

type RemoteCollector Uses

type RemoteCollector struct {

    // Log is the logger to use for errors and warnings. If nil, a new
    // logger is created.
    Log *log.Logger

    // Debug is whether to log debug messages.
    Debug bool
    // contains filtered or unexported fields

A RemoteCollector sends data to a collector server (created with NewServer).

func NewRemoteCollector Uses

func NewRemoteCollector(addr string) *RemoteCollector

NewRemoteCollector creates a collector that sends data to a collector server (created with NewServer). It sends data immediately when Collect is called. To send data in chunks, use a ChunkedCollector.

func NewTLSRemoteCollector Uses

func NewTLSRemoteCollector(addr string, tlsConfig *tls.Config) *RemoteCollector

NewTLSRemoteCollector creates a RemoteCollector that uses TLS.

func (*RemoteCollector) Close Uses

func (rc *RemoteCollector) Close() error

Close closes the connection to the server.

func (*RemoteCollector) Collect Uses

func (rc *RemoteCollector) Collect(span SpanID, anns ...Annotation) error

Collect implements the Collector interface by sending the events that occurred in the span to the remote collector server (see CollectorServer).

type Span Uses

type Span struct {
    // ID probabilistically uniquely identifies this span.
    ID  SpanID


Span is a span ID and its annotations.

func (*Span) Name Uses

func (s *Span) Name() string

Name returns a span's name if it has a name annotation, and "" otherwise.

func (*Span) String Uses

func (s *Span) String() string

String returns the Span as a formatted string.

type SpanID Uses

type SpanID struct {
    // Trace is the root ID of the tree that contains all of the spans
    // related to this one.
    Trace ID

    // Span is an ID that probabilistically uniquely identifies this
    // span.
    Span ID

    // Parent is the ID of the parent span, if any.
    Parent ID

A SpanID refers to a single span.

func NewRootSpanID Uses

func NewRootSpanID() SpanID

NewRootSpanID generates a new span ID for a root span. This should only be used to generate entries for spans caused exclusively by spans which are outside of your system as a whole (e.g., a root span for the first time you see a user request).

func NewSpanID Uses

func NewSpanID(parent SpanID) SpanID

NewSpanID returns a new ID for an span which is the child of the given parent ID. This should be used to track causal relationships between spans.

func ParseSpanID Uses

func ParseSpanID(s string) (*SpanID, error)

ParseSpanID parses the given string as a slash-separated set of parameters.

func (SpanID) Format Uses

func (id SpanID) Format(s string, args ...interface{}) string

Format formats according to a format specifier and returns the resulting string. The receiver's string representation is the first argument.


// Assume we're connected to a database.
var (
    event  SpanID
    db     *sql.DB
    userID int
// This passes the root ID and the parent event ID to the database, which
// allows us to correlate, for example, slow queries with the web requests
// which caused them.
query := event.Format(`/* %s/%s */ %s`, `SELECT email FROM users WHERE id = ?`)
r := db.QueryRow(query, userID)
if r == nil {
    panic("user not found")
var email string
if err := r.Scan(&email); err != nil {
    panic("couldn't read email")
fmt.Printf("User's email: %s\n", email)

func (SpanID) IsRoot Uses

func (id SpanID) IsRoot() bool

IsRoot returns whether id is the root ID of a trace.

func (SpanID) String Uses

func (id SpanID) String() string

String returns the SpanID as a slash-separated, set of hex-encoded parameters (root, ID, parent). If the SpanID has no parent, that value is elided.

type SpanNameEvent Uses

type SpanNameEvent struct{ Name string }

A SpanNameEvent event sets a span's name.

func (SpanNameEvent) Schema Uses

func (SpanNameEvent) Schema() string

type Store Uses

type Store interface {

    // Trace gets a trace (a tree of spans) given its trace ID. If no
    // such trace exists, ErrTraceNotFound is returned.
    Trace(ID) (*Trace, error)

A Store stores and retrieves spans.

func MultiStore Uses

func MultiStore(s ...Store) Store

MultiStore returns a Store whose operations occur on the multiple given stores.

type Timespan Uses

type Timespan struct {
    S   time.Time `trace:"Span.Start"`
    E   time.Time `trace:"Span.End"`

Timespan is an event that satisfies the appdash.TimespanEvent interface. This is used to show its beginning and end times of a span.

func (Timespan) End Uses

func (s Timespan) End() time.Time

func (Timespan) Schema Uses

func (s Timespan) Schema() string

func (Timespan) Start Uses

func (s Timespan) Start() time.Time

type TimespanEvent Uses

type TimespanEvent interface {
    Start() time.Time
    End() time.Time

A TimespanEvent is an Event with a start and an end time.

type TimestampedEvent Uses

type TimestampedEvent interface {
    Timestamp() time.Time

A TimestampedEvent is an Event with a timestamp.

type Trace Uses

type Trace struct {
    Span          // Root span
    Sub  []*Trace // Children

A Trace is a tree of spans.

func (*Trace) FindSpan Uses

func (t *Trace) FindSpan(spanID ID) *Trace

FindSpan recursively searches for a span whose Span ID is spanID in t and its descendants. If no such span is found, nil is returned.

func (*Trace) String Uses

func (t *Trace) String() string

String returns the Trace as a formatted string.

func (*Trace) TimespanEvent Uses

func (t *Trace) TimespanEvent() (TimespanEvent, error)

func (*Trace) TreeString Uses

func (t *Trace) TreeString() string

TreeString returns the Trace as a formatted string that visually represents the trace's tree.

type TracesOpts Uses

type TracesOpts struct {
    // Timespan specifies a time range values which can be used as input for filtering traces.
    Timespan Timespan

    // TraceIDs filters the returned traces to just the ones with the given IDs.
    TraceIDs []ID

TraceOpts bundles the options used for list of traces.


cmd/appdashCommand appdash runs the Appdash web UI from the command-line.
examples/cmd/webappwebapp: a standalone example Negroni / Gorilla based webapp.
examples/cmd/webapp-opentracingwebapp: a standalone example Negroni / Gorilla based webapp.
httptracePackage httptrace implements support for tracing HTTP applications.
internal/wirePackage wire is a generated protocol buffer package.
opentracingPackage opentracing provides an Appdash implementation of the OpenTracing API.
sqltracePackage sqltrace implements utility types for tracing SQL queries.
traceappPackage traceapp implements the Appdash web UI.
traceapp/tmplPackage tmpl contains template data used by Appdash.

Package appdash imports 24 packages (graph) and is imported by 97 packages. Updated 2019-08-01. Refresh now. Tools for package owners.