appdash

package module
v0.0.0-...-84f0569 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 11, 2016 License: MIT Imports: 24 Imported by: 0

README

appdash

Appdash is an application tracing system for Go, based on Google's Dapper and Twitter's Zipkin.

Appdash allows you to trace the end-to-end handling of requests and operations in your application (for perf and debugging). It displays timings and application-specific metadata for each step, and it displays a tree and timeline for each request and its children.

To use appdash, you must instrument your application with calls to an appdash recorder. You can record any type of event or operation. Recorders and schemas for HTTP (client and server) and SQL are provided, and you can write your own.

Usage

To install appdash, run:

go get -u sourcegraph.com/sourcegraph/appdash/cmd/...

A standalone example using Negroni and Gorilla packages is available in the examples/cmd/webapp folder.

A demo / pure net/http application (which is slightly more verbose) is also available at cmd/appdash/example_app.go, and it can be ran easily using appdash demo on the command line.

Community

Questions or comments? Join us on Slack!

Development

Appdash uses vfsgen to package HTML templates with the appdash binary for distribution. This means that if you want to modify the template data in traceapp/tmpl you can first build using the dev build tag, which makes the template data be reloaded from disk live.

After you're finished making changes to the templates, always run go generate sourcegraph.com/sourcegraph/appdash/traceapp/tmpl so that the data_vfsdata.go file is updated for normal Appdash users that aren't interested in modifying the template data.

Components

Appdash follows the design and naming conventions of Google's Dapper. You should read that paper if you are curious about why certain architectural choices were made.

There are 4 main components/concepts in appdash:

  • Spans: A span refers to an operation and all of its children. For example, an HTTP handler handles a request by calling other components in your system, which in turn make various API and DB calls. The HTTP handler's span includes all downstream operations and their descendents; likewise, each downstream operation is its own span and has its own descendents. In this way, appdash constructs a tree of all of the operations that occur during the handling of the HTTP request.
  • Event: Your application records the various operations it performs (in the course of handling a request) as Events. Events can be arbitrary messages or metadata, or they can be structured event types defined by a Go type (such as an HTTP ServerEvent or an SQLEvent).
  • Recorder: Your application uses a Recorder to send events to a Collector (see below). Each Recorder is associated with a particular span in the tree of operations that are handling a particular request, and all events sent via a Recorder are automatically associated with that context.
  • Collector: A Collector receives Annotations (which are the encoded form of Events) sent by a Recorder. Typically, your application's Recorder talks to a local Collector (created with NewRemoteCollector. This local Collector forwards data to a remote appdash server (created with NewServer that combines traces from all of the services that compose your application. The appdash server in turn runs a Collector that listens on the network for this data, and it then stores what it receives.

Acknowledgments

appdash was influenced by, and uses code from, Coda Hale's lunk.

Documentation

Overview

Package appdash provides a Go app performance tracing suite.

Appdash allows you to trace the end-to-end performance of hierarchically structured applications. You can, for example, measure the time and see the detailed information of each HTTP request and SQL query made by an entire distributed web application.

Web Front-end

The cmd/appdash tool launches a web front-end which displays a web UI for viewing collected app traces. It is effectively a remote collector which your application can connect and send events to.

Timing and application-specific metadata information can be viewed in a nice timeline view for each span (e.g. HTTP request) and it's children.

The web front-end can also be embedded in your own Go HTTP server by utilizing the traceapp sub-package, which is effectively what cmd/appdash serves internally.

HTTP and SQL tracing

Sub-packages for HTTP and SQL event tracing are provided for use with appdash, which allows it to function equivalently to Google's Dapper and Twitter's Zipkin performance tracing suites.

Appdash Structure

The most high-level structure is a Trace, which represents the performance of an application from start to finish (in an HTTP application, for example, the loading of a web page).

A Trace is a tree structure that is made up of several spans, which are just IDs (in an HTTP application, these ID's are passed through the stack via a few special headers).

Each span ID has a set of Events that directly correspond to it inside a Collector. These events can be any combination of message, log, time-span, or time-stamped events (the cmd/appdash web UI displays these events as appropriate).

Inside your application, a Recorder is used to send events to a Collector, which can be a remote HTTP(S) collector, a local in-memory or persistent collector, etc. Additionally, you can implement the Collector interface yourself and store events however you like.

Index

Examples

Constants

View Source 
const (
	// SpanIDDelimiter is the delimiter used to concatenate an
	// SpanID's components.
	SpanIDDelimiter = "/"
)

Variables

View Source 
var (
	// ErrBadSpanID is returned when the span ID cannot be parsed.
	ErrBadSpanID = errors.New("bad span ID")
)
View Source 
var (
	// ErrTraceNotFound is returned by Store.GetTrace when no trace is
	// found with the given ID.
	ErrTraceNotFound = errors.New("trace not found")
)

Functions

func PersistEvery 

func PersistEvery(s PersistentStore, interval time.Duration, file string) error

PersistEvery persists s's data to a file periodically.

func RegisterEvent 

func RegisterEvent(e Event)

RegisterEvent registers an event type for use with UnmarshalEvents.

Events must be registered with this package in order for unmarshaling to work. Much like the image package, sometimes blank imports will be used for packages that register Appdash events with this package: 

import(
    _ "sourcegraph.com/sourcegraph/appdash/httptrace"
    _ "sourcegraph.com/sourcegraph/appdash/sqltrace"
)

func UnmarshalEvent 

func UnmarshalEvent(as Annotations, e Event) error

UnmarshalEvent unmarshals annotations into an event.

func UnmarshalEvents 

func UnmarshalEvents(anns Annotations, events *[]Event) error

UnmarshalEvents unmarshals all events found in anns into events. Any schemas found in anns that were not registered (using RegisterEvent) are ignored; missing a schema is not an error.

Types

type AggregateEvent 

type AggregateEvent struct {
	// The root span name of every item in this aggregated set of timespan events.
	Name string

	// Trace IDs for the slowest of the above times (useful for inspection).
	Slowest []ID
}

AggregateEvent represents an aggregated set of timespan events. This is the only type of event produced by the AggregateStore type.

func (AggregateEvent) MarshalEvent 

func (e AggregateEvent) MarshalEvent() (Annotations, error)

MarshalEvent implements the EventMarshaler interface.

func (AggregateEvent) Schema 

func (e AggregateEvent) Schema() string

Schema implements the Event interface.

func (AggregateEvent) UnmarshalEvent 

func (e AggregateEvent) UnmarshalEvent(as Annotations) (Event, error)

UnmarshalEvent implements the EventUnmarshaler interface.

type AggregateStore 

type AggregateStore struct {
	// MinEvictAge is the minimum age of group data before it is evicted.
	MinEvictAge time.Duration

	// MaxRate is the maximum expected rate of incoming traces.
	//
	// Multiple traces can be collected at once, and they must be queued up in
	// memory until the entire trace has been collected, otherwise the N-slowest
	// traces cannot be stored.
	//
	// If the number is too large, a lot of memory will be used (to store
	// MaxRate number of traces), and if too small some then some aggregate
	// events will not have the full N-slowest traces associated with them.
	MaxRate int

	// NSlowest is the number of slowest traces to fully keep for inspection for
	// each group.
	NSlowest int

	// Debug is whether or not to log debug messages.
	Debug bool

	// MemoryStore is the memory store were aggregated traces are saved to and
	// deleted from. It is the final destination for traces.
	*MemoryStore
	// contains filtered or unexported fields
}

AggregateStore aggregates timespan events into groups based on the root span name. Much like a RecentStore, it evicts aggregated events after a certain time period.

func NewAggregateStore 

func NewAggregateStore() *AggregateStore

NewAggregateStore is short-hand for: 

store := &AggregateStore{
    MinEvictAge: 72 * time.Hour,
    MaxRate: 4096,
    NSlowest: 5,
    MemoryStore: NewMemoryStore(),
}

func (*AggregateStore) Collect 

func (as *AggregateStore) Collect(id SpanID, anns ...Annotation) error

Collect calls the underlying store's Collect, deleting the oldest trace if the capacity has been reached.

type Annotation 

type Annotation struct {
	// Key is the annotation's key.
	Key string

	// Value is the annotation's value, which may be either human or
	// machine readable, depending on the schema of the event that
	// generated it.
	Value []byte
}

An Annotation is an arbitrary key-value property on a span.

func (Annotation) Important 

func (a Annotation) Important() bool

Important determines if this annotation's key is considered important to any of the registered event types.

type Annotations 

type Annotations []Annotation

Annotations is a list of annotations (on a span).

func MarshalEvent 

func MarshalEvent(e Event) (Annotations, error)

MarshalEvent marshals an event into annotations.

func (Annotations) String 

func (as Annotations) String() string

String returns a formatted list of annotations.

func (Annotations) StringMap 

func (as Annotations) StringMap() map[string]string

StringMap returns the annotations as a key-value map. Only one annotation for a key appears in the map, and it is chosen arbitrarily among the annotations with the same key.

type ChunkedCollector 

type ChunkedCollector struct {
	// Collector is the underlying collector that spans are sent to.
	Collector

	// MinInterval is the minimum time period between calls to the
	// underlying collector's Collect method.
	MinInterval time.Duration
	// contains filtered or unexported fields
}

A ChunkedCollector groups annotations together that have the same span and calls its underlying collector's Collect method with the chunked data periodically (instead of immediately).

func (*ChunkedCollector) Collect 

func (cc *ChunkedCollector) Collect(span SpanID, anns ...Annotation) error

Collect adds the span and annotations to a local buffer until the next call to Flush (or when MinInterval elapses), at which point they are sent (grouped by span) to the underlying collector.

func (*ChunkedCollector) Flush 

func (cc *ChunkedCollector) Flush() error

Flush immediately sends all pending spans to the underlying collector.

func (*ChunkedCollector) Stop 

func (cc *ChunkedCollector) Stop()

Stop stops the collector. After stopping, no more data will be sent to the underlying collector and calls to Collect will fail.

type Collector 

type Collector interface {
	Collect(SpanID, ...Annotation) error
}

A Collector collects events that occur in spans.

func NewLocalCollector 

func NewLocalCollector(s Store) Collector

NewLocalCollector returns a Collector that writes directly to a Store.

type CollectorServer 

type CollectorServer struct {

	// Log is the logger to use for errors and warnings. If nil, a new
	// logger is created.
	Log *log.Logger

	// Debug is whether to log debug messages.
	Debug bool

	// Trace is whether to log all data that is received.
	Trace bool
	// contains filtered or unexported fields
}

A CollectorServer listens for spans and annotations and adds them to a local collector.

func NewServer 

func NewServer(l net.Listener, c Collector) *CollectorServer

NewServer creates and starts a new server that listens for spans and annotations on l and adds them to the collector c.

Call the CollectorServer's Start method to start listening and serving.

func (*CollectorServer) Start 

func (cs *CollectorServer) Start()

Start starts the server.

type DeleteStore 

type DeleteStore interface {
	Store

	// Delete deletes traces given their trace IDs.
	Delete(...ID) error
}

A DeleteStore is a Store that can delete traces.

type Event 

type Event interface {
	// Schema should return the event's schema, a constant string, for example
	// the sqltrace package defines SQLEvent which returns just "SQL".
	Schema() string
}

An Event is a record of the occurrence of something.

func Log 

func Log(msg string) Event

Log returns an Event whose timestamp is the current time that contains only a human-readable message.

func Msg 

func Msg(msg string) Event

Msg returns an Event that contains only a human-readable message.

type EventMarshaler 

type EventMarshaler interface {
	// MarshalEvent should marshal this event itself into a set of annotations, or
	// return an error.
	MarshalEvent() (Annotations, error)
}

EventMarshaler is the interface implemented by an event that can marshal a representation of itself into annotations.

type EventSchemaUnmarshalError 

type EventSchemaUnmarshalError struct {
	Found  []string // schemas found in the annotations
	Target string   // schema of the target event
}

An EventSchemaUnmarshalError is when annotations are attempted to be unmarshaled into an event object that does not match any of the schemas in the annotations.

func (*EventSchemaUnmarshalError) Error 

func (e *EventSchemaUnmarshalError) Error() string

type EventUnmarshaler 

type EventUnmarshaler interface {
	// UnmarshalEvent should unmarshal the given annotations into a event of the
	// same type, or return an error.
	UnmarshalEvent(Annotations) (Event, error)
}

EventUnmarshaler is the interface implemented by an event that can unmarshal an annotation representation of itself.

type ID 

type ID uint64

An ID is a unique, uniformly distributed 64-bit ID.

func ParseID 

func ParseID(s string) (ID, error)

ParseID parses the given string as a hexadecimal string.

func (ID) MarshalJSON 

func (id ID) MarshalJSON() ([]byte, error)

MarshalJSON encodes the ID as a hex string.

func (ID) String 

func (id ID) String() string

String returns the ID as a hex string.

func (*ID) UnmarshalJSON 

func (id *ID) UnmarshalJSON(data []byte) error

UnmarshalJSON decodes the given data as either a hexadecimal string or JSON integer.

type ImportantEvent 

type ImportantEvent interface {
	Important() []string
}

ImportantEvent is an event that can describe in particular which annotation keys it finds important. Only important annotation keys are displayed in the web UI by default.

type LimitStore 

type LimitStore struct {
	// Max is the maximum number of traces that the store should keep.
	Max int

	// DeleteStore is the underlying store that spans are saved to and
	// deleted from.
	DeleteStore
	// contains filtered or unexported fields
}

A LimitStore wraps another store and deletes the oldest trace when the number of traces reaches the capacity (Max).

func (*LimitStore) Collect 

func (ls *LimitStore) Collect(id SpanID, anns ...Annotation) error

Collect calls the underlying store's Collect, deleting the oldest trace if the capacity has been reached.

type MemoryStore 

type MemoryStore struct {
	sync.Mutex // protects trace
	// contains filtered or unexported fields
}

A MemoryStore is an in-memory Store that also implements the PersistentStore interface.

func NewMemoryStore 

func NewMemoryStore() *MemoryStore

NewMemoryStore creates a new in-memory store

func (*MemoryStore) Collect 

func (ms *MemoryStore) Collect(id SpanID, as ...Annotation) error

Collect implements the Collector interface by collecting the events that occured in the span in-memory.

func (*MemoryStore) Delete 

func (ms *MemoryStore) Delete(traces ...ID) error

Delete implements the DeleteStore interface by deleting the traces given by their span ID's from this in-memory store.

func (*MemoryStore) ReadFrom 

func (ms *MemoryStore) ReadFrom(r io.Reader) (int64, error)

ReadFrom implements the PersistentStore interface by using gob-decoding to load ms's internal data structures from the reader r.

func (*MemoryStore) Trace 

func (ms *MemoryStore) Trace(id ID) (*Trace, error)

Trace implements the Store interface by returning the Trace (a tree of spans) for the given trace span ID or, if no such trace exists, by returning ErrTraceNotFound.

func (*MemoryStore) Traces 

func (ms *MemoryStore) Traces() ([]*Trace, error)

Traces implements the Queryer interface.

func (*MemoryStore) Write 

func (ms *MemoryStore) Write(w io.Writer) error

Write implements the PersistentStore interface by gob-encoding and writing ms's internal data structures out to w.

type PersistentStore 

type PersistentStore interface {
	Write(io.Writer) error
	ReadFrom(io.Reader) (int64, error)
	Store
}

PersistentStore is a Store that can persist its data and read it back in.

type Queryer 

type Queryer interface {
	// Traces returns an implementation-defined list of traces. It is
	// a placeholder method that will be removed when other, more
	// useful methods are added to Queryer.
	Traces() ([]*Trace, error)
}

A Queryer indexes spans and makes them queryable.

func MultiQueryer 

func MultiQueryer(q ...Queryer) Queryer

MultiQueryer returns a Queryer whose Traces method returns a union of all traces across each queryer.

type RecentStore 

type RecentStore struct {
	// MinEvictAge is the minimum age of a trace before it is evicted.
	MinEvictAge time.Duration

	// DeleteStore is the underlying store that spans are saved to and
	// deleted from.
	DeleteStore

	// Debug is whether to log debug messages.
	Debug bool
	// contains filtered or unexported fields
}

A RecentStore wraps another store and deletes old traces after a specified amount of time.

func (*RecentStore) Collect 

func (rs *RecentStore) Collect(id SpanID, anns ...Annotation) error

Collect calls the underlying store's Collect and records the time that this trace was first seen.

type Recorder 

type Recorder struct {
	SpanID // the span ID that annotations are about
	// contains filtered or unexported fields
}

A Recorder is associated with a span and records annotations on the span by sending them to a collector.

func NewRecorder 

func NewRecorder(span SpanID, c Collector) *Recorder

NewRecorder creates a new recorder for the given span and collector. If c is nil, NewRecorder panics.

func (*Recorder) Annotation 

func (r *Recorder) Annotation(as ...Annotation)

Annotation records raw annotations on the span.

func (*Recorder) Child 

func (r *Recorder) Child() *Recorder

Child creates a new Recorder with the same collector and a new child SpanID whose parent is this recorder's SpanID.

func (*Recorder) Errors 

func (r *Recorder) Errors() []error

Errors returns all errors encountered by the Recorder since the last call to Errors. After calling Errors, the Recorder's list of errors is emptied.

func (*Recorder) Event 

func (r *Recorder) Event(e Event)

Event records any event that implements the Event, TimespanEvent, or TimestampedEvent interfaces.

func (*Recorder) Log 

func (r *Recorder) Log(msg string)

Log records a Log event (an event with the current timestamp and a human-readable message) on the span.

func (*Recorder) Msg 

func (r *Recorder) Msg(msg string)

Msg records a Msg event (an event with a human-readable message) on the span.

func (*Recorder) Name 

func (r *Recorder) Name(name string)

Name sets the name of this span.

type RemoteCollector 

type RemoteCollector struct {

	// Log is the logger to use for errors and warnings. If nil, a new
	// logger is created.
	Log *log.Logger

	// Debug is whether to log debug messages.
	Debug bool
	// contains filtered or unexported fields
}

A RemoteCollector sends data to a collector server (created with NewServer).

func NewRemoteCollector 

func NewRemoteCollector(addr string) *RemoteCollector

NewRemoteCollector creates a collector that sends data to a collector server (created with NewServer). It sends data immediately when Collect is called. To send data in chunks, use a ChunkedCollector.

func NewTLSRemoteCollector 

func NewTLSRemoteCollector(addr string, tlsConfig *tls.Config) *RemoteCollector

NewTLSRemoteCollector creates a RemoteCollector that uses TLS.

func (*RemoteCollector) Close 

func (rc *RemoteCollector) Close() error

Close closes the connection to the server.

func (*RemoteCollector) Collect 

func (rc *RemoteCollector) Collect(span SpanID, anns ...Annotation) error

Collect implements the Collector interface by sending the events that occured in the span to the remote collector server (see CollectorServer).

type Span 

type Span struct {
	// ID probabilistically uniquely identifies this span.
	ID SpanID

	Annotations
}

Span is a span ID and its annotations.

func (*Span) Name 

func (s *Span) Name() string

Name returns a span's name if it has a name annotation, and "" otherwise.

func (*Span) String 

func (s *Span) String() string

String returns the Span as a formatted string.

type SpanID 

type SpanID struct {
	// Trace is the root ID of the tree that contains all of the spans
	// related to this one.
	Trace ID

	// Span is an ID that probabilistically uniquely identifies this
	// span.
	Span ID

	// Parent is the ID of the parent span, if any.
	Parent ID
}

A SpanID refers to a single span.

func NewRootSpanID 

func NewRootSpanID() SpanID

NewRootSpanID generates a new span ID for a root span. This should only be used to generate entries for spans caused exclusively by spans which are outside of your system as a whole (e.g., a root span for the first time you see a user request).

func NewSpanID 

func NewSpanID(parent SpanID) SpanID

NewSpanID returns a new ID for an span which is the child of the given parent ID. This should be used to track causal relationships between spans.

func ParseSpanID 

func ParseSpanID(s string) (*SpanID, error)

ParseSpanID parses the given string as a slash-separated set of parameters.

func (SpanID) Format 

func (id SpanID) Format(s string, args ...interface{}) string

Format formats according to a format specifier and returns the resulting string. The receiver's string representation is the first argument.

Example 
// Assume we're connected to a database.
var (
	event  SpanID
	db     *sql.DB
	userID int
)
// This passes the root ID and the parent event ID to the database, which
// allows us to correlate, for example, slow queries with the web requests
// which caused them.
query := event.Format(`/* %s/%s */ %s`, `SELECT email FROM users WHERE id = ?`)
r := db.QueryRow(query, userID)
if r == nil {
	panic("user not found")
}
var email string
if err := r.Scan(&email); err != nil {
	panic("couldn't read email")
}
fmt.Printf("User's email: %s\n", email)

Output:

func (SpanID) IsRoot 

func (id SpanID) IsRoot() bool

IsRoot returns whether id is the root ID of a trace.

func (SpanID) String 

func (id SpanID) String() string

String returns the SpanID as a slash-separated, set of hex-encoded parameters (root, ID, parent). If the SpanID has no parent, that value is elided.

type Store 

type Store interface {
	Collector

	// Trace gets a trace (a tree of spans) given its trace ID. If no
	// such trace exists, ErrTraceNotFound is returned.
	Trace(ID) (*Trace, error)
}

A Store stores and retrieves spans.

func MultiStore 

func MultiStore(s ...Store) Store

MultiStore returns a Store whose operations occur on the multiple given stores.

type TimespanEvent 

type TimespanEvent interface {
	Event
	Start() time.Time
	End() time.Time
}

A TimespanEvent is an Event with a start and an end time.

type TimestampedEvent 

type TimestampedEvent interface {
	Timestamp() time.Time
}

A TimestampedEvent is an Event with a timestamp.

type Trace 

type Trace struct {
	Span          // Root span
	Sub  []*Trace // Children
}

A Trace is a tree of spans.

func (*Trace) Aggregated 

func (t *Trace) Aggregated() (*AggregateEvent, []TimespanEvent, error)

Aggregated returns the aggregate event (or nil if none is found) along with all of the TimespanEvents found in this trace.

func (*Trace) FindSpan 

func (t *Trace) FindSpan(spanID ID) *Trace

FindSpan recursively searches for a span whose Span ID is spanID in t and its descendants. If no such span is found, nil is returned.

func (*Trace) IsAggregate 

func (t *Trace) IsAggregate() bool

IsAggregate tells if the trace contains any AggregateEvents (it is therefor said to be a set of aggregated traces).

func (*Trace) String 

func (t *Trace) String() string

String returns the Trace as a formatted string.

func (*Trace) TreeString 

func (t *Trace) TreeString() string

TreeString returns the Trace as a formatted string that visually represents the trace's tree.

Source Files

View all Source files

Directories

Path Synopsis
cmd
appdash
Command appdash runs the Appdash web UI from the command-line.
 Command appdash runs the Appdash web UI from the command-line.
examples
cmd/webapp
webapp: a standalone example Negroni / Gorilla based webapp.
 webapp: a standalone example Negroni / Gorilla based webapp.
Package httptrace implements support for tracing HTTP applications.
 Package httptrace implements support for tracing HTTP applications.
internal
wire
Package wire is a generated protocol buffer package.
 Package wire is a generated protocol buffer package.
Package sqltrace implements utility types for tracing SQL queries.
 Package sqltrace implements utility types for tracing SQL queries.
Package traceapp implements the Appdash web UI.
 Package traceapp implements the Appdash web UI.
tmpl
Package tmpl contains template data used by Appdash.
 Package tmpl contains template data used by Appdash.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.