README
¶
prombench
Benchmark and integration testing tool for Prometheus 2.x
Prombench generates load with synthetic exporters and then queries Prometheus to compare what we think we sent with what we hope was stored.
The eventual goal is to have Prombench monitor Prometheus internal metrics and increase the load until Prometheus can't keep up, then throttle back until it's just below the tipping point. For now it just generates a static load.
Usage
If the prometheus binary is in your PATH, you can run it without any arguments.
It won't run if the test dir ("prombench" by default) already exists, so either
remove it yourself between invocations or use the -rmtestdir option.
Each exporter generates 200 metrics with 100 labels, or 20000 real metrics. I can run about 30 such exporters on my system (8GB RAM, AMD FX(tm)-6300) at 1s scrape interval for a total load of 300K samples/second. Going above 300K samples/second requires more RAM and will probably require tuning Prometheus, see the tips in the storage part of the docs. You can override the default Prometheus command ("prometheus") with a path and command-line arguments at the end of the prombench command line, after --, e.g.
prombench -exporters inc:20 -- ~/src/prometheus/prometheus
Do not provide the -storage.local.retention Prometheus argument, use rather the prombench -test-duration argument. This allows the verification query to be scaled to how much data should still be present by the time it's run.
Exporters
The -exporters flag is a comma-separated list specifying which load exporters
to use and how many, e.g. inc:2,static:3 would launch 2 inc exporters and 3
static exporters.
The inc exporter increments the value of each metric on each scrape.
The static exporter exports unchanging metrics.
The randcyclic exporter exports semi-random values.
The oscillate exporter toggles between two sets of values on each cycle.
Unlike the others it doesn't actually go through the standard Prometheus client
library except during initialization, so it has much lower CPU needs.
Scheduled tasks
The -run-every flag is a comma-separated list of commands to invoke at fixed
intervals. For example, if you've installed go-torch
and have setup your path appropriately, you might run
prombench -run-every '60s:go-torch -u http://localhost:9090 -f torch`date "+%s"`.svg'
to create per-minute flamegraphs based on debug/pprof data.
Dashboards
I've put up a rudimentary dashboard at Grafana.net which is helpful for seeing what's going on under the hood.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( QueryTime *prometheus.HistogramVec = prometheus.NewHistogramVec( prometheus.HistogramOpts{ Namespace: "prombench", Subsystem: "query", Name: "latency_seconds", Help: "time to execute query", ConstLabels: prometheus.Labels{ "benchmark": "insert-then-sum", }, }, []string{"run_name", "query"}, ) )
Functions ¶
Types ¶
type Config ¶
type Config struct {
TestDirectory string
RmTestDirectory bool
FirstPort int
PrometheusPath string
ScrapeInterval time.Duration
TestDuration time.Duration
TestRetention time.Duration
ExtraArgs []string
Exporters ExporterSpecList
RunIntervals RunIntervalSpecList
MaxDeltaRatio float64
MaxQueryRetries int
AdaptiveInterval time.Duration
PrombenchListenAddress string
PrometheusListenAddress string
}
func (Config) PrometheusInstance ¶
TODO check for errors when the Config is created
type ExporterSpec ¶
type ExporterSpec struct {
Exporter LoadExporterKind
Count int
}
func (*ExporterSpec) Get ¶
func (e *ExporterSpec) Get() interface{}
func (*ExporterSpec) Set ¶
func (e *ExporterSpec) Set(v string) error
func (*ExporterSpec) String ¶
func (e *ExporterSpec) String() string
type ExporterSpecList ¶
type ExporterSpecList []ExporterSpec
func (*ExporterSpecList) Get ¶
func (esl *ExporterSpecList) Get() interface{}
func (*ExporterSpecList) Set ¶
func (esl *ExporterSpecList) Set(v string) error
func (*ExporterSpecList) String ¶
func (esl *ExporterSpecList) String() string
type LoadExporterKind ¶
type LoadExporterKind int
const ( ExporterInc LoadExporterKind = iota ExporterStatic ExporterRandCyclic ExporterOscillate )
func (LoadExporterKind) String ¶
func (i LoadExporterKind) String() string
type RunIntervalSpec ¶
func (*RunIntervalSpec) Get ¶
func (r *RunIntervalSpec) Get() interface{}
func (*RunIntervalSpec) Set ¶
func (r *RunIntervalSpec) Set(v string) error
func (*RunIntervalSpec) String ¶
func (r *RunIntervalSpec) String() string
type RunIntervalSpecList ¶
type RunIntervalSpecList []RunIntervalSpec
func (*RunIntervalSpecList) Get ¶
func (rsl *RunIntervalSpecList) Get() interface{}
func (*RunIntervalSpecList) Set ¶
func (rsl *RunIntervalSpecList) Set(v string) error
func (*RunIntervalSpecList) String ¶
func (rsl *RunIntervalSpecList) String() string
Source Files
Directories