testsuite

package module
v0.0.0-...-12cbdf9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 15, 2022 License: Apache-2.0 Imports: 33 Imported by: 0

README

Prometheus Alert Generator Compliance

Specification

You can find the specification for this compliance here.

Running the tests

Step 1

Clone the repo

$ git clone https://github.com/prometheus/compliance.git
$ cd compliance/alert_generator
Step 2

Check that the rules file is up-to-date with

$ make check-rules
go run ./cmd/rule_config_builder/main.go -rules-file-path="./rules.yaml"
level=info ts=2022-02-24T14:14:19.596Z caller=main.go:53 msg="Rules file successfully generated" path=/home/ganesh/go/src/github.com/prometheus/compliance/alert_generator/rules.yaml

$ echo $?
0

Run make rules otherwise to generate fresh rules file.

Feed the rules.yaml file present in the directory into the software that you are testing with any additional tweaks required.

It is suggested to wait until one evaluation of all rule groups is done before starting the test.

Step 3

Check test-example.yaml for the example config to setup the test suite. That file has all the documentation on how to configure the test suite.

Feel free to modify the example config itself. Config for some softwares are provided as an example, more will be added.

Step 4

If you are running your software in a local environment, you can set the alertmanager URL to http://<host>:<port> with port being the one set in the test suite config. For example http://localhost:8080.

If you are testing a cloud offering, or if the local software setup cannot access the test suite's network, there are two alternatives:

Step 4a

You can run ./cmd/alert_generator_compliance_tester in docker and run it inside your infrastructure as a batch job. To do so, build the image with make docker and instead of step 6, simply run alert_generator_compliance_tester:latest with -config-file=<config file you have done in step 3>.

See example here

Step 4b
  1. Open https://webhook.site/. It will give a unique link to use as a webhook. Let's take https://webhook.site/12345 as an example.
  2. Set https://webhook.site/12345 to be the alertmanager URL in your software - hence all alerts will be set to this webhook.
  3. On the webhook.site page, enable XHR Redirect on top, and in its settings, set the Target to http://<host>:<port> from the above, for example http://localhost:8080.

Note:

  • The webhook.site page must be open at all times. It redirects the request locally, hence able to redirect to local network.
  • In the test logs you might see "error in unmarshaling" error immediately followed by "alerts received" log. You can ignore the error log. It is something to do with how webhook.site redirects.
  • Disable any kind of browser blockers for webhook.site if you are not getting alerts redirected to the test suite.
Step 5

If your software send alerts in a format that is not parsable by any of the provided parsers in alert_message_parsers.go, you can extend that file to include your custom parser and mention that name in the config file.

Step 6

Now that everything is set up, you can run the test as follows

$ make run CONFIG=./test-example.yaml

A successful test run will exit with Congrats! All tests passed message. An unsuccessful run will end with a description of what went wrong.

The test takes roughly 40 mins to run. But if all rule groups face any errors, the test will exit immediately.

Running on Prometheus as an example

  1. Get the rules file from step 2 above and put it in the Prometheus config.
  2. Set the alertmanager URL to localhost:8080.
  3. Run Prometheus with --web.enable-remote-write-receiver flag to accept remote write.
  4. Run the test with make run CONFIG=./test-prometheus.yaml

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var AlertMessageParsers = map[string]AlertMessageParser{

	"default": func(b []byte) ([]notifier.Alert, error) {
		var alerts []notifier.Alert
		err := json.Unmarshal(b, &alerts)
		return alerts, err
	},

	"alertmanager": func(b []byte) ([]notifier.Alert, error) {
		msg := webhook.Message{}
		err := json.Unmarshal(b, &msg)
		if err != nil {
			return nil, err
		}
		alerts := make([]notifier.Alert, 0, len(msg.Alerts))
		for _, al := range msg.Alerts {
			alerts = append(alerts, notifier.Alert{
				Labels:       labels.FromMap(al.Labels),
				Annotations:  labels.FromMap(al.Annotations),
				StartsAt:     al.StartsAt,
				EndsAt:       al.EndsAt,
				GeneratorURL: al.GeneratorURL,
			})
		}
		return alerts, nil
	},
}

AlertMessageParsers is a map of the parser name to the parser function. The parser name is the one to be used in the config file. You can extend this map to include your custom parser and it will be matched with the config file.

Functions

func DoGetRequest 

func DoGetRequest(u string, auth config.AuthConfig) ([]byte, error)

TODO: add retries and set some timeouts.

func NewMulti 

func NewMulti(errs ...error) multiError

NewMulti returns multiError with provided errors added if not nil.

func ParseAndGroupAlerts 

func ParseAndGroupAlerts(b []byte) (map[string][]v1.Alert, error)

ParseAndGroupAlerts parses the alerts and groups by the rule group name. The alerts are assumed to have a `rulegroup` label.

func ParseAndGroupMetrics 

func ParseAndGroupMetrics(b []byte) (map[string][]promql.Sample, error)

ParseAndGroupMetrics parses samples and groups by the rule group name. The metrics are assumed to have a `rulegroup` label.

func ParseAndGroupRules 

func ParseAndGroupRules(b []byte) (map[string]*v1.RuleGroup, error)

ParseAndGroupRules parses the rules and groups by the rule group name. The rules are assumed to have a `rulegroup` label.

Types

type AlertMessageParser 

type AlertMessageParser func(b []byte) ([]notifier.Alert, error)

type Alerts 

type Alerts struct {
	Alerts []v1.Alert `json:"alerts"`
}

type Data 

type Data struct {
	RuleGroups []*RuleGroup `json:"groups"`
}

type GETAlertsResponse 

type GETAlertsResponse struct {
	Status string `json:"status"`
	Data   Alerts `json:"data"`
}

type GETMetricsResponse 

type GETMetricsResponse struct {
	Status string  `json:"status"`
	Data   Metrics `json:"data"`
}

type GETRulesResponse 

type GETRulesResponse struct {
	Status string `json:"status"`
	Data   Data   `json:"data"`
}

type Metrics 

type Metrics struct {
	ResultType string   `json:"resultType"`
	Result     []Vector `json:"result"`
}

type RemoteWriter 

type RemoteWriter struct {
	// contains filtered or unexported fields
}

RemoteWriter remote writes the time series provided AddTimeSeries() in sorted fashion w.r.t. the timestamps.

func NewRemoteWriter 

func NewRemoteWriter(cfg agconfig.Config, logger log.Logger) (*RemoteWriter, error)

func (*RemoteWriter) AddTimeSeries 

func (rw *RemoteWriter) AddTimeSeries(ts []prompb.TimeSeries)

AddTimeSeries adds more timeseries to the queue. The timestamp of the samples should be 0 based. It should not be called after calling Start().

func (*RemoteWriter) Error 

func (rw *RemoteWriter) Error() error

func (*RemoteWriter) Start 

func (rw *RemoteWriter) Start() time.Time

Start starts remote-writing the given timeseries. It returns the time corresponding to the 0 timestamp.

func (*RemoteWriter) Stop 

func (rw *RemoteWriter) Stop()

func (*RemoteWriter) Wait 

func (rw *RemoteWriter) Wait()

type RuleGroup 

type RuleGroup struct {
	Name           string            `json:"name"`
	File           string            `json:"file"`
	Rules          []v1.AlertingRule `json:"rules"`
	Interval       float64           `json:"interval"`
	EvaluationTime float64           `json:"evaluationTime"`
	LastEvaluation time.Time         `json:"lastEvaluation"`
}

type TestSuite 

type TestSuite struct {
	// contains filtered or unexported fields
}

TestSuite runs the entire test suite from start to end.

func NewTestSuite 

func NewTestSuite(opts TestSuiteOptions) (*TestSuite, error)

func (*TestSuite) Error 

func (ts *TestSuite) Error() error

Error() returns any error occured during execution of test and does not tell if the tests passed or failed.

func (*TestSuite) Start 

func (ts *TestSuite) Start()

func (*TestSuite) Stop 

func (ts *TestSuite) Stop()

func (*TestSuite) TestUntil 

func (ts *TestSuite) TestUntil() time.Time

func (*TestSuite) Wait 

func (ts *TestSuite) Wait()

func (*TestSuite) WasTestSuccessful 

func (ts *TestSuite) WasTestSuccessful() (yes bool, describe string)

WasTestSuccessful tells if all the tests passed. It returns an explanation if any test failed. Before calling this method:

  • Error() should be checked for no errors.
  • The test should have finished (i.e. Wait() is not blocking).

type TestSuiteOptions 

type TestSuiteOptions struct {
	Logger log.Logger
	// All the test cases to test.
	Cases []cases.TestCase

	Config config.Config

	AlertMessageParser AlertMessageParser
}

type Vector 

type Vector struct {
	Metric labels.Labels  `json:"metric"`
	Value  [2]interface{} `json:"value"`
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
alert_generator_compliance_tester
rule_config_builder

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.