greenbay

package
v0.0.0-...-a129325 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 23, 2023 License: Apache-2.0 Imports: 26 Imported by: 0

README 

===================================
``greenbay`` -- System Testing Tool
===================================

Overview
--------

We use ``greenbay`` tests to verify that build images in `Evergreen
<https://evergreen.mongodb.com/>`_, our Continuous Integration system, are
configured as we expect. The ``greenbay`` tool (`API documentation
<https://godoc.org/github.com/mongodb/greenbay>`_) reads a configuration
file and executes the tests it describes.

Writing Tests
-------------

A test must include a name, a list of suites to run, a test type, and some
args. For example, this verifies that pip is installed: ::

  tests:
    - name: pip_installed
      suites:
        - archlinux
      type: shell-operation
      args:
        command: "pip --version"

Save this as ``test.yaml``, and run it by referring to either the test or the
suite.
::

  greenbay run --conf test.yaml --test pip_installed
  greenbay run --conf test.yaml --suite archlinux 

You can pass ``--test`` or ``--suite`` multiple times. The ``--suite``
argument is useful for bundling tests together. If you don't pass either
``--test`` or ``--suite``, ``greenbay`` will run the "all" suite.

Common Test Types
-----------------

Every command must succeed:

::

  - name: command_test
    suites:
      - all
    type: command-group-all
    args:
      commands:
        - command: "run this command"
        - command: "and also this command"
        - command: "all should exit 0"

Check if a package is installed with dpkg:

::

  - name: dpkg_test
    suites:
      - all
    type: dpkg-installed
    args:
      package: package-name

Check if a file exists:

::

  - name: file_test
    suites:
      - all
    type: file-exists
    args:
      name: "/path/to/file"

Run a bash script:

::

  - name: bash_test
    suites:
      - all
    type: run-bash-script
    args:
      source: |
        # some bash to run
      output: "foo"

Run a python script:

::

  - name: python_test
    suites:
      - all
    type: run-program-system-python
    args:
      source: |
        print("howdy")
      output: "howdy"

Run a single command:

::

  - name: pip_test
    suites:
      - all
    type: shell-operation
    args:
      command: "pip --version"

At least one of the yum packages must be installed:

::

  - name: yum_group_test
    suites:
      - all
    type: yum-group-any
    args:
      packages:
        - glibc-devel.i386
        - glibc-devel.i686

Check if a package is installed with yum:

::

  - name: yum_test
    suites:
      - all
    type: yum-installed
    args:
      package: package-name

Greenbay Test Types
-------------------

You can list all ``greenbay`` test types with the following command: ::

  greenbay list

This will output a list of tests like this one: ::

  address-size
  brew-group-all
  brew-group-any
  brew-group-none
  brew-group-one
  brew-installed
  brew-not-installed
  command-group-all
  command-group-any
  command-group-none
  command-group-one
  compile-and-run-gcc-auto
  compile-and-run-gcc-system
  compile-and-run-go-auto
  compile-and-run-opt-go-default
  compile-and-run-toolchain-gccgo-v2
  compile-and-run-toolchain-v0
  compile-and-run-toolchain-v1
  compile-and-run-toolchain-v2
  compile-and-run-user-local-go
  compile-and-run-usr-local-go
  compile-and-run-visual-studio
  compile-gcc-auto
  compile-gcc-system
  compile-go-auto
  compile-opt-go-default
  compile-toolchain-gccgo-v2
  compile-toolchain-v0
  compile-toolchain-v1
  compile-toolchain-v2
  compile-user-local-go
  compile-usr-local-go
  compile-visual-studio
  dpkg-group-all
  dpkg-group-any
  dpkg-group-none
  dpkg-group-one
  dpkg-installed
  dpkg-not-installed
  file-does-not-exist
  file-exists
  file-group-all
  file-group-any
  file-group-none
  file-group-one
  gem-group-all
  gem-group-any
  gem-group-none
  gem-group-one
  gem-installed
  gem-not-installed
  irp-stack-size
  lxc-containers-configured
  open-files
  pacman-group-all
  pacman-group-any
  pacman-group-none
  pacman-group-one
  pacman-installed
  pacman-not-installed
  pip-group-all
  pip-group-any
  pip-group-none
  pip-group-one
  pip-installed
  pip-not-installed
  python-module-version
  run-bash-script
  run-bash-script-succeeds
  run-dash-script
  run-dash-script-succeeds
  run-program-gcc-auto
  run-program-gcc-system
  run-program-go-auto
  run-program-opt-go-default
  run-program-python-auto
  run-program-system-python
  run-program-system-python2
  run-program-system-python3
  run-program-toolchain-gccgo-v2
  run-program-toolchain-v0
  run-program-toolchain-v1
  run-program-toolchain-v2
  run-program-user-local-go
  run-program-usr-bin-pypy
  run-program-usr-local-go
  run-program-usr-local-python
  run-program-visual-studio
  run-sh-script
  run-sh-script-succeeds
  run-zsh-script
  run-zsh-script-succeeds
  shell-operation
  shell-operation-error
  yum-group-all
  yum-group-any
  yum-group-none
  yum-group-one
  yum-installed
  yum-not-installed

Documentation

Overview

Package greenbay provides the core greenbay application functionality, distinct from user interfaces.

The core greenbay test execution code is available here to support better testing and alternate interfaces. Currently the only interface is a command line interface, but we could wrap this functionality in a web service to support easier integration with monitoring tools or other health-check services.

The core functionality of the application is in the Application structure which stores application and facilitates the integration of output production, test running, and test configuration.

Package greenbay contains the basic type definition used in the greenbay application.

Overview

The Greenbay application is a system integration testing and validation tool. It contains definitions of some generic test functions, such as "does this file exist" and "do these commands succeed." Specific tests are defined using these functions in a configuration file, and tests are run on hosts to ensure that the system is correctly configured.

The Checker interface is a superset of the amboy.Job interface. In most cases, specific check implementations inculde the check.Base type, which contains implementations of most methods (except for Run()) required by this interface.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AddFactory 

func AddFactory(name string, factory ResultsFactory)

AddFactory provides a mechanism for adding additional results output to output registry.

func SetupLogging 

func SetupLogging(format string, fileName string) error

SetupLogging is a helper to configure the global grip logging instance, and is used in the main package to configure logging for the Greebay Service. Reconfigures the logging backend for the process' default "grip" logging instance.

Types

type Application 

type Application struct {
	Output     *OutputOptions
	Conf       *Configuration
	NumWorkers int
	Tests      []string
	Suites     []string
}

Application encapsulates the execution of a greenbay run. You can construct the object, either with NewApplication(), or by building a Application structure yourself.

func NewApplication 

func NewApplication(confPath, outFn, format string, quiet bool, jobs int, suite, tests []string) (*Application, error)

NewApplication configures the greenbay application and manages the construction of the main config object as well as the output configuration structure. Returns an error if there are problems constructing either the main config or the output configuration objects.

func (*Application) Run 

func (a *Application) Run(ctx context.Context) error

Run executes all tasks defined in the application, and produces results as described by the output configuration. Returns an error if any test failed and/or if there were any problems with test execution.

type Builder 

type Builder struct {
	// contains filtered or unexported fields
}

Builder provides an interface to build a Configuration object programmatically. Primarily useful in testing or potentially for building out a greenbay-based test service.

func NewBuilder 

func NewBuilder() *Builder

NewBuilder constructs a fully initialized Builder object.

func (*Builder) AddCheck 

func (b *Builder) AddCheck(check Checker) error

AddCheck takes a greenbay.Checker implementation and adds it to the underlying config object.

func (*Builder) Conf 

func (b *Builder) Conf() (*Configuration, error)

Conf returns a fully constructed and resolved Configuration, returning an error if there are any parsing or validation errors.

Be aware that the config object returned is a *copy* of the config object in the builder, which allows you to generate and mutate generated multiple configs without impacting the builder's config. However, this operation does the correct but peculiar operation of copying a mutex in the config object.

Additionally, in most cases you'll want a *pointer* to this object produced by this method.

func (*Builder) Len 

func (b *Builder) Len() int

Len returns the number of checks represented in the builder config object.

type CheckOutput 

type CheckOutput struct {
	Completed bool       `bson:"completed" json:"completed" yaml:"completed"`
	Passed    bool       `bson:"passed" json:"passed" yaml:"passed"`
	Check     string     `bson:"check_type" json:"check_type" yaml:"check_type"`
	Name      string     `bson:"name" json:"name" yaml:"name"`
	Message   string     `bson:"message,omitempty" json:"message,omitempty" yaml:"message,omitempty"`
	Error     string     `bson:"error,omitempty" json:"error,omitempty" yaml:"error,omitempty"`
	Suites    []string   `bson:"suites" json:"suites" yaml:"suites"`
	Timing    TimingInfo `bson:"timing" json:"timing" yaml:"timing"`
}

CheckOutput provides a standard report format for tests that includes their result status and other metadata that may be useful in reporting data to users.

type Checker 

type Checker interface {
	// SetID modifies the ID reported by the ID() method in the
	// amboy.Job interface.
	SetID(string)

	// Output returns a common output format for all greenbay checks.
	Output() CheckOutput

	// Suites are a list of test suites associated with this check.
	SetSuites([]string)
	Suites() []string

	// Name returns the name of the checker. Use ID(), in the
	// amboy.Job interface to get a unique identifier for the
	// task. This is typically the same as the
	// amboy.Job.Type().Name value.
	Name() string

	// Checker includes the amboy.Job interface.
	amboy.Job
}

Checker is a superset of amboy.Job that includes several other features unique to Greenbay checks. These methods, in addition to all methods in the amboy.Job interface, except for Run(), are implemented by the check.Base type, which specific jobs can compose.

type Client 

type Client struct {
	Conf   *Configuration
	Output *OutputOptions

	Tests  []string
	Suites []string
	// contains filtered or unexported fields
}

Client provides all of the core greenbay operations by doing requests against a remote service, provided by GreenbayService.

func NewClient 

func NewClient(confPath, host string, port int, outFn, format string, quiet bool, suite, tests []string) (*Client, error)

NewClient constructs a greenbay client, with a signature that is roughly analogous to the NewApp constructor used for local greenbay operations.

func (*Client) Run 

func (c *Client) Run(ctx context.Context) error

Run executes all greenbay checks on the remote system. Currently waits for all checks to complete for the default (20 seconds,) or until the context expires. Control the timeout using the context.

type Configuration 

type Configuration struct {
	Options  *options  `bson:"options" json:"options" yaml:"options"`
	RawTests []rawTest `bson:"tests" json:"tests" yaml:"tests"`
	// contains filtered or unexported fields
}

Configuration defines the structure for a single greenbay test run, including execution behavior (options) and check definitions.

func ReadConfig 

func ReadConfig(fn string) (*Configuration, error)

ReadConfig takes a path name to a configuration file (yaml formatted,) and returns a configuration format.

func (*Configuration) GetAllTests 

func (c *Configuration) GetAllTests(tests, suites []string) <-chan JobWithError

GetAllTests returns a channel that produces tests given lists of tests and suites.

func (*Configuration) Reload 

func (c *Configuration) Reload() error

Reload reparses the local test file, and makes it possible to use greenbay as a service and change the test definition without restarting the service.

func (*Configuration) TestsByName 

func (c *Configuration) TestsByName(names ...string) <-chan JobWithError

TestsByName is a generator takes one or more names of tests (as strings) and returns a channel of result objects that contain errors (if those names do not exist) and job objects.

func (*Configuration) TestsForSuites 

func (c *Configuration) TestsForSuites(names ...string) <-chan JobWithError

TestsForSuites takes the name of a suite and then produces a sequence of jobs that are part of that suite.

type GoTest 

type GoTest struct {
	// contains filtered or unexported fields
}

GoTest defines a ResultsProducer implementation that generates output in the format of "go test -v"

func (*GoTest) Populate 

func (r *GoTest) Populate(jobs <-chan amboy.Job) error

Populate generates output, based on the content (via the Results() method) of an amboy.Queue instance. All jobs processed by that queue must also implement the greenbay.Checker interface.

func (*GoTest) Print 

func (r *GoTest) Print() error

Print writes the "go test -v" output to standard output.

func (*GoTest) SkipPassing 

func (r *GoTest) SkipPassing()

SkipPassing causes the reporter to skip all passing tests in the report.

func (*GoTest) ToFile 

func (r *GoTest) ToFile(fn string) error

ToFile writes the "go test -v" output to a file.

type GripOutput 

type GripOutput struct {
	// contains filtered or unexported fields
}

GripOutput provides a ResultsProducer implementation that writes the results of a greenbay run to logging using the grip logging package.

func (*GripOutput) Populate 

func (r *GripOutput) Populate(jobs <-chan amboy.Job) error

Populate generates output messages based on a sequence of amboy.Jobs. All jobs must also implement the greenbay.Checker interface. Returns an error if there are any invalid jobs.

func (*GripOutput) Print 

func (r *GripOutput) Print() error

Print logs, to standard output, the results of the greenbay operation. If any tasks failed, this operation returns an error.

func (*GripOutput) SkipPassing 

func (r *GripOutput) SkipPassing()

SkipPassing causes the reporter to skip all passing tests in the report.

func (*GripOutput) ToFile 

func (r *GripOutput) ToFile(fn string) error

ToFile logs, to the specified file, the results of the greenbay operation. If any tasks failed, this operation returns an error.

type JSONResults 

type JSONResults struct {
	// contains filtered or unexported fields
}

JSONResults provides a structured output JSON format.

func (*JSONResults) Populate 

func (r *JSONResults) Populate(jobs <-chan amboy.Job) error

Populate generates output messages based on a sequence of amboy.Jobs. All jobs must also implement the greenbay.Checker interface. Returns an error if there are any invalid jobs.

func (*JSONResults) Print 

func (r *JSONResults) Print() error

Print logs, to standard output, the results of the greenbay operation. If any tasks failed, this operation returns an error.

func (*JSONResults) SkipPassing 

func (r *JSONResults) SkipPassing()

SkipPassing causes the reporter to skip all passing tests in the report.

func (*JSONResults) ToFile 

func (r *JSONResults) ToFile(fn string) error

ToFile logs, to the specified file, the results of the greenbay operation. If any tasks failed, this operation returns an error.

type JobWithError 

type JobWithError struct {
	Job amboy.Job
	Err error
}

JobWithError is a type used by the test generators and contains an amboy.Job and an error message.

type OutputOptions 

type OutputOptions struct {
	// contains filtered or unexported fields
}

OutputOptions represents all operations for output generation, and provides methods for accessing and producing results using that configuration regardless of underlying output format.

func NewOutputOptions 

func NewOutputOptions(fn, format string, quiet bool) (*OutputOptions, error)

NewOutputOptions provides a constructor to generate a valid OutputOptions structure. Returns an error if the specified format is not valid or registered.

func (*OutputOptions) CollectResults 

func (o *OutputOptions) CollectResults(jobs <-chan amboy.Job) error

CollectResults takes a channel that produces jobs and produces results according to the options specified in the OutputOptions structure. ProduceResults returns an error if any of the tests failed in the operation.

func (*OutputOptions) GetResultsProducer 

func (o *OutputOptions) GetResultsProducer() (ResultsProducer, error)

GetResultsProducer returns the ResultsProducer implementation specified in the OutputOptions structure, and returns an error if the format specified in the structure does not refer to a registered type.

func (*OutputOptions) ProduceResults 

func (o *OutputOptions) ProduceResults(ctx context.Context, q amboy.Queue) error

ProduceResults takes an amboy.Queue object and produces results according to the options specified in the OutputOptions structure. ProduceResults returns an error if any of the tests failed in the operation.

func (*OutputOptions) Report 

func (o *OutputOptions) Report(jobs <-chan amboy.Job) (map[string]*CheckOutput, error)

Report produces the results of a test run in a parseable map structure for programmatic use.

type Report 

type Report struct {
	// contains filtered or unexported fields
}

Report implements a single machine-parsable json format for results, for use in the rest API

func (*Report) Populate 

func (r *Report) Populate(jobs <-chan amboy.Job) error

Populate generates output, based on the content (via the Results() method) of an amboy.Queue instance. All jobs processed by that queue must also implement the greenbay.Checker interface.

func (*Report) Print 

func (r *Report) Print() error

Print generates the JSON report and writes the output via a fmt.Println operation.

func (*Report) SkipPassing 

func (r *Report) SkipPassing()

SkipPassing causes the reporter to skip all passing tests in the report.

func (*Report) ToFile 

func (r *Report) ToFile(fn string) error

ToFile writes a JSON report to the specified file.

type Results 

type Results struct {
	// contains filtered or unexported fields
}

Results defines a ResultsProducer implementation for the Evergreen results.json output format.

func (*Results) Populate 

func (r *Results) Populate(jobs <-chan amboy.Job) error

Populate generates output, based on the content (via the Results() method) of an amboy.Queue instance. All jobs processed by that queue must also implement the greenbay.Checker interface.

func (*Results) Print 

func (r *Results) Print() error

Print writes, to standard output, the results.json data.

func (*Results) SkipPassing 

func (r *Results) SkipPassing()

SkipPassing causes the reporter to skip all passing tests in the report.

func (*Results) ToFile 

func (r *Results) ToFile(fn string) error

ToFile writes results.json output output to the specified file.

type ResultsFactory 

type ResultsFactory func() ResultsProducer

ResultsFactory defines the signature used by constructor functions for implementations of the ResultsProducer interface.

func GetResultsFactory 

func GetResultsFactory(name string) (ResultsFactory, bool)

GetResultsFactory provides a public mechanism for accessing constructors for result formats.

type ResultsProducer 

type ResultsProducer interface {
	// Populate takes an amboy.Queue instance that contains
	// completed greenbay.Checker instances to produce
	// output. Returns an error if the queue contained Job
	// instances that do not implement
	// greenbay.Checker. Implementations are not required to
	// de-duplicate tasks in the case that the Populate() method
	// is called multiple times on
	Populate(<-chan amboy.Job) error

	// ToFile takes a string, for a file name, and writes the
	// results to a file with that name. Returns an error if any
	// of the tasks did not pass. You may call this method
	// multiple times.
	ToFile(string) error

	// Print prints, to standard output, the results in a given
	// format. Returns an error if the results in the format have
	// any failed checks.
	Print() error

	// SkipPassing causes the reporter to skip all passing tests
	// in the report.
	SkipPassing()
}

ResultsProducer defines a common interface for generating results in different formats.

type Service 

type Service struct {
	DisableStats bool
	// contains filtered or unexported fields
}

Service holds the configuration and operations for running a Greenbay service.

func NewService 

func NewService(confPath string, host string, port int) (*Service, error)

NewService constructs a Service, but does not start the service. You will need to run Open to start the underlying workers and Run to start the HTTP service. You can set the host to the empty string, to bind the service to all interfaces.

func (*Service) Close 

func (s *Service) Close()

Close wraps the Close method from amboy.rest.Service, and releases all resources used by the queue.

func (*Service) Open 

func (s *Service) Open(ctx context.Context, opts rest.QueueServiceOptions) error

Open starts the service, using the configuration structure from the amboy.rest package to set the queue size, number of workers, and timeout when restarting the service.

func (*Service) Run 

func (s *Service) Run(ctx context.Context)

Run wraps the Run method from amboy.rest.Service, and is responsible for starting the service. This method blocks until the service terminates.

type TimingInfo 

type TimingInfo struct {
	Start time.Time `bson:"start_time" json:"start_time" yaml:"start_time"`
	End   time.Time `bson:"end_time" json:"end_time" yaml:"end_time"`
}

TimingInfo tracks the start and end time for a task.

func (TimingInfo) Duration 

func (t TimingInfo) Duration() time.Duration

Duration returns a time.Duration for the timing information stored in the TimingInfo object.

Source Files

View all Source files

Directories

Path Synopsis
Package check provides implementation of check functions or jobs that are used in system validation.
 Package check provides implementation of check functions or jobs that are used in system validation.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.