Documentation
¶
Overview ¶
Package scenario provides a high-level testing framework for running Bacalhau jobs in different configurations and making assertions against the results.
The unit of measure is the `Scenario` which describes a Bacalhau network, a job to be submitted to it, and a set of checks to exercise that the job was executed as expected.
Scenarios can be used in standalone way (see `pkg/test/executor/test_runner.go`) or using the provided `ScenarioRunner` can be used.
As well as executing jobs against real executors, a Scenario can instead use the NoopExecutor to implement a mocked out job. This makes is easier to test network features without needing to invent a real job.
Example (Basic) ¶
package main
import (
"testing"
"github.com/bacalhau-project/bacalhau/pkg/downloader"
"github.com/stretchr/testify/suite"
legacy_job "github.com/bacalhau-project/bacalhau/pkg/legacyjob"
"github.com/bacalhau-project/bacalhau/pkg/model"
testutils "github.com/bacalhau-project/bacalhau/pkg/test/utils"
)
func basicScenario(t testing.TB) Scenario {
return Scenario{
Inputs: ManyStores(
StoredText("hello, world!", "/inputs"),
StoredFile("../../../testdata/wasm/cat/main.wasm", "/job"),
),
Outputs: []model.StorageSpec{},
ResultsChecker: FileEquals(downloader.DownloadFilenameStdout, "hello, world!\n"),
JobCheckers: WaitUntilSuccessful(1),
Spec: testutils.MakeSpecWithOpts(t,
legacy_job.WithEngineSpec(
// TODO(forrest): [correctness] this isn't a valid wasm engine spec - it needs an entry module
// but leaving as is to preserve whatever behaviour this test is after.
model.NewWasmEngineBuilder(model.StorageSpec{}).
WithEntrypoint("_start").
Build(),
),
),
}
}
type ExampleTest struct {
ScenarioRunner
}
func main() {
// In a real example, use the testing.T passed to the TestXxx method.
suite.Run(&testing.T{}, new(ExampleTest))
}
func (suite *ExampleTest) TestRun() {
suite.RunScenario(basicScenario(suite.T()))
}
Output:
Example (Noop) ¶
package main
import (
"context"
"strings"
"testing"
"github.com/bacalhau-project/bacalhau/pkg/downloader"
"github.com/stretchr/testify/suite"
"github.com/bacalhau-project/bacalhau/pkg/models"
"github.com/bacalhau-project/bacalhau/pkg/executor"
"github.com/bacalhau-project/bacalhau/pkg/executor/noop"
"github.com/bacalhau-project/bacalhau/pkg/system"
testutils "github.com/bacalhau-project/bacalhau/pkg/test/utils"
)
func noopScenario(t testing.TB) Scenario {
return Scenario{
Stack: &StackConfig{
ExecutorConfig: noop.ExecutorConfig{
ExternalHooks: noop.ExecutorConfigExternalHooks{
JobHandler: func(ctx context.Context, jobID string, resultsDir string) (*models.RunCommandResult, error) {
return executor.WriteJobResults(resultsDir, strings.NewReader("hello, world!\n"), nil, 0, nil, executor.OutputLimits{
MaxStdoutFileLength: system.MaxStdoutFileLength,
MaxStdoutReturnLength: system.MaxStdoutReturnLength,
MaxStderrFileLength: system.MaxStderrFileLength,
MaxStderrReturnLength: system.MaxStderrReturnLength,
}), nil
},
},
},
},
Spec: testutils.MakeSpecWithOpts(t),
ResultsChecker: FileEquals(downloader.DownloadFilenameStdout, "hello, world!\n"),
JobCheckers: WaitUntilSuccessful(1),
}
}
type NoopTest struct {
ScenarioRunner
}
func main() {
// In a real example, use the testing.T passed to the TestXxx method.
suite.Run(&testing.T{}, new(NoopTest))
}
func (suite *NoopTest) TestRunNoop() {
suite.RunScenario(noopScenario(suite.T()))
}
Output:
Index ¶
- func GetAllScenarios(t testing.TB) map[string]Scenario
- func InlineData(data []byte) model.StorageSpec
- func WaitUntilSuccessful(nodes int) []legacy_job.CheckStatesFunction
- type CheckResults
- type CheckSubmitResponse
- type Scenario
- func AwkFile(t testing.TB) Scenario
- func CatFileToStdout(t testing.TB) Scenario
- func CatFileToVolume(t testing.TB) Scenario
- func GrepFile(t testing.TB) Scenario
- func SedFile(t testing.TB) Scenario
- func WasmCsvTransform(t testing.TB) Scenario
- func WasmDynamicLink(t testing.TB) Scenario
- func WasmEnvVars(t testing.TB) Scenario
- func WasmExitCode(t testing.TB) Scenario
- func WasmHelloWorld(t testing.TB) Scenario
- func WasmLogTest(t testing.TB) Scenario
- type ScenarioRunner
- type ScenarioTestSuite
- type SetupStorage
- func ManyStores(stores ...SetupStorage) SetupStorage
- func PartialAdd(numberOfNodes int, store SetupStorage) SetupStorage
- func StoredFile(filePath string, mountPath string) SetupStorage
- func StoredText(fileContents string, mountPath string) SetupStorage
- func URLDownload(server *httptest.Server, urlPath string, mountPath string) SetupStorage
- type StackConfig
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func InlineData ¶
func InlineData(data []byte) model.StorageSpec
InlineFile will store the file directly inline in the storage spec. Unlike the other storage set-ups, this function loads the file immediately. This makes it possible to store things deeper into the Spec object without the test system needing to know how to prepare them.
func WaitUntilSuccessful ¶
func WaitUntilSuccessful(nodes int) []legacy_job.CheckStatesFunction
WaitUntilSuccessful returns a set of job.CheckStatesFunctions that will wait until the job they are checking reaches the Completed state on the passed number of nodes. The checks will fail if any job errors.
Types ¶
type CheckResults ¶
A CheckResults is a function that will examine job output that has been written to storage and assert something about it. If the condition it is checking is false, it returns an error, else it returns nil.
func FileContains ¶
func FileContains( outputFilePath string, expectedStrings []string, expectedLines int, ) CheckResults
FileContains returns a CheckResults that asserts that the expected string is in the output file and that the file itself is of the correct size. If expectedLine is set to -1 then a line-check is not performed.
func FileEquals ¶
func FileEquals( outputFilePath string, expectedString string, ) CheckResults
FileEquals returns a CheckResults that asserts that the expected string is exactly equal to the full contents of the output file.
func ManyChecks ¶
func ManyChecks(checks ...CheckResults) CheckResults
ManyCheckes returns a CheckResults that runs the passed checkers and returns an error if any of them fail.
type CheckSubmitResponse ¶
A CheckSubmitResponse is a function that will examine and validate submitJob response. Useful when validating that a job should be rejected.
func SubmitJobErrorContains ¶
func SubmitJobErrorContains(msg string) CheckSubmitResponse
func SubmitJobFail ¶
func SubmitJobFail() CheckSubmitResponse
SubmitJobFail returns a CheckSubmitResponse that asserts an error was returned when submitting a job.
func SubmitJobSuccess ¶
func SubmitJobSuccess() CheckSubmitResponse
SubmitJobSuccess returns a CheckSubmitResponse that asserts no error was returned when submitting a job.
type Scenario ¶
type Scenario struct {
// An optional set of configuration options that define the network of nodes
// that the job will be run against. When unspecified, the Stack will
// consist of one node with requester and compute nodes set up according to
// their default configuration, and without a Noop executor.
Stack *StackConfig
// Setup routines which define data available to the job.
// If nil, no storage will be set up.
Inputs SetupStorage
// Output volumes that must be available to the job. If nil, no output
// volumes will be attached to the job.
Outputs []model.StorageSpec
// The job specification
Spec model.Spec
// The job deal. If nil, concurrency will default to 1.
Deal model.Deal
// A function that will assert submitJob response is as expected.
// if nil, will use SubmitJobSuccess by default.
SubmitChecker CheckSubmitResponse
// A function that will decide whether or not the job was successful. If
// nil, no check will be performed on job outputs.
ResultsChecker CheckResults
// A set of checkers that will decide when the job has completed, and maybe
// whether it was successful or not. If empty, the job will not be waited
// for once it has been submitted.
JobCheckers []legacy_job.CheckStatesFunction
}
A Scenario represents a repeatable test case of submitting a job against a Bacalhau network.
The Scenario defines:
- the topology and configuration of network that is required
- the job that will be submitted
- the conditions for the job to be considered successful or not
Most of the fields in a Scenario are optional and sensible defaults will be used if they are not present. All that is really required is the Spec which details what job to run.
func CatFileToStdout ¶
func CatFileToVolume ¶
func WasmCsvTransform ¶
func WasmDynamicLink ¶ added in v0.3.26
func WasmEnvVars ¶
func WasmExitCode ¶ added in v0.3.24
func WasmHelloWorld ¶
func WasmLogTest ¶ added in v0.3.26
type ScenarioRunner ¶
The ScenarioRunner is an object that can run a Scenario.
It will spin up an appropriate Devstack for the Scenario, submit and wait for the job to complete, and then make assertions against the results of the job.
ScenarioRunner implements a number of testify/suite interfaces making it appropriate as the basis for a test suite. If a test suite composes itself from the ScenarioRunner then default set up and tear down methods that instrument and configure the test will be used. Test suites should not define their own set up or tear down routines.
func (*ScenarioRunner) RunScenario ¶
func (s *ScenarioRunner) RunScenario(scenario Scenario) string
RunScenario runs the Scenario.
Spin up a devstack, execute the job, check the results, and tear down the devstack.
func (*ScenarioRunner) SetupTest ¶
func (s *ScenarioRunner) SetupTest()
type ScenarioTestSuite ¶
type ScenarioTestSuite interface {
suite.SetupTestSuite
suite.TearDownTestSuite
suite.TestingSuite
}
type SetupStorage ¶
type SetupStorage func( ctx context.Context, driverName model.StorageSourceType, ipfsClients ...ipfs.Client, ) ([]model.StorageSpec, error)
A SetupStorage is a function that return a model.StorageSpec representing some data that has been prepared for use by a job. It is the responsibility of the function to ensure that the data has been set up correctly.
func ManyStores ¶
func ManyStores(stores ...SetupStorage) SetupStorage
ManyStores runs all of the passed setups and returns the model.StorageSpecs associated with all of them. If any of them fail, the error from the first to fail will be returned.
func PartialAdd ¶
func PartialAdd(numberOfNodes int, store SetupStorage) SetupStorage
PartialAdd will only store data on a subset of the nodes that it is passed. So if there are 5 IPFS nodes configured and PartialAdd is defined with 2, only the first two nodes will have data loaded.
func StoredFile ¶
func StoredFile( filePath string, mountPath string, ) SetupStorage
StoredFile will store the file at the passed path on an IPFS node, and return the file name and CID in the model.StorageSpec.
func StoredText ¶
func StoredText( fileContents string, mountPath string, ) SetupStorage
StoredText will store the passed string as a file on an IPFS node, and return the file name and CID in the model.StorageSpec.
func URLDownload ¶
func URLDownload( server *httptest.Server, urlPath string, mountPath string, ) SetupStorage
URLDownload will return a model.StorageSpec referencing a file on the passed HTTP test server.
type StackConfig ¶
type StackConfig struct {
*devstack.DevStackOptions
node.ComputeConfig
node.RequesterConfig
noop.ExecutorConfig
}
All the information that is needed to uniquely define a devstack.
Source Files