temporalio-omes

Omes - a load generator for Temporal

This project is for testing load generation scenarios against Temporal. This is primarily used by the Temporal team to benchmark features and situations. Backwards compatibility may not be maintained.

Why the weird name?

Omes (pronounced oh-mess) is the Hebrew word for "load" (עומס).

Prerequisites

• Go 1.20+
• Node 16+
• Python 3.10+
  • Poetry: poetry install (More TBD when we support workers in other languages)

Installation

There's no need to install anything to use this, it's a self-contained Go project.

Usage

Define a scenario

Scenarios are defined using plain Go code. They are located in the scenarios folder. There are already multiple defined that can be used.

A scenario must select an Executor. The most common is the KitchenSinkExecutor which is a wrapper on the GenericExecutor specific for executing the Kitchen Sink workflow. The Kitchen Sink workflow accepts actions and is implemented in every worker language.

For example, here is scenarios/workflow_with_single_noop_activity.go:

func init() {
	loadgen.MustRegisterScenario(loadgen.Scenario{
		Description: "Each iteration executes a single workflow with a noop activity.",
		Executor: loadgen.KitchenSinkExecutor{
			WorkflowParams: kitchensink.NewWorkflowParams(kitchensink.NopActionExecuteActivity),
		},
	})
}

NOTE: The file name where the Register function is called, will be used as the name of the scenario.

The executor has other options such as altering the workflow parameters based on

Scenario Authoring Guidelines

1. Use snake case for scenario file names.
2. Use KitchenSinkExecutor for most basic scenarios, adding common/generic actions as need, but for unique scenarios use GenericExecutor.
3. When using GenericExecutor, use methods of *loadgen.Run in your Execute as much as possible.
4. Liberally add helpers to the loadgen package that will be useful to other scenario authors.

Run a worker for a specific language SDK

$ go run ./cmd run-worker --scenario workflow_with_single_noop_activity --run-id local-test-run --language go

Notes:

• --embedded-server can be passed here to start an embedded localhost server
• --task-queue-suffix-index-start and --task-queue-suffix-index-end represent an inclusive range for running the worker on multiple task queues. The process will create a worker for every task queue from <task-queue>-<start> through <task-queue>-end. This only applies to multi-task-queue scenarios.

Run a test scenario

$ go run ./cmd run-scenario --scenario workflow_with_single_noop_activity --run-id local-test-run

Notes:

• Run ID is used to derive ID prefixes and the task queue name, it should to start a worker on the correct task queue and by the cleanup script
• By default the number of iterations or duration is specified in the scenario config, those can be overridden with CLI flags
• See help output for available flags

Cleanup after scenario run

$ go run ./cmd cleanup-scenario --scenario workflow_with_single_noop_activity --run-id local-test-run

Run scenario with worker - Start a worker, an optional dev server, and run a scenario

$ go run ./cmd run-scenario-with-worker --scenario workflow_with_single_noop_activity --language go --embedded-server

Notes:

• Cleanup is not automatically performed here
• Accepts combined flags for run-worker and run-scenario commands

Building and publishing docker images

For example, to build a go worker image using v1.24.0 of the Temporal Go SDK:

$ go run ./cmd build-worker-image --language go --version v1.24.0

This will produce an image tagged like <current git commit hash>-go-v1.24.0.

Publishing images is typically done via CI, using the push-images command. See the GHA workflows for more.

Design decisions

Kitchen Sink Workflow

The Kitchen Sink workflows accepts a DSL generated by the kitchen-sink-gen Rust tool, allowing us to test a wide variety of scenarios without having to imagine all possible edge cases that could come up in workflows. Input may be saved for regression testing, or hand written for specific cases.

Scenario Failure

A scenario can only fail if an Execute method returns an error, that means the control is fully in the scenario authors's hands. For enforcing a timeout for a scenario, use options like workflow execution timeouts or write a workflow that waits for a signal for a configurable amount of time.

TODO

• Nicer output that includes resource utilization for the worker (when running all-in-one)
• More lang workers

Fuzzer trophy case

• Python upsert SA with no initial attributes: PR