e2e/

End-to-end Tests

Structure

e2e Package

The e2e package defines an integration testing suite used for full end-to-end testing functionality. This package is decoupled from depending on the Mokita codebase. It initializes the chains for testing via Docker files. As a result, the test suite may provide the desired Mokita version to Docker containers during the initialization. This design allows for the opportunity of testing chain upgrades in the future by providing an older Mokita version to the container, performing the chain upgrade, and running the latest test suite. When testing a normal upgrade, the e2e test suite submits an upgrade proposal at an upgrade height, ensures the upgrade happens at the desired height, and then checks that operations that worked before still work as intended. If testing a fork, the test suite instead starts the chain a few blocks before the set fork height and ensures the chain continues after the fork triggers the upgrade. Note that a regular upgrade and a fork upgrade are mutually exclusive.

The file e2e_setup_test.go defines the testing suite and contains the core bootstrapping logic that creates a testing environment via Docker containers. A testing network is created dynamically with 2 test validators.

The file e2e_test.go contains the actual end-to-end integration tests that utilize the testing suite.

Currently, there is a single IBC test in e2e_test.go.

Additionally, there is an ability to disable certain components of the e2e suite. This can be done by setting the environment variables. See "Environment variables" section below for more details.

How It Works

Conceptually, we can split the e2e setup into 2 parts:

1. Chain Initialization

   The chain can either be initialized off of the current branch, or off the prior mainnet release and then upgraded to the current branch.

   If current, we run chain initialization off of the current Git branch by calling chain.Init(...) method in the configurer/current.go.

   If with the upgrade, the same chain.Init(...) function is run inside a Docker container of the previous Mokita version, inside configurer/upgrade.go. This is needed to initialize chain configs and the genesis of the previous version that we are upgrading from.

   The decision of what configuration type to use is decided by the Configurer. This is an interface that has CurrentBranchConfigurer and UpgradeConfigurer implementations. There is also a BaseConfigurer which is shared by the concrete implementations. However, the user of the configurer package does not need to know about this detail.

   When the desired configurer is created, the caller may configure the chain in the desired way as follows:

   conf, _ := configurer.New(..., < isIBCEnabled bool >, < isUpgradeEnabled bool >)
   
   conf.ConfigureChains()
   

   The caller (e2e setup logic), does not need to be concerned about what type of configurations is hapenning in the background. The appropriate logic is selected depending on what the values of the arguments to configurer.New(...) are.

   The configurer constructor is using a factory design pattern to decide on what kind of configurer to return. Factory design pattern is used to decouple the client from the initialization details of the configurer. More on this can be found here

   The rules for deciding on the configurer type are as follows:

   • If only isIBCEnabled, we want to have 2 chains initialized at the current branch version of Mokita codebase

   • If only isUpgradeEnabled, that's invalid (we can decouple upgrade testing from IBC in a future PR)

   • If both isIBCEnabled and isUpgradeEnabled, we want 2 chain with IBC initialized at the previous Mokita version

   • If none are true, we only need one chain at the current branch version of the Mokita code

2. Setting up e2e components

   Currently, there exist the following components:

   • Base logic
     • This is the most basic type of setup where a single chain is created
     • It simply spins up the desired number of validators on a chain.
   • IBC testing
     • 2 chains are created connected by Hermes relayer
     • Upgrade Testing
     • 2 chains of the older Mokita version are created, and connected by Hermes relayer
   • Upgrade testing
     • CLI commands are run to create an upgrade proposal and approve it
     • Old version containers are stopped and the upgrade binary is added
     • Current branch Mokita version is spun up to continue with testing
   • State Sync Testing (WIP)
     • An additional full node is created after a chain has started.
     • This node is meant to state sync with the rest of the system.

   This is done in configurer/setup_runner.go via function decorator design pattern where we chain the desired setup components during configurer creation. Example

initialization Package

The initialization package introduces the logic necessary for initializing a chain by creating a genesis file and all required configuration files such as the app.toml. This package directly depends on the Mokita codebase.

upgrade Package

The upgrade package starts chain initialization. In addition, there is a Dockerfile init-e2e.Dockerfile. When executed, its container produces all files necessary for starting up a new chain. These resulting files can be mounted on a volume and propagated to our production mokita container to start the mokitad service.

The decoupling between chain initialization and start-up allows to minimize the differences between our test suite and the production environment.

containers Package

Introduces an abstraction necessary for creating and managing Docker containers. Currently, validator containers are created with a name of the corresponding validator struct that is initialized in the chain package.

Running From Current Branch

To build chain initialization image

Please refer to tests/e2e/initialization/README.md

To build the debug Mokita image

    make docker-build-e2e-debug

### Environment variables

Some tests take a long time to run. Sometimes, we would like to disable them
locally or in CI. The following are the environment variables to disable
certain components of e2e testing.

- `MOKITA_E2E_SKIP_UPGRADE` - when true, skips the upgrade tests.
If MOKITA_E2E_SKIP_IBC is true, this must also be set to true because upgrade
tests require IBC logic.

- `MOKITA_E2E_SKIP_IBC` - when true, skips the IBC tests tests.

- `MOKITA_E2E_SKIP_STATE_SYNC` - when true, skips the state sync tests.

- `MOKITA_E2E_SKIP_CLEANUP` - when true, avoids cleaning up the e2e Docker
containers.

- `MOKITA_E2E_FORK_HEIGHT` - when the above "IS_FORK" env variable is set to true, this is the string
of the height in which the network should fork. This should match the ForkHeight set in constants.go

- `MOKITA_E2E_UPGRADE_VERSION` - string of what version will be upgraded to (for example, "v10")

- `MOKITA_E2E_DEBUG_LOG` - when true, prints debug logs from executing CLI commands
via Docker containers. Set to trus in CI by default.

#### VS Code Debug Configuration

This debug configuration helps to run e2e tests locally and skip the desired tests.

```json
{
    "name": "E2E IntegrationTestSuite",
    "type": "go",
    "request": "launch",
    "mode": "test",
    "program": "${workspaceFolder}/tests/e2e",
    "args": [
        "-test.timeout",
        "30m",
        "-test.run",
        "IntegrationTestSuite",
        "-test.v"
    ],
    "buildFlags": "-tags e2e",
    "env": {
        "MOKITA_E2E_SKIP_IBC": "true",
        "MOKITA_E2E_SKIP_UPGRADE": "true",
        "MOKITA_E2E_SKIP_CLEANUP": "true",
        "MOKITA_E2E_SKIP_STATE_SYNC": "true",
        "MOKITA_E2E_UPGRADE_VERSION": "v10",
        "MOKITA_E2E_DEBUG_LOG": "true",
        "MOKITA_E2E_FORK_HEIGHT": "4713065" # this is v10 fork height.
    }
}

Common Problems

Please note that if the tests are stopped mid-way, the e2e framework might fail to start again due to duplicated containers. Make sure that containers are removed before running the tests again: docker containers rm -f $(docker containers ls -a -q).

Additionally, Docker networks do not get auto-removed. Therefore, you can manually remove them by running docker network prune.