Running a test suite

Running a test suite executes a resolved set of Tests against the same Environment.

The individual tests use the same execution flow described in Running a test. A test suite adds test resolution, a shared runtime environment, and an aggregate result.

Execution flow

At a high level, a test suite runs through the following stages:

Environment + TestSuite
        │
        ▼
Resolve selected Tests
        │
        ▼
Create TestSuiteExecution
      scheduled
        │
        ▼
Create child TestExecutions
      scheduled
        │
        ▼
Schedule Tests
        │
        ▼
Acquire shared runtime environment
        │
        ▼
TestSuiteExecution
       running
        │
        ├── TestExecution A ──▶ passed
        ├── TestExecution B ──▶ failed
        └── TestExecution C ──▶ passed
        │
        ▼
Aggregate results
        │
        ▼
Record suite result
 passed / failed / error
        │
        ▼
Release runtime environment

Resolving tests

Before execution begins, E2Engine resolves the TestSuite selectors into concrete Tests.

For example:

spec:
  selectors:
    tags:
      - smoke

Selectors by ID, name, and tag are combined as a union. Tests selected more than once are deduplicated by resource ID.

The resolved Tests are ordered deterministically by resource ID before execution.

If no Tests are resolved, the suite is not started.

The number of resolved Tests is also limited by the configured maximum.

See Test suites for the complete selector semantics.

Creating executions

E2Engine creates a TestSuiteExecution for the suite and a separate TestExecution for every resolved Test.

Initially, all executions have the scheduled status.

Conceptually:

TestSuiteExecution
        │
        ├── TestExecution A
        ├── TestExecution B
        └── TestExecution C

Each child TestExecution has its own execution ID and records its own request, response, observed calls, deviations, and final status.

Shared runtime environment

Tests in a suite execute against the same runtime instance of the selected Environment.

                  TestSuiteExecution
                         │
                         ▼
                Runtime Environment
                  /      |       \
                 /       |        \
                ▼        ▼         ▼
             Test A   Test B    Test C

The runtime environment is created when execution begins and is identified by the suite execution.

Its services, routing, and call store are therefore shared by the suite, while observed calls remain associated with the individual TestExecution that produced them.

This allows each Test to evaluate its own interaction expectations independently while using the same runtime environment.

Executing tests

Each resolved Test follows the normal test execution flow:

1. its TestExecution changes to running;
2. its request is executed;
3. the response is evaluated;
4. service interactions are evaluated;
5. the result is recorded as passed, failed, or error.

Tests may be executed concurrently according to the runner configuration.

See Running a test for details of the individual test execution flow.

Completing the suite

The TestSuiteExecution remains active until every child TestExecution has completed.

Once all Tests have reached a terminal status, E2Engine creates the suite summary:

Field	Meaning
total	Total number of resolved Tests
passed	Tests with status passed
failed	Tests with status failed
errors	Tests with status error

The final suite status is derived from the child executions:

Child results	Suite status
All Tests passed	passed
At least one Test failed and none ended with an error	failed
At least one Test ended with an error	error

An error therefore takes precedence over failed, and failed takes precedence over passed.

The suite completion time corresponds to the latest completion time among its child TestExecutions.

Once the suite is complete, E2Engine releases the shared runtime environment.

See Executions for the TestSuiteExecution model and its summary.