Test suites

A TestSuite describes a set of tests that should be executed together.

Rather than embedding test definitions, a test suite selects existing tests by:

ID;
name;
tag;
or a combination of these selectors.

This keeps test definitions independent while allowing them to be organized into reusable groups such as smoke tests, regression suites, or subsystem-specific test sets.

The test suite model

At a high level, a test suite is a set of selectors:

TestSuite
    │
    └── Selectors
          ├── IDs
          ├── Names
          └── Tags

For example:

kind: TestSuite
version: 1.0.0
name: smoke

spec:
  selectors:
    tags:
      - smoke

This suite selects tests carrying the smoke tag.

If the following tests exist:

successful-payment    tags: [smoke]
fraud-rejection       tags: [smoke]
account-rejection     tags: [smoke]
large-regression      tags: [regression]

the suite resolves to:

smoke
 │
 ├── successful-payment
 ├── fraud-rejection
 └── account-rejection

The test definitions themselves remain separate resources.

TestSuite resource

A test suite is a versioned E2Engine resource:

kind: TestSuite
version: 1.0.0
name: smoke
description: payment demo smoke tests

spec:
  selectors:
    tags:
      - smoke

Resource fields

Field	Type	Required	Description
`kind`	string	yes	Resource kind. Must be `TestSuite`.
`id`	string	no	E2Engine resource ID. Assigned when the resource is created.
`name`	string	yes	Test suite name. Between 3 and 200 characters.
`version`	string	yes	Test suite specification version. Must be a semantic version.
`description`	string	no	Human-readable description. Between 3 and 2000 characters when specified.
`spec`	object	yes	Test suite-specific configuration.
`created_at`	timestamp	no	Creation timestamp assigned to the stored resource.
`updated_at`	timestamp	no	Last-update timestamp assigned to the stored resource.

When defining a test suite in a file, you normally provide kind, name, version, description, and spec.

Fields such as id, created_at, and updated_at belong to the stored resource and are managed by E2Engine.

Selectors

spec.selectors determines which tests belong to the suite.

spec:
  selectors:
    ids:
      # ...

    names:
      # ...

    tags:
      # ...

At least one selector must be specified.

Selector fields

Field	Type	Required	Description
`ids`	array	no	Test IDs to select.
`names`	array	no	Test names to select.
`tags`	array	no	Test tags to select.

The three selector types can be used independently or together.

Selecting by tag

Tags are useful when a suite represents a logical category of tests.

For example, the E2Engine demo tests all carry the smoke tag:

kind: Test
version: 1.0.0
name: successful-payment

spec:
  tags:
    - smoke

  # ...

The suite can select them with:

kind: TestSuite
version: 1.0.0
name: smoke

spec:
  selectors:
    tags:
      - smoke

This keeps suite membership in the test metadata rather than requiring the suite to enumerate every test.

Conceptually:

                     tag: smoke
                         │
            ┌────────────┼────────────┐
            │            │            │
            ▼            ▼            ▼
      successful-     fraud-       account-
        payment      rejection     rejection
            ▲            ▲            ▲
            └────────────┼────────────┘
                         │
                    smoke suite

A new test can participate in the same logical group by using the same tag.

Selecting by name

Tests can also be selected explicitly by name:

spec:
  selectors:
    names:
      - successful-payment
      - fraud-rejection
      - account-rejection

This is useful when the suite should contain a specific set of named tests rather than every test sharing a classification.

For example:

kind: TestSuite
version: 1.0.0
name: payment-core

spec:
  selectors:
    names:
      - successful-payment
      - fraud-rejection
      - account-rejection

The suite now expresses its membership directly:

payment-core
    │
    ├── successful-payment
    ├── fraud-rejection
    └── account-rejection

Selecting by ID

Tests can be selected by resource ID:

spec:
  selectors:
    ids:
      - ae619f9418f89041a5f6e0e952eae9fb9495e2c271b66739ffd6fae46f9ff30a

ID selection identifies a particular stored test resource directly.

This is useful when exact resource identity matters more than a human-readable name or logical tag.

Combining selectors

Selector types can be combined in the same suite:

spec:
  selectors:
    ids:
      - ae619f9418f89041a5f6e0e952eae9fb9495e2c271b66739ffd6fae46f9ff30a

    names:
      - successful-payment

    tags:
      - smoke

Selectors use OR semantics.

A test is included when it matches any configured selector:

selected by ID
      OR
selected by name
      OR
selected by tag
      │
      ▼
   TestSuite

The resulting tests are combined into one set.

If the same test is selected more than once—for example, explicitly by name and also through a tag—it is included only once.

For example:

selectors:
  names:
    - successful-payment

  tags:
    - smoke

If successful-payment also has the smoke tag, it is still executed only once.

Conceptually:

name: successful-payment ──┐
                           ├──▶ successful-payment
tag: smoke ────────────────┘

This makes it possible to combine explicitly selected tests with broader groups without creating duplicate executions.

Why suites use selectors

A test suite does not contain copies of its tests:

TestSuite
    │
    ├── Test definition
    ├── Test definition
    └── Test definition

Instead, it contains selection criteria:

TestSuite
    │
    └── Selectors
           │
           ▼
       existing Tests

This separation has several useful properties.

Tests remain independently addressable E2Engine resources:

Test
├── name
├── version
├── request
└── expectations

while a suite only describes how tests are grouped:

TestSuite
└── selectors

The same test can therefore be selected by different suites without duplicating its definition.

For example:

successful-payment
       ▲
       │
   ┌───┴──────────┐
   │              │
 smoke        payments
 suite           suite

Tags and suites

Tags are particularly useful when suite membership represents a property of the test.

For example:

tags:
  - smoke
  - payments

allows the same test to participate in different logical groups.

A smoke suite can select:

selectors:
  tags:
    - smoke

while another suite can select:

selectors:
  tags:
    - payments

The test itself remains unchanged.

This provides a lightweight way to organize a growing collection of tests without introducing dependencies between test resources and suite resources.

Test resolution

When a test suite is executed, E2Engine resolves its selectors into a concrete set of tests.

Resolution follows these rules:

IDs resolve individual tests;
names resolve individual tests;
tags resolve all tests carrying that tag;
all resolved tests are combined using OR semantics;
duplicate tests are removed by resource ID;
at least one test must resolve;
the total number of resolved tests cannot exceed the configured runtime limit.

For example:

selectors:
  names:
    - successful-payment

  tags:
    - smoke

might resolve as:

name: successful-payment
          │
          └──────────────┐
                         │
tag: smoke               │
    │                    │
    ├── successful-payment
    ├── fraud-rejection  │
    └── account-rejection
                         │
                         ▼
               deduplicated set
                         │
            ┌────────────┼────────────┐
            ▼            ▼            ▼
       successful-    fraud-       account-
         payment     rejection     rejection

Although successful-payment was found through two selectors, it appears only once in the resolved set.

After resolution, E2Engine orders the tests deterministically by resource ID before starting the suite execution.

Resolution happens at execution time

A test suite stores selectors, not a resolved list of tests.

For example:

selectors:
  tags:
    - smoke

does not permanently store the tests currently carrying the smoke tag.

Instead:

TestSuite
    │
    │ selectors
    ▼
tag: smoke
    │
    │ resolve when executed
    ▼
current matching Tests

This means that tag-based suite membership can change as tests are created or updated.

The suite continues to describe the selection rule rather than a snapshot of its previous matches.

No tests resolved

A test suite must contain at least one selector, but a valid selector does not necessarily guarantee that a test will resolve when the suite is executed.

For example:

selectors:
  tags:
    - nonexistent-tag

is structurally valid.

If no tests currently have that tag, however, the suite resolves to an empty set and cannot be executed.

tag: nonexistent-tag
          │
          ▼
      no matches
          │
          ▼
   execution rejected

Resolution limit

E2Engine limits the number of tests that a suite can resolve.

If the combined selector result exceeds the configured runtime limit, the suite execution is rejected rather than silently dropping tests.

This protects suite execution from unexpectedly broad selectors.

For example, a tag intended for a small smoke suite could accidentally match a much larger collection of tests. The configured limit provides an explicit boundary on the resulting execution.

Complete example

The E2Engine demo contains three tests:

successful-payment
fraud-rejection
account-rejection

Each is tagged:

spec:
  tags:
    - smoke

The complete smoke suite is therefore:

kind: TestSuite
version: 1.0.0
name: smoke
description: payment demo smoke tests

spec:
  selectors:
    tags:
      - smoke

When the suite is executed, the tag is resolved:

                       smoke
                     TestSuite
                        │
                        │ tag: smoke
                        ▼
                  resolve tests
                        │
             ┌──────────┼──────────┐
             │          │          │
             ▼          ▼          ▼
       successful-   fraud-     account-
         payment    rejection   rejection

All three tests remain independent resources. The suite provides the selection that groups them for execution.

Validation rules

E2Engine validates test suite specifications before they are used.

The main resource-level rules are:

kind must be TestSuite;
name must contain between 3 and 200 characters;
version must be a semantic version;
description, when present, must contain between 3 and 2000 characters.

For selectors:

at least one of ids, names, or tags must contain a value;
every ID must contain between 1 and 200 characters;
every name must contain between 1 and 200 characters;
every tag must contain between 1 and 200 characters;
values within ids must be unique;
values within names must be unique;
values within tags must be unique.

For example, this is invalid because it contains no selectors:

spec:
  selectors: {}

This is also invalid:

spec:
  selectors:
    tags:
      - smoke
      - smoke

because selector values within the same selector type must be unique.

A valid suite must identify at least one selection criterion:

spec:
  selectors:
    tags:
      - smoke

Validation and resolution are separate steps.

A suite such as:

spec:
  selectors:
    tags:
      - unknown

can pass specification validation because it contains a valid selector, but still fail to execute if that selector resolves to no tests.

Test suites and tests

A Test answers:

What behavior should E2Engine execute and verify?

A TestSuite answers:

Which tests should be executed together?

The distinction is important.

A test owns its request and expectations:

Test
 │
 ├── Request
 └── Expectations

A suite owns only the selection:

TestSuite
 │
 └── Selectors
       │
       ▼
      Tests

This keeps execution behavior in tests and grouping behavior in suites.

Test suites and environments

A suite does not define an environment.

The environment is selected when the suite is executed:

              TestSuite
                  │
                  │ selects
                  ▼
                Tests
                  │
                  │ execute against
                  ▼
              Environment

This means the suite describes a collection of tests independently of the environment used for a particular execution.

The Executions model brings these resources together:

Environment + TestSuite
          │
          ▼
 TestSuiteExecution
          │
          ▼
   TestExecutions

The environment defines the topology.

The tests define the behavior.

The test suite defines the selection.

The execution brings them together.