@probitas-test/runner

Test execution and orchestration engine for Probitas scenarios.

This package provides the ScenarioRunner class that executes scenario definitions built with @probitas-test/builder. It handles the complete test lifecycle including resource initialization, setup/cleanup hooks, step execution with retry logic, and result reporting.

Links

GitHub Repository
@probitas-test/probitas - Main package (recommended for most users)

Package	Description
@probitas-test/builder	Build scenario definitions to execute
@probitas-test/core	Core type definitions
@probitas-test/reporter	Output formatters for results
probitas	CLI that uses these reporters

Key Features

Concurrent execution: Run multiple scenarios in parallel with configurable concurrency
Failure control: Stop after N failures or continue through all tests
Resource lifecycle: Automatic initialization and disposal of resources
Retry logic: Built-in retry with linear/exponential backoff strategies
Reporter integration: Pluggable reporters for custom output formatting
Abort support: Cancel running tests via AbortSignal

Core Exports

ScenarioRunner - Main class for executing scenarios
Skip - Exception class to skip scenarios conditionally
Reporter - Interface for observing test execution events
RunOptions - Configuration options for test runs
RunResult - Aggregated results from a test run
ScenarioResult - Result from executing a single scenario
StepResult - Result from executing a single step

Error Types

StepTimeoutError - Thrown when a step exceeds its timeout
ScenarioTimeoutError - Thrown when a scenario exceeds its timeout

Installation

npm install @probitas-test/runner

Classes

class

#Runner

class Runner

Top-level test runner that orchestrates execution of multiple scenarios.

The Runner manages:

Parallel execution with concurrency control (maxConcurrency)
Early stopping on failures (maxFailures)
Test lifecycle events through the Reporter interface
Aggregated results in RunResult

Examples

import { Runner } from "@probitas-test/runner";
import type { Reporter } from "@probitas-test/runner";
import type { ScenarioDefinition } from "@probitas-test/core";

const reporter: Reporter = {};
const scenarios: ScenarioDefinition[] = [];
const runner = new Runner(reporter);

const result = await runner.run(scenarios, {
  maxConcurrency: 4,
  maxFailures: 1,
});
console.log(result);

Constructor

new Runner(reporter: Reporter)

Create a new Runner with the given reporter.

Name	Description
`reporter`	Reporter instance for lifecycle events
`run`()	Execute all scenarios and return aggregated results.

Properties

readonlyreporterReporter
Reporter instance for lifecycle events

Methods

run(
  scenarios: readonly ScenarioDefinition[],
  options?: RunOptions,
): Promise<RunResult>

Execute all scenarios and return aggregated results.

Emits events through the Reporter:

onRunStart - Before execution starts
For each scenario (concurrently):
- Delegates to ScenarioRunner for scenario-level execution
- ScenarioRunner emits scenario and step level events
onRunEnd - After all scenarios complete with RunResult

Parameters

scenariosreadonly ScenarioDefinition[]
Scenarios to execute
options?RunOptions
Execution options (maxConcurrency, maxFailures, signal)

class

#ScenarioTimeoutError

class ScenarioTimeoutError extends Error

ExtendsError

Error thrown when a scenario execution times out.

Contains structured information about the timeout including the scenario name, timeout duration, total elapsed time, and optionally the step being executed when timeout occurred.

Constructor

new ScenarioTimeoutError(
  scenarioName: string,
  timeoutMs: number,
  elapsedMs: number,
  options?: ErrorOptions & { currentStepIndex?: number; currentStepName?: string },
)

Name	Description
`currentStepIndex`	—
`currentStepName`	—
`elapsedMs`	—
`name`	—
`scenarioName`	—
`timeoutMs`	—
`timeoutSignal`()	Creates an AbortSignal that aborts with ScenarioTimeoutError after the specified timeout.

Properties

readonlycurrentStepIndex?number
readonlycurrentStepName?string
readonlyelapsedMsnumber
readonlyname"ScenarioTimeoutError"
readonlyscenarioNamestring
readonlytimeoutMsnumber

Methods

static timeoutSignal(
  timeout: number,
  params: { scenarioName: string },
): AbortSignal & Disposable

Creates an AbortSignal that aborts with ScenarioTimeoutError after the specified timeout.

Parameters

timeoutnumber
Timeout duration in milliseconds
params{ scenarioName: string }
Scenario information for the error

class

#Skip

class Skip extends Error

ExtendsError

Exception class for conditionally skipping scenario execution.

Throw Skip from any step, setup, or resource fn to skip the entire scenario. Skipped scenarios are counted separately in the summary and don't count as failures.

Common use cases:

Feature flags or environment checks
Prerequisites not met
Platform-specific tests
Temporary test disabling with clear reason

Examples

Skip based on environment

import { scenario } from "@probitas-test/builder";
import { Skip } from "@probitas-test/runner";

const def = scenario("Production Only")
  .step("Check environment", () => {
    if (process.env.ENV !== "production") {
      throw new Skip("Only runs in production");
    }
  })
  .step("Production test", () => {
    console.log("Running in production");
  })
  .build();
console.log(def.name);

Skip based on resource availability

import { scenario } from "@probitas-test/builder";
import { Skip } from "@probitas-test/runner";

const Database = { connect: async () => ({ query: (sql: string) => sql }) };

const def = scenario("Database Test")
  .resource("db", async () => {
    try {
      return await Database.connect();
    } catch {
      throw new Skip("Database not available");
    }
  })
  .step("Query data", (ctx) => ctx.resources.db.query("SELECT 1"))
  .build();
console.log(def.name);

Skip with feature flag

import { scenario } from "@probitas-test/builder";
import { Skip } from "@probitas-test/runner";

const def = scenario("Beta Feature")
  .step("Check feature flag", (ctx) => {
    if (!ctx.store.get("betaEnabled")) {
      throw new Skip("Beta feature not enabled");
    }
  })
  .build();
console.log(def.name);

Constructor

new Skip(reason?: string)

Create a Skip exception.

Name	Description
`reason`	Human-readable reason for skipping.

Properties

readonlyreason?string
Human-readable reason for skipping.

When provided, this reason is displayed in test reports. If not provided, defaults to undefined (the error message will still show "Skipped").

class

#StepTimeoutError

class StepTimeoutError extends Error

ExtendsError

Error thrown when a step execution times out.

Contains structured information about the timeout including the step name, timeout duration, retry context, and total elapsed time.

Constructor

new StepTimeoutError(
  stepName: string,
  timeoutMs: number,
  attemptNumber: number,
  elapsedMs: number,
  options?: ErrorOptions,
)

Name	Description
`attemptNumber`	—
`elapsedMs`	—
`name`	—
`stepName`	—
`timeoutMs`	—
`timeoutSignal`()	Creates an AbortSignal that aborts with StepTimeoutError after the specified timeout.

Properties

readonlyattemptNumbernumber
readonlyelapsedMsnumber
readonlyname"StepTimeoutError"
readonlystepNamestring
readonlytimeoutMsnumber

Methods

static timeoutSignal(
  timeout: number,
  params: { attemptNumber: number; stepName: string },
): AbortSignal & Disposable

Creates an AbortSignal that aborts with StepTimeoutError after the specified timeout.

Parameters

timeoutnumber
Timeout duration in milliseconds
params{ attemptNumber: number; stepName: string }
Step information for the error

Interfaces

interface

#Reporter

interface Reporter

Reporter interface for receiving test execution events.

All callbacks receive serializable metadata types (ScenarioMetadata, StepMetadata) instead of definition types, enabling use across process/thread boundaries (e.g., Web Workers).

Name	Description
`onRunEnd`()	Called when test run completes
`onRunStart`()	Called when the test run starts, before any scenarios execute.
`onScenarioEnd`()	Called when scenario completes
`onScenarioStart`()	Called when a scenario begins execution.
`onStepEnd`()	Called when step completes (passed, failed, or skipped).
`onStepStart`()	Called when step starts

Methods

onRunEnd?(
  scenarios: readonly ScenarioMetadata[],
  result: RunResult,
): void | Promise<void>

Called when test run completes

Parameters

scenariosreadonly ScenarioMetadata[]
resultRunResult

onRunStart?(scenarios: readonly ScenarioMetadata[]): void | Promise<void>

Called when the test run starts, before any scenarios execute.

Parameters

scenariosreadonly ScenarioMetadata[]

onScenarioEnd?(
  scenario: ScenarioMetadata,
  result: ScenarioResult,
): void | Promise<void>

Called when scenario completes

Parameters

scenarioScenarioMetadata
resultScenarioResult

onScenarioStart?(scenario: ScenarioMetadata): void | Promise<void>

Called when a scenario begins execution.

Parameters

scenarioScenarioMetadata

onStepEnd?(
  scenario: ScenarioMetadata,
  step: StepMetadata,
  result: StepResult,
): void | Promise<void>

Called when step completes (passed, failed, or skipped).

The result contains status-specific information:

If status is "passed": contains value from step execution
If status is "failed" or "skipped": contains error information

Parameters

scenarioScenarioMetadata
stepStepMetadata
resultStepResult

onStepStart?(
  scenario: ScenarioMetadata,
  step: StepMetadata,
): void | Promise<void>

Called when step starts

Parameters

scenarioScenarioMetadata
stepStepMetadata

interface

#RunFilter

interface RunFilter

Filter configuration for selecting scenarios to run.

Allows filtering scenarios by tags and/or name pattern. Both conditions must match if both are specified (AND logic).

Examples

Filter by tags

const filter: RunFilter = {
  tags: ["api", "integration"]  // Must have BOTH tags
};

Filter by name pattern

const filter: RunFilter = {
  pattern: /login/i  // Name must contain "login"
};

Name	Description
`pattern`	Pattern to match against scenario name
`tags`	Tags that scenarios must have (all must match)

Properties

readonlypattern?string | RegExp
Pattern to match against scenario name
readonlytags?readonly string[]
Tags that scenarios must have (all must match)

interface

#RunOptions

interface RunOptions

Configuration options for the scenario runner.

Controls execution behavior including concurrency, failure handling, and reporting.

Examples

Sequential execution with fail-fast

import type { RunOptions } from "@probitas-test/runner";

const options: RunOptions = {
  maxConcurrency: 1,  // Run one at a time
  maxFailures: 1      // Stop after first failure
};
console.log(options);

Parallel execution with limit

import type { RunOptions } from "@probitas-test/runner";

const options: RunOptions = {
  maxConcurrency: 4   // Run up to 4 scenarios at once
};
console.log(options);

With stepOptions

import type { RunOptions } from "@probitas-test/runner";

const options: RunOptions = {
  stepOptions: {
    timeout: 60000,
    retry: { maxAttempts: 3, backoff: "exponential" }
  }
};
console.log(options);

Name	Description
`maxConcurrency`	Maximum number of scenarios to run in parallel.
`maxFailures`	Maximum number of failures before stopping the run.
`signal`	Abort signal for external cancellation.
`stepOptions`	Default step options applied when step uses default values.
`timeout`	Timeout for scenario execution in milliseconds.

Properties

readonlymaxConcurrency?number
Maximum number of scenarios to run in parallel.
- undefined or 0: Unlimited (all scenarios run in parallel)
- 1: Sequential execution (one at a time)
- n: Run up to n scenarios concurrently
readonlymaxFailures?number
Maximum number of failures before stopping the run.
- undefined: Continue all scenarios regardless of failures
- 1: Fail-fast (stop immediately on first failure)
- n: Stop after n failures
readonlysignal?AbortSignal
Abort signal for external cancellation.

When aborted, running scenarios complete but no new ones start. If both timeout and signal are provided, either one can trigger cancellation.
readonlystepOptions?StepOptions
Default step options applied when step uses default values.

These options override builder defaults but are themselves overridden by scenario-level or step-level options.

Priority: step options > scenario.stepOptions > config.stepOptions > defaults
readonlytimeout?number
Timeout for scenario execution in milliseconds.
- undefined: No timeout (unlimited execution time)
- 0: No timeout
- n: Timeout after n milliseconds
When timeout occurs, a ScenarioTimeoutError is thrown. This is independent from step-level timeouts.

interface

#RunResult

interface RunResult

Summary of all scenario executions in a test run.

Provides aggregate statistics and detailed results for the entire run. Passed to Reporter.onRunEnd after all scenarios complete.

Examples

import type { RunResult } from "@probitas-test/runner";

const summary: RunResult = {
  total: 10,
  passed: 8,
  failed: 1,
  skipped: 1,
  duration: 5432,
  scenarios: []
};

console.log(`${summary.passed}/${summary.total} passed`);
// → "8/10 passed"

Name	Description
`duration`	Total execution time in milliseconds
`failed`	Number of scenarios that failed
`passed`	Number of scenarios that passed
`scenarios`	Detailed result for each scenario
`skipped`	Number of scenarios that were skipped
`total`	Total number of scenarios in the run

Properties

readonlydurationnumber
Total execution time in milliseconds
readonlyfailednumber
Number of scenarios that failed
readonlypassednumber
Number of scenarios that passed
readonlyscenariosreadonly ScenarioResult[]
Detailed result for each scenario
readonlyskippednumber
Number of scenarios that were skipped
readonlytotalnumber
Total number of scenarios in the run

interface

#ScenarioContext

interface ScenarioContext

Runtime context for scenario execution.

Provides access to scenario metadata, accumulated results, shared storage, and resources during scenario execution. This is the "live" counterpart to the static ScenarioDefinition.

Examples

ScenarioContext structure

import type { ScenarioContext } from "@probitas-test/runner";

// ScenarioContext is created by the runner for each scenario execution
const ctx: ScenarioContext = {
  name: "My Test",
  tags: ["api", "integration"],
  results: [],
  store: new Map(),
  resources: {},
  signal: undefined,
};
console.log(`Running: ${ctx.name}`);
console.log(`Tags: ${ctx.tags.join(", ")}`);
console.log(`Previous results: ${ctx.results.length}`);

Name	Description
`name`	Human-readable scenario name
`resources`	Named resources registered with `.resource()`
`results`	Array of all exec step results so far
`signal`	Abort signal (fires on timeout or manual cancellation)
`store`	Shared key-value storage for cross-step communication
`tags`	—

Properties

readonlynamestring
Human-readable scenario name
readonlyresourcesRecord<string, unknown>
Named resources registered with .resource()
readonlyresultsunknown[]
Array of all exec step results so far
readonlysignal?AbortSignal
Abort signal (fires on timeout or manual cancellation)
readonlystoreMap<string, unknown>
Shared key-value storage for cross-step communication
readonlytagsreadonly string[]

Functions

function

#toScenarioMetadata

function toScenarioMetadata(scenario: ScenarioDefinition): ScenarioMetadata

Convert ScenarioDefinition to serializable ScenarioMetadata

Removes the fn property from each step to make the result JSON-serializable.

Parameters

scenarioScenarioDefinition

function

#toStepMetadata

function toStepMetadata(step: StepDefinition): StepMetadata

Convert StepDefinition to serializable StepMetadata

Removes the fn property to make the result JSON-serializable.

Parameters

stepStepDefinition

Types

type

#ScenarioResult

type ScenarioResult = {
    readonly duration: number;
    readonly metadata: ScenarioMetadata;
    readonly status: "passed";
    readonly steps: readonly StepResult[];
  }
  | {
    readonly duration: number;
    readonly error: unknown;
    readonly metadata: ScenarioMetadata;
    readonly status: "failed" | "skipped";
    readonly steps: readonly StepResult[];
  }

Result from executing a complete scenario.

Contains the overall scenario status, timing, all step results, and any error or skip reason. Passed to reporters and included in the final RunResult.

Examples

Passed scenario

import type { ScenarioResult } from "@probitas-test/runner";

const result: ScenarioResult = {
  metadata: { name: "Login Flow", tags: [], steps: [], origin: { path: "", line: 0 } },
  status: "passed",
  duration: 1250,
  steps: [{ metadata: { kind: "step", name: "step1", timeout: 30000, retry: { maxAttempts: 1, backoff: "linear" } }, status: "passed", value: null, duration: 100 }]
};
console.log(result);

Skipped scenario

import type { ScenarioResult } from "@probitas-test/runner";

const result: ScenarioResult = {
  metadata: { name: "Premium Feature", tags: [], steps: [], origin: { path: "", line: 0 } },
  status: "skipped",
  duration: 5,
  steps: [],
  error: "Feature flag not enabled"
};
console.log(result);

type

#StepResult

type StepResult = {
    readonly duration: number;
    readonly metadata: StepMetadata;
    readonly status: "passed";
    readonly value: unknown;
  }
  | {
    readonly duration: number;
    readonly error: unknown;
    readonly metadata: StepMetadata;
    readonly status: "failed" | "skipped";
  }

Result from executing a single step.

StepResult is a discriminated union type - the fields available depend on the status:

"passed": Contains value (return value from step function)
"failed": Contains error (exception that was thrown)
"skipped": Contains error (skip reason)

This type design ensures type-safe field access. Reporters should check result.status to safely access status-specific fields.

Examples

Passed step result

import type { StepResult } from "@probitas-test/runner";
import type { StepMetadata } from "@probitas-test/core";

const metadata: StepMetadata = { kind: "step", name: "Create user", timeout: 30000, retry: { maxAttempts: 1, backoff: "linear" } };
const result: StepResult = {
  metadata,
  status: "passed",
  duration: 150,
  value: { userId: "123" }
};
console.log(result);

Failed step result

import type { StepResult } from "@probitas-test/runner";
import type { StepMetadata } from "@probitas-test/core";

const metadata: StepMetadata = { kind: "step", name: "Validate input", timeout: 30000, retry: { maxAttempts: 1, backoff: "linear" } };
const result: StepResult = {
  metadata,
  status: "failed",
  duration: 50,
  error: new Error("Invalid email format")
};
console.log(result);

Skipped step result

import type { StepResult } from "@probitas-test/runner";
import type { StepMetadata } from "@probitas-test/core";

const metadata: StepMetadata = { kind: "step", name: "Optional step", timeout: 30000, retry: { maxAttempts: 1, backoff: "linear" } };
const result: StepResult = {
  metadata,
  status: "skipped",
  duration: 0,
  error: "Prerequisite not met"
};
console.log(result);

Links

Related Packages

Key Features

Core Exports

Error Types

Installation

Classes

#Runner

Examples

Constructor

Properties

Methods

Parameters

Links

#ScenarioTimeoutError

Constructor

Properties

Methods

Parameters

Links

#Skip

Examples

Constructor

Properties

Links

#StepTimeoutError

Constructor

Properties

Methods

Parameters

Links

Interfaces

#Reporter

Methods

Parameters

Parameters

Parameters

Parameters

Parameters

Parameters

Links

#RunFilter

Examples

Properties

#RunOptions

Examples

Properties

Links

#RunResult

Examples

Properties

Links

#ScenarioContext

Examples

Properties

Functions

#toScenarioMetadata

Parameters

Links

#toStepMetadata

Parameters

Links

Types

#ScenarioResult

Examples

Links

#StepResult

Examples

Links