# @probitas/runner

> Version: 0.5.6

Test execution and orchestration engine for Probitas scenarios.

This package provides the {@linkcode ScenarioRunner} class that executes
scenario definitions built with `@probitas/builder`. It handles the complete
test lifecycle including resource initialization, setup/cleanup hooks,
step execution with retry logic, and result reporting.

## Links

- [GitHub Repository](https://github.com/probitas-test/probitas)
- [@probitas/probitas](https://jsr.io/@probitas/probitas) - Main package (recommended for most users)

## Related Packages

| Package | Description |
|---------|-------------|
| [@probitas/builder](https://jsr.io/@probitas/builder) | Build scenario definitions to execute |
| [@probitas/core](https://jsr.io/@probitas/core) | Core type definitions |
| [@probitas/reporter](https://jsr.io/@probitas/reporter) | Output formatters for results |
| [@probitas/cli](https://jsr.io/@probitas/cli) | CLI that uses this runner |

## Key Features

- **Concurrent execution**: Run multiple scenarios in parallel with configurable concurrency
- **Failure control**: Stop after N failures or continue through all tests
- **Resource lifecycle**: Automatic initialization and disposal of resources
- **Retry logic**: Built-in retry with linear/exponential backoff strategies
- **Reporter integration**: Pluggable reporters for custom output formatting
- **Abort support**: Cancel running tests via AbortSignal

## Core Exports

- {@linkcode ScenarioRunner} - Main class for executing scenarios
- {@linkcode Skip} - Exception class to skip scenarios conditionally
- {@linkcode Reporter} - Interface for observing test execution events
- {@linkcode RunOptions} - Configuration options for test runs
- {@linkcode RunResult} - Aggregated results from a test run
- {@linkcode ScenarioResult} - Result from executing a single scenario
- {@linkcode StepResult} - Result from executing a single step

## Error Types

- {@linkcode StepTimeoutError} - Thrown when a step exceeds its timeout
- {@linkcode ScenarioTimeoutError} - Thrown when a scenario exceeds its timeout


## Classes

### `Skip`

```typescript
class Skip extends Error
```

Exception class for conditionally skipping scenario execution.

Throw `Skip` from any step, setup, or resource fn to skip the
entire scenario. Skipped scenarios are counted separately in the
summary and don't count as failures.

Common use cases:
- Feature flags or environment checks
- Prerequisites not met
- Platform-specific tests
- Temporary test disabling with clear reason


**Constructor:**

```typescript
new Skip(reason?: string)
```

**Properties:**

- [readonly] `reason?`: `string` — Human-readable reason for skipping.

When provided, this reason is displayed in test reports.
If not provided, defaults to `undefined` (the error message
will still show "Skipped").

**Example:**

Skip based on environment
```ts
import { scenario } from "@probitas/builder";
import { Skip } from "@probitas/runner";

const def = scenario("Production Only")
  .step("Check environment", () => {
    if (Deno.env.get("ENV") !== "production") {
      throw new Skip("Only runs in production");
    }
  })
  .step("Production test", () => {
    console.log("Running in production");
  })
  .build();
console.log(def.name);
```

Skip based on resource availability
```ts
import { scenario } from "@probitas/builder";
import { Skip } from "@probitas/runner";

const Database = { connect: async () => ({ query: (sql: string) => sql }) };

const def = scenario("Database Test")
  .resource("db", async () => {
    try {
      return await Database.connect();
    } catch {
      throw new Skip("Database not available");
    }
  })
  .step("Query data", (ctx) => ctx.resources.db.query("SELECT 1"))
  .build();
console.log(def.name);
```

Skip with feature flag
```ts
import { scenario } from "@probitas/builder";
import { Skip } from "@probitas/runner";

const def = scenario("Beta Feature")
  .step("Check feature flag", (ctx) => {
    if (!ctx.store.get("betaEnabled")) {
      throw new Skip("Beta feature not enabled");
    }
  })
  .build();
console.log(def.name);
```

---

### `Runner`

```typescript
class Runner
```

Top-level test runner that orchestrates execution of multiple scenarios.

The Runner manages:
- Parallel execution with concurrency control (`maxConcurrency`)
- Early stopping on failures (`maxFailures`)
- Test lifecycle events through the Reporter interface
- Aggregated results in RunResult


**Constructor:**

```typescript
new Runner(reporter: Reporter)
```

**Methods:**

```typescript
run(): unknown
```

Execute all scenarios and return aggregated results.

Emits events through the Reporter:
1. `onRunStart` - Before execution starts
2. For each scenario (concurrently):
   - Delegates to ScenarioRunner for scenario-level execution
   - ScenarioRunner emits scenario and step level events
3. `onRunEnd` - After all scenarios complete with RunResult


**Example:**

```ts
import { Runner } from "@probitas/runner";
import type { Reporter } from "@probitas/runner";
import type { ScenarioDefinition } from "@probitas/core";

const reporter: Reporter = {};
const scenarios: ScenarioDefinition[] = [];
const runner = new Runner(reporter);

const result = await runner.run(scenarios, {
  maxConcurrency: 4,
  maxFailures: 1,
});
console.log(result);
```

---

### `ScenarioTimeoutError`

```typescript
class ScenarioTimeoutError extends Error
```

Error thrown when a scenario execution times out.

Contains structured information about the timeout including
the scenario name, timeout duration, total elapsed time,
and optionally the step being executed when timeout occurred.

**Constructor:**

```typescript
new ScenarioTimeoutError(
  scenarioName: string,
  timeoutMs: number,
  elapsedMs: number,
  options?: ErrorOptions & { currentStepName?: string; currentStepIndex?: number },
)
```

**Properties:**

- [readonly] `name`: `string`
- [readonly] `scenarioName`: `string`
- [readonly] `timeoutMs`: `number`
- [readonly] `elapsedMs`: `number`
- [readonly] `currentStepName?`: `string`
- [readonly] `currentStepIndex?`: `number`

**Methods:**

```typescript
static timeoutSignal(): unknown
```

Creates an AbortSignal that aborts with ScenarioTimeoutError after the specified timeout.


---

### `StepTimeoutError`

```typescript
class StepTimeoutError extends Error
```

Error thrown when a step execution times out.

Contains structured information about the timeout including
the step name, timeout duration, retry context, and total elapsed time.

**Constructor:**

```typescript
new StepTimeoutError(
  stepName: string,
  timeoutMs: number,
  attemptNumber: number,
  elapsedMs: number,
  options?: ErrorOptions,
)
```

**Properties:**

- [readonly] `name`: `string`
- [readonly] `stepName`: `string`
- [readonly] `timeoutMs`: `number`
- [readonly] `attemptNumber`: `number`
- [readonly] `elapsedMs`: `number`

**Methods:**

```typescript
static timeoutSignal(): unknown
```

Creates an AbortSignal that aborts with StepTimeoutError after the specified timeout.


---

## Interfaces

### `Reporter`

```typescript
interface Reporter
```

Reporter interface for receiving test execution events.

All callbacks receive serializable metadata types (ScenarioMetadata, StepMetadata)
instead of definition types, enabling use across process/thread boundaries
(e.g., Web Workers).

**Methods:**

```typescript
onRunStart(scenarios: readonly ScenarioMetadata[]): void | Promise<void>
```

Called when the test run starts, before any scenarios execute.

```typescript
onRunEnd(
  scenarios: readonly ScenarioMetadata[],
  result: RunResult,
): void | Promise<void>
```

Called when test run completes

```typescript
onScenarioStart(scenario: ScenarioMetadata): void | Promise<void>
```

Called when a scenario begins execution.

```typescript
onScenarioEnd(
  scenario: ScenarioMetadata,
  result: ScenarioResult,
): void | Promise<void>
```

Called when scenario completes

```typescript
onStepStart(
  scenario: ScenarioMetadata,
  step: StepMetadata,
): void | Promise<void>
```

Called when step starts

```typescript
onStepEnd(
  scenario: ScenarioMetadata,
  step: StepMetadata,
  result: StepResult,
): void | Promise<void>
```

Called when step completes (passed, failed, or skipped).

The result contains status-specific information:
- If status is "passed": contains `value` from step execution
- If status is "failed" or "skipped": contains `error` information

---

### `RunResult`

```typescript
interface RunResult
```

Summary of all scenario executions in a test run.

Provides aggregate statistics and detailed results for the entire run.
Passed to {@linkcode Reporter.onRunEnd} after all scenarios complete.


**Properties:**

- [readonly] `total`: `number` — Total number of scenarios in the run
- [readonly] `passed`: `number` — Number of scenarios that passed
- [readonly] `failed`: `number` — Number of scenarios that failed
- [readonly] `skipped`: `number` — Number of scenarios that were skipped
- [readonly] `duration`: `number` — Total execution time in milliseconds
- [readonly] `scenarios`: `readonly ScenarioResult[]` — Detailed result for each scenario

**Example:**

```ts
import type { RunResult } from "@probitas/runner";

const summary: RunResult = {
  total: 10,
  passed: 8,
  failed: 1,
  skipped: 1,
  duration: 5432,
  scenarios: []
};

console.log(`${summary.passed}/${summary.total} passed`);
// → "8/10 passed"
```

---

### `RunFilter`

```typescript
interface RunFilter
```

Filter configuration for selecting scenarios to run.

Allows filtering scenarios by tags and/or name pattern.
Both conditions must match if both are specified (AND logic).


**Properties:**

- [readonly] `tags?`: `readonly string[]` — Tags that scenarios must have (all must match)
- [readonly] `pattern?`: `string | RegExp` — Pattern to match against scenario name

**Example:**

Filter by tags
```ts
const filter: RunFilter = {
  tags: ["api", "integration"]  // Must have BOTH tags
};
```

Filter by name pattern
```ts
const filter: RunFilter = {
  pattern: /login/i  // Name must contain "login"
};
```

---

### `RunOptions`

```typescript
interface RunOptions
```

Configuration options for the scenario runner.

Controls execution behavior including concurrency, failure handling,
and reporting.


**Properties:**

- [readonly] `maxConcurrency?`: `number` — Maximum number of scenarios to run in parallel.

- `undefined` or `0`: Unlimited (all scenarios run in parallel)
- `1`: Sequential execution (one at a time)
- `n`: Run up to n scenarios concurrently
- [readonly] `maxFailures?`: `number` — Maximum number of failures before stopping the run.

- `undefined`: Continue all scenarios regardless of failures
- `1`: Fail-fast (stop immediately on first failure)
- `n`: Stop after n failures
- [readonly] `timeout?`: `number` — Timeout for scenario execution in milliseconds.

- `undefined`: No timeout (unlimited execution time)
- `0`: No timeout
- `n`: Timeout after n milliseconds

When timeout occurs, a {@linkcode ScenarioTimeoutError} is thrown.
This is independent from step-level timeouts.
- [readonly] `signal?`: `AbortSignal` — Abort signal for external cancellation.

When aborted, running scenarios complete but no new ones start.
If both `timeout` and `signal` are provided, either one can trigger cancellation.
- [readonly] `stepOptions?`: `StepOptions` — Default step options applied when step uses default values.

These options override builder defaults but are themselves overridden
by scenario-level or step-level options.

Priority: step options > scenario.stepOptions > config.stepOptions > defaults

**Example:**

Sequential execution with fail-fast
```ts
import type { RunOptions } from "@probitas/runner";

const options: RunOptions = {
  maxConcurrency: 1,  // Run one at a time
  maxFailures: 1      // Stop after first failure
};
console.log(options);
```

Parallel execution with limit
```ts
import type { RunOptions } from "@probitas/runner";

const options: RunOptions = {
  maxConcurrency: 4   // Run up to 4 scenarios at once
};
console.log(options);
```

With stepOptions
```ts
import type { RunOptions } from "@probitas/runner";

const options: RunOptions = {
  stepOptions: {
    timeout: 60000,
    retry: { maxAttempts: 3, backoff: "exponential" }
  }
};
console.log(options);
```

---

### `ScenarioContext`

```typescript
interface ScenarioContext
```

Runtime context for scenario execution.

Provides access to scenario metadata, accumulated results, shared storage,
and resources during scenario execution. This is the "live" counterpart
to the static {@linkcode ScenarioDefinition}.


**Properties:**

- [readonly] `name`: `string` — Human-readable scenario name
- [readonly] `tags`: `readonly string[]`
- [readonly] `results`: `unknown[]` — Array of all exec step results so far
- [readonly] `store`: `Map<string, unknown>` — Shared key-value storage for cross-step communication
- [readonly] `resources`: `Record<string, unknown>` — Named resources registered with `.resource()`
- [readonly] `signal?`: `AbortSignal` — Abort signal (fires on timeout or manual cancellation)

**Example:**

ScenarioContext structure
```ts
import type { ScenarioContext } from "@probitas/runner";

// ScenarioContext is created by the runner for each scenario execution
const ctx: ScenarioContext = {
  name: "My Test",
  tags: ["api", "integration"],
  results: [],
  store: new Map(),
  resources: {},
  signal: undefined,
};
console.log(`Running: ${ctx.name}`);
console.log(`Tags: ${ctx.tags.join(", ")}`);
console.log(`Previous results: ${ctx.results.length}`);
```

---

## Functions

### `toScenarioMetadata`

```typescript
function toScenarioMetadata(scenario: ScenarioDefinition): ScenarioMetadata
```

Convert ScenarioDefinition to serializable ScenarioMetadata

Removes the `fn` property from each step to make the result JSON-serializable.

**Parameters:**

- `scenario`: `ScenarioDefinition`

**Returns:** `ScenarioMetadata`

---

### `toStepMetadata`

```typescript
function toStepMetadata(step: StepDefinition): StepMetadata
```

Convert StepDefinition to serializable StepMetadata

Removes the `fn` property to make the result JSON-serializable.

**Parameters:**

- `step`: `StepDefinition`

**Returns:** `StepMetadata`

---

## Types

### `ScenarioResult`

```typescript
type ScenarioResult = {
    status: "passed";
    metadata: ScenarioMetadata;
    duration: number;
    steps: readonly StepResult[];
  }
  | {
    status: "failed" | "skipped";
    metadata: ScenarioMetadata;
    duration: number;
    steps: readonly StepResult[];
    error: unknown;
  }
```

Result from executing a complete scenario.

Contains the overall scenario status, timing, all step results,
and any error or skip reason. Passed to reporters and included
in the final {@linkcode RunResult}.


---

### `StepResult`

```typescript
type StepResult = {
    status: "passed";
    metadata: StepMetadata;
    duration: number;
    value: unknown;
  }
  | {
    status: "failed" | "skipped";
    metadata: StepMetadata;
    duration: number;
    error: unknown;
  }
```

Result from executing a single step.

StepResult is a discriminated union type - the fields available depend on
the status:
- "passed": Contains `value` (return value from step function)
- "failed": Contains `error` (exception that was thrown)
- "skipped": Contains `error` (skip reason)

This type design ensures type-safe field access. Reporters should check
`result.status` to safely access status-specific fields.


---

## Related Links

### This Package

- [`Reporter`](https://probitas-test.github.io/documents/api/runner#Reporter)
- [`RunResult`](https://probitas-test.github.io/documents/api/runner#RunResult)
- [`ScenarioResult`](https://probitas-test.github.io/documents/api/runner#ScenarioResult)
- [`StepResult`](https://probitas-test.github.io/documents/api/runner#StepResult)

### Other Packages

- [`ScenarioDefinition`](https://probitas-test.github.io/documents/api/scenario#ScenarioDefinition) (@probitas/scenario)
- [`ScenarioMetadata`](https://probitas-test.github.io/documents/api/scenario#ScenarioMetadata) (@probitas/scenario)
- [`StepDefinition`](https://probitas-test.github.io/documents/api/scenario#StepDefinition) (@probitas/scenario)
- [`StepMetadata`](https://probitas-test.github.io/documents/api/scenario#StepMetadata) (@probitas/scenario)
- [`StepOptions`](https://probitas-test.github.io/documents/api/scenario#StepOptions) (@probitas/scenario)

### Built-in Types

- [`Map`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map)
- [`Promise`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise)
- [`Record`](https://www.typescriptlang.org/docs/handbook/utility-types.html#recordkeys-type)
- [`RegExp`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp)

---

*Last updated: 2026-01-12*