API Test Generator

Generates Playwright integration test suites from the Camunda REST API OpenAPI specification. Analyses the spec's semantic type annotations (x-semantic-type) to build an operation dependency graph, then emits scenario-driven test files with full request/response synthesis. Also emits negative request-validation tests (intended HTTP 400) covering ~24 distinct malformed-request scenario kinds.

Architecture

┌──────────────────────────┐
│  camunda-schema-bundler  │  Fetches & bundles the upstream multi-file OpenAPI spec
│  (devDependency)         │  → spec/bundled/rest-api.bundle.json
└────────────┬─────────────┘
             │
             ├──────────────────────────────────────────┐
             ▼                                          ▼
┌──────────────────────────┐               ┌──────────────────────────┐
│ semantic-graph-extractor │               │   request-validation     │
│                          │               │ (negative-test generator)│
│ Parses bundled spec,     │               │                          │
│ extracts semantic types  │               │ Synthesizes ~24 kinds of │
│ & operations             │               │ malformed-request tests  │
│ → operation-dependency-  │               │ expecting HTTP 400       │
│   graph.json             │               │ → request-validation/    │
└────────────┬─────────────┘               │   generated/*.spec.ts    │
             │                             └──────────────────────────┘
             ▼
┌──────────────────────────┐
│     path-analyser        │  Reads graph + spec, generates positive scenarios
│                          │  per endpoint, emits Playwright test suites
│                          │  → path-analyser/dist/generated-tests/*.spec.ts
└──────────────────────────┘

Prerequisites

Node.js ≥ 22
npm ≥ 10 (ships with Node 22+)
Camunda 8 server — either via c8ctl (recommended) or Docker Compose

Quick Start

Starting the Camunda Server

Option A — `c8ctl` (recommended)

c8ctl is the Camunda 8 CLI. It manages a local cluster lifecycle for you and is the fastest path from clone to a running server:

# One-off install (Node ≥ 22)
npm install -g @camunda8/c8ctl

# Start a cluster pinned to a specific Camunda version
c8ctl cluster start 8.9

# Stop it when you're done
c8ctl cluster stop

The Camunda REST API will be available at http://localhost:8080.

Option B — Docker Compose

If you prefer to run Compose directly:

cd docker
docker compose up -d
docker compose logs camunda          # health check
docker compose down                  # stop

Override the REST port with CAMUNDA_REST_PORT:

CAMUNDA_REST_PORT=9080 docker compose up -d

Running the Test Generator

# Install all workspace dependencies (runs in all sub-packages)
npm install

# Fetch the upstream OpenAPI spec and bundle it
npm run fetch-spec

# Run the positive pipeline: extract graph → generate scenarios → emit Playwright tests
npm run pipeline

# Generate the negative request-validation suite (HTTP 400 tests, all supported scenario kinds)
npm run generate:request-validation

# Run the generated tests (requires running Camunda server)
npm run test:pw                       # both suites (path-analyser + request-validation)
npm run test:pw:path-analyser         # positive scenarios only
npm run test:pw:request-validation    # negative request-validation only

Project Structure

This is an npm workspaces monorepo with four packages:

api-test-generator/
├── package.json              ← root workspace orchestrator
├── spec/                     ← (gitignored) bundled OpenAPI spec output
│   └── bundled/
│       ├── rest-api.bundle.json
│       └── spec-metadata.json
├── semantic-graph-extractor/ ← workspace: graph extraction from OpenAPI
├── path-analyser/            ← workspace: positive scenario generation & Playwright codegen
├── request-validation/       ← workspace: negative request-validation test generator (HTTP 400)
└── optional-responses/       ← workspace: optional response field analyser

npm Workspaces

All four sub-packages are registered as npm workspaces. A single npm install at the root installs dependencies for every package.

Managing Dependencies

# Add a dependency to a specific workspace
npm install <pkg> -w semantic-graph-extractor

# Add a dev dependency to a specific workspace
npm install -D <pkg> -w path-analyser

# Add a dependency to the root
npm install -D <pkg> -w .

# Remove a dependency
npm uninstall <pkg> -w semantic-graph-extractor

Running Scripts in Workspaces

# Run a script in a specific workspace
npm run build -w semantic-graph-extractor
npm run build -w path-analyser

# Run a script in all workspaces that define it
npm run build --workspaces --if-present

Available Root Scripts

Script	Description
`npm run fetch-spec`	Fetch and bundle the upstream OpenAPI spec (from `main` branch)
`npm run fetch-spec:ref`	Fetch a specific branch/tag: `SPEC_REF=stable/8.8 npm run fetch-spec:ref`
`npm run extract-graph`	Build the semantic graph extractor and extract the dependency graph
`npm run generate:scenarios`	Build the path analyser and generate scenario JSON files
`npm run codegen:playwright`	Build and emit a Playwright test for a single endpoint
`npm run codegen:playwright:all`	Build and emit Playwright tests for all endpoints
`npm run build:request-validation`	Build the request-validation generator
`npm run generate:request-validation`	Emit negative request-validation tests with all supported scenario kinds (deep coverage by default)
`npm run generate:request-validation:shallow`	Emit only the core kinds (`missing-required`, `type-mismatch`, `union`) — fast iteration
`npm run test:pw`	Run both generated Playwright suites (path-analyser + request-validation)
`npm run test:pw:path-analyser`	Run only the positive path-analyser suite
`npm run test:pw:request-validation`	Run only the negative request-validation suite
`npm run testsuite:generate`	Full positive-generation pipeline: extract graph → scenarios → Playwright tests
`npm run testsuite:observe:run`	Generate tests, run them, and aggregate runtime observations
`npm run observe:aggregate`	Aggregate runtime observation data
`npm run optional-responses`	Run the optional response field analyser
`npm run pipeline`	End-to-end: fetch spec + generate entire test suite
`npm run lint`	Lint all workspaces with Biome
`npm run lint:fix`	Lint and apply safe Biome fixes
`npm run format`	Format all workspaces with Biome
`npm test`	Run the regression test suite (extractor + planner fixtures, bundled-spec invariants)

Code Quality Tooling

This repo uses Biome for both linting and formatting, configured at the root in biome.json. The recommended ruleset is enabled with three additional escalations to error:

suspicious/noExplicitAny
suspicious/noImplicitAnyLet
suspicious/noEvolvingTypes

A custom GritQL plugin (plugins/no-unsafe-type-assertion.grit) bans as T type assertions outside of imports and as const. Use type guards, narrowing, or satisfies instead — and only suppress with // biome-ignore lint/plugin: <reason> when genuinely unavoidable.

Regression Testing

The pipeline emits hundreds of generated files (semantic graph, scenario JSON, Playwright tests, validation tests). The regression strategy is layered (see #36):

Layer 1 — extractor construct fixtures (tests/fixtures/extractor/). Hand-curated minimal OpenAPI snippets paired with property assertions. Each it block is one regression statement; failures point at one construct, not at hundreds of hashed files.
Layer 2 — planner contract fixtures (tests/fixtures/planner/). Tiny dependency-graph fixtures paired with chain-shape assertions on generateScenariosForEndpoint.
Layer 3 — bundled-spec invariants (tests/regression/bundled-spec-invariants.test.ts). Named, human-readable invariants over the real bundled spec output.

There is no Layer 4 end-to-end snapshot. The previous SHA-256 manifest guard was retired in favour of the layered strategy: a 412-file diff is not a useful signal, whereas a fixture or invariant failure names the broken property directly.

Standing rule: every bug fix to the extractor or planner should land with (a) a fixture demonstrating the bug BEFORE the fix, and (b) an invariant if the property is observable at the chain or graph level. See CONTRIBUTING.md.

Determinism

Generator output is byte-reproducible by default. The seeding module (path-analyser/src/codegen/support/seeding.ts) uses TEST_SEED to seed all deterministicSuffix(...) calls; if unset, it falls back to the constant 'snapshot-baseline', so npm run pipeline produces identical output across runs and machines without needing TEST_SEED to be set explicitly.

To opt out (for example, when generating a one-off suite for live-broker exploration where unique-per-run identifiers are useful), set:

TEST_SEED=random npm run pipeline

Any other non-empty value is treated as a custom deterministic seed.

Spec pin

The bundled-spec invariants test the real upstream spec output, so it is only meaningful against a fixed upstream spec content. tests/regression/spec-pin.json records the expectedSpecHash plus the specRef CI fetches. A vitest globalSetup (tests/regression/spec-pin.setup.ts) aborts the entire run with a single actionable error if the bundled spec drifts from that hash, so reviewers don't have to debug a confusing invariant failure when the real cause is upstream drift.

To bump the spec:

# 1. Fetch the new spec. SPEC_REF can be a branch, tag, or commit SHA;
#    a branch/tag is fine for convenience here, but step 3 must record
#    the resolved 40-char commit SHA in spec-pin.json (branches drift).
SPEC_REF=stable/8.10 npm run fetch-spec:ref

# 2. Regenerate the pipeline so the invariants run against fresh output
npm run testsuite:generate
npm run generate:request-validation

# 3. Update tests/regression/spec-pin.json:
#    - specRef:          the resolved 40-char commit SHA (NOT the branch/tag)
#    - expectedSpecHash: the `specHash` printed in spec/bundled/spec-metadata.json
# 4. Update any invariants whose values legitimately changed, then commit
#    spec-pin.json alongside the invariant updates.

Continuous integration

.github/workflows/ci.yml runs on every PR to main (and on pushes to main). It executes:

npm run lint — Biome
tsc --noEmit against each workspace tsconfig
Builds (build:analyser, build:request-validation)
npm run fetch-spec:ref at the pinned specRef
Full pipeline regeneration with TEST_SEED=snapshot-baseline
npm test — spec-pin guard, layered regression, and unit tests

On failure the generated outputs are uploaded as the pipeline-outputs artifact for inspection.

Fetching the OpenAPI Spec

The bundled spec is produced by camunda-schema-bundler, installed as a dev dependency. It fetches the multi-file upstream spec from camunda/camunda and produces a single normalised JSON file.

# Fetch from main (default)
npm run fetch-spec

# Fetch a specific branch or tag
SPEC_REF=stable/8.9 npm run fetch-spec:ref

# Or use the CLI directly for more options
npx camunda-schema-bundler --help

The output lands in spec/bundled/ (gitignored).

Using a Custom Spec

All spec-reading code supports the OPENAPI_SPEC_PATH environment variable:

OPENAPI_SPEC_PATH=/path/to/my-spec.yaml npm run testsuite:generate

Workspace Package Details

semantic-graph-extractor

Analyses the OpenAPI spec and builds an operation dependency graph based on x-semantic-type annotations. The graph captures which operations produce and consume which semantic types, enabling dependency-aware test ordering.

npm run build -w semantic-graph-extractor
npm run extract-graph              # build + extract
npm run analyze-graph -w semantic-graph-extractor   # human-readable analysis report
npm run validate-graph -w semantic-graph-extractor  # validate graph integrity

path-analyser

Reads the dependency graph and the OpenAPI spec to generate positive test scenarios for every endpoint — happy paths, oneOf variant selection, dependency chaining, response-shape assertions and artifact deployment coverage. Then emits executable Playwright test files. Negative-request scenarios (missing required fields, wrong types, etc.) are owned exclusively by request-validation.

npm run build -w path-analyser
npm run generate:scenarios         # build + generate scenario JSON
npm run codegen:playwright:all     # build + emit all Playwright tests
npm run test:pw                    # run the generated tests
npm run observe:aggregate          # aggregate runtime observations

Pluggable test emitters

Suite generation is layered behind a small Emitter strategy interface (path-analyser/src/codegen/emitter.ts). The CLI selects an emitter via --target=<id> and falls back to playwright when omitted:

node path-analyser/dist/src/codegen/index.js --target=playwright createWidget
node path-analyser/dist/src/codegen/index.js --target=playwright --all

The current built-in is playwright. Additional targets (e.g. SDK-based suites — see #8) register themselves through registerEmitter() and are listed in --help. The emitter contract is experimental and may change while the SDK strategies land.

request-validation

A spec-driven generator that synthesizes negative Playwright tests targeting request-validation surfaces — every test sends a deliberately malformed request and asserts the server responds with HTTP 400. Covers ~24 scenario kinds including missing required fields (single + combinations), wrong primitive types, root-body type mismatches, oneOf ambiguity / no-match / cross-bleed, discriminator mismatches, enum / format / multipleOf / uniqueItems / length / pattern violations, allOf conflicts, additional-property rejection and multipart-only adaptation.

npm run build:request-validation             # compile the generator
npm run generate:request-validation          # all supported scenario kinds (deep coverage by default)
npm run generate:request-validation:shallow  # only missing-required, type-mismatch, union

Output lands in request-validation/generated/:

File	Description
`<resource>-validation-api-tests.spec.ts`	Playwright specs grouped by resource
`MANIFEST.json`	Global counts per scenario kind + generation options
`COVERAGE.json` / `COVERAGE.md`	Per-operation coverage matrix and missing-kind list

The generator consumes the bundled spec produced by npm run fetch-spec. To point it at a different OpenAPI document, set REQUEST_VALIDATION_SPEC to the absolute or repo-relative path.

optional-responses

A lightweight utility that scans the OpenAPI spec for response schemas with optional fields — useful for understanding which response fields may be absent.

npm run optional-responses

Configuration Files (path-analyser)

File	Purpose
`domain-semantics.json`	Domain-level semantic requirements, runtime states, and value bindings
`filter-providers.json`	Maps fields to value providers (`ctx`, `const`, `enumFirst`, etc.)
`request-defaults.json`	Default values for request body fields per operation
`fixtures/`	BPMN, DMN, and form files used by deployment tests

Environment Variables

Variable	Description
`OPENAPI_SPEC_PATH`	Override the path to the OpenAPI spec file (consumed by `semantic-graph-extractor` / `path-analyser`)
`OPERATION_GRAPH_PATH`	Override the path to the dependency graph JSON
`SPEC_REF`	Git ref for `fetch-spec:ref` (branch, tag, or SHA)
`CAMUNDA_BASE_URL`	Base URL of the Camunda instance for Playwright tests
`REQUEST_VALIDATION_SPEC`	Override the spec path consumed by the `request-validation` generator

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

API Test Generator

Architecture

Prerequisites

Quick Start

Starting the Camunda Server

Option A — `c8ctl` (recommended)

Option B — Docker Compose

Running the Test Generator

Project Structure

npm Workspaces

Managing Dependencies

Running Scripts in Workspaces

Available Root Scripts

Code Quality Tooling

Regression Testing

Determinism

Spec pin

Continuous integration

Fetching the OpenAPI Spec

Using a Custom Spec

Workspace Package Details

semantic-graph-extractor

path-analyser

Pluggable test emitters

request-validation

optional-responses

Configuration Files (path-analyser)

Environment Variables

About

Uh oh!

Releases

Packages

Uh oh!

Contributors

Uh oh!

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 140 Commits
.github/workflows		.github/workflows
docker		docker
optional-responses		optional-responses
path-analyser		path-analyser
plugins		plugins
request-validation		request-validation
semantic-graph-extractor		semantic-graph-extractor
tests		tests
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
README.md		README.md
biome.json		biome.json
package-lock.json		package-lock.json
package.json		package.json
vitest.config.ts		vitest.config.ts

Folders and files

Latest commit

History

Repository files navigation

API Test Generator

Architecture

Prerequisites

Quick Start

Starting the Camunda Server

Option A — c8ctl (recommended)

Option B — Docker Compose

Running the Test Generator

Project Structure

npm Workspaces

Managing Dependencies

Running Scripts in Workspaces

Available Root Scripts

Code Quality Tooling

Regression Testing

Determinism

Spec pin

Continuous integration

Fetching the OpenAPI Spec

Using a Custom Spec

Workspace Package Details

semantic-graph-extractor

path-analyser

Pluggable test emitters

request-validation

optional-responses

Configuration Files (path-analyser)

Environment Variables

About

Resources

Contributing

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Option A — `c8ctl` (recommended)

Packages