Skip to content

feat: suppress feature specs covered by scenario-template ontology#332

Merged
jwulf merged 2 commits into
mainfrom
feat/scenario-coverage-suppression-331
May 21, 2026
Merged

feat: suppress feature specs covered by scenario-template ontology#332
jwulf merged 2 commits into
mainfrom
feat/scenario-coverage-suppression-331

Conversation

@jwulf
Copy link
Copy Markdown
Member

@jwulf jwulf commented May 21, 2026

Implements coverage-driven suppression of per-endpoint feature specs for operations already exercised by a scenario-template instantiation (EdgeLifecycle, EntityLifecycle, UpdatedFieldVisibleOnReadBack, StateTransitionVisibleAfterAction).

What

  • New materializer/src/coverage.ts walks emitted template scenario JSONs and collects every operationId bound to an Invoke / Observe step.
  • Per-template opt-out via ScenarioTemplate.suppressesFeatureTest on the ABox (defaults to true). Schema TS source + regenerated JSON Schema artefact updated in lockstep.
  • Orchestrator filters the --all feature loop by the resulting suppression set and writes a stable, sorted coverage.json alongside the suite so the suppression is diffable in PRs.

Why

A scenario-template instantiation is the canonical functional test for the operationIds bound to its Invoke / Observe steps. Emitting a parallel per-endpoint feature spec is either strictly weaker (EntityLifecycle case) or structurally malformed (EdgeLifecycle case: a key-only prereq chain cannot encode the establish-before-revoke precondition the feature spec needs). See #331 for the full motivation.

The implementation is template-agnostic — adding a new ScenarioTemplate to configs/<config>/ontology/scenario-templates.json (and wiring its output dir into templateOutputDirs) extends suppression automatically.

Numbers

In the camunda-oca config: 73 feature specs suppressed across 26 lifecycle scenarios (4 templates).

L3 invariants

Updated to reflect the intentional behaviour change:

Test layers touched

  • Layer-3: 4 invariants in configs/camunda-oca/regression-invariants.test.ts (3 updated, 1 added).
  • Schema drift-detector: regenerated ontology/vocabulary/scenario-template.schema.json (mandatory pair with the TS source).

Pre-push gate

  • npm run build:analyser, npm run build:emitter-sdk
  • tsc --noEmit across all 6 workspace tsconfigs ✅
  • TEST_SEED=snapshot-baseline npm run testsuite:generate && npm run generate:request-validation
  • npm test — 627 passed, 4 skipped, 0 failed ✅
  • npm run lint — 0 errors ✅

Closes #331

For every Invoke/Observe step in a scenario-template instantiation
(EdgeLifecycle, EntityLifecycle, UpdatedFieldVisibleOnReadBack,
StateTransitionVisibleAfterAction) the materializer now omits the
per-endpoint feature spec for the operationId those steps target. The
lifecycle spec is the canonical functional test for those operations;
emitting a parallel feature spec is either strictly weaker (EntityLifecycle
case) or structurally malformed (EdgeLifecycle case: a key-only prereq
chain cannot encode the establish-before-revoke precondition).

The behaviour is template-agnostic and coverage-driven, not hard-coded:

- New module `materializer/src/coverage.ts` walks the on-disk template
  scenario JSONs under `generated/<config>/scenarios/templates/<Template>/`
  and collects every operationId bound to an Invoke/Observe step. Per
  template, the opt-out lives on the ABox as
  `ScenarioTemplate.suppressesFeatureTest` (defaults to `true`); flipping
  it to `false` lets a future non-functional template (smoke / chaos /
  load) emit a spec without claiming coverage of the operations it
  touches.
- The orchestrator filters the `--all` feature loop by the resulting
  suppression set and writes a stable, sorted `coverage.json` artefact
  next to the suite so the suppression is diffable in PRs.

L3 invariants updated:

- Partition-cut #162 PR 4 now subtracts suppressed opIds.
- awaitEventually #106 skips suppressed opIds (their reads are wrapped
  by the template emitter).
- #305 Phase 5a getIncident chain-shape assertions move from
  `getIncident.feature.spec.ts` to
  `state-transitions/Incident.resolveIncident.lifecycle.spec.ts`; the
  emitted chain (deploys incident-script-task.bpmn, runs searchIncidents
  → extracts incidentKey → consumes via runtime binding) is unchanged.
- New positive invariant: every suppressed opId has a coverage entry
  pointing at an emitted lifecycle spec that exists on disk. Catches
  the regression class where an empty/stale coverage map silently
  deletes the only spec for an operation.

73 feature specs are suppressed in the camunda-oca config (covered by
the 26 lifecycle scenarios across the 4 active templates).

Closes #331
Copilot AI review requested due to automatic review settings May 21, 2026 04:32
@jwulf jwulf self-assigned this May 21, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a coverage-derived suppression mechanism so per-endpoint Playwright *.feature.spec.ts files are not emitted for operations already exercised by emitted scenario-template lifecycle/state-transition specs, and writes a coverage.json artefact to make this suppression auditable and testable.

Changes:

  • Introduces materializer/src/coverage.ts to derive covered/suppressed operationIds by scanning planner-emitted template scenario JSONs.
  • Updates the materializer --all feature emission loop to skip suppressed operations and to emit a stable coverage.json alongside generated suites.
  • Extends the scenario-template ontology schema with suppressesFeatureTest and updates L3 regression invariants to account for suppression and to assert suppressed ops are backed by an on-disk lifecycle spec.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
path-analyser/src/ontology/scenarioTemplateSchema.ts Adds suppressesFeatureTest to the TS schema for scenario templates.
ontology/vocabulary/scenario-template.schema.json Regenerates the JSON schema to include suppressesFeatureTest.
materializer/src/index.ts Builds coverage before the --all feature loop, skips suppressed ops, and writes coverage.json.
materializer/src/coverage.ts New coverage extractor that walks template scenario JSONs to derive suppression + coverage entries.
configs/camunda-oca/regression-invariants.test.ts Adjusts invariants to subtract suppressed ops and adds a new invariant validating coverage entries point to existing lifecycle specs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread materializer/src/coverage.ts
@jwulf
Copy link
Copy Markdown
Member Author

jwulf commented May 21, 2026

Reviewer note: hardcoded template registry in materializer/src/index.ts

During self-review, the templateOutputDirs map introduced in this PR (around materializer/src/index.ts:485):

templateOutputDirs: {
  EdgeLifecycle: 'edges',
  EntityLifecycle: 'entities',
  UpdatedFieldVisibleOnReadBack: 'runtime-entities',
  StateTransitionVisibleAfterAction: 'state-transitions',
}

duplicates template-name → output-dir knowledge that is already hardcoded a few lines below in the template emission loop (lines ~562–599). Adding a 5th scenario template therefore requires editing index.ts in three places (imports, emission loop, this coverage map) on top of the legitimate ontology + emitter additions.

The suppression mechanism itself is already template-agnostic — buildCoverage() only cares about step.kind ∈ {'invoke', 'observe'} and the bound operationId; nothing in it knows what an EdgeLifecycle is. What's coupled is purely the discovery of where on disk the template scenario JSONs live.

Filed as #333 to hoist a single TEMPLATE_REGISTRY in materializer/src/templateRegistry.ts so both the emission loop and buildCoverage() consume it. Post-#333, adding a template will be one entry in one file (plus the ontology and, if the shape is new, a new emitter implementation) — index.ts will stop mentioning template names entirely.

Keeping that out of this PR to preserve a clean separation: #331 introduces the policy (suppress feature specs for opIds covered by a template); #333 cleans up the plumbing it touched.

…nomy

Adds tests/codegen/coverage.test.ts with four focused assertions on
materializer/src/coverage.ts:

  1. default (no ABox) treats every template as suppressing
  2. suppressesFeatureTest: false honours the opt-out
  3. only invoke/observe contribute; prereqChain is scaffolding
  4. templates absent from templateOutputDirs are skipped

Closes a guard gap surfaced during PR #332 review: the false-opt-out
path was previously exercised only at the ABox level (no row has
suppressesFeatureTest: false today) and a future rename of the
schema field or the reader-side property access could silently
invert the default. Test 2 fails red on either drift.

Refs #331, PR #332.
@jwulf jwulf merged commit 6367987 into main May 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

materializer: suppress feature specs for operations covered by scenario-template ontology

2 participants