Close coverage gap vs upstream e2e suite (negative-path + search-refinement emitters)

[esraagamal6]

Context

Generator coverage assessment in PR #278 (kept open as living scaffolding). Current state vs upstream's hand-written OCA suite (c8-orchestration-cluster-e2e-test-suite/coverage-analysis/, snapshot camunda/camunda#53387 head 7cf8bc1):

• Generator: 1617 unique tests vs upstream 1001 (+616). Per-bucket comparison and per-category detail in PR chore: add coverage analysis for generated tests (#275) #278.
• Real capability gaps (verified by grepping every generated test file):
  • expect(status).toBe(401) — 0 generator tests vs 165 upstream
  • expect(status).toBe(403) — 0 generator tests vs 29 upstream
  • expect(status).toBe(409) — 0 generator tests vs 31 upstream
  • expect(status).toBe(404) for a fake/never-existing ID — 0 generator tests vs 127 upstream (generator does assert 404 in 48 places, but those are observe-absence after-delete tests, not fake-ID tests — see PR chore: add coverage analysis for generated tests (#275) #278 README note)
  • Pagination/filter behaviour assertions (e.g. "page 2 returns the next N items") — 0 generator (the variant emitter sends page.after cursor and filter: {} in request bodies, but only asserts status 200; upstream's 53 pagination + 85 filter tests assert results)

Prioritisation and "should we build all of these" flows from the coverage-analyzer work in #277.

See coverage-analysis/gaps.md on the PR #278 branch for the categorised per-entity list.

Already done — not in this backlog

These were listed in earlier versions of this issue; current assessment shows they're already covered by other emitters or have been added since:

• 400 bad-request emitter — the request-validation emitter produces 1071 tests across 17 violation kinds (additional-prop, constraint-violation, enum-violation, missing-body, missing-required, oneof-ambiguous/cross-bleed/none-match, param-missing, type-mismatch, union, unique-items-violation, etc.). Generator exceeds upstream's 195 occurrences by 5.5×.
• observe-absence-after-delete — the EntityLifecycle (feat: EntityLifecycle scenario template (create→present→delete→absent) — refines #279 item 3 #280) and EdgeLifecycle (Planner + emitter consumption of edge-lifecycle scenario template (Phase 2 of #268) #270) templates produce 22 lifecycle tests, each ending with expect(status).toBe(404) after the delete step. 48 observe-absence label occurrences in total.

Checklist (cheapest first; LLM-pacing estimates)

• 404 fake-ID emitter (~1 day) — for every endpoint with a path param, emit a test that replaces the ID with a generated invalid-but-syntactically-valid value (e.g. 999999999999) and asserts status === 404. Distinct from observe-absence (post-delete) which lifecycle templates already cover. Computable from path params + ontology/ semantic types (already encode key formats).
• Pagination/sort behaviour assertions (~1–2 days) — for every search endpoint, emit a test that: seeds N items, fetches with page: { limit: K }, asserts exactly K items returned; fetches next page, asserts no overlap with first page; emits a sort variant and asserts result order. The variant emitter already exercises the request shape — this adds the assertions.
• Filter behaviour assertions (~3–5 days) — for every search endpoint and every filter field, emit a test that: seeds two items differing in that field, filters by one value, asserts only that item is returned. Needs filter-schema introspection + ABox knowledge of which fields are filterable per entity.
• 401 unauthorized emitter — depends on a deployment-mode encoding axis (which APIs / test-kinds are applicable per deployment mode). See camunda/camunda#52511. Strip auth header from every endpoint, expect 401. Computable once the deployment-mode axis lands.
• 409 conflict emitter — depends on duplicatePolicy ABox slice (idempotent / conflict / replace; designed in 8.8, never landed). See How do we calculate the test coverage of an API? #277. Hold until the ABox slice lands; then the emitter is just "call create twice, assert second returns 409 if duplicatePolicy=conflict".
• 403 forbidden emitter — needs three things, none of which exist yet:
  1. An RBAC ABox slice mapping { operationId → requiredPermission(s) }. No such slice exists in configs/camunda-oca/ontology/ today (verified). Spec-coverage-analyzer (feat: spec-coverage-analyzer spike (#277) #300) flags 190 endpoints that would need an entry — upper bound for "one 403 test per endpoint". Could be seeded from upstream's 29 hand-written forbidden tests, which encode endpoint → required permission in test names like "Search Tenant - 403 Forbidden - When User Lacks Required Permission".
  2. Test infrastructure for restricted tokens — ability to mint a token with a known permission set (or strip a specific permission), call the endpoint, assert 403.
  3. Deployment-mode wiring — relates to Encode deployment-mode availability as a first-class axis of the semantic graph camunda#52511; only meaningful when running against an auth-enabled deployment.
     Largest infra cost in the backlog; lowest test count. Defer until (1) lands as an ABox slice.

Doing items 1–3 (404 fake-ID + pagination/filter behaviour) covers the behaviour-assertion gap for the slices the generator already sends request shapes for. Items 4–6 are blocked on ABox/infra work in #277.

Caveat

Generator-emitted negatives can verify status code + response shape, not specific error semantics. Hand-written tests can assert which 404 is returned ("Create Authorization - 404 - not existing ownerId" vs the generic 404). Numeric coverage and semantic depth are different things.

How to use this issue

Split checklist items into standalone issues when picked up. Re-prioritise based on #277's findings as they land. This umbrella stays as the planning record.

[jwulf]

LLM time estimates are based on its training data: so "humans doing the thing".

1 week = 1 day or less for an LLM.

@esraagamal6 does the negative test coverage take into account the request-validation negative test suite?

observe-absence-after-delete emitter (~30 tests, ~1 day) — after every delete in the feature emitter, GET expecting 404.

This one we can close quickly. The new test scenario ontology introduced for edge lifecycles is a more sophisticated test scenario planner. My estimate: 1 hour.

A lot of these (401, 403) require encoding whether the test server is running with auth or not. Relates to camunda/camunda#52511.

That issue encodes which APIs are available depending on the deployment mode. This new axis encodes what kinds of tests are applicable, and introduces a new kind of test: unauthorized.

[jwulf]

@esraagamal6 we now have systematic coverage of entity lifecycles — 10 new lifecycle tests.

Can you please analyse the disjoint between the 10 new generated tests and the ~30 tests in the OCA suite.

[esraagamal6]

@jwulf — addressed all four points; updated the issue description to match:

• Time estimates retuned to LLM pacing (1 week → 1 day where relevant).
• Negative coverage / request-validation — yes, included now. Our original analysis missed it; once we scanned generated/camunda-oca/request-validation/, the picture flipped. The generator emits 1071 bad-request tests across 17 violation kinds, which exceeds upstream's 195. The 400-emitter line is now under "Already done" rather than in the backlog.
• observe-absence-after-delete — confirmed done via the EntityLifecycle (feat: EntityLifecycle scenario template (create→present→delete→absent) — refines #279 item 3 #280) and EdgeLifecycle (Planner + emitter consumption of edge-lifecycle scenario template (Phase 2 of #268) #270) templates: 22 lifecycle tests, each ending with expect(status).toBe(404) after the delete step. 48 observe-absence label occurrences across the suite. Removed from the backlog.
• 401/403 deployment-mode axis — noted on the 401/403 backlog items, pinned to Encode deployment-mode availability as a first-class axis of the semantic graph camunda#52511.

For your second comment (the 10-test disjoint), the analysis lives in coverage-analysis/lifecycle_disjoint.md on the PR #278 branch. TL;DR: 10 generator lifecycle tests vs 333 upstream A-CRUD tests for the same 9 entities; generator catches "full lifecycle in a single atomic flow" + post-delete observe-absence that upstream barely has (only global-task-listener has 1); upstream catches per-entity create/update variants (e.g. authorization × 5 owner types = 10 create tests) + 47 search-related happy-path + 194 negatives (400 covered by the generator's request-validation; 401/403/404-fake-ID/409 are the real residual gap).