[Hackathon] Runs return verdict: blocked on verbose/multi-branch assertions even when the page is correct

[lxcario]

Summary

Frontend runs intermittently finish with verdict: blocked / executionStatus: completed even when the page under test rendered correctly. The trigger is the shape of the plan/assertion, not a product defect: verbose "either A or B … and not broken" OR-assertions, or plans that navigate many items sequentially, exhaust the agent's step/runway budget before it emits a clean pass/fail verdict.

Environment

• CLI: testsprite (latest)
• Test type: frontend (--plan-from)

Reproduction pattern

1. Author a plan whose final assertion is a two-branch OR, e.g. "verify either an interactive graph canvas OR a clear empty-state message, rather than an empty or broken container."
2. Run against a page that clearly satisfies one branch (heading + empty-state button visible).
3. The agent re-checks both branches; its own run summary states the page is correct ("PASS"), but the verdict comes back blocked because the budget ran out before a decisive verdict was emitted.

The same failure mode reproduces with plans that walk many nav items in a single run (e.g. 11 sidebar links sequentially — every executed step passed, but the run blocked before finishing).

Expected

• Verbose / conditional assertions shouldn't consume enough budget to block a verdict on a correct page, or
• The runner should surface a clearer signal that the block was budget/plan-shape driven (distinct from an infra block), and/or
• The practical assertion-complexity ceiling should be documented so authors know the limit up front.

Actual

verdict: blocked with no product defect. The only reliable fix we found was to rewrite each assertion to be single and decisive ("verify heading X and button Y") — a plan-authoring workaround rather than a runner behavior.

Impact

Correct pages report as blocked, which is easy to misread as a product failure. Authors have to learn by trial that FE assertions must be minimal — this isn't documented anywhere.

Suggested fix

Either raise (and clearly report) the per-run step budget, distinguish "blocked (budget / plan-shape)" from "blocked (infra)", or document an assertion-complexity guideline for frontend plans.

---

Filed as part of TestSprite S3 Hackathon dogfooding (project: Nora — https://github.com/lxcario/Nora). Happy to share the exact plan JSON, the test plan put fix, and run IDs for the blocked→green arcs.

[lxcario]

Maintainers: could you please add the hackathon and needs-triage labels to this issue? It was filed via gh --body-file, which bypasses the issue-template auto-labeling, and external contributors can't self-apply labels. Thanks!

[zeshi-du]

Confirmed and appreciated — this is one of the better-characterized reports we've had on verdict quality. The budget-exhaustion-reads-as-blocked behavior and the missing distinction between a budget block and an infra block are both real, engine-side, and now tracked internally. Two practical notes meanwhile: keep FE assertions single and decisive (your workaround is the current best practice), and if you can share the run IDs for the blocked→green arcs here, they'll go straight into the investigation. Leaving this open as the public tracking issue.

[lxcario]

Thanks — glad it's useful! Run IDs for the blocked→green arcs below. Project 4ba5d8f8-310d-41bc-bbf4-b85208bb6d44.

Verbose two-branch OR-assertion → blocked even though the page rendered correctly (the core of this issue). Fix in both cases was test plan put tightening the assertion to a single decisive check ("heading X + button Y"), then a rerun:

• Knowledge Web (test 6191b308-c665-4a17-a80a-46bab6cf8f08)

  • blocked → runId d3109a89-90f3-4b53-92ea-2edd3c70898a (2026-07-03 14:55Z) — page was correct (heading + "Generate Knowledge Web" empty-state button visible), agent burned runway re-checking both OR branches.
  • passed → runId 2464187a-3751-454d-b703-17c793578303 (2026-07-03 15:20Z) after tightening the assertion.

• Eureka (test ceb34635-e9f4-4b00-8583-79c3afd1b3c6)

  • blocked → runId 1a269881-0cc1-46aa-9d85-a0a2bf777f51 (2026-07-03 14:56Z) — identical pattern.
  • passed → runId 2a520929-4838-48a5-912a-76e3bf345324 (2026-07-03 15:17Z).

Budget-exhaustion → blocked (data-dependent / many-step plans), for the "budget block vs infra block" distinction:

• Review card full flow (test 452f36a7-1067-448b-b7bb-02cd7bf30647)

  • blocked runs: f0417199-e1c9-46af-91c7-91ce4e606a55 (07-02 13:00Z), ebfa229e-7442-4510-948f-d68ce9e09b79 (13:26Z), c341079e-3e2d-4b48-8455-7d277a7f676a (13:56Z) — agent graded cards until the deck emptied, then kept looking.
  • passed → 1d69263a-1362-4c96-9805-664a735ac12b (07-02 14:04Z) after accepting the session-complete/empty state as a valid outcome.

• Feynman validation-error (test e13c538f-3570-43b4-bebb-fe91fd8288d8)

  • blocked: 2aa9940e-718f-454a-9cf5-fa601e135386 (07-04 09:14Z), 085e0c00-9f09-41e4-945e-606a941added (09:37Z) → passed ec49921e-f514-4e4c-bb8f-a666108626eb (10:47Z).

In every case the run summary itself reported the page as correct, so the block was plan-shape/budget driven rather than a product defect — which is exactly why surfacing "blocked (budget)" vs "blocked (infra)" would help. Happy to share failure bundles for any of these if useful.