Skip to content

fix(dry-run): include failed run-scoped step sample#198

Open
JerryNee wants to merge 2 commits into
TestSprite:mainfrom
JerryNee:fix/dry-run-run-step-sample
Open

fix(dry-run): include failed run-scoped step sample#198
JerryNee wants to merge 2 commits into
TestSprite:mainfrom
JerryNee:fix/dry-run-run-step-sample

Conversation

@JerryNee

@JerryNee JerryNee commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • enrich the getRun dry-run sample with a failed run-scoped assertion step carrying non-null error text
  • align the sample run metadata (status, failedStepIndex, failureKind, stepSummary) with that failed step
  • add fixture and command-level coverage so test steps --run-id ... --dry-run keeps surfacing the failed step error and contributor flag

Context

Follow-up to the maintainer request on #136 for the dry-run sample enrichment slice that #167 did not cover.

Fixes #197

Tests

  • npm test -- src/lib/dry-run/samples.test.ts src/lib/client-factory.test.ts src/commands/test.test.ts -t "runSteps|findSample|makeHttpClient"
  • npm run typecheck
  • npm run lint
  • git diff --check

Summary by CodeRabbit

  • Bug Fixes
    • Improved dry-run get run results to correctly represent terminal failed runs, including the correct failed step details, failure kind, and associated error text.
    • Updated the dry-run sample mapping so requesting the failed-run scenario returns the failed-run response shape instead of the passed one.
  • Tests
    • Added coverage for dry-run getRun responses and stdout JSON for the failed-run path, ensuring the failing and non-failing steps are classified correctly.

@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: a94185da-7354-456a-9862-208faa112fee

📥 Commits

Reviewing files that changed from the base of the PR and between 4b2e45b and 9179281.

📒 Files selected for processing (3)
  • src/commands/test.test.ts
  • src/lib/dry-run/samples.test.ts
  • src/lib/dry-run/samples.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/commands/test.test.ts

Walkthrough

The dry-run getRun sample now resolves to either a passed or failed run fixture, with a sentinel run id selecting the failed run shape. Tests were added to verify the failed sample response and the runSteps dry-run mapping for the failing step.

Changes

Dry-run failed step sample

Layer / File(s) Summary
getRun sample data and selection
src/lib/dry-run/samples.ts
The getRun dry-run fixture is split into passed and failed samples, the passed sample is registered for the default run response, and the sample resolver returns the failed fixture for the sentinel failed run id.
Tests validating failed sample and runSteps mapping
src/lib/dry-run/samples.test.ts, src/commands/test.test.ts
The dry-run sample test and the runSteps test assert the failed run shape, the failing assertion step, and the CLI JSON mapping for the failing step.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • TestSprite/testsprite-cli#167: Related to the runSteps handling of per-step error and stepType mapping exercised by this PR's new dry-run test.

Suggested reviewers: ruili-testsprite, zeshi-du

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed Title accurately summarizes the dry-run failed run sample change.
Linked Issues check ✅ Passed The added sentinel failed run sample and tests satisfy the linked issue's failure-shape and metadata requirements.
Out of Scope Changes check ✅ Passed The changes stay within the requested dry-run sample and test scope.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@zeshi-du zeshi-du left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Appreciate the intent (and the companion issue #197 makes the case well), but this reintroduces a defect we already fixed once, so I can't take it in this shape.

History: until 2026-05-21 the GET /runs/{runId} dry-run sample WAS failed-shaped, and it was a bug — test wait --dry-run and test run --wait --dry-run map terminal status through exitCodeForRunStatus (passed → 0, anything else → 1), so every offline happy-path smoke exited 1 and agents learned the wrong canned response. That's why the sample was deliberately flipped to passed-shape (see the comment this PR deletes — "defect-2"). This PR flips it back: with it merged, testsprite test wait run_x --dry-run && next-step breaks again. The rewritten snapshot test makes CI green, but green-by-rewriting-the-guard is exactly what that guard was for.

The underlying tension is real: one route-keyed sample serves two consumers (test wait wants passed; test steps --run-id wants an instructive failure). Two ways to resolve it that I'd accept:

  1. Sentinel run-id: teach findSample to special-case a documented sample id (e.g. run_failed_sample) that returns the failed shape, keeping the default passed. test steps --run-id run_failed_sample --dry-run then demonstrates the failure mapping, and wait flows stay green.
  2. Route the failure demo elsewhere: the failed-test sample already exists on the testId-scoped result surface (SAMPLE_TEST_ID_FAILED); extend that entry with the step-level error detail instead of touching the run-scoped default.

Either keeps both consumers correct. Want to take one of these? (If you go with (1), please also restore the passed-shape guard test alongside the new failed-shape one for the sentinel id.)

@JerryNee

JerryNee commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed context. I updated this PR to take option 1 (sentinel run-id):

  • restored the default GET /runs/{runId} dry-run sample to the passed shape, preserving the test wait --dry-run happy path
  • added a documented sentinel run id, run_failed_sample, that returns the failed run-scoped step sample
  • restored the default passed-shape guard and added coverage for the sentinel failed sample
  • updated the runSteps dry-run test to use run_failed_sample so the failure mapping remains demonstrated without changing the default run sample

Validation run locally:

  • npm test -- src/lib/dry-run/samples.test.ts src/commands/test.test.ts
  • npm run typecheck
  • npm run format:check
  • npm run lint
  • NO_COLOR= npm test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Hackathon] dry-run run sample should include a failed run-scoped step

2 participants