feat(test): add failure triage for batch root-cause grouping#44
Open
SahilRakhaiya05 wants to merge 8 commits into
Open
feat(test): add failure triage for batch root-cause grouping#44SahilRakhaiya05 wants to merge 8 commits into
SahilRakhaiya05 wants to merge 8 commits into
Conversation
Contributor
|
@coderabbitai review |
✅ Action performedReview finished.
|
WalkthroughThis PR adds a client-side "test failure triage" CLI command that clusters a project's failed tests by root cause using existing analysis fields (fix target reference, failure kind, hypothesis) without downloading failure bundles. Includes a new clustering library, CLI wiring, dry-run sample data, tests, and documentation updates. ChangesFailure Triage Feature
Estimated code review effort: 3 (Moderate) | ~30 minutes Sequence Diagram(s)sequenceDiagram
participant User
participant CLI
participant runFailureTriage
participant TestSpriteAPI
participant buildFailureClusters
User->>CLI: testsprite test failure triage --project <id>
CLI->>runFailureTriage: parsed options (type, filter, maxConcurrency)
alt dry-run mode
runFailureTriage-->>CLI: sampleFailureTriageResult (canned)
else live mode
runFailureTriage->>TestSpriteAPI: fetch failed tests (paginated, filtered)
loop per failed test (bounded concurrency)
runFailureTriage->>TestSpriteAPI: GET /tests/{testId}/failure/summary
alt NOT_FOUND
TestSpriteAPI-->>runFailureTriage: 404 NOT_FOUND
runFailureTriage->>runFailureTriage: record skipped test
else success
TestSpriteAPI-->>runFailureTriage: failure summary fields
end
end
runFailureTriage->>buildFailureClusters: FailureTriageInput[]
buildFailureClusters-->>runFailureTriage: FailureTriageResult (clusters, summary)
end
runFailureTriage-->>CLI: render text or JSON
CLI-->>User: cluster report
Possibly related issues
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Add testsprite test failure triage --project <id> to group failed tests into root-cause clusters using existing M2.1 analysis fields. Returns a representative test per cluster, confidence score, and fix priority without downloading failure bundles. Includes grouping library, command wiring, unit/integration tests, docs, CHANGELOG entry, agent skill update, and help snapshot.
…iage - Fix test failure triage help snapshot default value quoting - Run prettier on changed files - Add filter and max-concurrency validation tests - Remove draft issue/PR markdown files from repo
SahilRakhaiya05
force-pushed
the
feat/failure-triage
branch
from
62af870 to
9f7c9e4
Compare
SahilRakhaiya05
requested review from
ruili-testsprite and
zeshi-du
as code owners
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (2)
src/lib/failure-triage.test.ts (1)
1-212: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winConsider adding coverage for
buildClusterLabel,renderFailureTriageText, and representative tie-breaks.Current tests exercise the grouping/priority/confidence heuristics well but skip
buildClusterLabel/renderFailureTriageTextentirely and only test the "prefers hypothesis" branch ofpickRepresentativeTestId(not the recency/testId tie-breaks). A test assertingcluster.labelis derived consistently withcluster.canonicalRootCause/representativeTestIdfor a multi-member hypothesis cluster would have caught the label/representative mismatch noted infailure-triage.ts.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lib/failure-triage.test.ts` around lines 1 - 212, Add tests covering the missing failure-triage paths: verify buildClusterLabel and renderFailureTriageText produce labels/text that stay aligned with cluster.canonicalRootCause and representativeTestId, especially for multi-member hypothesis clusters. Also extend pickRepresentativeTestId coverage to include the recency and testId tie-break cases, since the current suite only checks the hypothesis-preference branch. Use the existing failure-triage.test.ts helpers and symbols buildFailureClusters, buildClusterLabel, renderFailureTriageText, and pickRepresentativeTestId to locate the relevant assertions.src/commands/test.test.ts (1)
3454-3482: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a command-level missing-
--projecttest fortest failure triage.These cases only call
runFailureTriage()directly, so they won't catch the new.requiredOption('--project')parse path returning Commander's generic failure instead of the documented validation exit code. Please exercise the CLI command itself here and assert the missing-project exit behavior.As per path instructions, "Tests must be deterministic and offline" and "Cover new behavior, including error and exit-code paths."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/commands/test.test.ts` around lines 3454 - 3482, Add a command-level test for missing --project in test failure triage by exercising the CLI parser path instead of only calling runFailureTriage(), so the new .requiredOption('--project') behavior is covered. Update the existing validation tests in test.test.ts to invoke the command entrypoint for the missing-project case and assert the exact exit behavior/exit code returned by Commander, while keeping the test deterministic and offline. Use the existing runFailureTriage and test failure triage command symbols to locate the right place to extend coverage.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/commands/test.ts`:
- Around line 4067-4141: The `opts.dryRun` branch in `test.ts` is hard-coding a
triage response that should live in the shared dry-run sample registry instead.
Move this canned `FailureTriageResult` payload into
`src/lib/dry-run/samples.ts`, update the `dry-run` sample lookup/use path so the
command reads the fixture from the registry, and add or adjust the corresponding
test to cover the shared sample shape and keep `renderFailureTriageText` output
aligned.
- Around line 8920-8926: The triage command currently uses Commander’s
requiredOption for --project, which short-circuits before runFailureTriage() and
bypasses the VALIDATION_ERROR envelope and exit-5 mapping. Update the command
definition in failure.command('triage') to make --project a plain option, then
keep validation inside requireProjectId() so the existing createTestCommand()
error-path behavior and documented exit code handling remain intact.
In `@src/lib/failure-triage.ts`:
- Around line 191-212: buildClusterLabel is re-selecting its own representative
member, which can diverge from the representative already chosen in
buildFailureClusters via representativeTestId. Update buildClusterLabel to
accept and use that existing representative (or look it up by ID) instead of
falling back to members[0], so hypothesis/fix-target labels always come from the
same test as cluster.canonicalRootCause. Keep the existing label formatting
logic, but ensure the rep/testName/rootCauseHypothesis source is consistent with
the cluster’s chosen representative.
- Around line 214-221: `slugifyClusterId` is truncating cluster IDs in a way
that can cause collisions between distinct clusters. Update `slugifyClusterId`
in `failure-triage` to keep the readable slug but append a short stable
disambiguator, such as a hash of the full `groupKey`, so different `hyp:`
entries do not end up with the same `clusterId` after truncation. Make sure the
`groups`/cluster creation flow continues to use this new unique `clusterId`
shape consistently wherever `normalizeHypothesis` and downstream consumers rely
on it.
---
Nitpick comments:
In `@src/commands/test.test.ts`:
- Around line 3454-3482: Add a command-level test for missing --project in test
failure triage by exercising the CLI parser path instead of only calling
runFailureTriage(), so the new .requiredOption('--project') behavior is covered.
Update the existing validation tests in test.test.ts to invoke the command
entrypoint for the missing-project case and assert the exact exit behavior/exit
code returned by Commander, while keeping the test deterministic and offline.
Use the existing runFailureTriage and test failure triage command symbols to
locate the right place to extend coverage.
In `@src/lib/failure-triage.test.ts`:
- Around line 1-212: Add tests covering the missing failure-triage paths: verify
buildClusterLabel and renderFailureTriageText produce labels/text that stay
aligned with cluster.canonicalRootCause and representativeTestId, especially for
multi-member hypothesis clusters. Also extend pickRepresentativeTestId coverage
to include the recency and testId tie-break cases, since the current suite only
checks the hypothesis-preference branch. Use the existing failure-triage.test.ts
helpers and symbols buildFailureClusters, buildClusterLabel,
renderFailureTriageText, and pickRepresentativeTestId to locate the relevant
assertions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: c17d6b2c-df3b-47d1-88af-90fadf5de85a
⛔ Files ignored due to path filters (1)
test/__snapshots__/help.snapshot.test.ts.snapis excluded by!**/*.snap,!**/*.snap
📒 Files selected for processing (11)
.github/workflows/ci.ymlCHANGELOG.mdDOCUMENTATION.mdREADME.mdskills/testsprite-verify.skill.mdsrc/commands/test.test.tssrc/commands/test.tssrc/lib/failure-triage.test.tssrc/lib/failure-triage.tstest/help.snapshot.test.tsvitest.config.ts
- Move --dry-run payload to sampleFailureTriageResult in samples.ts - Use .option for --project so requireProjectId emits exit 5 - Build cluster labels from the chosen representative test - Append short hash suffix to clusterId to avoid slug collisions
16 tasks
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/commands/test.ts (1)
4520-4562: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winStop fan-out after the first fatal error.
reject(err)still falls through toinFlight--andstartNext(), so a non-NOT_FOUNDfailure can keep launching more/failure/summaryrequests after the operation has already failed. Add an abort guard or return immediately on fatal errors.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/commands/test.ts` around lines 4520 - 4562, Stop launching more triage requests after a fatal `/failure/summary` error. In the `startNext` flow inside `test.ts`, a non-`NOT_FOUND` failure currently calls `reject(err)` and then still decrements `inFlight` and recurses, so add a fatal-error/aborted guard or return immediately from the `.catch` path after rejecting. Make sure `startNext`, the `inFlight` bookkeeping, and the `triageInputs` fan-out all stop once a real error has been hit.
🧹 Nitpick comments (1)
src/lib/failure-triage.test.ts (1)
213-244: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winTest doesn't actually distinguish representative-based label from insertion-order label.
Both
t_firstandt_recentuse the identicalsharedHypstring, socluster.label/canonicalRootCausewould equalsharedHypregardless of which member is picked as representative — even under a buggy insertion-order implementation. The test name promises to catch that regression but the assertions can't actually detect it.♻️ Suggested fix: use distinct hypothesis text per member
it('labels hypothesis clusters from the representative test, not map-insertion order', () => { const sharedHyp = 'Shared login validation failure across checkout flows.'; + const olderHyp = 'shared login validation failure across checkout flows.'; // normalized-equal groupKey, distinct display text const result = buildFailureClusters('proj_hyp', [ makeInput({ testId: 't_first', updatedAt: '2026-06-20T00:00:00.000Z', summary: { status: 'failed', failureKind: 'assertion', snapshotId: 'snap_first', - rootCauseHypothesis: sharedHyp, + rootCauseHypothesis: 'Shared login validation failure across checkout flows (older wording).', recommendedFixTarget: null, }, }),Note the raw hypothesis text must still normalize to the same
groupKey(vianormalizeHypothesis) for both members to land in the same cluster, so any replacement text needs to collapse to the same key while differing enough in display casing/whitespace to prove selection-order matters, or simply differ in a way that keeps grouping intact while makinglabel/canonicalRootCausedistinguishable per representative.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lib/failure-triage.test.ts` around lines 213 - 244, The cluster-label test in failure-triage.test.ts does not prove representative-based labeling because both inputs share the same hypothesis text, so a buggy insertion-order implementation would still pass. Update the test data in buildFailureClusters to give each member a distinct rootCauseHypothesis that still normalizes to the same groupKey via normalizeHypothesis, then keep the assertions on representativeTestId, label, and canonicalRootCause so the chosen representative actually affects the expected value.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lib/dry-run/samples.ts`:
- Around line 355-419: The canned dry-run fixtures are using human-readable
cluster IDs instead of the hashed wire-format produced by
buildFailureClusters(). Update the sample data in samples.ts so the clusterId
values follow the cluster_${slug}_${8hex} pattern, and make the corresponding
dry-run test expect the same hashed shape. Use the existing groupKey/fix-target
entries to keep the fixture semantics unchanged while aligning the IDs with the
real response format.
---
Outside diff comments:
In `@src/commands/test.ts`:
- Around line 4520-4562: Stop launching more triage requests after a fatal
`/failure/summary` error. In the `startNext` flow inside `test.ts`, a
non-`NOT_FOUND` failure currently calls `reject(err)` and then still decrements
`inFlight` and recurses, so add a fatal-error/aborted guard or return
immediately from the `.catch` path after rejecting. Make sure `startNext`, the
`inFlight` bookkeeping, and the `triageInputs` fan-out all stop once a real
error has been hit.
---
Nitpick comments:
In `@src/lib/failure-triage.test.ts`:
- Around line 213-244: The cluster-label test in failure-triage.test.ts does not
prove representative-based labeling because both inputs share the same
hypothesis text, so a buggy insertion-order implementation would still pass.
Update the test data in buildFailureClusters to give each member a distinct
rootCauseHypothesis that still normalizes to the same groupKey via
normalizeHypothesis, then keep the assertions on representativeTestId, label,
and canonicalRootCause so the chosen representative actually affects the
expected value.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: ca395e38-f7b5-4a21-a4f2-3e16c35cdeb5
⛔ Files ignored due to path filters (1)
test/__snapshots__/help.snapshot.test.ts.snapis excluded by!**/*.snap,!**/*.snap
📒 Files selected for processing (10)
CHANGELOG.mdDOCUMENTATION.mdREADME.mdsrc/commands/test.test.tssrc/commands/test.tssrc/lib/dry-run/samples.test.tssrc/lib/dry-run/samples.tssrc/lib/failure-triage.test.tssrc/lib/failure-triage.tstest/help.snapshot.test.ts
✅ Files skipped from review due to trivial changes (3)
- README.md
- CHANGELOG.md
- DOCUMENTATION.md
🚧 Files skipped from review as they are similar to previous changes (3)
- test/help.snapshot.test.ts
- src/lib/failure-triage.ts
- src/commands/test.test.ts
Comment on lines
+355
to
+419
| clusterId: 'cluster_kind_network_timeout', | ||
| label: 'Environment issue (network_timeout)', | ||
| groupKey: 'kind:network_timeout', | ||
| groupReason: 'failure_kind', | ||
| failureKind: 'network_timeout', | ||
| representativeTestId: 'test_dryrun_a', | ||
| memberTestIds: ['test_dryrun_a', 'test_dryrun_b'], | ||
| members: [ | ||
| { | ||
| testId: 'test_dryrun_a', | ||
| testName: 'Dry-run checkout flow', | ||
| testType: 'frontend', | ||
| updatedAt: '2026-06-26T12:00:00.000Z', | ||
| status: 'failed', | ||
| failureKind: 'network_timeout', | ||
| snapshotId: 'snap_dryrun_a', | ||
| rootCauseHypothesis: null, | ||
| recommendedFixTarget: null, | ||
| }, | ||
| { | ||
| testId: 'test_dryrun_b', | ||
| testName: 'Dry-run profile update', | ||
| testType: 'frontend', | ||
| updatedAt: '2026-06-26T12:01:00.000Z', | ||
| status: 'failed', | ||
| failureKind: 'network_timeout', | ||
| snapshotId: 'snap_dryrun_b', | ||
| rootCauseHypothesis: null, | ||
| recommendedFixTarget: null, | ||
| }, | ||
| ], | ||
| canonicalRootCause: null, | ||
| confidence: 0.88, | ||
| fixPriority: 1, | ||
| }, | ||
| { | ||
| clusterId: 'cluster_ref_src_components_checkoutform_tsx_412', | ||
| label: 'Shared fix target: src/components/CheckoutForm.tsx:412', | ||
| groupKey: 'ref:src/components/CheckoutForm.tsx:412', | ||
| groupReason: 'fix_target', | ||
| failureKind: 'assertion', | ||
| representativeTestId: 'test_dryrun_c', | ||
| memberTestIds: ['test_dryrun_c'], | ||
| members: [ | ||
| { | ||
| testId: 'test_dryrun_c', | ||
| testName: 'Dry-run submit checkout', | ||
| testType: 'frontend', | ||
| updatedAt: '2026-06-26T12:02:00.000Z', | ||
| status: 'failed', | ||
| failureKind: 'assertion', | ||
| snapshotId: 'snap_dryrun_c', | ||
| rootCauseHypothesis: | ||
| 'Submit button is disabled because the credit-card field is empty.', | ||
| recommendedFixTarget: { | ||
| kind: 'code', | ||
| reference: 'src/components/CheckoutForm.tsx:412', | ||
| rationale: 'Disabled state originates from `isFormValid()`.', | ||
| }, | ||
| }, | ||
| ], | ||
| canonicalRootCause: 'Submit button is disabled because the credit-card field is empty.', | ||
| confidence: 0.7, | ||
| fixPriority: 3, | ||
| }, |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Inspect the real clusterId construction to see if the hash suffix is unconditional.
ast-grep run --pattern 'clusterId: $_' --lang typescript src/lib/failure-triage.ts
rg -n -A5 -B5 'slugifyClusterId' src/lib/failure-triage.tsRepository: TestSprite/testsprite-cli
Length of output: 958
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Read the relevant implementation and sample lines.
sed -n '215,280p' src/lib/failure-triage.ts
printf '\n--- sample ---\n'
sed -n '355,419p' src/lib/dry-run/samples.ts
printf '\n--- expected cluster ids from group keys ---\n'
python3 - <<'PY'
import hashlib
keys = [
'kind:network_timeout',
'ref:src/components/CheckoutForm.tsx:412',
]
for key in keys:
slug = ''.join(ch if ch.isalnum() else '_' for ch in key.lower())
while '__' in slug:
slug = slug.replace('__', '_')
slug = slug.strip('_')
h = hashlib.sha256(key.encode()).hexdigest()[:8]
print(key, '=>', f'cluster_{slug}_{h}')
PYRepository: TestSprite/testsprite-cli
Length of output: 5111
Align dry-run cluster IDs with the hashed wire format
buildFailureClusters() always emits cluster_${slug}_${8hex}, but these canned clusters still use unhashed IDs. Update the sample (and the matching dry-run test) so --dry-run matches the real response shape.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/lib/dry-run/samples.ts` around lines 355 - 419, The canned dry-run
fixtures are using human-readable cluster IDs instead of the hashed wire-format
produced by buildFailureClusters(). Update the sample data in samples.ts so the
clusterId values follow the cluster_${slug}_${8hex} pattern, and make the
corresponding dry-run test expect the same hashed shape. Use the existing
groupKey/fix-target entries to keep the fixture semantics unchanged while
aligning the IDs with the real response format.
Source: Path instructions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a new CLI command:
The command groups failed tests into root-cause clusters instead of returning a flat list of unrelated failures. This helps agents and developers quickly identify the highest-priority issue, investigate one representative test first, and avoid fixing the same underlying problem multiple times.
Each cluster includes:
The command uses existing TestSprite APIs only, so no backend changes are required. It fetches lightweight failure summary data per test and does not download screenshots, videos, or full failure bundles.
Problem
Today, when a batch run fails many tests, the CLI and agents only see individual failures:
This makes agents and developers:
The CLI already has strong per-test analysis through
test failure get,test failure summary,rootCauseHypothesis,recommendedFixTarget, andfailureKind.What was missing is cross-test grouping after a batch failure.
Solution
test failure triageworks in three steps:The grouping algorithm uses the following signals:
recommendedFixTarget.referencefailureKind, such asnetwork_timeoutorinfrarootCauseHypothesisClusters are ordered by fix priority first, then by member count.
Command surface
Supported options:
--project <id>— required project ID--type frontend|backend— filter failed tests by type--filter <substr>— filter tests by name substring--max-concurrency <n>— parallel summary fetches, default 5--output json|text— machine or human output--endpoint-url <url>— override API hostAlso supports global flags such as
--dry-run,--profile,--verbose, and--debug.Recommended agent workflow
The agent skill was also updated to recommend triage before downloading bundles when multiple tests fail.
Implementation details
Added new grouping logic in:
This includes:
normalizeHypothesis()computeGroupKey()pickRepresentativeTestId()computeClusterConfidence()computeFixPriority()buildFailureClusters()renderFailureTriageText()Added command implementation in:
The command validates inputs, paginates failed tests, applies filters, fetches summaries with bounded concurrency, handles stale failed rows, and emits JSON or text output through the existing output system.
Test coverage
This PR adds 18 automated tests:
src/lib/failure-triage.test.tssrc/commands/test.test.tsCoverage includes:
Future work
Out of scope for this PR:
GET /projects/{id}/failures/clustersAPIrootCauseHypothesis--rerun-representatives --waitorchestration flagChecklist
The Pr solving issue #116
Summary by CodeRabbit
New Features
test failure triagecommand to group failed tests into root-cause clusters and show a representative test, affected test IDs, confidence, and fix priority.Documentation