[Hackathon] feat: test failure triage — group batch failures by root cause

[SahilRakhaiya05]

Discord Username / User ID

uxdev

What does this improvement do?

New CLI command to triage failed tests by grouping related failures into deterministic clusters.

When many tests fail in the same project, such as during test run --all, regression reruns, or MCP execution, agents and humans currently see a flat list of individual failures. In practice, many of those failures often share the same root cause, such as an expired auth token, staging environment outage, broken routing, shared code defect, or cascading test data failure.

Today, the CLI only supports per-test failure analysis through commands like test failure get and test failure summary. This forces agents to download many bundles or guess which failed test to inspect first, wasting AI tokens, testing credits, and debugging time.

This improvement adds:

testsprite test failure triage --project <id>

The command lists failed tests, fetches lightweight failure summaries, and groups related failures client-side using deterministic heuristics over existing failure metadata. This helps users quickly identify the most likely root cause and choose the best representative test to investigate first.

Details / implementation notes

Implement a CLI-first failure triage command:

testsprite test failure triage --project <id>

Behavior:

1. List all failed tests using the existing failed-test API surface.

2. Fetch lightweight failure or summary information for each failed test.

3. Do not download screenshots, videos, or full bundles.

4. Group failures client-side using deterministic heuristics based on existing M2.1 fields.

5. Return clusters containing:

   • representativeTestId
   • memberTestIds
   • confidence
   • fixPriority

Suggested grouping heuristics:

• Shared recommendedFixTarget.reference

• Environment-wide failure kinds such as:

  • infra
  • network
  • network_timeout
  • routing_404

• Normalized rootCauseHypothesis prefix

• Singleton fallback when no grouping signal is found

Supported flags:

--project <id>
--output json
--type
--filter
--max-concurrency
--dry-run

Acceptance criteria:

• testsprite test failure triage --project <id> --output json returns clustered failure output.
• Clusters include representativeTestId, memberTestIds, confidence, and fixPriority.
• Command uses existing public APIs only.
• Command does not require backend changes.
• Command avoids downloading heavy artifacts such as screenshots and videos.
• --type is supported.
• --filter is supported.
• --max-concurrency is supported.
• --dry-run returns a canned sample.
• Unit tests cover grouping heuristics and singleton fallback.
• Integration tests cover command behavior.
• Documentation is updated.
• CHANGELOG is updated.
• Agent skill documentation is updated.

Why CLI-first:

• Uses only existing public APIs.
• Reduces duplicate investigation immediately.
• Avoids unnecessary bundle downloads.
• Saves AI tokens, testing credits, and debugging time.
• Provides a natural read surface for future backend-native clustering.

Future backend follow-up:

Add a native clustering API using semantic embeddings, wave/cascade graph analysis, and optional --rerun-representatives orchestration.

Confirmations

• I have searched existing issues and this is not a duplicate.
• I have provided my Discord identity above for reward coordination.

[zeshi-du]

Triage: parked for design — client-side heuristic clustering can actively mislead agents when it groups wrong, and this overlaps the project-report surface we are redesigning against the V3 schema. Leaving the issue open as the anchor; PR #44 will wait on that design.