Skip to content

feat(cli): --retry <n> to rerun failed tests (flaky mitigation) #737

Description

@Chemaclass

Summary

Add a --retry <n> flag (and BASHUNIT_RETRY env var) that re-runs each failed test up to n extra times before marking it failed. Mitigates flaky tests in CI without hiding real failures.

Motivation

Flaky tests (timing, network, filesystem races) fail intermittently and poison CI. A bounded retry lets a genuinely-flaky test recover while a consistently-broken test still fails after n attempts. --stop-on-failure already exists but does the opposite; there is no retry today (verified: grep -- --retry src/*.sh → none).

Proposed API

bashunit --retry 2 tests/            # each failed test retried up to 2 more times (3 total)
BASHUNIT_RETRY=2 bashunit tests/     # env equivalent
  • Default: 0 (current behaviour, no retry).
  • A test is reported passed if any attempt passes; failed only if all attempts fail.
  • Output should note when a test passed only on retry (e.g. PASSED (retry 2/2)), so flakiness stays visible.
  • Retries apply per-test, and must work with --parallel.

Files

  • Flag parsing + env: src/globals.sh, src/console_header.sh/src/helpers.sh (follow how --test-timeout / BASHUNIT_TEST_TIMEOUT is wired — added in Test timeouts #721).
  • Execution loop: src/runner.sh (per-test run/report path).
  • Docs: docs/command-line.md (new flag section) + docs/configuration.md (env var).
  • Tests: tests/acceptance/bashunit_test.sh (CLI end-to-end), tests/unit/runner_test.sh if unit-testable.

TDD steps

  1. RED: acceptance test — a fixture test that fails on attempt 1 and passes on attempt 2 (use a temp-file counter) is reported PASSED with --retry 1; the same test with --retry 0 is reported FAILED.
  2. RED: a test that always fails is still FAILED after --retry 2 and shows 3 attempts.
  3. GREEN: minimal retry loop around the per-test run.
  4. REFACTOR: keep parallel path working; verify ./bashunit --parallel --retry 1 tests/.

Acceptance criteria

  • --retry <n> and BASHUNIT_RETRY parsed; default 0.
  • Failed test retried up to n times; passes if any attempt passes.
  • Retry-passed tests visibly annotated in output.
  • Works with --parallel.
  • Assertion/test counters remain correct (no double-counting).
  • Docs + CHANGELOG (Unreleased → Added) updated.

Constraints

  • Bash 3.0+ only (no declare -A, no [[ ]], no ${var,,}). See .claude/rules/bash-style.md.
  • Must pass make sa, make lint, shfmt -w ., and both ./bashunit tests/ and ./bashunit --parallel tests/.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Fields

No fields configured for issues without a type.

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions