feat(forge): mutation testing by gakonst · Pull Request #13091 · foundry-rs/foundry

gakonst · 2026-01-15T01:15:52Z

Summary

Implements parallel mutation testing for Foundry, building on PR #11996 from @emo-eth.

Key Features

Parallel execution via rayon thread pool with configurable --mutation-jobs
Isolated TempDir workspaces per mutant for safe concurrent execution
Adaptive span skipping shared across workers (skip mutations in spans where a mutation already survived).
Note: the number of skipped/invalid mutants may vary with worker count since more parallelism means more mutants start testing before any complete and mark spans as survived.
Fail-fast per mutant (stops on first test failure)
Symlinks for lib directories to avoid expensive copies
Preserves project layout (custom src/test/libs paths supported)

Safety Improvements

Path traversal protection (rejects ../ components)
catch_unwind prevents single panic from aborting entire run
Symlinked directories skipped in copy to prevent traversal attacks
16MB stack size for rayon threads to avoid overflow

Performance

~2.5x speedup observed with 4 workers on 150 mutants (Vault.sol test contract):

4 workers: 15.3s
1 worker: 37.9s

Usage

# Run mutation testing with 4 parallel workers
forge test --mutate src/Contract.sol --mutation-jobs 4

# Auto-detect parallelism (defaults to available CPUs)
forge test --mutate src/Contract.sol --mutation-jobs 0

Changes

New file: crates/forge/src/mutation/runner.rs (526 lines) - parallel runner
Modified: crates/forge/src/cmd/test/mod.rs - CLI integration
Tests: 31 unit tests + 2 CLI integration tests

Closes #478
Closes OSS-1

Built on the excellent foundation from #11996

Regular run

Run with progress

Final report:

* fix(mutation): key cached results by execution inputs * fix(mutation): include execution inputs in cache key * fix: clean-up

* fix(mutation): key cached results by execution inputs * fix(mutation): include execution inputs in cache key * fix(mutation): honor test filters/isolation, tighten cache key, drop compound assignment mutation, reject incompatible flags - runner: thread FilterArgs and --isolate through MutationRunConfig into compile_and_test_inner; rebuild ProjectPathsAwareFilter against the per-mutant temp config so --match-test/--match-contract/--match-path work and mutants exercise the same test set/execution model as baseline - mod: add runtime_context_digest folded into the per-file cache key so cached results aren't reused when filters, isolation, fork URL/block, sender, or initial_balance change - orchestrator: compute runtime_context_digest once per run and bind it to every MutationHandler - binary_op_mutator: stop mutating compound assignments (a += b would silently be rewritten to a - b dropping the assignment); add regression tests - cmd/test: bail when --mutate is combined with --list/--debug/ --flamegraph/--flamechart/--junit instead of silently mixing modes - fs_permissions guard now narrowed to permissions whose path can reach symlinked dependency trees (lib/node_modules/dependencies) Amp-Thread-ID: https://ampcode.com/threads/T-019e4567-e7ca-717e-bcc0-bb67a3667c4d Co-authored-by: Amp <amp@ampcode.com> * fix(mutation): address fresh oracle review blockers and mediums Blockers - filter parity: pass materialized baseline filter (filter.args().clone()) into MutationRunConfig so positional 'forge test <path>' and --rerun reach mutant runs too; raw self.filter dropped them silently - bail when baseline matched zero tests; without it every compileable mutant was reported as 'Alive' - negative literal mutation: add OwnedLiteral::NegatedNumber(U256) formatted as '-{val}'; the old Number(-*val) wrapped via two's complement and produced wrong-source mutants - mutator test harness: wrap snippets in valid Solidity, panic on parse failure, and check emitted replacement text. Fixed 6 stale test expectations and 1 unmatched delegate-pattern case the old vacuous harness was hiding Mediums - survived-span resume: load retrieve_survived_spans BEFORE mutant generation/cached load and filter cached mutants through should_skip_span so cross-run adaptive skipping actually works - runtime context digest: hash the full serialized EvmOpts (networks, env, gas-limit toggles, fork, sender, balance, ...) instead of a hand-picked 4-field subset that missed several mutation-affecting knobs - deterministic output: BTreeMap for JSON survived_mutants with sorted entries; reporter sorts survived list by (path, line, col, span, mutation); persisted results_vec sorted by span - --mutate-contract: intersect with explicit --mutate <paths> instead of silently overriding them; explicit paths are now respected - workspace: copy 'script/' when present and distinct from src/test so projects that import script-side helpers don't see false Invalid mutants Amp-Thread-ID: https://ampcode.com/threads/T-019e4567-e7ca-717e-bcc0-bb67a3667c4d Co-authored-by: Amp <amp@ampcode.com> * fix: `BTreeMap` import * fix mutation compound assignment handling * test(mutation): compare full mutator output sets * fix: clean-up * fix mutation testing review feedback * test: update require mutator snapshot * fix: mutation cache filtering and resumed skips * fix(mutation): preserve survivors when resuming * fix: test output * fix(forge): harden mutation testing cache and isolation Reject incompatible coverage/showmap mutation runs, make dependency fs_permissions checks path-aware through symlinked ancestors, validate result caches against the current mutant set, and restore live same-span adaptive skipping while keeping resume semantics strict. * fix: mutation compile filters --------- Co-authored-by: Mablr <59505383+mablr@users.noreply.github.com> Co-authored-by: Amp <amp@ampcode.com>

# Conflicts: # crates/forge/src/cmd/test/mod.rs

fix(forge): reject mutation under machine mode

fix(forge): reject empty mutation selections

* fix(forge): preserve mutation workspace config * fix: make clippy happy

fix(forge): use source spans for unary targets

* fix(forge): surface mutator generation errors * fix: surface mutator generation errors with best-effort model

* fix(forge): bound timed-out mutation cleanup * fix: `Default` impl for `SharedMutationState`

* fix(forge): filter no-op mutation outputs * fix: bless * fix require condition inversion

* fix(forge): copy mutation compiler paths * fix: copy mutation compiler paths

* fix(mutation): handle cancellation without progress UI * fix: according review * refactor mutation state constructor

mablr

Given all previous fixes and thorough local testing, I think it's ready for a final review pass.

Let's consider it "experimental" like symbolic testing.

grandizzy

re reviewed and gtg for the MVP 🚀

grandizzy · 2026-06-03T07:57:27Z

@mablr could oyu pls fix merge conflicts? ty!

mablr

Approved for MVP, LFG!

simon-something added 30 commits February 27, 2025 12:54

feat: init

c0ffee9

chore: lexing/parsing

c0ffeea

chore: visiting contracts

c0ffeef

chore: wip

c0ffeea

chore: wip

c0ffee4

chore: wip

c0ffee0

chore: visitor not visiting anything

c0ffee0

chore: visitor visiting

c0ffee1

chore: quick refactor before its too late

c0ffee0

chore: quick refactor

c0ffee1

chore: mutation gen part1

c0ffee1

feat: mutation collection

c0ffeed

feat: visitor refactor

c0ffeee

feat: visitor refactor

c0ffeeb

feat: temp folder mgmt

c0ffeef

feat: temp file creation logic

c0ffeec

feat: compiling mutants

c0ffee6

chore: wip future multithread

c0ffee2

feat: mutation set building

c0ffeeb

feat: multithread compile

c0ffee1

chore: fmt

c0ffee8

chore: fmt

c0ffeeb

chore: fmt

c0ffeef

chore: fmt

c0ffeee

feat: refactor for test runner

c0ffeec

chore: test runner wip

c0ffee4

feat: working poc

c0ffee1

chore: doc

c0ffee2

chore: fmt

c0ffee5

chore: fmt

c0ffeec

mablr and others added 15 commits May 29, 2026 19:46

Merge branch 'master' into mutation-testing-fast

101c29e

fix(mutation): key cached results by execution inputs (#14971)

6c0d06e

* fix(mutation): key cached results by execution inputs * fix(mutation): include execution inputs in cache key * fix: clean-up

Merge branch 'master' into mutation-testing-fast

3841222

Merge branch 'master' into mutation-testing-fast

dac38d9

# Conflicts: # crates/forge/src/cmd/test/mod.rs

fix(mutation): reject mutation under machine mode (#15009)

3ce9b66

fix(forge): reject mutation under machine mode

fix(mutation): reject empty mutation selections (#15010)

fc77e99

fix(forge): reject empty mutation selections

fix(forge): preserve mutation workspace config (#15011)

eaaa755

* fix(forge): preserve mutation workspace config * fix: make clippy happy

fix(mutation): use source spans for unary targets (#15014)

273331f

fix(forge): use source spans for unary targets

fix(mutation): surface mutator generation errors (#15015)

7dba35b

* fix(forge): surface mutator generation errors * fix: surface mutator generation errors with best-effort model

fix(mutation): bound timed-out mutation cleanup (#15016)

65180ab

* fix(forge): bound timed-out mutation cleanup * fix: `Default` impl for `SharedMutationState`

fix(mutation): filter no-op mutation outputs (#15013)

b340423

* fix(forge): filter no-op mutation outputs * fix: bless * fix require condition inversion

fix(mutation): copy mutation compiler paths (#15012)

0ae6090

* fix(forge): copy mutation compiler paths * fix: copy mutation compiler paths

fix(mutation): handle cancellation without progress UI (#15030)

44c32b4

* fix(mutation): handle cancellation without progress UI * fix: according review * refactor mutation state constructor

Merge branch 'master' into mutation-testing-fast

29dc2b6

mablr marked this pull request as ready for review June 2, 2026 18:53

mablr reviewed Jun 2, 2026

View reviewed changes

grandizzy requested review from figtracer and mablr June 3, 2026 07:36

grandizzy previously approved these changes Jun 3, 2026

View reviewed changes

Merge branch 'master' into mutation-testing-fast

f2594d3

mablr dismissed grandizzy’s stale review via f2594d3 June 3, 2026 08:07

grandizzy self-requested a review June 3, 2026 08:08

grandizzy approved these changes Jun 3, 2026

View reviewed changes

mablr approved these changes Jun 3, 2026

View reviewed changes

mablr enabled auto-merge (squash) June 3, 2026 08:10

mablr merged commit 8f6a9da into master Jun 3, 2026
18 checks passed

github-project-automation Bot moved this to Done in Foundry Jun 3, 2026

mablr deleted the mutation-testing-fast branch June 3, 2026 08:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(forge): mutation testing#13091

feat(forge): mutation testing#13091
mablr merged 166 commits into
masterfrom
mutation-testing-fast

gakonst commented Jan 15, 2026 •

edited by mablr

Loading

Uh oh!

mablr left a comment

Uh oh!

grandizzy left a comment

Uh oh!

grandizzy commented Jun 3, 2026

Uh oh!

mablr left a comment •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Uh oh!

Conversation

gakonst commented Jan 15, 2026 • edited by mablr Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Features

Safety Improvements

Performance

Usage

Changes

Uh oh!

mablr left a comment

Choose a reason for hiding this comment

Uh oh!

grandizzy left a comment

Choose a reason for hiding this comment

Uh oh!

grandizzy commented Jun 3, 2026

Uh oh!

mablr left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

gakonst commented Jan 15, 2026 •

edited by mablr

Loading

mablr left a comment •

edited

Loading