[DRAFT] - Updates to Mutation Testing#11996
Closed
emo-eth wants to merge 100 commits into
Closed
Conversation
emo-eth
requested review from
0xrusowsky,
DaniPopes,
grandizzy,
mattsse,
onbjerg,
yash-atreya and
zerosnacks
as code owners
3 tasks
1 task
gakonst
added a commit
that referenced
this pull request
Jan 15, 2026
Implements parallel mutation testing for Foundry, building on PR #11996. Key features: - Parallel execution via rayon thread pool with configurable --mutation-jobs - Isolated TempDir workspaces per mutant for safe concurrent execution - Adaptive span skipping shared across workers (skip mutations in spans where a mutation already survived) - Fail-fast per mutant (stops on first test failure) - Symlinks for lib directories to avoid expensive copies - Preserves project layout (custom src/test/libs paths supported) Safety improvements: - Path traversal protection (rejects .. components) - catch_unwind prevents single panic from aborting entire run - Symlinked directories skipped in copy to prevent traversal attacks - 16MB stack size for rayon threads to avoid overflow Performance: 2.5x speedup observed with 4 workers on 150 mutants Closes #478
Contributor
|
Closing in favor of #13091, commits have been included |
mablr
added a commit
that referenced
this pull request
Jun 3, 2026
* feat: init * chore: lexing/parsing * chore: visiting contracts * chore: wip * chore: wip * chore: wip * chore: visitor not visiting anything * chore: visitor visiting * chore: quick refactor before its too late * chore: quick refactor * chore: mutation gen part1 * feat: mutation collection * feat: visitor refactor * feat: visitor refactor * feat: temp folder mgmt * feat: temp file creation logic * feat: compiling mutants * chore: wip future multithread * feat: mutation set building * feat: multithread compile * chore: fmt * chore: fmt * chore: fmt * chore: fmt * feat: refactor for test runner * chore: test runner wip * feat: working poc * chore: doc * chore: fmt * chore: fmt * chore: fmt * feat: assign mut gen * feat: unary mut * feat: binary mut * feat: members unary mut * feat: members unary mut * feat: rm delegatecall mut * chore: refactor modular mut wip * chore: refactor modular mut wip * chore: refactor modular mut wip * feat: mutator assign trait * chore: fmt * chore: fmt * feat: bin mutator * feat: other mutator mod * feat: visitor refactor registry * chore: merge fix * chore: comment * fix: solar visitor use * chore: clippy and min refactors * feat: assign mutator tests * feat: test gen mut ident * feat: test binop and delete expr * feat: test delegate unaryop mut * feat: test unaryop mut * feat: wip generic test * refactor: add mutate-path and mutate-contract optional args, keep mutate args working like before * feat: enable contract match filter in mutation tests * feat: optional mutators * feat: generic mutator test(length) * feat: generic mutator test(content) * fix: unary mut for bool * test: unary mut * test: all mutators * test: add neg case * chore: refactor visitor * chore: clippy * chore: typos * feat: integ test (hacky) * feat: basic reporting * fix: copy only src * fix: copy only src * fix: rel path * fix: add walker in visitor * fix: assign mutator * chore: dbg cleanup * fix: dyn test linking * fix: missing mutant fmt * fix: path in mut ctxt * feat: save and resume mutation tests * chore: typo and doc * chore: remove cache file * merge master * clippy * update * use serde instead of manual dto for json caching checkpoint * fix cache filenames * initial pass at adaptive testing - skip surviving sibling or child spans * clippy && fmt * feat(forge): parallel mutation testing with isolated workspaces Implements parallel mutation testing for Foundry, building on PR #11996. Key features: - Parallel execution via rayon thread pool with configurable --mutation-jobs - Isolated TempDir workspaces per mutant for safe concurrent execution - Adaptive span skipping shared across workers (skip mutations in spans where a mutation already survived) - Fail-fast per mutant (stops on first test failure) - Symlinks for lib directories to avoid expensive copies - Preserves project layout (custom src/test/libs paths supported) Safety improvements: - Path traversal protection (rejects .. components) - catch_unwind prevents single panic from aborting entire run - Symlinked directories skipped in copy to prevent traversal attacks - 16MB stack size for rayon threads to avoid overflow Performance: 2.5x speedup observed with 4 workers on 150 mutants Closes #478 * fix: clippy, rustfmt, and config test for mutation_dir * feat(mutation): add progress display with Ctrl+C support - Add progress bar showing runs/total and job count - Show relative file paths instead of absolute paths - Add spinner for active mutants with line number and mutation diff - Capture original expression context in Mutant struct - Center code truncation around mutated operator - Graceful Ctrl+C: cancel pending, show report for completed, exit 130 - Enhanced reporter with Solidity diffs and security suggestions - Add mutation_testing_with_show_progress CLI test Amp-Thread-ID: https://ampcode.com/threads/T-019be5ee-0615-7048-97dc-7ba7b354d384 Co-authored-by: Amp <amp@ampcode.com> * chore: rename 'runs' to 'mutation runs' in progress bar * fix: mark doc code block as text to fix doctest * refactor: address grandizzy review comments - Move mutation logic from test/mod.rs to mutation/orchestrator.rs - Implement test helper for mutator tests - Add snapbox assertions to CLI mutation tests with full output - Update Solar LitKind comment (still valid - no int/uint distinction) - Use 4 workers in show-progress test for consistency Amp-Thread-ID: https://ampcode.com/threads/T-019be9ee-1c7c-734d-8e72-7e9b837b3945 Co-authored-by: Amp <amp@ampcode.com> * Add time and report legend * feat(mutation): add RequireMutator and source hash cache validation - Add RequireMutator for security-critical require/assert mutations - require(x) -> require(true) - always passes - require(x) -> require(false) - always fails - require(x) -> require(!x) - inverted condition - Add source content hash to SurvivedSpans cache - Prevents stale span reuse when source changes - Invalidates cache on hash mismatch - Handles legacy cache format migration - Add comprehensive integration test (mutation_testing_require_mutator) - Demonstrates two-phase testing workflow - First run: basic tests (84% score) - Second run: comprehensive tests (98.1% score) - Shows only equivalent mutant survives Amp-Thread-ID: https://ampcode.com/threads/T-019beb12-9720-740b-b4b6-de4f92eb10d6 Co-authored-by: Amp <amp@ampcode.com> * Add unary_op_mutator_test module to tests * fmt imports * Update crates/forge/src/mutation/mutators/mod.rs * nits * Update crates/forge/src/mutation/mutators/assignment_mutator.rs * nits * refactor: replace ctrlc with tokio signal, update rstest to 0.26 Addresses review comments from @zerosnacks: - Remove ctrlc dependency, use tokio's built-in signal handling instead - Update rstest from 0.25.0 to 0.26 - Add tokio signal feature for Ctrl+C handling The ctrlc crate is replaced with a background thread running a tokio runtime that waits on tokio::signal::ctrl_c(). This avoids adding the ctrlc dependency which is on the deny list. Amp-Thread-ID: https://ampcode.com/threads/T-019bec4d-844c-7624-9054-1b7e6b056c2c Co-authored-by: Amp <amp@ampcode.com> * feat(mutation): add --json support for mutation testing When --json flag is used with --mutate, outputs JSON with: - Summary: total, killed, survived, invalid, skipped, mutation_score, duration_secs - Survived mutants grouped by file with line, column, original expression, and mutant All progress messages are suppressed in JSON mode for clean output. Amp-Thread-ID: https://ampcode.com/threads/T-019bf8c4-8b62-741e-9226-e5cf23a050db Co-authored-by: Amp <amp@ampcode.com> * fix(mutation): symlink node_modules and soldeer dependencies in temp workspace (#13221) Projects using npm (node_modules/) or Soldeer (dependencies/) for external dependencies fail mutation testing because these directories are not copied to the temporary mutation workspace. This fix adds symlinking for these common external dependency directories, following the same pattern already used for lib directories. Fixes compilation errors like: error: 'node_modules/@uniswap/v3-core/...' not found Tested with: - Uniswap/universal-router (npm dependencies) - morpho-org/morpho-blue (lib dependencies) Amp-Thread-ID: https://ampcode.com/threads/T-019bf98e-1307-72f9-ac5f-a00d33518d40 Co-authored-by: Amp <amp@ampcode.com> * fix(mutation): recursively symlink nested lib directories in temp workspace (#13224) * fix(mutation): recursively symlink nested lib directories in temp workspace Projects with nested git submodules (e.g., lib/euler-earn/lib/openzeppelin-contracts) fail mutation testing because the temp workspace only symlinks top-level lib directories. When libraries have their own dependencies in nested lib/ folders, and remappings reference them with context-specific prefixes like: lib/euler-earn:@OpenZeppelin=lib/euler-earn/lib/openzeppelin-contracts/ These paths need to exist in the mutation workspace for compilation to succeed. This fix adds symlink_nested_libs() which recursively walks through each library and symlinks any nested lib/ directories, ensuring deeply nested submodules are accessible in the temp workspace. Tested against euler-xyz/evk-periphery which has multiple levels of nested submodules. Amp-Thread-ID: https://ampcode.com/threads/T-019bfa2f-d0b0-703a-9822-945c87b1e2ab Co-authored-by: Amp <amp@ampcode.com> * test(mutation): add unit tests for symlink and copy functions Tests cover: - symlink_dir: basic symlink creation - symlink_nested_libs: single level, deeply nested (3 levels), no lib dir, skip existing - copy_dir_recursive: basic copy, skip symlinked dirs, nonexistent source - relative_to_root: basic, same path, outside root Amp-Thread-ID: https://ampcode.com/threads/T-019bfa2f-d0b0-703a-9822-945c87b1e2ab Co-authored-by: Amp <amp@ampcode.com> * style: format with nightly rustfmt Amp-Thread-ID: https://ampcode.com/threads/T-019bfa2f-d0b0-703a-9822-945c87b1e2ab Co-authored-by: Amp <amp@ampcode.com> * fix(mutation): support custom lib directories in nested submodules Read each nested library's foundry.toml config to get its actual lib paths instead of hardcoding 'lib'. Falls back to default 'lib' if no config exists. Amp-Thread-ID: https://ampcode.com/threads/T-019bfab1-a4b5-7008-8fb4-735aa3a560ac Co-authored-by: Amp <amp@ampcode.com> * clippy fix --------- Co-authored-by: Amp <amp@ampcode.com> Co-authored-by: zerosnacks <zerosnacks@protonmail.com> * feat(mutation): add Yul opcode mutator (#13341) * feat: make mutation test operators configurable (#13410) make mutators configurable Amp-Thread-ID: https://ampcode.com/threads/T-019c477b-60f0-71af-853e-f2b5902f0906 Co-authored-by: Amp <amp@ampcode.com> * refactor: rename UnaryOperatorMutator to UnaryOpMutator (#13411) * make mutators configurable Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-019c477b-60f0-71af-853e-f2b5902f0906 * refactor: rename UnaryOperatorMutator to UnaryOpMutator Amp-Thread-ID: https://ampcode.com/threads/T-019c477b-60f0-71af-853e-f2b5902f0906 Co-authored-by: Amp <amp@ampcode.com> --------- Co-authored-by: Amp <amp@ampcode.com> * fix: address mutation testing issues (#13412) * fix: address critical mutation testing issues - Rename orchestrator MutationConfig to MutationRunConfig to avoid naming collision with foundry_config::MutationConfig - Make generate_mutated_solidity and restore_original_source return Result instead of panicking, use byte-based splicing to avoid UTF-8 boundary panics - Validate relative paths in copy_project_for_mutation with is_safe_relative_path to prevent path escape when config paths are not under the project root - Fix cache key collisions by including a hash of the full contract path in cache filenames (prevents src/A.sol vs contracts/A.sol from sharing cache entries) - Make generate_ast return Result and propagate parse errors instead of silently swallowing them Amp-Thread-ID: https://ampcode.com/threads/T-019c477b-60f0-71af-853e-f2b5902f0906 Co-authored-by: Amp <amp@ampcode.com> * fix: harden mutation testing with path validation and diagnostics - Validate user-provided --mutate paths: normalize relative paths against config.root, check existence/is_file/.sol/non-test - Fix --mutate-contract to match any contract in a multi-contract file (use .any() instead of .find().is_some_and()) - Add span bounds check in generate_mutated_solidity to prevent panics on invalid span positions - Replace unwrap() with expect() in MutationContextBuilder calls with descriptive messages for each AST node type - Fix cross-platform relative_path: use Path::components() with forward-slash joining for stable output across OSes Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-019c47bc-5807-7721-9051-7028de802b89 * refactor: address review feedback for mutation testing Extract cache_file_path helper to consolidate repeated cache dir and filename prefix logic across persist/retrieve methods. Extract ensure_safe_relative_path helper to replace repeated validate-and-bail pattern for path safety checks. Remove dead code: generate_mutated_solidity and restore_original_source are unused (runner.rs uses its own apply_mutation with checked slicing and temp directories). Amp-Thread-ID: https://ampcode.com/threads/T-019c47ec-22ab-75db-9c35-35e0e08ac396 Co-authored-by: Amp <amp@ampcode.com> --------- Co-authored-by: Amp <amp@ampcode.com> * chore: remove dead code from mutation module (#13413) * fix: address critical mutation testing issues - Rename orchestrator MutationConfig to MutationRunConfig to avoid naming collision with foundry_config::MutationConfig - Make generate_mutated_solidity and restore_original_source return Result instead of panicking, use byte-based splicing to avoid UTF-8 boundary panics - Validate relative paths in copy_project_for_mutation with is_safe_relative_path to prevent path escape when config paths are not under the project root - Fix cache key collisions by including a hash of the full contract path in cache filenames (prevents src/A.sol vs contracts/A.sol from sharing cache entries) - Make generate_ast return Result and propagate parse errors instead of silently swallowing them Amp-Thread-ID: https://ampcode.com/threads/T-019c477b-60f0-71af-853e-f2b5902f0906 Co-authored-by: Amp <amp@ampcode.com> * fix: harden mutation testing with path validation and diagnostics - Validate user-provided --mutate paths: normalize relative paths against config.root, check existence/is_file/.sol/non-test - Fix --mutate-contract to match any contract in a multi-contract file (use .any() instead of .find().is_some_and()) - Add span bounds check in generate_mutated_solidity to prevent panics on invalid span positions - Replace unwrap() with expect() in MutationContextBuilder calls with descriptive messages for each AST node type - Fix cross-platform relative_path: use Path::components() with forward-slash joining for stable output across OSes Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-019c47bc-5807-7721-9051-7028de802b89 * refactor: address review feedback for mutation testing Extract cache_file_path helper to consolidate repeated cache dir and filename prefix logic across persist/retrieve methods. Extract ensure_safe_relative_path helper to replace repeated validate-and-bail pattern for path safety checks. Remove dead code: generate_mutated_solidity and restore_original_source are unused (runner.rs uses its own apply_mutation with checked slicing and temp directories). Amp-Thread-ID: https://ampcode.com/threads/T-019c47ec-22ab-75db-9c35-35e0e08ac396 Co-authored-by: Amp <amp@ampcode.com> * chore: remove dead code from mutation module Remove unused symbols superseded by the orchestrator and parallel runner architecture: - MutationsSummary: update_valid_mutant, dead/survived/invalid/skipped string formatters, get_skipped, get_report_mut - Mutant: line_number(source) method (shadowed by field), format_diff, description_with_context - ParallelMutationRunner struct and run_mutations_parallel wrapper (replaced by orchestrator + run_mutations_parallel_with_progress) - Stale re-exports and unused imports Amp-Thread-ID: https://ampcode.com/threads/T-019c47ec-22ab-75db-9c35-35e0e08ac396 Co-authored-by: Amp <amp@ampcode.com> --------- Co-authored-by: Amp <amp@ampcode.com> * refactor(mutation): extract workspace utilities into shared module (#13423) Move project-copying logic from mutation/runner.rs into a new crates/forge/src/workspace.rs module for reuse by other features that need isolated temp workspaces (e.g., brutalization). Extracted functions: copy_project, copy_dir_recursive, symlink_dir, symlink_nested_libs, relative_to_root, ensure_safe_relative_path. Tests moved from mutation/runner.rs to workspace.rs. Amp-Thread-ID: https://ampcode.com/threads/T-019c4d7b-acac-75ef-9f69-f3619f6941b6 Co-authored-by: Amp <amp@ampcode.com> * fix: resolve clippy warnings in mutation module Co-authored-by: zerosnacks <95942363+zerosnacks@users.noreply.github.com> Amp-Thread-ID: https://ampcode.com/threads/T-019d873c-bf66-7018-96d5-0d0e7d34e540 * fix: use then_some instead of then with closure Co-authored-by: zerosnacks <95942363+zerosnacks@users.noreply.github.com> Amp-Thread-ID: https://ampcode.com/threads/T-019d873c-bf66-7018-96d5-0d0e7d34e540 * fix: make default_excluded const fn Co-authored-by: zerosnacks <95942363+zerosnacks@users.noreply.github.com> Amp-Thread-ID: https://ampcode.com/threads/T-019d873c-bf66-7018-96d5-0d0e7d34e540 * fix: add const to eligible mutation module functions Co-authored-by: zerosnacks <95942363+zerosnacks@users.noreply.github.com> Amp-Thread-ID: https://ampcode.com/threads/T-019d873c-bf66-7018-96d5-0d0e7d34e540 * fix: revert with_path to non-const (PathBuf has destructor) Co-authored-by: zerosnacks <95942363+zerosnacks@users.noreply.github.com> Amp-Thread-ID: https://ampcode.com/threads/T-019d873c-bf66-7018-96d5-0d0e7d34e540 * fix: make builder() const fn Co-authored-by: zerosnacks <95942363+zerosnacks@users.noreply.github.com> Amp-Thread-ID: https://ampcode.com/threads/T-019d873c-bf66-7018-96d5-0d0e7d34e540 * fix: address review feedback - Fix typo: 'Pathe' -> 'Path' in mutation_dir doc comment - Add depth limit (max 10) to symlink_nested_libs to prevent infinite recursion - Support all network types (Eth/OP/Tempo) in mutation testing, not just EthEvmNetwork Co-authored-by: zerosnacks <95942363+zerosnacks@users.noreply.github.com> Amp-Thread-ID: https://ampcode.com/threads/T-019d873c-bf66-7018-96d5-0d0e7d34e540 * fix: rustfmt and add depth arg to test calls Co-authored-by: zerosnacks <95942363+zerosnacks@users.noreply.github.com> Amp-Thread-ID: https://ampcode.com/threads/T-019d873c-bf66-7018-96d5-0d0e7d34e540 * test(mutation): replace rstest with mutator_tests! macro Each parameterized case becomes a standalone #[test] (parallel execution, individual failure reporting, IDE run buttons) without pulling in rstest. Also fixes a pre-existing clippy::question-mark warning in assignment_mutator.rs surfaced by make lint. Amp-Thread-ID: https://ampcode.com/threads/T-019e0314-c110-735c-8858-123078e60e37 Co-authored-by: Amp <amp@ampcode.com> * fix(mutation): address PR review feedback - mutant.rs: add missing Pow/Rem variants to BinOpKind deserializer - elim_delegate_mutator: narrow mutation span to the 'delegatecall' identifier so replacement does not clobber the surrounding call - unary_op_mutator: fix Member operand swap (e.g. 'a.b++' was producing '++ba') - require_mutator: use AST argument spans instead of string-splitting on the first comma, so 'require(foo(a, b))' is handled correctly - mutation/mod.rs: include hash of enabled operators in the cache file name so changes to include/exclude_operators invalidate the cache - orchestrator: only load survived-spans cache after mutants are successfully obtained - visitor: add per-contract name filter; visit_item_contract toggles an 'in_allowed_contract' flag that gates expr/var/yul mutation collection, so --mutate-contract filters per contract rather than per file - cmd/test: add clap conflicts_with between --mutate-path and --mutate-contract; add runtime error when --mutate has explicit paths combined with --mutate-path Amp-Thread-ID: https://ampcode.com/threads/T-019e4020-a7ef-70a8-84e0-a44a82f56780 Co-authored-by: Amp <amp@ampcode.com> * feat(mutation): add --mutation-timeout + richer progress UX (WIP) Mirrors invariant.timeout for mutation campaigns: - MutationConfig.timeout / --mutation-timeout SECS CLI flag - MutationResult::TimedOut variant (separate from Invalid/Alive) - Per-mutant enforcement via std::thread + mpsc::recv_timeout in runner::run_compile_and_test_with_timeout; worker slot freed immediately on timeout (background work unwinds via gas limit) - Adaptive survived-span cache only marked for genuine Alive results - Cache key folds in timeout so changed budget invalidates results - Reporter row + legend + JSON 'timed_out' field Progress UX (needs further iteration on formatting): - Live tally on overall bar (k:/s:/i:/t:/sk:) + elapsed_precise - Keyed active-mutant tracking so parallel completions remove the correct row instead of FIFO - Per-result completion lines printed above bars via multi.suspend Tested end-to-end: 70-mutant project with --mutation-timeout 2 sees 4 mutants TimedOut at exactly 2.0s, score correctly excludes them. Unit tests + CLI mutation snapshot tests pass. Amp-Thread-ID: https://ampcode.com/threads/T-019e4020-a7ef-70a8-84e0-a44a82f56780 Co-authored-by: Amp <amp@ampcode.com> * fix(mutation): enforce per-mutant timeout in fuzz/invariant harness; clean up progress UX - runner: propagate config.mutation.timeout into temp_config.fuzz.timeout and temp_config.invariant.timeout so the inner FuzzTestTimer bails out at the deadline. Previously the outer recv_timeout returned TimedOut but the leaked worker thread kept running expensive fuzz/invariant runs and starved the pool — a 200k-runs fuzz test would not respect --mutation-timeout at all. Never raises a user-configured fuzz/invariant timeout, only lowers it. - progress: drop the fake static '[0.0s]' prefix from in-flight spinner messages. Replace the emoji-prefixed completion lines (✗/⚠/⏱) with a plain color-coded label (KILLED green, SURVIVED red, TIMED OUT yellow) padded to 9 chars so columns align after ANSI escapes. - reporter: strip prescriptive prose. Drops the 'These mutations were NOT caught…' preamble, the entire Security Implications section, and the Suggestions to improve test coverage block. Shortens the legend to factual definitions. Removes the emoji (⚠/✓/ℹ/⏱) and parenthetical opinions from the survived/killed/invalid/timed-out section headers. - Disambiguate MutationProgress::clear via UFCS to silence yansi Paint::clear trait shadow warning. - Snapshot tests regenerated for the new reporter shape. Amp-Thread-ID: https://ampcode.com/threads/T-019e4061-d738-70bd-8f43-3fc319258ed2 Co-authored-by: Amp <amp@ampcode.com> * fix(mutation): rustfmt, no-default-features, config snapshot - runner: gate OpEvmNetwork import + dispatch behind #[cfg(feature = "optimism")] so cargo check --no-default-features builds. Mirrors the pattern already used in cmd/test/mod.rs. - tests/cli/config.rs: update test_default_config snapshot for the new mutation.timeout field. - Re-run cargo +nightly fmt across the mutation module touched by previous commits. Amp-Thread-ID: https://ampcode.com/threads/T-019e4061-d738-70bd-8f43-3fc319258ed2 Co-authored-by: Amp <amp@ampcode.com> * fix(mutation): address oracle review blockers and highs Blockers: - runner: move TempDir ownership into worker thread so timed-out workers can no longer race against a deleted workspace; park JoinHandles in SharedMutationState.pending_workers and join them at the end of the parallel run to actually reclaim cleanup - workspace: add ensure_within_root containment check (canonicalize + starts_with) for src/test/lib/node_modules/dependencies so a symlinked root cannot escape the project; validate nested lib dirs from untrusted dependency foundry.toml (reject .., absolute paths, symlinked nested roots); use entry.file_type() to skip symlinked entries instead of following them Highs: - orchestrator: do not persist cached results on a cancelled or short run; would otherwise be reloaded as the authoritative answer for the file - mutation/mod: include --mutate-contract regex in cache key so cached results from a different filter aren't silently reused - cmd/test: bail when mutation is requested with inline per-test network overrides (single-pass runner can't honor them) or with ffi/write fs_permissions (shared symlinked dep trees aren't safe to mutate from tests) Adds 4 workspace tests for the new symlink-escape protections. Amp-Thread-ID: https://ampcode.com/threads/T-019e4567-e7ca-717e-bcc0-bb67a3667c4d Co-authored-by: Amp <amp@ampcode.com> * fix(mutation): key cached results by execution inputs (#14971) * fix(mutation): key cached results by execution inputs * fix(mutation): include execution inputs in cache key * fix: clean-up * fix(mutation): honor filters and isolation (#14848) * fix(mutation): key cached results by execution inputs * fix(mutation): include execution inputs in cache key * fix(mutation): honor test filters/isolation, tighten cache key, drop compound assignment mutation, reject incompatible flags - runner: thread FilterArgs and --isolate through MutationRunConfig into compile_and_test_inner; rebuild ProjectPathsAwareFilter against the per-mutant temp config so --match-test/--match-contract/--match-path work and mutants exercise the same test set/execution model as baseline - mod: add runtime_context_digest folded into the per-file cache key so cached results aren't reused when filters, isolation, fork URL/block, sender, or initial_balance change - orchestrator: compute runtime_context_digest once per run and bind it to every MutationHandler - binary_op_mutator: stop mutating compound assignments (a += b would silently be rewritten to a - b dropping the assignment); add regression tests - cmd/test: bail when --mutate is combined with --list/--debug/ --flamegraph/--flamechart/--junit instead of silently mixing modes - fs_permissions guard now narrowed to permissions whose path can reach symlinked dependency trees (lib/node_modules/dependencies) Amp-Thread-ID: https://ampcode.com/threads/T-019e4567-e7ca-717e-bcc0-bb67a3667c4d Co-authored-by: Amp <amp@ampcode.com> * fix(mutation): address fresh oracle review blockers and mediums Blockers - filter parity: pass materialized baseline filter (filter.args().clone()) into MutationRunConfig so positional 'forge test <path>' and --rerun reach mutant runs too; raw self.filter dropped them silently - bail when baseline matched zero tests; without it every compileable mutant was reported as 'Alive' - negative literal mutation: add OwnedLiteral::NegatedNumber(U256) formatted as '-{val}'; the old Number(-*val) wrapped via two's complement and produced wrong-source mutants - mutator test harness: wrap snippets in valid Solidity, panic on parse failure, and check emitted replacement text. Fixed 6 stale test expectations and 1 unmatched delegate-pattern case the old vacuous harness was hiding Mediums - survived-span resume: load retrieve_survived_spans BEFORE mutant generation/cached load and filter cached mutants through should_skip_span so cross-run adaptive skipping actually works - runtime context digest: hash the full serialized EvmOpts (networks, env, gas-limit toggles, fork, sender, balance, ...) instead of a hand-picked 4-field subset that missed several mutation-affecting knobs - deterministic output: BTreeMap for JSON survived_mutants with sorted entries; reporter sorts survived list by (path, line, col, span, mutation); persisted results_vec sorted by span - --mutate-contract: intersect with explicit --mutate <paths> instead of silently overriding them; explicit paths are now respected - workspace: copy 'script/' when present and distinct from src/test so projects that import script-side helpers don't see false Invalid mutants Amp-Thread-ID: https://ampcode.com/threads/T-019e4567-e7ca-717e-bcc0-bb67a3667c4d Co-authored-by: Amp <amp@ampcode.com> * fix: `BTreeMap` import * fix mutation compound assignment handling * test(mutation): compare full mutator output sets * fix: clean-up * fix mutation testing review feedback * test: update require mutator snapshot * fix: mutation cache filtering and resumed skips * fix(mutation): preserve survivors when resuming * fix: test output * fix(forge): harden mutation testing cache and isolation Reject incompatible coverage/showmap mutation runs, make dependency fs_permissions checks path-aware through symlinked ancestors, validate result caches against the current mutant set, and restore live same-span adaptive skipping while keeping resume semantics strict. * fix: mutation compile filters --------- Co-authored-by: Mablr <59505383+mablr@users.noreply.github.com> Co-authored-by: Amp <amp@ampcode.com> * fix(mutation): reject mutation under machine mode (#15009) fix(forge): reject mutation under machine mode * fix(mutation): reject empty mutation selections (#15010) fix(forge): reject empty mutation selections * fix(forge): preserve mutation workspace config (#15011) * fix(forge): preserve mutation workspace config * fix: make clippy happy * fix(mutation): use source spans for unary targets (#15014) fix(forge): use source spans for unary targets * fix(mutation): surface mutator generation errors (#15015) * fix(forge): surface mutator generation errors * fix: surface mutator generation errors with best-effort model * fix(mutation): bound timed-out mutation cleanup (#15016) * fix(forge): bound timed-out mutation cleanup * fix: `Default` impl for `SharedMutationState` * fix(mutation): filter no-op mutation outputs (#15013) * fix(forge): filter no-op mutation outputs * fix: bless * fix require condition inversion * fix(mutation): copy mutation compiler paths (#15012) * fix(forge): copy mutation compiler paths * fix: copy mutation compiler paths * fix(mutation): handle cancellation without progress UI (#15030) * fix(mutation): handle cancellation without progress UI * fix: according review * refactor mutation state constructor --------- Co-authored-by: drgorillamd <83670532+drgorillamd@users.noreply.github.com> Co-authored-by: Simon Something /DrGoNoGo <83670532+simon-something@users.noreply.github.com> Co-authored-by: 0xchin <alanracciatti1220@gmail.com> Co-authored-by: emo.eth <emodoteth@gmail.com> Co-authored-by: grandizzy <38490174+grandizzy@users.noreply.github.com> Co-authored-by: grandizzy <grandizzy.the.egg@gmail.com> Co-authored-by: Amp <amp@ampcode.com> Co-authored-by: zerosnacks <95942363+zerosnacks@users.noreply.github.com> Co-authored-by: zerosnacks <zerosnacks@protonmail.com> Co-authored-by: Derek Cofausper <256792747+decofe@users.noreply.github.com> Co-authored-by: Mablr <59505383+mablr@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Updates to #10193 - made it a separate pr into master because I have merged a lot of changes from master.
Updates
Improvements:
Spans already have surviving mutants and skips running tests on Mutants of duplicate or sub-SpansThings to check/explicitly test
Proposed "v0" todos:
hir) for fewer invalid mutations.mutations-snapshotProposed "v1" todos: