Skip to content

Reduce integration test CI time from 2h37m by removing redundant test executions #971

@Stevengre

Description

@Stevengre

Problem

Integration tests currently take 2h 37m (example run), which is too long for PR iteration.

Root Cause: Redundant Test Executions

Analysis of the test suite reveals significant overlap:

Test Backend Count Wall Time Purpose
test_exec_smir[*-haskell] Haskell 35 ~70 min Concrete execution + snapshot
test_exec_smir[*-llvm] LLVM 35 ~55 min Same, different backend
test_prove_termination Haskell 19 ~25 min Prove termination (same 19 programs as exec_smir depth=None subset)
test_prove_rs Haskell 49 long tail Symbolic verification

Key redundancies:

  1. exec_smir runs every test twice (LLVM + Haskell) — both verify the same snapshot output
  2. prove_termination tests the exact same 19 programs already covered by exec_smir[*-haskell]
  3. 9 out of 10 overlapping .rs files between prove-rs/ and exec-smir/ are byte-identical duplicates

Proposed Phased Approach

Phase 1: Remove redundant test runs (small PR, ~80 min savings)

  1. exec_smir: keep Haskell only, remove LLVM parametrization
    • Haskell is the backend used for proving — bugs here have higher impact
    • Removing LLVM saves ~35 tests
  2. Remove test_prove_termination entirely
    • exec_smir[*-haskell] already runs these 19 programs through the Haskell backend
    • prove_rs already covers symbolic verification

Expected result: ~2h37m → ~1h20m

Phase 2: Consolidate Rust test programs (structural refactor)

  • Merge prove-rs/, exec-smir/, run-rs/ source files into a single programs/ directory
  • Remove 9 duplicated .rs files
  • Test logic references the unified directory

Phase 3: Declarative test configuration

  • Introduce a config file (e.g., test-config.toml) mapping each program to its test modes (prove, run-haskell, parse-only)
  • Single test function reads config and dispatches accordingly

More: Consolidate redundant tests

Data

134 unique Rust programs across 4 directories:

  • prove-rs/: 84 .rs + 1 .json
  • exec-smir/: 39 .rs
  • run-rs/: 34 .rs (only used for smir-parse stability, not integration tests)
  • run-smir-random/: 2 .rs

File overlaps: prove-rs ∩ exec-smir = 10 (9 identical), prove-rs ∩ run-rs = 13

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions