Skip to content

feat(exceptions): support external client exception mappers#2776

Draft
shrirajpawar4 wants to merge 2 commits into
ethereum:forks/amsterdamfrom
shrirajpawar4:feat/yaml-exception-mappers
Draft

feat(exceptions): support external client exception mappers#2776
shrirajpawar4 wants to merge 2 commits into
ethereum:forks/amsterdamfrom
shrirajpawar4:feat/yaml-exception-mappers

Conversation

@shrirajpawar4
Copy link
Copy Markdown

🗒️ Description

Implements YAML-backed external exception mappers for client-owned error-string mappings.

Today, client exception mappings live in EEST Python source, so client teams need an execution-specs PR whenever their error strings change. This adds a small external mapper layer so client repos can
keep those mappings next to their client code while EEST keeps the built-in mappers as defaults.

External mappings are additive only:

  • built-in Python mappers remain unchanged
  • external YAML mappings extend the built-in mapper
  • duplicate mapped exceptions are de-duplicated
  • unknown messages still fall back to UndefinedException

YAML v1 format:

version: 1
name: geth-ci
substring:
  TransactionException.INSUFFICIENT_ACCOUNT_FUNDS:
    - "insufficient funds for gas * price + value"
regex:
  BlockException.INVALID_GASLIMIT:
    - 'child gas_limit \d+ .*'

Integration points:

- validate.toml:

  [geth]
  bin = "../go-ethereum/build/bin/evm"
  exception-mapper = "../go-ethereum/eest-exceptions.yaml"
- Hive consume:

  uv run consume engine --exception-mapper geth=../go-ethereum/eest-exceptions.yaml

This is stacked on #2622 because that PR introduces the new direct validate workflow where client-owned exception mappings are most useful for CI.

🔗 Related Issues or PRs

Fixes #2631.
Depends on #2622.

✅ Checklist

  • All: Ran fast static checks to avoid unnecessary CI fails:

    just static

  • All: PR title adheres to the repo standard and starts type(scope):.

  • All: Considered updating the online docs in the ./docs/ directory.

  • All: Set appropriate labels for the changes (only maintainers can apply labels).

Focused checks run:

uv run ruff check packages/testing/src/execution_testing/exceptions/external.py packages/testing/src/execution_testing/exceptions/__init__.py packages/testing/src/execution_testing/exceptions/tests/
test_external.py packages/testing/src/execution_testing/cli/pytest_commands/plugins/consume/simulators/exceptions.py packages/testing/src/execution_testing/cli/pytest_commands/plugins/consume/
simulators/sync/conftest.py packages/testing/src/execution_testing/cli/pytest_commands/plugins/consume/tests/test_exception_mappers.py packages/testing/src/execution_testing/cli/pytest_commands/
plugins/validate/conftest.py packages/testing/src/execution_testing/cli/pytest_commands/plugins/validate/tests/test_exception_mappers.py

uv run pytest packages/testing/src/execution_testing/exceptions/tests packages/testing/src/execution_testing/cli/pytest_commands/plugins/consume/tests packages/testing/src/execution_testing/cli/
pytest_commands/plugins/validate/tests/test_exception_mappers.py

Result: 41 passed, 5 warnings. The warnings are existing collect-only dummy consumer warnings.

spencer-tb and others added 2 commits April 8, 2026 20:46
New `validate` CLI command for running EEST fixtures directly against
client EVM binaries, replacing Hive for execution correctness testing.

Usage:
  validate health                    # health check all clients
  validate engine --client geth      # engine tests
  validate state --client besu       # state tests
  validate block --client nethermind # block tests

Features:
- 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus
- Per-type Pydantic result models: StateTestResult, BlockTestResult,
  EngineTestResult with type-specific fields
- Exception matching: maps client error strings to EEST exception
  types via ExceptionMapper, verifies correct exception for every
  invalid test (--no-exception-check to disable)
- Cross-validation: lastBlockHash against fixture, lastPayloadStatus
  (VALID/INVALID) for engine tests
- validate.toml config for client binary paths with per-type overrides
  (state-bin, block-bin, engine-bin)
- Auto bin-workers and xdist tuning per client
- Bundled Frontier sanity fixtures for health checks
- Shared validate_helpers.py for validation logic

Client binary PRs:
- geth: ethereum/go-ethereum#34650
- erigon: erigontech/erigon#20315
- besu: besu-eth/besu#10184
- nethermind: NethermindEth/nethermind#11035
- reth: paradigmxyz/reth#23361
- ethrex: lambdaclass/ethrex#6445
- nimbus: status-im/nimbus-eth1#4101
- revm: bluealloy/revm#3544

Tracking issue: ethereum#2319
@felix314159
Copy link
Copy Markdown
Contributor

this pr is impossible to review right now. you should wait until the pr it depends on is merged and then rebase.

@felix314159 felix314159 marked this pull request as draft April 29, 2026 09:21
@shrirajpawar4
Copy link
Copy Markdown
Author

yes okay

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable definition of client exception mappers via "external" file

3 participants