feat: Docker-based E2E test framework with chaos testing

[AlexCheema]

Motivation

We had no end-to-end testing for exo clusters. Unit tests can't catch issues in node discovery, master election, or multi-node coordination. We also need a framework for chaos testing (network failures, partitions, etc.) to build confidence in cluster resilience.

Changes

Adds a Python/asyncio E2E test framework that spins up 2-node exo clusters in Docker Compose:

• e2e/Dockerfile — Multi-stage build: Node.js dashboard → Rust nightly + Python 3.13. Cleans up Rust build artifacts to keep the image small. Includes a g++ wrapper for MLX CPU JIT compatibility with GCC 14.
• e2e/conftest.py — Cluster class wrapping docker compose: build, start, stop, logs, exec, place_model, chat. Async context manager with automatic cleanup.
• e2e/run_all.py — Test runner discovering test_*.py files. Supports --slow / E2E_SLOW=1 to include inference tests.
• 3 tests:
  • test_cluster_formation — Nodes discover each other via mDNS, elect a master, API responds.
  • test_no_internet — iptables blocks all outbound traffic except private subnets and multicast. Verifies cluster forms without internet, confirms connectivity is actually blocked (curl + exo's own "Internet connectivity: False" log).
  • test_inference_snapshot (slow) — Launches mlx-community/Qwen3-0.6B-4bit, sends a chat completion with seed=42, temperature=0, verifies output matches a committed snapshot. Skipped in CI (x86 MLX CPU too slow), runs on Apple Silicon with --slow.
• .github/workflows/e2e.yml — CI workflow on push/PR. Frees disk space before Docker build (Rust compilation is heavy).

Why It Works

• mDNS discovery works on Docker bridge networks — multicast 224.0.0.251 stays within the bridge.
• No-internet isolation uses iptables (not Docker internal: true, which blocks multicast). NET_ADMIN capability lets containers set up firewall rules before starting exo.
• .venv/bin/exo is called directly instead of uv run, avoiding PyPI resolution at container startup.
• Deterministic inference — MLX respects mx.random.seed() with temperature=0 for reproducible output.

Test Plan

Manual Testing

All 3 tests pass locally on macOS (Apple Silicon, Docker Desktop):

$ python3 e2e/run_all.py --slow
PASSED: cluster_formation
PASSED: inference_snapshot
PASSED: no_internet
3/3 tests passed

Automated Testing

CI runs cluster_formation and no_internet (2/3 passed, 1 skipped):
https://github.com/exo-explore/exo/actions/runs/21961324819

[AlexCheema]

Code Review: Docker-based E2E test framework with chaos testing

Large, well-engineered PR (24 files, +1399/-5, 23 commits) adding E2E test infrastructure plus RunnerSupervisor health check.

CI Failures

e2e jobs (FAILING): test_inference_snapshot fails — no x86_64 snapshot baseline is committed. The test is NOT marked slow, so it runs in CI. Fix: either generate and commit x86_64 baselines (UPDATE_SNAPSHOTS=1) or mark the test as slow.

aarch64-darwin (HANGING ~6 hours): Root cause is the MpReceiver.close() thread hang bug — _forward_events() → receive_async() → abandoned thread blocked on queue.get() that never receives _MpEndOfStream. Fixed in PR #1511. Once #1511 is merged and this PR is rebased, the hang should be resolved.

E2E Framework — Well-Designed

• Cluster class (conftest.py): Clean async wrapper around docker compose with automatic cleanup, smart CI optimization (skips build if image exists), good failure output
• Dockerfile: Multi-stage build, aggressive Rust cleanup, BLAS/LAPACK for MLX CPU on Linux. The g++ -fpermissive wrapper is acceptable for a test image but should note which MLX version requires it
• Snapshot infrastructure: Architecture-aware (x86_64/arm64), explicit baseline requirement, clear diff output on mismatch
• docker-compose: Two services, same image, isolated namespace. No-internet override uses iptables REJECT (not DROP) for fast test feedback

E2E Tests — Good Coverage

Test	Verdict
test_cluster_formation	Good — minimal smoke test
test_no_internet	Good — validates iptables blocks + exo detects no connectivity
test_inference_snapshot	Needs x86_64 baseline committed
test_runner_chaos	Good — validates health check E2E via EXO_RUNNER_MUST_DIE
Snapshot variants (5 files)	Good coverage, somewhat repetitive boilerplate

RunnerSupervisor Health Check — Well-Designed, Minor Concerns

Good:

• Counter-based heartbeat (avoids clock skew), daemon thread, 0.5s interval
• _death_handled flag prevents race between health check and event forwarding
• Comprehensive unit tests (6 test cases, real multiprocessing)

Concerns:

1. Sync shutdown() in async context: _handle_process_exit() and _handle_unresponsive() are sync methods that call self.shutdown() (which does runner_process.join(1) — blocking for up to 1s). These are called from _health_check() which is an async coroutine, so they block the event loop. Consider wrapping in to_thread.run_sync().

2. Two overlapping failure-handling paths: _check_runner() (from _forward_events) and _handle_process_exit() (from _health_check) both handle process death. The _death_handled flag coordinates them, but having two code paths for the same failure is fragile. Consider consolidating.

3. Pre-heartbeat freeze gap: If the process freezes before the heartbeat thread starts (e.g., during import), heartbeat.value stays at 0, and the if current > 0 guard causes the health check to ignore it. The is_alive() check won't detect a live-but-frozen process. Known limitation, acceptable for now.

Security Note

Debug triggers (EXO_RUNNER_MUST_DIE, MUST_FAIL, MUST_OOM, MUST_TIMEOUT) are gated on prompt content, not environment variables. Any API user can kill a runner by sending a specific prompt. Consider gating on EXO_ENABLE_DEBUG_TRIGGERS=1 for production safety.

Minor

• Hard-coded port 52415 in conftest.py — extract to a constant
• run_all.py slow detection parses docstrings — fragile, consider filename convention or pytest markers
• 23 commits, many incremental CI fixes — squash before merge

Verdict

Substantial, well-engineered addition to the project. The E2E framework is clean and the RunnerSupervisor health check is sound. Two blockers:

1. Fix e2e CI: Generate x86_64 snapshot baselines or mark test_inference_snapshot as slow
2. Rebase onto fix: unblock MpReceiver.close() to prevent shutdown hang #1511: The aarch64-darwin hang is caused by the MpReceiver.close() bug — fixed in PR fix: unblock MpReceiver.close() to prevent shutdown hang #1511

Once those are resolved, recommend squashing the 23 commits and merging.

---

Review only — not a merge approval.