test(e2e): repro reth-ahead-of-consensus cross-store desync (audit #703/#704)

[keanji-x]

Summary

Adds a deterministic single-node e2e suite (gravity_e2e/cluster_test_cases/cross_store_desync_703_704/) that reproduces the non-atomic cross-store commit bug and the recovery-path crash it triggers — findings F1/F2 in galxe/RESTART_RECOVERY_FINDINGS.md.

reth (execution rocksdb) and the aptos consensus DB persist on independent stores with no write-ordering barrier. A crash in the commit window can leave reth at block N while consensus is at N-1. On restart, recovery anchors on reth's height (bin/gravity_node/src/main.rs:123 recover_block_number) but the consensus index no longer contains it, so recovery cannot find the root and the node dies — there is no reconcile / state-sync fallback.

Reproduction (no power-loss race needed)

1. Bring the node live, commit past height H, node.stop() cleanly.
2. gravity_cli unwind the consensus DB back to ~H/2, leaving reth at H.
   • Gotcha encoded in the test: --consensus-db-path must be the parent <data> dir, because ConsensusDB::new joins consensus_db internally (aptos-core/consensus/src/consensusdb/mod.rs:112). Passing <data>/consensus_db operates on an empty nested DB and silently no-ops.
3. node.start() and observe recovery.

Observed failure (current code)

ERROR aptos-core/consensus/src/persistent_liveness_storage.rs:549
  Failed to construct recovery data {"error":"unable to find root: 80dce1e8 ...
  LedgerRecoveryData::find_root ... RecoveryData::new ..."}
panicked at aptos-core/consensus/src/persistent_liveness_storage.rs:550:17
  (panic!(""))   -> node process dead (crash)

This is F1 (#703): the PartialRecoveryData / state-sync fallback at :551 is commented out and recovery_manager.rs:104 is todo!(), so any inconsistent recovery data is a panic!(""). The sibling F2 (#704) silent-stall path (crates/block-buffer-manager/src/block_buffer_manager.rs:343 early-return leaving the buffer Uninitialized → get_ordered_blocks awaits ready_notifier forever) is also asserted as a recovery failure.

Test status

The test asserts the correct post-fix behaviour (node detects reth_height > consensus_index, reconciles, and resumes producing blocks), so it is @pytest.mark.xfail(strict=True) on current code:

1 xfailed in 6.48s

Remove the xfail once startup auto-reconciles instead of panicking/stalling.

Notes

• Test-only; no production code changed.
• Unique base_dir=/tmp/gravity-cluster-desync and unique ports (rpc 8945, validator 6480, vfn 6490, authrpc 8953, reth_p2p 12324, metrics 9050, inspection 10300) — no collision with existing suites.
• Cross-repo refs: Galxe/gravity-audit#703, Galxe/gravity-audit#704.

🤖 Generated with Claude Code

[Lchangliang]

Thanks for adding the repro test. I think the premise of this test does not match the actual commit ordering, so this case is probably not something we need to cover.

In the real commit path, consensus is committed before reth. Therefore the crash window should not leave us with reth = N and consensus = N - 1. The possible inconsistent state is the opposite one: consensus = N while reth = N - 1.

That latter case is expected and already recoverable: on restart, the node can replay / recover the missing reth side from consensus. So intentionally unwinding only the consensus DB creates an artificial state that should not be produced by the normal system failure model.

Because of that, I don’t think this e2e test is necessary as-is. If we want to keep a test around recovery ordering, it should model the realistic direction instead: consensus ahead of reth, not reth ahead of consensus. But since that scenario already recovers normally, there may not be much value in adding a new xfail test here.