perf(wal): fix the high-stream-cardinality write cliff (1M streams → 1.11M ops/s @ 16 vCPU)

[balegas]

Stacked on #4675 (perf/combined-t1a-t1c-t2a) — this diff contains only the cardinality work.

Summary

At high stream cardinality (200k–1M streams) write throughput fell off a cliff even after #4675's coordination fixes — at a fixed local load, going 20k→400k streams cost −65% throughput while server CPU dropped. This PR removes the four costs behind that cliff. Local repro at 400k streams: 16.0k → ~44k ops/s, p99 144 → 27 ms; on GKE at 16 vCPU, 1M streams goes 862k → 1,114,644 ops/s (ladder still unsaturated at 64 pods) and worst-case latency at the baseline rung drops 405 ms → 150 ms. The 500k→1M degradation at equal load is now −17% (was a cliff).

Root cause — why cardinality broke the amortization

The write path amortizes several costs to "once per stream per ~3 s checkpoint interval". At high cardinality, ops/stream/interval < 1, so every such cost silently becomes a per-op cost — and some of them ran under locks the hot path needs. Concretely (each confirmed with WAL_CKPT phase telemetry added here, plus perf):

1. Checkpoint drain held the shard dirty mutex during O(touched) work (shard.rs). The drain captured each touched stream's tail/file (shared.read() + Arc clone, ~8–20k streams/shard/tick) inside the mutex — 25–140 ms holds. Meanwhile the Tier-1a "lock-free" fast path is dead at high cardinality: nearly every append is its stream's first touch of the epoch, so every append takes the transition path through that same mutex. Measured: dirty_wait_load 0.01 → 0.30 cores as streams went 20k → 400k; client p99 (144 ms) ≈ max drain hold (140 ms).
2. Checkpoint phases ran on async runtime threads, serially across shards. Only the per-stream fsyncs were spawn_blocking; the capture, the cumulative tails-file rewrite (re-read + re-parse + re-sort + re-format of every stream ever touched, every 3 s), and recycle all stalled runtime workers.
3. Per-append meta sidecar flush — the biggest, flattest tax. Every producer append debounce-scheduled write_meta_sync (JSON + File::create(.meta.tmp) + rename). The 100 ms debounce only coalesces when a stream is re-appended within 100 ms — never, at high cardinality — so the server did one create+rename in the shared data directory per append, and all workers spun on that directory's inode rwsem. perf: osq_lock + rwsem_spin_on_owner under write_meta_sync = 38–46% of all server CPU, at every cardinality.
4. Two registry lookups per append: handle_append did a store.get(&path) just for the is_json metric label, then handle_append_inner did the real one — 2× SipHash + cold DashMap walk per op at 1M keys.

What unlocked the performance

• O(1) dirty-drain critical section: take the Vec + bump the epoch under the lock, nothing else; the O(touched) capture runs after release. The Tier-1a safety argument is preserved — the epoch bump and take still happen in the same critical section register_dirty's push uses, and the recycle invariant was never anchored on capture position (it's protected by snapshotting checkpoint_lsn before the drain).
• Whole checkpoint body in one spawn_blocking (capture → per-stream fdatasyncs → tails persist → checkpoint_lsn persist → recycle, same hard ordering), with shards checkpointing concurrently (JoinSet) instead of serially.
• Resident tails map: seeded from disk once, merged + serialized from memory each tick — no more O(total-streams) file re-read/re-parse/re-sort every 3 s.
• Meta sidecar flush moved to the checkpoint: WAL-staged appends just set meta_dirty (one atomic store — also drops a tokio::spawned debounce timer per append); the checkpoint writes sidecars for drained streams after recycle, off the hot path. Memory-mode appends keep the debounced flush (no checkpoint exists to pick them up).
• Single registry lookup: handle_append_inner returns is_json for the metric label.
• WAL_CKPT telemetry (--wal-stats): per-shard checkpoint line with touched/tails/meta counts and capture/fsync/tails/rest/meta phase timings — this is what attributed the cliff, and it's what a future run reads to decide the next fix.

Key invariants (unchanged)

• Recycle safety: per-stream file fsyncs and the durable-tails persist still strictly precede WAL segment recycling; checkpoint_lsn is snapshotted before the drain, so any record durable at snapshot time is in the drained set (its register_dirty happened-before the commit that made it durable).
• Close durability: stream closes still write the sidecar synchronously + fsynced before readers observe EOF.
• Producer-state contract: sidecar producer/access flushes were already documented as non-durable and lagging (producers bump epoch on restart). The lag bound moves from the 100 ms debounce to the checkpoint cadence.

Non-goals / follow-ups

• Per-shard producer-state journal: sidecar writes/s still ≈ ops/s at full cardinality — now off the hot path, but it stretches checkpoint cadence and staleness. The structural fix (one cumulative per-shard file per tick, recovery overlays producers by max(epoch, seq)) touches recovery and is deliberately not in this PR.
• Read-path work, T1b/T3 from the original investigation, and the true 16 vCPU ceiling (ladder was still climbing at 64 pods) are out of scope. Details in packages/durable-streams-rust/CARDINALITY_1M.md.

Verification

• cargo test --release — 95 unit/e2e tests pass (incl. WAL recovery + checkpoint ordering tests).
• pnpm run test:conformance — 326 conformance tests pass.
• Local A/B: scripts/contention-repro-linux.sh --shards 6 --connections 256 --streams 400000 --tmpfs 4g before/after (16.0k → ~44k ops/s; p99 144 → 27 ms; 20k-stream rung also ~2× from the meta-flush fix alone).
• Remote: ds-bench suite run-durable-cpu16-1m-card (results + provenance + caveats in ds-bench/results/run-durable-cpu16-1m-card/FINDINGS.md): 1M streams 899k @ 32 pods (max 150 ms) → 1.115M @ 64 pods, unsaturated; 500k @ 48 pods 1.110M.

Files

• src/wal/shard.rs — O(1) drain; checkpoint body in one blocking task; resident tails map; checkpoint-time meta flush; WAL_CKPT timings.
• src/main.rs — concurrent per-shard checkpoint ticker.
• src/handlers.rs — mark-meta_dirty instead of per-append flush (WAL mode); single registry lookup.
• scripts/contention-repro-linux.sh — --tmpfs/--wal-stats knobs + WAL_CKPT summary.
• CARDINALITY_1M.md — investigation + results write-up.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WzV7avRUMhPqmm1SgU4z1J