perf(server): remove the WAL coordination ceiling + 1M-stream cliff; memory mode 4× faster; crash-sim + 3 recovery fixes

.changeset/wal-write-path-perf-and-recovery-fixes.md

@@ -0,0 +1,9 @@
+---
+"@electric-ax/durable-streams-server-rust": patch
+---
+
+Write-path performance overhaul plus three crash-recovery correctness fixes.
+
+Performance: removed the WAL group-commit coordination ceiling (+55–60% saturated write throughput, p99 41→12 ms), fixed the 200k–1M stream-cardinality write cliff (1M streams now sustains 1.11M ops/s on 16 vCPU), and made `--durability memory` the buffered append path (4× faster, no longer Linux-only). New flags: `--wal-stats <secs>` and `--worker-threads <n>`.
+
+Fixes (found by the new seeded crash/fault simulation): multi-segment WAL recovery no longer drops acked records after the first segment; a torn, never-acked tail on a quiet stream is truncated instead of becoming reader-visible; an acked DELETE is now durable before the 204.

packages/durable-streams-rust/ARCHITECTURE.md

@@ -35,10 +35,10 @@ flowchart LR
     W2 --> W3["encode_wire<br/>(JSON flatten, append delimiter)"]
     W3 --> W4[["per-stream appender mutex"]]
     W4 --> W5["write_all → data file<br/>(lands in page cache)"]
-    W5 --> W6["update tail + resident cache"]
-    W6 --> W8["publish tail<br/>(watch channel)"]
-    W8 --> W7["group-commit fsync<br/>(WAL shard committer)"]
-    W7 --> W9["204 / 200 — only after durable"]
+    W5 --> W6["advance writer tail<br/>+ stage into WAL shard"]
+    W6 --> W7["group-commit fsync<br/>(WAL shard committer)"]
+    W7 --> W8["publish durable tail + resident cache<br/>(watch channel)"]
+    W8 --> W9["204 / 200 — only after durable"]
   end
 
   subgraph READ["READ  ·  GET"]
@@ -72,10 +72,10 @@ The dotted edges are the only coupling between writers and readers: publishing t
 1. **Parse idempotency headers** — `Producer-Id` / `Producer-Epoch` / `Stream-Seq`. Duplicate `(producer, epoch, seq)` is acknowledged without re-appending (exactly-once producers).
 2. **`encode_wire`** — turn the request body into the contiguous wire representation. In JSON mode this flattens arrays and appends the `,` delimiter so the on-disk bytes are already a valid stream fragment.
 3. **Acquire the per-stream appender mutex** (`AsyncMutex<Appender>`). This is the _only_ serialization point, and it's per-stream — different streams never contend.
-4. **`write_wire`** — `write_all` the bytes to the data file (they land in the OS page cache immediately), advance the tail under a short `RwLock` write, update the **resident tail cache**, then **publish the new tail** on the `watch` channel. Note the order: the cache is populated _before_ the wake, so a woken subscriber reliably hits it.
-5. **Durability** — in `wal` mode the append is staged into the stream's assigned WAL shard, and the response returns only after the shard's group-commit committer `fdatasync`s the segment covering this record. See [Durability](#durability) below. In `wal` mode everything above and the entire read path are identical regardless of workload. In `memory` mode binary appends take a separate socket→file splice path (zero-copy `splice(2)`, no WAL, no `fdatasync`) that bypasses `handle_append` / `encode_wire` via the engine zero-copy intercept and acks immediately after the page-cache write.
+4. **`write_wire`** — `write_all` the bytes to the data file (they land in the OS page cache immediately) and advance the _writer_ tail (`Shared.tail`) under a short `RwLock` write. The reader-observable tail does **not** move yet.
+5. **Durability, then visibility** — in `wal` mode the append is staged into the stream's assigned WAL shard (under the appender mutex, so per-stream LSN order matches byte order) and the handler awaits the shard's group-commit `fdatasync`. Only after the record is durable does `publish_durable_tail` advance the **reader-observable `durable_tail`** (monotonically), populate the **resident tail cache**, and **publish on the `watch` channel** — cache before wake, so a woken subscriber reliably hits it. Then the 2xx is returned. See [Durability](#durability) below. In `memory` mode the same buffered path runs with the WAL stage/wait skipped: the page-cache write is the ack.
 
-Visibility vs durability are deliberately decoupled: the bytes are in the page cache (and the tail is published) before durability resolves, so a live reader sees data with minimal latency, while the _appender_ doesn't get its 2xx until the data is durable.
+Visibility is gated on durability (PROTOCOL.md §4.1): a live reader never observes bytes (or an EOF) that a crash could roll back. The writer tail runs ahead in `Shared.tail`; readers see `durable_tail`, which follows it as group commits resolve.
 
 ## Durability
 
@@ -85,7 +85,7 @@ The server supports two durability modes, chosen at startup via `--durability`.
 
 **`wal` (default)** — durable, single-node no-loss durability via a sharded write-ahead log. An append acks only after its record is durable in the WAL (group-commit `fdatasync`). This is the safe default for any deployment where local disk loss must not cause data loss. See the `wal` mode section below for the design.
 
-**`memory` (Linux-only)** — no WAL, no `fsync`: binary appends move `socket→file` via `splice(2)` (zero-copy in the kernel); JSON appends are buffered writes; ack fires on the page-cache write. The per-stream files are the only durable-enough record, and recovery is the existing sidecar pass (rebuild stream state from the per-stream files + `.meta` sidecars). **NOT locally crash-durable** — a power loss or kernel panic can lose any un-fsynced page. Durability is delegated to (future) replication. Exits with status 2 on non-Linux.
+**`memory`** — no WAL, no `fsync`: appends take the same buffered write path as `wal` mode with the WAL stage/wait skipped; ack fires on the page-cache write. The per-stream files are the only durable-enough record, and recovery is the existing sidecar pass (rebuild stream state from the per-stream files + `.meta` sidecars). **NOT locally crash-durable** — a power loss or kernel panic can lose any un-fsynced page. Durability is delegated to (future) replication. Refuses to start over a WAL left by a previous `wal` run (replay it with `--durability wal` first, or delete the `wal/` directory to discard it deliberately).
 
 | Mode     | ack after        | fsync                  | WAL | crash-safe?          |
 | -------- | ---------------- | ---------------------- | --- | -------------------- |
@@ -105,11 +105,11 @@ Every append is written to the per-stream data file (page cache, no hot fsync 
 
 Per-stream files are `fdatasync`'d off the ack path at a periodic **checkpoint**, after which the bounded WAL is recycled. On boot, recovery replays the WAL from its oldest retained segment, reconciles each stream's durable tail (torn-tail repair via truncation + `fdatasync`), then resets the WAL for fresh appends.
 
-The invariant: **visibility is never gated on durability**. Bytes land in the page cache and the tail is published before the WAL `fdatasync`, so live readers see an append at memory latency while the appender's acknowledgement waits for the WAL commit.
+The invariant: **readers only ever observe durable bytes** (PROTOCOL.md §4.1). Bytes land in the page cache immediately, but the reader-observable `durable_tail` (and the `watch` wake) advances only after the WAL `fdatasync` covering the record — the same barrier that releases the appender's acknowledgement. A crash therefore never rolls back anything a reader has seen.
 
 ### `memory` mode
 
-In `memory` mode no WAL is created or attached. Appends write directly to the per-stream file (buffered write for JSON; zero-copy `socket→file` splice for binary) and ack immediately after the page-cache write — no `fdatasync`, no WAL staging. The per-stream file is the data; the `.meta` sidecar records the stream configuration and tail. On restart, the server runs the same sidecar pass it runs in `wal` mode (rebuild each stream from its file + sidecar) — there is no WAL to replay. Durability is delegated to replication (not yet built).
+In `memory` mode no WAL is created or attached. Appends write directly to the per-stream file (the same buffered write as `wal` mode) and ack immediately after the page-cache write — no `fdatasync`, no WAL staging. The per-stream file is the data; the `.meta` sidecar records the stream configuration and tail. On restart, the server runs the same sidecar pass it runs in `wal` mode (rebuild each stream from its file + sidecar) — there is no WAL to replay. Durability is delegated to replication (not yet built).
 
 ## Read path in detail
 
@@ -140,8 +140,8 @@ flowchart TB
   end
 
   A3 ==> PC[("OS page cache<br/>— the hot tier")]
-  A3 ==> RC["resident tail cache<br/>(last chunk, configurable via --tail-cache-bytes,<br/>in heap)"]
-  A3 -.-> TW[/"tail watch channel<br/>(one notify per append — before fsync)"/]
+  A4 ==> RC["resident tail cache<br/>(last chunk, configurable via --tail-cache-bytes,<br/>in heap)"]
+  A4 -.-> TW[/"tail watch channel<br/>(one notify per append — after the group commit)"/]
 
   subgraph FAN["② Fan-out"]
     direction TB
@@ -159,7 +159,7 @@ The techniques, each with its mechanism and payoff:
 1. **Contiguous wire-byte storage.** The file _is_ the response. Reads are byte ranges with no reframing and no per-message copy — and this is what makes `sendfile` zero-copy possible at all.
 2. **Group-commit coalesced fsync.** The durability contract ("return after fsync") is the expensive part of an append. Concurrent appenders share a single in-flight barrier fsync, so throughput scales with the _batch size_ per fsync rather than one fsync per message. This is why unbatched appends hit ~30k/s where a per-append-fsync server (Node) does ~130/s.
 3. **Per-stream single writer, lock-free reads.** One async mutex orders a stream's appends; there is no global lock (streams live in a `DashMap`). Reads take a brief tail snapshot and do positioned reads — they never block the writer and never wait on each other.
-4. **Pre-fsync visibility.** Appended bytes are in the page cache and the tail is published before the fsync resolves, so live readers see data at memory latency; only the appender's acknowledgement waits for durability.
+4. **Durable-gated visibility, group-commit-amortized.** The reader-observable tail is published only after the record's group-commit fsync (readers never see bytes a crash could roll back — PROTOCOL.md §4.1). Because N concurrent appends share one barrier fsync, the visibility latency cost is one amortized group commit, not one fsync per append.
 5. **`watch`-channel wakeups.** Live readers park on a per-stream `watch`; an append is one `send_replace` that wakes all of them. No polling loop, no timer churn.
 6. **Resident tail cache (fan-out de-duplication).** Without it, N caught-up SSE/long-poll subscribers each re-read (and re-encode) the _same_ just-appended bytes — N× duplicated work that grows with audience size. The cache keeps the last chunk in the heap so all N share one read (and SSE encodes once per subscriber off that shared buffer). For small hot reads it's also fewer syscalls than `sendfile` and skips the read-offload pool hop.
 7. **Zero-copy egress.** `FileRange` reads are served with `sendfile` (page cache → socket, no userspace copy → ~5× less CPU per byte than a buffered copy). The **`--read-offload`** strategy keeps a cold backfill's disk fault off the async workers so one slow read can't stall unrelated requests.

packages/durable-streams-rust/CARDINALITY_1M.md

@@ -0,0 +1,54 @@
+# 1M-stream cardinality fixes — findings + results (2026-07-02)
+
+Follow-up to `WRITE_BOTTLENECKS_1M.md` (bottleneck #2: stream cardinality) and
+`CONTENTION_INVESTIGATION.md`. Server commit: `662b0c845` on
+`perf/combined-t1a-t1c-t2a`. **Outcome: 1M streams reaches 1,114,644 ops/s on a
+16 vCPU `c4d-standard-16-lssd` (ladder unsaturated), and the 500k→1M degradation at
+equal load is −17% (was a cliff).**
+
+## Root causes (evidence-first, local Linux repro at 20k→400k streams)
+
+The key mechanism: **at high cardinality, ops/stream/checkpoint-interval drops below
+1, so every "amortized once-per-stream-per-interval" cost becomes a per-op cost.**
+
+| #   | cost                                                                                                                                                                                                                                  | evidence                                                                                                           | fix                                                                                                                                                                                                                                                  |
+| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| 1   | Checkpoint drain did O(touched) capture (`shared.read()` + Arc clone per stream) **while holding the shard `dirty` mutex** — the same mutex every append's epoch-transition takes; at 400k streams every append is a transition       | `WAL_CKPT drain_us` 25–140 ms/tick; `dirty_wait_load` 0.01→0.30 cores as streams 20k→400k; p99 ≈ max drain         | O(1) critical section: take Vec + bump epoch under the lock, capture after release                                                                                                                                                                   |
+| 2   | Checkpoint capture / cumulative tails-file re-read+re-sort+rewrite / recycle ran **on async runtime threads**, serially across shards                                                                                                 | `WAL_CKPT` capture 31 ms + tails 24 ms per tick per shard on runtime threads                                       | whole checkpoint body in one `spawn_blocking`; tails map memory-resident; shards checkpoint concurrently (`JoinSet`)                                                                                                                                 |
+| 3   | **Per-append meta sidecar flush**: with inter-append gap > the 100 ms debounce (always, at high cardinality) every producer append did JSON + `File::create(.meta.tmp)` + `rename` → all workers spin on the **data-dir inode rwsem** | perf: `osq_lock`+`rwsem_spin_on_owner` under `write_meta_sync` = **38–46% of ALL server CPU at every cardinality** | WAL-staged appends only mark `meta_dirty`; checkpoint writes sidecars for drained streams after recycle (memory-mode keeps the debounced flush). Producer/access staleness bound: 100 ms debounce → checkpoint cadence (contract already allows lag) |
+| 4   | Two registry lookups per append (`handle_append` metric label + `_inner`) — 2× SipHash + cold DashMap walk at 1M keys                                                                                                                 | code inspection                                                                                                    | `_inner` returns `is_json`                                                                                                                                                                                                                           |
+
+## Local A/B (Linux harness, 6 srv cores, conn=256, shards=6)
+
+| streams | before        | after      | p99          |
+| ------- | ------------- | ---------- | ------------ |
+| 20k     | ~43–46k ops/s | **80.4k**  | 41 → 7.7 ms  |
+| 200k    | 32.3k         | **50.6k**  | 53 → 17.9 ms |
+| 400k    | 16.0k         | **36–44k** | 144 → ~28 ms |
+
+Correctness: 95 crate tests + 326 conformance tests pass. New telemetry: `WAL_CKPT`
+per-shard checkpoint phase line (`--wal-stats`), and the repro script grew
+`--tmpfs` / `--wal-stats` knobs + WAL_CKPT summarizing.
+
+## Remote validation (GKE, 16 vCPU, pool client, 256 B, batch 1)
+
+Suite `ds-bench/suites/run-durable-cpu16-1m-card.json`, image
+`durable-streams:combined-card@sha256:d74840bd…`; full detail + caveats in
+`ds-bench/results/run-durable-cpu16-1m-card/FINDINGS.md`.
+
+| streams | pods | ops/s         | p50 / p99 / max                                              |
+| ------- | ---- | ------------- | ------------------------------------------------------------ |
+| 1M      | 32   | 898,582       | 3.5 / 30.5 / **149.6 ms** (baseline 862k, 3.3 / 32 / 405 ms) |
+| 1M      | 64   | **1,114,644** | 3.4 / 60.3 / 211 ms — still climbing (+21%/+16 pods)         |
+| 500k    | 48   | 1,110,268     | 3.1 / 42.7 / 145 ms                                          |
+
+## Open follow-ups
+
+1. True 16 vCPU ceiling at 1M (ladder past 64 pods) and a 32 vCPU 1M run.
+2. **Per-shard producer-state journal**: sidecar writes/s ≈ ops/s at full cardinality
+   (off the hot path now, but it stretches checkpoint cadence — locally ~1.6 s/shard
+   meta phase at 400k — and bounds producer-state staleness). One cumulative
+   per-shard file per tick, recovery overlays producers by max(epoch, seq).
+3. Read+write mix at 1M streams (everything here is write-only).
+4. `--wal-stats` cell on NVMe (`run-durable-cpu16-1m-card-stats.json`, never ran) to
+   confirm checkpoint fsync/meta phase behavior at 1M on real disks.