feat(durable-streams-rust): replicated durability — quorum-replication acks via Raft (openraft)

[balegas]

Summary

Adds a third durability mode to the Rust durable-streams server: --durability replicated — Kafka-style durability where an append acks once a quorum of replicas has committed and applied it (openraft), with no fsync anywhere on the hot path. A 3-node cluster survives loss of any minority with zero acked-data loss, any node accepts writes, every node serves reads, and failed nodes restart-rejoin or get replaced live.

Full design/ops doc: packages/durable-streams-rust/REPLICATION.md. Deploy kit under deploy/replicated/ (local cluster, docker compose, k8s, smoke).

Reviewer guidance

Approach — log-first apply. Every state-mutating op (PUT/POST/DELETE) is a LogOp proposed into a single Raft log; every node, including the leader, mutates its store only from committed entries, inside the state machine. That makes each replica a deterministic function of the log prefix — no divergence, no truncation/repair on fail-over. The ack is the state machine's apply outcome (client_write returns it), so HTTP responses are byte-identical to single-node modes, including producer dedup/fencing evaluated authoritatively in log order.

Key mechanics, in review order:

• src/replication/sm.rs — the state machine. Applies are sharded by stream path within each batch (per-stream order preserved; fork-creates barrier): one slow fs write must not stall every other stream's ack — we measured 100–450 ms apply stalls without this. Snapshots are manifests (per-stream metadata, O(streams) to build); the installing node wipes and pulls each stream's [0, tail) prefix over the mesh, exact because prefixes are immutable under append-only.
• src/replication/net.rs — TCP RPC mesh (correlation ids, auto-reconnect) carrying the three Raft RPCs plus Forward (proposal to leader) and FetchStream (snapshot byte pull). Clients are a dynamic registry fed by membership addresses, so added nodes need no restarts.
• src/replication/core.rs — propose path. On a follower: forward to leader, then wait for the local apply to cover the entry (read-your-writes on the node you wrote to).
• src/replication/log_store.rs — in-memory log (deliberate: durability comes from replication), durable vote (fsync at election time only; also serves as the initialized marker that makes restarts skip bootstrap).
• handlers.rs — apply_replicated_* are the semantic twins of the single-node paths (same check order: closed → producer → seq), kept next to the code they mirror; replicated branches in PUT/POST/DELETE; /_repl/status, /_repl/add-learner, /_repl/change-membership.

Key invariants

1. An acked append is applied on a quorum and survives any minority loss; replicas are byte-identical at equal applied indexes (tests assert full-content equality).
2. Client retries after 503 are safe: producer dedup is replicated state, evaluated at apply time.
3. Trim/purge never outruns state transfer: manifests are cheap so snapshots always exist ahead of purge; a fetch of a deleted+recreated path fails loudly (stream-id check) rather than installing wrong bytes.
4. The vote is durable before it is visible — restart cannot grant conflicting elections.

Trade-offs (accepted, documented)

• Quorum-memory durability: simultaneous power loss of a majority loses unsynced data — the same trade as Kafka acks=all with OS-managed flush. Zone-spread replicas are the mitigation.
• One consensus group (single ordering pipeline, batched); sharding across groups is roadmap.
• Follower fast-fail pre-checks are sequentially consistent: an append racing its stream's create on a lagging follower can see a retryable 404 (~1 in 10⁵ under round-robin load).
• Tiering (--tier) unsupported in this mode.

Non-goals here: ds-bench harness integration (single-server assumption), sharded consensus groups, large-payload copy reduction — all in the REPLICATION.md roadmap.

Why openraft (and not the original OmniPaxos implementation, visible mid-branch): a deterministic fault-injection simulation we built caught a consensus safety bug in the last published omnipaxos release (stale cached Promise after AcceptSync → decided-log divergence, acked entry lost on leader change; fixed upstream Jan 2024 but never published), and the project is unmaintained. openraft is active and production-proven; the swap cost only the consensus adapter layer and made large payloads 18–39% faster with 3–4× lower memory (entry-count purge vs time-based trim). The simulation harness lives in git history pending an openraft port (roadmap).

Verification

cd packages/durable-streams-rust
cargo test                                # 106 tests: unit + 3-node TCP cluster tests
                                          # (convergence, forwarding, dedup, close/delete,
                                          #  forks, learner-join-past-purge, vote durability)
PROFILE=release deploy/replicated/smoke.sh  # e2e: writes via all nodes, byte-identical reads,
                                            # leader kill + fail-over, restart-rejoin

Local 3-node benchmarks (one 10-core machine, so conservative; ds-bench multi-stream/sustained, details + sizing rules in REPLICATION.md): 256 B appends 36–52k ops/s (p99 2–7 ms) vs 96–100k single-node memory mode; 16 KB 14.9k ops/s (244 MB/s); 5-minute sustained runs hold RSS flat (20 MB @ 256 B; ≤143 MB @ 16 KB) with zero ack timeouts and all replicas byte-identical.

Files changed

• packages/durable-streams-rust/src/replication/ — the new module: types (Raft type wiring), entry (LogOp/outcomes), sm (state machine, sharded apply, manifest snapshots), net (RPC mesh), core (propose/ack, stats), log_store (in-mem log + durable vote), tests
• packages/durable-streams-rust/src/handlers.rs — replicated branches + apply twins + admin endpoints
• packages/durable-streams-rust/src/{main,store}.rs — --repl-* flags, boot wipe/resync, create_with_meta_durability, wipe_all
• packages/durable-streams-rust/deploy/replicated/ — local-cluster.sh (up/status/down/restart), smoke.sh, monitor.sh, Dockerfile, docker-compose.yml, k8s.yaml
• packages/durable-streams-rust/{REPLICATION,README}.md — design, ops, benchmarks
• .changeset/replicated-durability.md — minor bump

🤖 Generated with Claude Code