From 24896c3d081cdb1ac288f8c21886616603fd1716 Mon Sep 17 00:00:00 2001 From: Jared Lunde Date: Thu, 11 Jun 2026 19:33:22 -0700 Subject: [PATCH 1/3] =?UTF-8?q?feat:=20verified=20snapshot=20transport=20?= =?UTF-8?q?=E2=80=94=20pointer-swap=20protocol,=20model-checked,=20v0.6.0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three real bugs found and fixed, each caught by a new verification layer: 1. NATS silently clamps a resume below first_seq (no error), so the entire CursorExpired -> resync path was dead code: every expired resume silently skipped the gap, evicted delete markers included. Expiry is now detected proactively (first_sequence comparison, re-checked after consumer creation), pinned by a live regression test. 2. A failed stale-key resync degraded to a warning, silently leaving deleted keys in the fold forever. Resync failure now fails the watch (fail-stop); the model proves degrade semantics break the convergence theorem. 3. The transport's two-object layout allowed a stalled exporter to regress "latest" (last-write-wins clobber) and a mid-upload crash to tear the payload/manifest pair. Replaced with content-addressed write-once payloads + a single pointer published last via monotonic conditional swap (If-Match CAS; fail-closed on non-CAS stores with explicit dev-only opt-in). The model checker additionally caught a dangling-pointer race between prune and an in-flight publish; the fix is the strictly-below- the-pointer prune rule, with cursors embedded in payload keys. Proof stack: - src/protocol.rs: shared decision kernels — production code and the model checker execute the same guard source, so guards cannot drift from proof. - tests/model.rs: Stateright model, exhaustive within bounds; theorems (no regression, no torn pairs, no dangling pointers, no mixed imports, no silent divergence, terminal liveness) over multi-round fleets at N=2/3; legacy configurations kept as machine-checked records of each bug; mutation tests prove every kernel guard is load-bearing; deep tier (~250M unique states) for release runs. - tests/multi_export.rs: prevention proofs on live NATS + real fjall/ RocksDB folds (slow-exporter refused, crash window keeps bootstrap available, post-compaction multi-SST fidelity under source churn). - tests/resync.rs: live expiry->resync chain end-to-end; NATS clamp pinned; no-reader divergence pinned. - tests/transport_s3.rs: conditional-put axiom verified on live MinIO (Create, If-Match swap, refusal, crash window, prune). Wire format change (payload location): upgrade exporters before bootstrap-dependent nodes. v0.6.0. Co-Authored-By: Claude Fable 5 --- Cargo.lock | 84 +++- Cargo.toml | 3 +- src/applied.rs | 56 ++- src/export_lease.rs | 11 +- src/lib.rs | 3 +- src/nats.rs | 87 ++++ src/protocol.rs | 151 +++++++ src/transport.rs | 420 ++++++++++++++++--- tests/common/mod.rs | 87 ++++ tests/model.rs | 934 ++++++++++++++++++++++++++++++++++++++++++ tests/multi_export.rs | 830 +++++++++++++++++++++++++++++++++++++ tests/resync.rs | 476 +++++++++++++++++++++ tests/transport.rs | 166 +++++++- tests/transport_s3.rs | 138 +++++++ 14 files changed, 3371 insertions(+), 75 deletions(-) create mode 100644 src/protocol.rs create mode 100644 tests/model.rs create mode 100644 tests/multi_export.rs create mode 100644 tests/resync.rs diff --git a/Cargo.lock b/Cargo.lock index b269c91..e4e659b 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2,6 +2,19 @@ # It is not intended for manual editing. version = 4 +[[package]] +name = "ahash" +version = "0.8.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" +dependencies = [ + "cfg-if", + "getrandom 0.3.4", + "once_cell", + "version_check", + "zerocopy", +] + [[package]] name = "aho-corasick" version = "1.1.4" @@ -50,6 +63,12 @@ version = "0.7.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" +[[package]] +name = "ascii" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d92bec98840b8f03a5ff5413de5293bfcd8bf96467cf5452609f939ec6f5de16" + [[package]] name = "async-nats" version = "0.46.0" @@ -124,7 +143,7 @@ checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" [[package]] name = "beyond-slipstream" -version = "0.5.0" +version = "0.6.0" dependencies = [ "aho-corasick", "async-nats", @@ -139,6 +158,7 @@ dependencies = [ "rust-rocksdb", "serde", "serde_json", + "stateright", "tar", "tempfile", "thiserror 2.0.18", @@ -281,6 +301,12 @@ dependencies = [ "rand_core 0.10.1", ] +[[package]] +name = "choice" +version = "0.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3b71fc821deaf602a933ada5c845d088156d0cdf2ebf43ede390afe93466553" + [[package]] name = "chrono" version = "0.4.45" @@ -293,6 +319,12 @@ dependencies = [ "windows-link", ] +[[package]] +name = "chunked_transfer" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e4de3bc4ea267985becf712dc6d9eed8b04c953b3fcfb339ebc87acd9804901" + [[package]] name = "ciborium" version = "0.2.2" @@ -986,6 +1018,12 @@ version = "1.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" +[[package]] +name = "httpdate" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9" + [[package]] name = "humantime" version = "2.3.0" @@ -1163,6 +1201,12 @@ version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954" +[[package]] +name = "id-set" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9633fadf6346456cf8531119ba4838bc6d82ac4ce84d9852126dd2aa34d49264" + [[package]] name = "idna" version = "1.1.0" @@ -1423,6 +1467,12 @@ dependencies = [ "signatory", ] +[[package]] +name = "nohash-hasher" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2bf50223579dc7cdcfb3bfcacf7069ff68243f8c363f62ffa99cf000a6b9c451" + [[package]] name = "nom" version = "7.1.3" @@ -2359,6 +2409,26 @@ version = "1.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" +[[package]] +name = "stateright" +version = "0.31.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fd1157f21b11916f90fe1f2ac9a8d0e09a8813b28701584141060f414eedf6ba" +dependencies = [ + "ahash", + "choice", + "crossbeam-utils", + "dashmap", + "id-set", + "log", + "nohash-hasher", + "parking_lot", + "rand 0.9.4", + "serde", + "serde_json", + "tiny_http", +] + [[package]] name = "subtle" version = "2.6.1" @@ -2491,6 +2561,18 @@ dependencies = [ "time-core", ] +[[package]] +name = "tiny_http" +version = "0.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "389915df6413a2e74fb181895f933386023c71110878cd0825588928e64cdc82" +dependencies = [ + "ascii", + "chunked_transfer", + "httpdate", + "log", +] + [[package]] name = "tinystr" version = "0.8.3" diff --git a/Cargo.toml b/Cargo.toml index 115bf6e..a301510 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -8,7 +8,7 @@ rust-version = "1.92" [package] name = "beyond-slipstream" -version = "0.5.0" +version = "0.6.0" edition.workspace = true license.workspace = true rust-version.workspace = true @@ -40,6 +40,7 @@ url = "2" [dev-dependencies] criterion = { version = "0.5", features = ["html_reports"] } +stateright = "0.31.0" tokio = { version = "1", features = ["macros", "rt-multi-thread", "time"] } [[bench]] diff --git a/src/applied.rs b/src/applied.rs index 8d8c2b1..a12affd 100644 --- a/src/applied.rs +++ b/src/applied.rs @@ -190,6 +190,12 @@ impl Default for BatchConfig { /// without a `store` to diff against) the fallback is re-list-only and a /// warning marks the possible stale keys. /// +/// A resync that was armed but FAILS (live-key listing or fold diff error) is +/// fatal to the watch — degrading to re-list-only would silently leave +/// deleted keys in the fold (`tests/model.rs` proves that divergence +/// reachable), so the error surfaces and the caller's restart retries the +/// resume → expiry → resync path from scratch. +/// /// See `ARCHITECTURE.md` ("Applied-Cursor Watch") for the invariant and its /// rationale. /// @@ -434,8 +440,20 @@ where .map(|e| e.key), ), Err(e) => { + // FATAL, not a degrade: an incomplete + // diff silently leaves deleted keys in + // the fold forever (tests/model.rs + // proves the divergence reachable + // under degrade semantics). Fail the + // watch; the restart re-runs the + // resume → expiry → resync from + // scratch. warn!(error = %e, prefix = %prefix, - "resync fold range failed; stale keys under this prefix may persist"); + "resync fold range failed; aborting watch rather than diverging"); + handle.abort(); + return Err(KvError::WatchError(format!( + "cursor-expired resync failed listing fold prefix {prefix:?}: {e}" + ))); } } } @@ -604,7 +622,7 @@ async fn run_watch( warn!( "watch cursor expired; resyncing, then falling back to full watch_all" ); - resync_stale_keys(scope, &resync).await; + resync_stale_keys(scope, &resync).await?; watcher.watch_all(tx).await } other => other, @@ -620,7 +638,7 @@ async fn run_watch( warn!( "watch cursor expired; resyncing, then falling back to full watch_prefix" ); - resync_stale_keys(scope, &resync).await; + resync_stale_keys(scope, &resync).await?; watcher.watch_prefix(prefix, tx).await } other => other, @@ -640,7 +658,7 @@ async fn run_watch( warn!( "watch cursor expired; resyncing, then falling back to full watch_prefixes" ); - resync_stale_keys(scope, &resync).await; + resync_stale_keys(scope, &resync).await?; watcher.watch_prefixes(&refs, tx).await } other => other, @@ -659,24 +677,37 @@ async fn run_watch( /// makes a delete-then-recreate during the gap converge: the synthetic delete /// always lands before the re-list put. /// -/// Best-effort: with no reader/store wired (`resync` is `None`), or a failed -/// listing, this degrades to the warn-and-relist-only fallback — keys deleted -/// during the gap stay in the fold until their next update. -async fn resync_stale_keys(scope: &WatchScope, resync: &Option) { +/// With no reader/store wired (`resync` is `None`) the caller explicitly opted +/// out: warn and fall back re-list-only (keys deleted during the gap stay in +/// the fold — `tests/model.rs` pins this divergence as reachable). +/// +/// A FAILED listing, by contrast, is **fatal** — it fails the watch rather +/// than degrading. The resync is load-bearing for the "stale, never corrupt" +/// convergence guarantee: a silently degraded resync leaves the fold holding +/// keys the bucket deleted, with one warn line as the only witness +/// (`tests/model.rs` proves this divergence reachable under degrade +/// semantics). Failing the watch turns the violated guarantee into a visible +/// error; the caller's restart re-resumes, hits `CursorExpired` again, and +/// retries the resync from scratch. +async fn resync_stale_keys( + scope: &WatchScope, + resync: &Option, +) -> Result<(), KvError> { let Some((reader, resync_tx)) = resync else { warn!( "no reader wired for cursor-expired resync; keys deleted during the gap may persist in the fold" ); - return; + return Ok(()); }; let mut live_keys = Vec::new(); for prefix in scope.prefixes() { match reader.keys(&prefix).await { Ok(keys) => live_keys.extend(keys), Err(e) => { - warn!(error = %e, prefix = %prefix, - "resync live-key listing failed; keys deleted during the gap may persist in the fold"); - return; + return Err(KvError::WatchError(format!( + "cursor-expired resync failed listing live keys under {prefix:?}: {e}; \ + failing the watch rather than silently keeping stale keys" + ))); } } } @@ -693,6 +724,7 @@ async fn resync_stale_keys(scope: &WatchScope, resync: &Option) { // is about to die with it; nothing to recover. let _ = ack_rx.await; } + Ok(()) } #[cfg(test)] diff --git a/src/export_lease.rs b/src/export_lease.rs index 5ee2e40..9933843 100644 --- a/src/export_lease.rs +++ b/src/export_lease.rs @@ -20,9 +20,14 @@ //! portable to any [`KvWriter`] backend and free of server-version/bucket-flag //! requirements. The cost is wall-clock comparison across nodes: with //! NTP-sane clocks and round periods measured in minutes, skew is noise — and -//! a premature steal is *safe* anyway (two exporters produce two identical -//! artifacts; the upload is last-write-wins on the same key). The lease is a -//! work-deduplication optimization, never a correctness gate. +//! a premature steal is *safe* anyway, though NOT because concurrent +//! exporters produce identical artifacts (they don't: each replica exports +//! at its own applied cursor). Safety comes from the transport: payloads are +//! content-addressed (concurrent uploads cannot collide or tear) and the +//! published pointer only moves forward (an overrun round's stale artifact +//! is refused, [`PublishOutcome::SupersededByNewer`](crate::PublishOutcome)), +//! machine-checked in `tests/model.rs` with NO lease in the model at all. +//! The lease is a work-deduplication optimization, never a correctness gate. //! //! ## Lifecycle //! diff --git a/src/lib.rs b/src/lib.rs index 48d8be1..cb95982 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -29,6 +29,7 @@ mod artifact; mod export_lease; mod kv; mod nats; +pub mod protocol; pub mod snapshot; #[cfg(feature = "fjall")] mod snapshot_fjall; @@ -54,4 +55,4 @@ pub use snapshot_fjall::{FjallConfig, FjallReader, FjallSnapshot}; pub use snapshot_rocksdb::{RocksDbConfig, RocksDbReader, RocksDbSnapshot}; pub use stores::{Connection, ConnectionCapabilities, KvStore, StorageType, StoreConfig}; #[cfg(feature = "transport")] -pub use transport::{ArtifactTransport, ObjectStoreTransport, run_export_round}; +pub use transport::{ArtifactTransport, ObjectStoreTransport, PublishOutcome, run_export_round}; diff --git a/src/nats.rs b/src/nats.rs index 5c446cd..1c4eee4 100644 --- a/src/nats.rs +++ b/src/nats.rs @@ -1001,6 +1001,14 @@ async fn stream_watch( /// Check if a NATS watch error indicates the requested start sequence is /// too old (compacted), meaning callers should fall back to a full watch. /// +/// SECOND line of defense only: live nats-server (2.14) does not error on a +/// below-head start sequence at all — it silently clamps to the first +/// retained message (pinned by `tests/resync.rs`), so the PRIMARY expiry +/// detection is [`NatsKvWatcher::check_resume_window`]'s proactive +/// `first_sequence` comparison. This matcher remains for server versions or +/// paths that do error, where mapping to [`KvError::CursorExpired`] keeps +/// the fallback reachable instead of stranding the caller. +/// /// async-nats has no granular error kind for this: `WatchErrorKind` is only /// `InvalidKey`/`TimedOut`/`ConsumerCreate`/`Other`, and "start sequence too old" /// arrives as `ConsumerCreate`/`Other` with the real reason buried in the source @@ -1085,6 +1093,54 @@ fn kv_message_to_update(msg: &async_nats::Message, kv_prefix: &str) -> Option Result<(), KvError> { + let stream = timed(self.js.get_stream(format!("KV_{}", self.bucket))) + .await? + .map_err(|e| { + KvError::OperationFailed(format!("get KV stream for resume check: {e}")) + })?; + let first = stream.cached_info().state.first_sequence; + // The shared protocol kernel — the same guard the model checker's + // Resume transition executes (`crate::protocol::resume_window_ok`). + if !crate::protocol::resume_window_ok(revision, first) { + warn!( + revision, + first_sequence = first, + "resume cursor is below the stream's first retained sequence; cursor expired" + ); + return Err(KvError::CursorExpired); + } + Ok(()) + } +} + #[async_trait] impl KvWatcher for NatsKvWatcher { async fn watch_all(&self, tx: Sender) -> Result<(), KvError> { @@ -1146,6 +1202,7 @@ impl KvWatcher for NatsKvWatcher { Some(rev) if rev > 0 => rev, _ => return self.watch_all(tx).await, }; + self.check_resume_window(revision).await?; let watcher = match timed(self.kv.watch_all_from_revision(revision + 1)).await? { Ok(w) => w, @@ -1158,6 +1215,12 @@ impl KvWatcher for NatsKvWatcher { return Err(KvError::WatchError(err_str)); } }; + // Re-check AFTER the consumer exists: head eviction in the window + // between the pre-flight check and consumer creation would otherwise + // clamp silently. Past this point the consumer delivers from retained + // messages; the residual exposure is the generic retention-vs-lag + // bound every live consumer carries (model axiom 5). + self.check_resume_window(revision).await?; info!(revision, "resumed watch from cursor"); stream_watch(watcher, &tx).await @@ -1173,6 +1236,7 @@ impl KvWatcher for NatsKvWatcher { Some(rev) if rev > 0 => rev, _ => return self.watch_prefix(prefix, tx).await, }; + self.check_resume_window(revision).await?; let nats_key = format!("{prefix}>"); let watcher = match timed(self.kv.watch_from_revision(&nats_key, revision + 1)).await? { @@ -1186,6 +1250,9 @@ impl KvWatcher for NatsKvWatcher { return Err(KvError::WatchError(err_str)); } }; + // Same post-create re-check as watch_all_from: close the + // check→create eviction window. + self.check_resume_window(revision).await?; info!(revision, prefix, "resumed prefix watch from cursor"); stream_watch(watcher, &tx).await @@ -1227,6 +1294,21 @@ impl KvWatcher for NatsKvWatcher { .await? .map_err(|e| KvError::WatchError(format!("get KV stream: {e}")))?; + // Same proactive expiry detection as `check_resume_window` (NATS + // silently clamps a below-head ByStartSequence; see that method's + // docs) — checked on the stream handle this path already fetched, + // via the shared protocol kernel. + let first = stream.cached_info().state.first_sequence; + if !crate::protocol::resume_window_ok(revision, first) { + warn!( + revision, + first_sequence = first, + ?prefixes, + "resume cursor is below the stream's first retained sequence; cursor expired" + ); + return Err(KvError::CursorExpired); + } + let consumer = match timed(stream.create_consumer(push::OrderedConfig { deliver_subject: self.client.new_inbox(), description: Some("kv multi-prefix resume consumer".to_string()), @@ -1253,6 +1335,11 @@ impl KvWatcher for NatsKvWatcher { } }; + // Re-check AFTER the consumer exists (fresh stream info, not the + // handle's cached copy): closes the check→create eviction window, + // same as the single-filter resume paths. + self.check_resume_window(revision).await?; + let mut messages = timed(consumer.messages()) .await? .map_err(|e| KvError::WatchError(e.to_string()))?; diff --git a/src/protocol.rs b/src/protocol.rs new file mode 100644 index 0000000..7424c62 --- /dev/null +++ b/src/protocol.rs @@ -0,0 +1,151 @@ +//! Protocol decision kernels — the load-bearing guards of the snapshot +//! export/import protocol, extracted as pure functions so the PRODUCTION +//! code and the exhaustive model checker (`tests/model.rs`) execute the +//! **same logic**, not two hand-synchronized copies of it. +//! +//! Every function here is a guard whose correctness the machine-checked +//! theorems depend on. The call sites: +//! +//! | kernel | production | model | +//! |----------------------------|-------------------------------------|--------------------| +//! | [`pointer_publish_allowed`]| `transport::swap_pointer` | `Act::Publish` | +//! | [`payload_prunable`] | `transport::ObjectStoreTransport::prune` | `Act::Prune` | +//! | [`resume_window_ok`] | `nats` resume paths (`check_resume_window`) | `Act::Resume` | +//! +//! Because the model transitions call these very functions, a change to any +//! guard is re-verified against the full bounded state space on the next +//! `cargo test --test model` — the guards cannot drift from the proof. The +//! mutation tests in `tests/model.rs` additionally prove each guard is +//! load-bearing: substituting a broken variant makes the checker produce a +//! counterexample. +//! +//! Kernels operate on plain `u64` ranks (a [`WatchCursor`](crate::WatchCursor)'s +//! revision, with revisionless cursors ranked 0 by the callers) so they stay +//! free of I/O types and usable from the checker's `u8`-bounded state space. + +/// What the publisher observed at the pointer key before deciding. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum PointerState { + /// No pointer object exists — the slot is open (create-only publish). + Absent, + /// A pointer object exists. `rank` is its cursor's rank, or `None` when + /// the object is unparseable — a corrupt pointer MUST be replaceable, or + /// one bad write wedges publishing forever (the same rule as the export + /// lease's corrupt-steal). + Present { + /// Rank of the existing pointer's cursor; `None` if unparseable. + rank: Option, + }, +} + +/// THE monotonic pointer guard: may `candidate_rank` be published over the +/// observed `current` pointer? +/// +/// `true` for an open slot, a corrupt pointer, or a candidate at or above +/// the existing cursor; `false` exactly when the existing pointer is +/// parseable and STRICTLY newer — the refusal that makes a slow exporter's +/// stale publish a no-op instead of a regression. +/// +/// Soundness of deciding on a read (before the conditional put): every +/// writer uses this guard with a compare-and-swap, so the pointer's rank is +/// monotone non-decreasing — once "strictly newer" is observed, it can never +/// become false, so refusal needs no CAS. Machine-checked as `published +/// cursor never regresses` in `tests/model.rs`. +pub fn pointer_publish_allowed(current: &PointerState, candidate_rank: u64) -> bool { + match current { + PointerState::Absent => true, + PointerState::Present { rank: None } => true, + PointerState::Present { + rank: Some(existing), + } => candidate_rank >= *existing, + } +} + +/// THE prune guard: may this payload object be deleted, given the current +/// pointer's rank? +/// +/// A payload is prunable only when ALL hold: +/// - it is not the pointer's own target; +/// - its rank is parseable AND **strictly below** the pointer's (an +/// unparseable rank is never deleted — unknown objects are not ours); +/// - its age has cleared the grace period (`aged_out`; the model passes +/// `true`, checking the harshest zero-grace timing). +/// +/// Strictly-below is what makes a dangling pointer impossible regardless of +/// timing: [`pointer_publish_allowed`] refuses any candidate below the +/// pointer, and the pointer is monotone — so anything this guard deletes +/// (rank < pointer-at-prune ≤ pointer-at-any-later-swap) can never be +/// successfully published afterward. The model checker FOUND the dangling +/// counterexample under the earlier age-only rule (a same-cursor payload +/// collected mid-publish, then published by the `>=` swap guard); this rule +/// is the structural fix, machine-checked as `pointer target always +/// fetchable` under zero-grace pruning. +pub fn payload_prunable( + payload_rank: Option, + pointer_rank: u64, + is_pointer_target: bool, + aged_out: bool, +) -> bool { + !is_pointer_target && aged_out && payload_rank.is_some_and(|rank| rank < pointer_rank) +} + +/// THE cursor-expiry guard: is resuming from `revision` sound, given the +/// stream's first retained sequence? +/// +/// A resume reads `revision + 1` onward; it is sound iff nothing at or below +/// `revision + 1`'s predecessor gap has been head-evicted — i.e. +/// `first_sequence <= revision + 1`. Interior (per-subject) eviction inside +/// the gap is safe for a last-write-wins fold (an overwrite-evicted revision +/// implies a later revision of the same subject exists and will be +/// delivered); lost DELETES come from head eviction, which is exactly what +/// advances `first_sequence`. +/// +/// This check must be performed by US: NATS does not error on a below-head +/// start sequence — it silently clamps to the first retained message +/// (pinned live by `tests/resync.rs::nats_silently_clamps_resume_below_first_seq`), +/// which would skip the gap's evicted delete markers with no fallback and no +/// resync. Machine-checked as `bootstrap never silently diverges` in +/// `tests/model.rs` (where the model's retention floor is +/// `first_sequence - 1`). +pub fn resume_window_ok(revision: u64, first_sequence: u64) -> bool { + first_sequence <= revision.saturating_add(1) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn publish_guard_boundaries() { + let absent = PointerState::Absent; + let corrupt = PointerState::Present { rank: None }; + let at = |r| PointerState::Present { rank: Some(r) }; + assert!(pointer_publish_allowed(&absent, 0)); + assert!(pointer_publish_allowed(&corrupt, 0)); + assert!(pointer_publish_allowed(&at(5), 5), "equal republishes"); + assert!(pointer_publish_allowed(&at(5), 6)); + assert!(!pointer_publish_allowed(&at(5), 4), "stale is refused"); + } + + #[test] + fn prune_guard_boundaries() { + // Strictly below, aged, not the target: prunable. + assert!(payload_prunable(Some(4), 5, false, true)); + // Equal rank is NOT prunable — it is still publishable (>= guard). + assert!(!payload_prunable(Some(5), 5, false, true)); + assert!(!payload_prunable(Some(6), 5, false, true)); + // The pointer's own target and unparseable ranks are never prunable. + assert!(!payload_prunable(Some(4), 5, true, true)); + assert!(!payload_prunable(None, 5, false, true)); + // Grace window holds everything. + assert!(!payload_prunable(Some(4), 5, false, false)); + } + + #[test] + fn resume_guard_boundaries() { + assert!(resume_window_ok(3, 4), "first retained == next read: sound"); + assert!(resume_window_ok(3, 1), "history intact"); + assert!(!resume_window_ok(3, 5), "gap head-evicted: expired"); + assert!(resume_window_ok(u64::MAX, u64::MAX), "saturating boundary"); + } +} diff --git a/src/transport.rs b/src/transport.rs index 179933d..1db8749 100644 --- a/src/transport.rs +++ b/src/transport.rs @@ -1,24 +1,51 @@ //! Artifact transport: ship export artifacts to object storage and fetch them //! back for bootstrap. Feature `transport`. //! -//! The wire format is a **plain tar** of the artifact directory -//! (`MANIFEST.json` + `data/…`) at `/`, with the manifest -//! duplicated as a sibling object `.manifest.json` so a node can peek at -//! an artifact's cursor/backend without downloading the payload. No +//! ## Wire format: content-addressed payload + monotonic pointer +//! +//! The payload is a **plain tar** of the artifact directory (`MANIFEST.json` +//! plus `data/…`) at a **content-addressed** key — +//! `/.payloads/.tar` — so payload +//! objects are write-once: two different artifacts can never collide on a +//! key, and re-uploading the same artifact is an idempotent overwrite. No //! compression layer: fjall/RocksDB payload files are already lz4/zstd -//! compressed. The sibling manifest is uploaded **last**, so its presence -//! means the payload object is complete — the remote twin of the local -//! artifact's manifest-written-last discipline. +//! compressed. +//! +//! The manifest doubles as the **pointer**: it is published at +//! `/.manifest.json` LAST, via a conditional put (create-only or +//! compare-and-swap on the object version) that only ever moves the cursor +//! FORWARD. This single atomic object is what readers trust; the payload key +//! is derived from its bytes. The discipline is machine-checked: the pointer +//! protocol is the `pointer_swap` model in `tests/model.rs`, where the +//! checker proves torn payload/pointer pairs and cursor regression are +//! structurally impossible — the two hazards the legacy two-register layout +//! (payload and manifest as independent last-write-wins objects at fixed +//! keys) provably had. +//! +//! Consequences, each pinned by `tests/multi_export.rs`: +//! - A slow exporter whose round overran its lease CANNOT clobber a newer +//! published artifact: its swap is refused +//! ([`PublishOutcome::SupersededByNewer`]). +//! - A crash between the payload upload and the pointer swap leaves the OLD +//! pointer fully consistent — bootstrap stays available throughout. +//! +//! Old payload objects linger after their pointer moves on; [`run_export_round`] +//! prunes unreferenced payloads older than a grace period (never the one the +//! current pointer targets, and never young objects a concurrent publisher or +//! an in-flight bootstrap may still reference). //! //! Transport is **untrusted**: [`download`](ArtifactTransport::download) -//! cross-checks the tar's embedded manifest against the sibling object, and -//! the backend `import` re-verifies every payload file hash regardless. +//! cross-checks the tar's embedded manifest against the pointer bytes (the +//! content address makes a mismatch unreachable short of hash breakage or +//! store corruption — kept as defense in depth), and the backend `import` +//! re-verifies every payload file hash regardless. //! //! [`run_export_round`] composes the whole at-most-once round: //! lease → export (through the [`watch_applied`](crate::watch_applied) -//! [`ExportRequest`] channel) → upload → publish completion → **delete the -//! local artifact** (artifacts hardlink fold files and pin storage if they -//! linger — transience is enforced here, not hoped for). +//! [`ExportRequest`] channel) → upload + pointer swap → publish completion → +//! prune stale payloads → **delete the local artifact** (artifacts hardlink +//! fold files and pin storage if they linger — transience is enforced here, +//! not hoped for). use std::path::{Path, PathBuf}; use std::sync::Arc; @@ -27,17 +54,21 @@ use std::time::Duration; use async_trait::async_trait; use futures::StreamExt; use object_store::path::Path as ObjPath; -use object_store::{ObjectStore, ObjectStoreExt, PutPayload, WriteMultipart}; +use object_store::{ + ObjectStore, ObjectStoreExt, PutMode, PutOptions, PutPayload, UpdateVersion, WriteMultipart, +}; use tokio::io::AsyncWriteExt; use tokio::sync::{mpsc, oneshot}; use tracing::warn; use crate::applied::ExportRequest; use crate::artifact::{ - ExportManifest, MANIFEST_FILE, check_dest_available, manifest_from_slice, rename_into_place, + ExportManifest, MANIFEST_FILE, check_dest_available, hex_encode, manifest_from_slice, + rename_into_place, }; use crate::export_lease::ExportLease; use crate::kv::WatchCursor; +use crate::protocol::{PointerState, payload_prunable, pointer_publish_allowed}; use crate::snapshot::SnapshotError; /// Buffered chunk size for uploads/downloads (also the multipart part size). @@ -60,6 +91,34 @@ const MAX_MANIFEST_BYTES: usize = 1 << 20; /// same hazard and same 30 s bound as the NATS layer's `timed()`. const OP_TIMEOUT: Duration = Duration::from_secs(30); +/// Bound on pointer-swap CAS retries. Each retry means another publisher won +/// the race in the read→swap window; with rounds minutes apart, more than a +/// couple of iterations indicates something pathological, and an unbounded +/// loop would livelock against it. +const MAX_SWAP_ATTEMPTS: usize = 8; + +/// Outcome of [`ArtifactTransport::upload`]'s pointer swap. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum PublishOutcome { + /// This artifact is now the published "latest": its payload is uploaded + /// and the pointer references it. + Published, + /// The pointer already references a STRICTLY newer artifact, so the + /// monotonic swap refused to regress it. This round's payload was + /// uploaded but is unreferenced (the next prune collects it). Routine + /// under lease overrun: the fleet has moved on, nothing was lost. + SupersededByNewer { + /// The newer published artifact's cursor. + remote_cursor: crate::kv::WatchCursor, + }, +} + +/// Total order for the monotonic pointer guard. Revisionless cursors rank 0: +/// a real cursor always supersedes an empty one, never the reverse. +fn cursor_rank(c: &WatchCursor) -> u64 { + c.as_u64().unwrap_or(0) +} + /// Bound one object-store await by `limit`. async fn timed_by( what: &str, @@ -86,16 +145,28 @@ async fn timed( /// for the wire format. #[async_trait] pub trait ArtifactTransport: Send + Sync { - /// Tar `artifact_dir` and upload it at `key`, then upload the manifest as - /// the sibling object `.manifest.json` (the completeness marker). - /// Re-uploading the same `key` overwrites — "latest" keys are - /// last-write-wins by design. - async fn upload(&self, key: &str, artifact_dir: &Path) -> Result<(), SnapshotError>; - - /// Fetch only the sibling manifest — peek at an artifact's cursor and + /// Tar `artifact_dir`, upload it at its content-addressed payload key, + /// then publish the manifest as the pointer `.manifest.json` via a + /// monotonic conditional swap. The pointer only ever moves the cursor + /// forward: an older artifact gets + /// [`PublishOutcome::SupersededByNewer`], never a regression. + async fn upload(&self, key: &str, artifact_dir: &Path) + -> Result; + + /// Fetch only the pointer manifest — peek at an artifact's cursor and /// backend before committing to a payload download. async fn manifest(&self, key: &str) -> Result; + /// Delete payload objects under `key` that the current pointer does not + /// reference and that are older than `grace`. Best-effort housekeeping — + /// returns the number deleted. The grace period protects concurrent + /// publishers mid-swap and in-flight bootstraps holding an older pointer + /// read. Default: no-op (transports with nothing to prune). + async fn prune(&self, key: &str, grace: Duration) -> Result { + let _ = (key, grace); + Ok(0) + } + /// Download and unpack the artifact at `key` into `dest_dir` (which must /// not exist or be an empty directory), returning its manifest. The /// unpacked directory is a local artifact, ready for the backend's @@ -109,6 +180,9 @@ pub trait ArtifactTransport: Send + Sync { pub struct ObjectStoreTransport { store: Arc, prefix: ObjPath, + /// Accept stores without conditional-put support (see + /// [`with_non_atomic_pointer_fallback`](Self::with_non_atomic_pointer_fallback)). + allow_non_atomic_pointer: bool, } impl ObjectStoreTransport { @@ -117,9 +191,27 @@ impl ObjectStoreTransport { Self { store, prefix: ObjPath::from(prefix.as_ref()), + allow_non_atomic_pointer: false, } } + /// Accept stores that lack conditional puts (`PutMode::Update` → + /// `NotImplemented`, e.g. `object_store`'s `LocalFileSystem`): the + /// pointer publish degrades to read-check-then-unconditional-put. + /// + /// OUTSIDE the verified protocol: the monotonic refusal still runs, but + /// the write is not atomic, so two publishers racing the read→write + /// window are last-write-wins — the legacy regression hazard + /// (`tests/model.rs`, legacy configuration) survives on such a store. + /// Without this opt-in, a swap on a non-CAS store FAILS the round + /// instead of silently degrading. Dev/test `file://` use only; every + /// deployment-grade store (S3/GCS/Azure/MinIO/R2) supports the atomic + /// path. + pub fn with_non_atomic_pointer_fallback(mut self) -> Self { + self.allow_non_atomic_pointer = true; + self + } + /// Build from a URL (`s3://bucket/prefix`, `file:///path`, …) plus /// explicit builder options (e.g. `aws_endpoint`, `aws_access_key_id`, /// `aws_virtual_hosted_style_request`). @@ -141,19 +233,176 @@ impl ObjectStoreTransport { Ok(Self { store: Arc::from(store), prefix, + allow_non_atomic_pointer: false, }) } // `Path::from` parses `/` separators, so multi-segment keys // (`edge-origins/us-east/latest`) land as real object hierarchy. - fn payload_path(&self, key: &str) -> ObjPath { - ObjPath::from(format!("{}/{key}", self.prefix)) + // + /// The directory of `key`'s content-addressed payload objects. + fn payloads_dir(&self, key: &str) -> ObjPath { + ObjPath::from(format!("{}/{key}.payloads", self.prefix)) + } + + /// A payload object's address: `-.tar`. + /// + /// The hash half is the content address (the manifest embeds a BLAKE3 + /// digest per payload file, so it commits to the artifact's content; 64 + /// bits — a birthday collision needs ~2^32 artifacts under ONE key, and + /// even then the embedded-manifest cross-check at download detects the + /// mix). The cursor half is LOAD-BEARING for prune: it lets prune apply + /// the strictly-below-the-pointer rule without fetching each payload + /// (see [`prune`](Self::prune) — the rule is what makes a + /// pruned-then-published dangling pointer structurally impossible, a + /// hazard the model checker found at zero grace). + fn payload_path(&self, key: &str, cursor: &WatchCursor, pointer_bytes: &[u8]) -> ObjPath { + let digest = blake3::hash(pointer_bytes); + let hex = hex_encode(&digest.as_bytes()[..8]); + ObjPath::from(format!( + "{}/{key}.payloads/{:016x}-{hex}.tar", + self.prefix, + cursor_rank(cursor) + )) } fn manifest_path(&self, key: &str) -> ObjPath { ObjPath::from(format!("{}/{key}.manifest.json", self.prefix)) } + /// Publish `pointer_bytes` (the manifest) at the pointer key with a + /// monotonic conditional swap: create-only when absent, compare-and-swap + /// against the observed object version when present, and REFUSED when + /// the present pointer's cursor is strictly newer. An unparseable + /// pointer is replaced (CAS against its version) — one corrupt object + /// must not wedge publishing, mirroring the lease's corrupt-steal rule. + /// + /// This is the `Publish` transition of the `pointer_swap` model in + /// `tests/model.rs`; the conditional-put semantics it relies on are + /// verified against real S3 (MinIO) by `tests/transport_s3.rs`. + async fn swap_pointer( + &self, + key: &str, + new_cursor: &WatchCursor, + pointer_bytes: &[u8], + ) -> Result { + let path = self.manifest_path(key); + for _ in 0..MAX_SWAP_ATTEMPTS { + // Read the current pointer (bytes + object version for the CAS). + let current = match timed("pointer get", self.store.get(&path)).await? { + Ok(get) => { + let meta = get.meta.clone(); + let mut stream = get.into_stream(); + let mut buf = Vec::new(); + while let Some(chunk) = timed("pointer read", stream.next()).await? { + let chunk = chunk.map_err(map_obj)?; + if buf.len() + chunk.len() > MAX_MANIFEST_BYTES { + return Err(SnapshotError::ArtifactInvalid(format!( + "remote pointer for {key:?} exceeds {MAX_MANIFEST_BYTES} bytes" + ))); + } + buf.extend_from_slice(&chunk); + } + Some((meta, buf)) + } + Err(object_store::Error::NotFound { .. }) => None, + Err(e) => return Err(map_obj(e)), + }; + + match current { + None => { + // Open slot: create-only — exactly one concurrent + // publisher can win it. + let opts = PutOptions::from(PutMode::Create); + match timed( + "pointer create", + self.store + .put_opts(&path, PutPayload::from(pointer_bytes.to_vec()), opts), + ) + .await? + { + Ok(_) => return Ok(PublishOutcome::Published), + Err(object_store::Error::AlreadyExists { .. }) => continue, // lost the race; re-read + Err(e) => return Err(map_obj(e)), + } + } + Some((meta, bytes)) => { + // THE monotonic guard — the shared protocol kernel, the + // same function the model checker's Publish transition + // executes (`crate::protocol`). An unparseable pointer is + // rank-less and replaced. + let existing = manifest_from_slice(&bytes).ok(); + let observed = PointerState::Present { + rank: existing.as_ref().map(|m| cursor_rank(&m.cursor)), + }; + if !pointer_publish_allowed(&observed, cursor_rank(new_cursor)) { + let existing = + existing.expect("refusal implies a parseable, newer pointer"); + return Ok(PublishOutcome::SupersededByNewer { + remote_cursor: existing.cursor, + }); + } + let opts = PutOptions::from(PutMode::Update(UpdateVersion { + e_tag: meta.e_tag, + version: meta.version, + })); + match timed( + "pointer swap", + self.store + .put_opts(&path, PutPayload::from(pointer_bytes.to_vec()), opts), + ) + .await? + { + Ok(_) => return Ok(PublishOutcome::Published), + Err(object_store::Error::Precondition { .. }) => continue, // raced; re-read + Err(object_store::Error::NotFound { .. }) => continue, // deleted under us; re-read + Err(object_store::Error::NotImplemented { .. }) => { + // Store without compare-and-swap (object_store's + // LocalFileSystem). FAIL CLOSED unless the caller + // explicitly opted in: a silently degraded swap + // would reintroduce the legacy regression hazard + // (tests/model.rs proves it reachable) with a log + // line as the only witness — the same violated- + // obligation pattern the resync fail-stop change + // eliminated. + if !self.allow_non_atomic_pointer { + return Err(SnapshotError::Backend(format!( + "object store lacks conditional puts (PutMode::Update \ + unimplemented); the pointer swap for {key:?} cannot be \ + atomic and this store is outside the verified protocol. \ + For dev/test stores (file://), opt in explicitly with \ + ObjectStoreTransport::with_non_atomic_pointer_fallback()" + ))); + } + // Opted in: the monotonic REFUSAL above still + // ran, but the write is unconditional — two + // publishers racing this window are + // last-write-wins, on this store only. + warn!( + key, + "non-atomic pointer fallback (explicit opt-in): publish is \ + read-check-then-put; concurrent publishers may race" + ); + timed( + "pointer put (non-atomic fallback)", + self.store + .put(&path, PutPayload::from(pointer_bytes.to_vec())), + ) + .await? + .map_err(map_obj)?; + return Ok(PublishOutcome::Published); + } + Err(e) => return Err(map_obj(e)), + } + } + } + } + Err(SnapshotError::Backend(format!( + "pointer swap for {key:?} lost {MAX_SWAP_ATTEMPTS} consecutive CAS races; giving up" + ))) + } + /// Fetch the sibling manifest object, enforcing [`MAX_MANIFEST_BYTES`]. async fn fetch_manifest_bytes(&self, key: &str) -> Result, SnapshotError> { let mut stream = timed("manifest get", self.store.get(&self.manifest_path(key))) @@ -176,14 +425,19 @@ impl ObjectStoreTransport { #[async_trait] impl ArtifactTransport for ObjectStoreTransport { - async fn upload(&self, key: &str, artifact_dir: &Path) -> Result<(), SnapshotError> { + async fn upload( + &self, + key: &str, + artifact_dir: &Path, + ) -> Result { // Read the manifest first — it doubles as the artifact-completeness - // check (export writes it last). + // check (export writes it last) and, as the pointer bytes, derives + // the payload's content address. let manifest_bytes = tokio::fs::read(artifact_dir.join(MANIFEST_FILE)) .await .map_err(SnapshotError::Io)?; // Validate before shipping: never upload an artifact we couldn't read back. - manifest_from_slice(&manifest_bytes)?; + let manifest = manifest_from_slice(&manifest_bytes)?; // Tar the artifact into a temp file on a blocking task. A temp file // (rather than streaming the tar straight into the upload) keeps the @@ -223,7 +477,8 @@ impl ArtifactTransport for ObjectStoreTransport { .map_err(SnapshotError::Io)?; let upload = timed( "multipart create", - self.store.put_multipart(&self.payload_path(key)), + self.store + .put_multipart(&self.payload_path(key, &manifest.cursor, &manifest_bytes)), ) .await? .map_err(map_obj)?; @@ -251,15 +506,10 @@ impl ArtifactTransport for ObjectStoreTransport { .await? .map_err(map_obj)?; - // Manifest sibling LAST: its presence marks the payload complete. - timed( - "manifest put", - self.store - .put(&self.manifest_path(key), PutPayload::from(manifest_bytes)), - ) - .await? - .map_err(map_obj)?; - Ok(()) + // Pointer LAST, by monotonic conditional swap: its presence marks the + // payload complete, and it can never regress past a newer round. + self.swap_pointer(key, &manifest.cursor, &manifest_bytes) + .await } async fn manifest(&self, key: &str) -> Result { @@ -279,8 +529,9 @@ impl ArtifactTransport for ObjectStoreTransport { )) })?; - // Sibling manifest first — it is the completeness marker and the value - // we cross-check the tar against. + // Pointer first — it is the completeness marker, the source of the + // payload's content address, and the value we cross-check the tar + // against. let sibling = self.fetch_manifest_bytes(key).await?; let manifest = manifest_from_slice(&sibling)?; @@ -289,10 +540,14 @@ impl ArtifactTransport for ObjectStoreTransport { let mut tar_writer = tokio::fs::File::create(tar_tmp.path()) .await .map_err(SnapshotError::Io)?; - let mut stream = timed("payload get", self.store.get(&self.payload_path(key))) - .await? - .map_err(map_obj)? - .into_stream(); + let mut stream = timed( + "payload get", + self.store + .get(&self.payload_path(key, &manifest.cursor, &sibling)), + ) + .await? + .map_err(map_obj)? + .into_stream(); while let Some(chunk) = timed("payload read", stream.next()).await? { let chunk = chunk.map_err(map_obj)?; tar_writer @@ -344,6 +599,54 @@ impl ArtifactTransport for ObjectStoreTransport { Ok(manifest) } + + async fn prune(&self, key: &str, grace: Duration) -> Result { + // No pointer (or an unparseable one — it will be replaced by the + // next publish) → nothing is provably stale, so prune nothing. + let pointer = match self.fetch_manifest_bytes(key).await { + Ok(b) => b, + Err(_) => return Ok(0), + }; + let Ok(current) = manifest_from_slice(&pointer) else { + return Ok(0); + }; + let keep = self.payload_path(key, ¤t.cursor, &pointer); + let pointer_rank = cursor_rank(¤t.cursor); + let cutoff_millis = std::time::SystemTime::now() + .checked_sub(grace) + .and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok()) + .map_or(0, |d| d.as_millis() as i64); + + let mut deleted = 0usize; + let mut listing = self.store.list(Some(&self.payloads_dir(key))); + while let Some(meta) = timed("payload list", listing.next()).await? { + let meta = meta.map_err(map_obj)?; + // THE prune guard — the shared protocol kernel, the same + // function the model checker's Prune transition executes + // (`crate::protocol::payload_prunable`, where the + // strictly-below-the-pointer rule and its dangling-pointer + // impossibility argument live). The grace period remains as + // defense in depth for in-flight downloads holding stale + // pointer reads. + let payload_rank = meta + .location + .filename() + .and_then(|f| f.split('-').next()) + .and_then(|h| u64::from_str_radix(h, 16).ok()); + if payload_prunable( + payload_rank, + pointer_rank, + meta.location == keep, + meta.last_modified.timestamp_millis() <= cutoff_millis, + ) { + timed("payload delete", self.store.delete(&meta.location)) + .await? + .map_err(map_obj)?; + deleted += 1; + } + } + Ok(deleted) + } } fn map_obj(e: object_store::Error) -> SnapshotError { @@ -432,10 +735,25 @@ pub async fn run_export_round( } }; - // Upload; only then publish completion. - if let Err(e) = transport.upload(key, &artifact_dir).await { - guard.abandon().await; - return Err(e); + // Upload + pointer swap; only then publish completion. + let outcome = match transport.upload(key, &artifact_dir).await { + Ok(o) => o, + Err(e) => { + guard.abandon().await; + return Err(e); + } + }; + if let PublishOutcome::SupersededByNewer { remote_cursor } = &outcome { + // Routine under lease overrun: another node published a newer round + // while this one ran. The monotonic swap refused the regression — + // the published "latest" is intact, this round's payload awaits the + // next prune. + warn!( + key, + local = ?manifest.cursor, + remote = ?remote_cursor, + "export round superseded by a newer published artifact; pointer left untouched" + ); } if let Err(e) = guard.complete(&manifest.cursor).await { // The artifact IS uploaded — the round succeeded. Losing the @@ -443,6 +761,16 @@ pub async fn run_export_round( warn!(key, error = %e, "export round uploaded but completion record failed"); } + if matches!(outcome, PublishOutcome::Published) { + // Housekeeping: collect payloads no pointer references. Grace of 4 + // round periods comfortably outlives any concurrent publisher's + // upload→swap window and any in-flight bootstrap holding an older + // pointer read (both are bounded by round-period timescales). + if let Err(e) = transport.prune(key, ttl.saturating_mul(4)).await { + warn!(key, error = %e, "stale payload prune failed; retried next round"); + } + } + drop(round_dir); // enforce artifact transience Ok(Some(manifest)) } diff --git a/tests/common/mod.rs b/tests/common/mod.rs index bdf1cf6..8f45940 100644 --- a/tests/common/mod.rs +++ b/tests/common/mod.rs @@ -195,3 +195,90 @@ impl Drop for TestMinio { let _ = self.child.wait(); } } + +// --- Crash injection for the transport tiers --------------------------------- + +/// An [`object_store::ObjectStore`] that, once armed, fails the next PUT of a +/// `*.manifest.json` object — the process crash in the window between +/// `upload`'s payload multipart (already complete) and its sibling-manifest +/// PUT. Everything else delegates to the wrapped store. +#[derive(Debug)] +pub struct ManifestPutCrash { + pub inner: std::sync::Arc, + pub armed: std::sync::atomic::AtomicBool, +} + +impl std::fmt::Display for ManifestPutCrash { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "ManifestPutCrash({})", self.inner) + } +} + +#[async_trait::async_trait] +impl object_store::ObjectStore for ManifestPutCrash { + async fn put_opts( + &self, + location: &object_store::path::Path, + payload: object_store::PutPayload, + opts: object_store::PutOptions, + ) -> object_store::Result { + if location.as_ref().ends_with(".manifest.json") + && self.armed.swap(false, std::sync::atomic::Ordering::SeqCst) + { + return Err(object_store::Error::Generic { + store: "ManifestPutCrash", + source: "injected crash between payload and manifest PUT".into(), + }); + } + self.inner.put_opts(location, payload, opts).await + } + + async fn put_multipart_opts( + &self, + location: &object_store::path::Path, + opts: object_store::PutMultipartOptions, + ) -> object_store::Result> { + self.inner.put_multipart_opts(location, opts).await + } + + async fn get_opts( + &self, + location: &object_store::path::Path, + options: object_store::GetOptions, + ) -> object_store::Result { + self.inner.get_opts(location, options).await + } + + fn delete_stream( + &self, + locations: futures::stream::BoxStream< + 'static, + object_store::Result, + >, + ) -> futures::stream::BoxStream<'static, object_store::Result> { + self.inner.delete_stream(locations) + } + + fn list( + &self, + prefix: Option<&object_store::path::Path>, + ) -> futures::stream::BoxStream<'static, object_store::Result> { + self.inner.list(prefix) + } + + async fn list_with_delimiter( + &self, + prefix: Option<&object_store::path::Path>, + ) -> object_store::Result { + self.inner.list_with_delimiter(prefix).await + } + + async fn copy_opts( + &self, + from: &object_store::path::Path, + to: &object_store::path::Path, + options: object_store::CopyOptions, + ) -> object_store::Result<()> { + self.inner.copy_opts(from, to, options).await + } +} diff --git a/tests/model.rs b/tests/model.rs new file mode 100644 index 0000000..706d4de --- /dev/null +++ b/tests/model.rs @@ -0,0 +1,934 @@ +//! Exhaustive model-check of the snapshot export/import protocol (Stateright). +//! +//! `tests/multi_export.rs` demonstrates specific bad interleavings; this file +//! PROVES properties over **every** interleaving of the modeled protocol, +//! within explicit bounds. The two layers are deliberately coupled: each +//! hazard demonstrated empirically appears here as a `sometimes` property the +//! checker must re-derive (so the model is faithful enough to express the +//! bugs), and each safety claim appears as an `always` property checked over +//! the full state space (so the claim is not an induction from sampled runs). +//! +//! ## What is modeled +//! +//! - N exporter replicas of one fold, each at its own applied cursor, racing +//! uploads of a shared "latest" key. A replica may **crash between any two +//! steps** — including between its payload upload and its manifest publish. +//! - The object store under BOTH transport layouts: the SHIPPED protocol +//! (`pointer_swap: true` — content-addressed write-once payloads plus a +//! single pointer object published last via monotonic conditional swap, +//! `transport.rs` as of 0.6), and the LEGACY pre-0.6 layout +//! (`pointer_swap: false` — payload tar and sibling manifest as two +//! independent atomic last-write-wins registers), kept as the +//! machine-checked record of why the protocol changed. +//! - **Prune** (shipped protocol): unreferenced payload objects can be +//! deleted at any moment — modeled at zero grace, a superset of every real +//! grace-period timing. The checker proves a prune racing a stale pointer +//! read costs a DETECTED fetch miss and a retry, never wrong data, and the +//! current pointer's target is never pruned. +//! - The source stream with **retention**: a floor that advances freely; +//! resuming below the floor is `CursorExpired`; delete markers at or below +//! the floor are evicted (the re-list cannot see them). +//! - A bootstrapping importer whose two reads (sibling manifest, then +//! payload) interleave with all of the above, whose cross-check compares +//! them, and whose post-import resume either replays the tail, or falls +//! back (expired cursor) under one of THREE resync modes: reader not wired +//! (`None`), reader wired with the pre-fix warn-and-continue failure +//! semantics (`Degrade`), or reader wired with fail-stop failure semantics +//! (`FailStop` — `applied.rs` as it ships). The checker proves `Degrade` +//! breaks the convergence theorem, which is the machine-checked +//! justification for the fail-stop change in `resync_stale_keys`. +//! +//! The empirical coupling runs both directions: the legacy configuration's +//! `sometimes` hazards are the interleavings `tests/multi_export.rs` drives +//! against the real code, where the shipped protocol's `always` theorems are +//! asserted as outcomes. +//! +//! ## What is deliberately NOT modeled, and why that is sound +//! +//! - **The export lease.** The lease only ever REMOVES interleavings (its own +//! docs: "a work-deduplication optimization, never a correctness gate"). +//! Exporters here act with no coordination at all, which checks a strict +//! SUPERSET of the behaviors any lease implementation (any ttl, any clock +//! skew, any takeover policy) can produce. Every `always` property proven +//! here therefore holds a fortiori with the lease present. This removes +//! clock skew from the proof obligation entirely. +//! - **Artifact bytes.** An artifact is its identity `(node, cursor, key-set)`; +//! "embedded manifest equals sibling manifest byte-for-byte" is modeled as +//! identity equality. Axiom: manifest bytes are equal iff they describe the +//! same artifact content (manifests embed a BLAKE3 digest per payload file; +//! collision resistance). Under that axiom the model's cross-check and the +//! code's byte-compare accept exactly the same pairs. +//! +//! ## Axioms (the environment obligations the proof is relative to) +//! +//! 1. Object PUTs are atomic per object, and conditional puts (create-only, +//! compare-and-swap on the object version) have one winner per slot — +//! S3/GCS/Azure/MinIO semantics, verified against live MinIO by +//! `tests/transport_s3.rs`. (`object_store`'s LocalFileSystem lacks CAS; +//! `swap_pointer` FAILS CLOSED there unless the caller explicitly opts in +//! via `with_non_atomic_pointer_fallback()` — `file://` is a dev +//! convenience outside the verified envelope, by signed waiver only.) +//! 2. BLAKE3 collision resistance (manifest equality ⟺ content identity). +//! 3. Cursor expiry is DETECTED: the model's `floor` is the stream's +//! `first_sequence`, and a resume below it takes the expired path, never +//! a silent skip. NATS does NOT provide this by erroring — it silently +//! clamps a below-head start sequence (pinned by +//! `tests/resync.rs::nats_silently_clamps_resume_below_first_seq`) — so +//! the code provides it proactively via `check_resume_window` +//! (first_sequence comparison), verified end-to-end against a live +//! nats-server by `tests/resync.rs`. +//! 4. The fold is the KV-mirror `SnapshotStore` (last-write-wins per key); +//! `import` verifies every declared file hash and rejects extras +//! (empirical tier: tampered-artifact and multi-SST round-trip tests). +//! NATS KV CAS semantics for the lease are unneeded here (see above); the +//! lease layer is verified by `integration.rs` contention tests. +//! 5. Retention outlives consumer lag: the stream's retention window exceeds +//! the seconds-scale lag between a consumer's expiry check and its +//! deliveries. The code closes the check→create window with a re-check +//! after consumer creation (`nats.rs`); the residual in-flight exposure +//! is the bound every log consumer (Kafka included) operates under, and +//! is an OPERATING requirement: configure retention in hours, not +//! seconds. The model's atomic delivery transitions encode this axiom. +//! +//! ## Bounds and the small-scope argument +//! +//! Default: 2 exporters (a const-generic parameter), revisions ≤ 3, one +//! importer, one deletable sentinel key — and **unbounded rounds**: a +//! publisher re-enters the pipeline whenever it has applied past its last +//! publish (`NextRound`), so every theorem quantifies over repeated rounds, +//! including a node racing its own previous publish. Every hazard class +//! needs at most: two distinct cursors (regression), one crash window (torn +//! pair), one delete + floor advance (stale key) — and every `sometimes` +//! witness fails loudly if a bound ever clips its scenario. +//! +//! The ignored deep tier (release mode, scheduled runs) pushes both axes: +//! revisions ≤ 5 (~154M unique states) and THREE exporters (~95M unique +//! states — three-way publish races, double-stalled rounds behind a +//! takeover, prune racing two concurrent uploads), plus the legacy layout at +//! fleet size 3 proving the hazards stay reachable at scale. +//! +//! ## Liveness +//! +//! Cycle-proof and Stateright-native: the `every maximal run ends with a +//! completed, synced bootstrap` invariant recomputes the enabled-action set +//! per state — a state with no enabled actions is the end of a maximal +//! execution and must hold a finished, converged bootstrap. Retry loops are +//! cycles, never terminal, so they cannot satisfy it vacuously; a protocol +//! change that could strand the importer (deadlock, unrecoverable failure +//! state, bootstrap that can never finish) fails this invariant. + +use stateright::{Checker, Model, Property}; + +/// Default bucket-revision bound (1..=MAX_REV). 3 suffices for every hazard +/// class and witness (two distinct export cursors plus a delete revision — +/// each `sometimes` property fails loudly if a bound ever clips its +/// scenario). The ignored deep tests push the bound and the fleet size +/// further in release mode. +const MAX_REV: u8 = 3; + +/// An export artifact's identity: who exported, at which applied cursor, and +/// whether the sentinel key was still present at that cursor. Two artifacts +/// are byte-identical iff this identity is equal (BLAKE3 axiom). +#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq, PartialOrd, Ord)] +struct Artifact { + node: u8, + cursor: u8, + has_key: bool, +} + +#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq, PartialOrd, Ord)] +enum ExporterPc { + Idle, + /// Fold exported to local scratch at this identity; nothing uploaded yet. + Exported(Artifact), + /// Payload object uploaded; sibling manifest / pointer not yet published. + /// Crashing HERE is the torn-pair window of the current protocol. + PayloadUp(Artifact), + /// Published at this cursor. `NextRound` re-enters the pipeline once + /// the replica has applied past it — fleets round forever. + Done(u8), +} + +#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq, PartialOrd, Ord)] +enum FoldStatus { + /// The bootstrapped fold converges to the bucket (tail replay, or + /// fallback with resync, or fallback where the re-list happens to cover). + Synced, + /// Silent divergence: the fold holds a key the bucket deleted, and + /// nothing will ever remove it (expired cursor + evicted marker + no + /// resync). + StaleKey, +} + +#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq, PartialOrd, Ord)] +enum ImporterPc { + Start, + /// Read the sibling manifest (or pointer) — first of the two reads. + GotManifest(Artifact), + /// Cross-check passed; fold installed at the artifact's cursor. + Imported(Artifact), + /// Cross-check FAILED (embedded manifest ≠ sibling): the torn pair was + /// detected and rejected. Retry returns to Start. + CrossCheckFailed, + /// Pointer-swap only: the held pointer's payload was PRUNED between the + /// pointer read and the payload fetch (in code: download's content + /// address dereferences to NotFound → `ArtifactInvalid`). Detected, + /// never silent; Retry re-reads the (necessarily newer) pointer. + FetchMissed, + /// Resume ran; final verdict on this bootstrap. + Resumed(Artifact, FoldStatus), +} + +#[derive(Clone, Debug, Hash, PartialEq, Eq)] +struct St { + /// Bucket high-water revision. + head: u8, + /// Revision at which the sentinel key was deleted, if it was. + delete_rev: Option, + /// Stream retention floor: resuming from cursor < floor is CursorExpired; + /// a delete marker at rev ≤ floor has been evicted. + floor: u8, + /// Each replica's applied cursor (≤ head). + applied: [u8; N], + exporters: [ExporterPc; N], + /// CURRENT protocol: the payload object — an atomic LWW register. + payload: Option, + /// CURRENT: the sibling manifest LWW register. FIXED: the pointer object, + /// published only via monotonic conditional swap. + manifest: Option, + /// FIXED protocol: content-addressed payload objects — write-once, never + /// overwritten. (Unused in the current protocol.) + uploaded: std::collections::BTreeSet, + /// Latched when a manifest/pointer publish replaced a strictly newer one. + regressed: bool, + /// FIXED: latched when the monotonic swap refused an older publish — + /// vacuity witness that the guard actually fires within the bounds. + refused: bool, + importer: ImporterPc, +} + +#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq)] +enum Act { + /// A new revision lands in the bucket. + Churn, + /// The sentinel key is deleted (consumes a revision). + DeleteKey, + /// Retention floor advances by one. + Compact, + /// Replica n applies the next revision. + Apply(u8), + /// Replica n snapshots its fold at its current applied cursor. + Export(u8), + /// Replica n uploads its payload object. + UploadPayload(u8), + /// Replica n publishes its sibling manifest (current) / swaps the + /// pointer (fixed). + Publish(u8), + /// Replica n crashes mid-round and restarts idle. + Crash(u8), + /// Replica n starts a fresh round after a successful publish (enabled + /// once it has applied past its last published cursor). + NextRound(u8), + ReadManifest, + ReadPayload, + /// Pointer-swap only: delete every payload the current pointer does not + /// reference — the harshest prune (zero grace, fires whenever anything + /// is unreferenced), a SUPERSET of every real grace-period timing. + Prune, + Retry, + Resume, + /// Degrade mode only: the resume completed but its resync failed + /// mid-flight and the code warned-and-continued re-list-only. + ResumeResyncDegraded, +} + +/// How the bootstrapping node handles the cursor-expired stale-key resync. +#[derive(Clone, Copy, PartialEq, Eq)] +enum ResyncMode { + /// No reader wired (`watch_applied(reader: None, ..)`): expiry falls back + /// re-list-only by explicit caller choice. + None, + /// Reader wired, but a resync I/O failure DEGRADES to re-list-only with a + /// warning — the code's semantics BEFORE the fail-stop fix. The checker + /// proves this breaks the convergence theorem, which is why the code + /// changed. + Degrade, + /// Reader wired, resync failure fails the watch (the caller's restart + /// retries resume → expiry → resync) — `applied.rs` as it ships now. A + /// failed attempt changes nothing observable, so in the model it is the + /// `Resume` action simply remaining enabled; only a SUCCESSFUL resync + /// completes the bootstrap. + FailStop, +} + +/// Deliberately broken guard variants. Each mutation test substitutes one +/// and asserts the checker PRODUCES A COUNTEREXAMPLE — proving every shared +/// kernel guard is load-bearing for the theorems, not incidentally safe. +/// (The unmutated configurations call the kernels themselves, so a kernel +/// regression fails the main theorems directly; these prove the properties +/// would catch it.) +#[derive(Clone, Copy, PartialEq, Eq)] +enum Mutation { + None, + /// Pointer publish ignores the monotonic guard (last-write-wins). + LwwPointer, + /// Prune ignores the strictly-below-the-pointer rule (age-only — the + /// rule the checker originally caught dangling). + PruneAgeOnly, + /// Expiry detection removed: an expired resume behaves like NATS's + /// silent clamp (gap skipped, resync never triggered) — the live bug + /// `tests/resync.rs` pinned. + SilentClamp, +} + +/// Model parameters: which protocol, the importer's resync mode, an optional +/// guard mutation, and the revision bound. +#[derive(Clone)] +struct SnapshotProtocol { + pointer_swap: bool, + resync: ResyncMode, + mutation: Mutation, + max_rev: u8, +} + +impl SnapshotProtocol { + fn shipped(resync: ResyncMode) -> Self { + Self { + pointer_swap: true, + resync, + mutation: Mutation::None, + max_rev: MAX_REV, + } + } + + fn legacy(resync: ResyncMode) -> Self { + Self { + pointer_swap: false, + resync, + mutation: Mutation::None, + max_rev: MAX_REV, + } + } + + fn key_present_at(s: &St, cursor: u8) -> bool { + s.delete_rev.is_none_or(|d| cursor < d) + } + + fn bucket_has_key(s: &St) -> bool { + s.delete_rev.is_none() + } + + /// The expiry guard — THE SHARED KERNEL (`slipstream::protocol`), the + /// same function `nats.rs`'s resume paths execute. The model's retention + /// floor is the stream's `first_sequence - 1`, so the first retained + /// sequence is `floor + 1`. + fn resume_ok(&self, s: &St, a: Artifact) -> bool { + slipstream::protocol::resume_window_ok(a.cursor as u64, s.floor as u64 + 1) + } + + /// Outcome of an expired-cursor fallback WITHOUT a working resync: the + /// re-list delivers current values only, so a key deleted during the gap + /// is covered iff its delete marker survived retention (delete_rev > + /// floor — the fallback watch replays retained history, markers + /// included). + fn relist_only_status(s: &St, a: Artifact) -> FoldStatus { + let marker_evicted = s.delete_rev.is_some_and(|d| d <= s.floor); + if a.has_key && !Self::bucket_has_key(s) && marker_evicted { + FoldStatus::StaleKey + } else { + FoldStatus::Synced + } + } +} + +impl Model for SnapshotProtocol { + type State = St; + type Action = Act; + + fn init_states(&self) -> Vec> { + vec![St { + head: 0, + delete_rev: None, + floor: 0, + applied: [0; N], + exporters: [ExporterPc::Idle; N], + payload: None, + manifest: None, + uploaded: Default::default(), + regressed: false, + refused: false, + importer: ImporterPc::Start, + }] + } + + fn actions(&self, s: &St, acts: &mut Vec) { + if s.head < self.max_rev { + acts.push(Act::Churn); + if s.delete_rev.is_none() { + acts.push(Act::DeleteKey); + } + } + if s.floor < s.head { + acts.push(Act::Compact); + } + for n in 0..N as u8 { + if s.applied[n as usize] < s.head { + acts.push(Act::Apply(n)); + } + match s.exporters[n as usize] { + ExporterPc::Idle if s.applied[n as usize] >= 1 => acts.push(Act::Export(n)), + // A new round once the replica has applied past its last + // publish — fleets round forever, so every theorem must hold + // across repeated rounds, including a node racing its OWN + // previous publish. + ExporterPc::Done(c) if s.applied[n as usize] > c => { + acts.push(Act::NextRound(n)); + } + ExporterPc::Exported(_) => { + acts.push(Act::UploadPayload(n)); + acts.push(Act::Crash(n)); + } + ExporterPc::PayloadUp(_) => { + acts.push(Act::Publish(n)); + acts.push(Act::Crash(n)); + } + _ => {} + } + } + if self.pointer_swap + && let Some(m) = s.manifest + && s.uploaded.iter().any(|a| a.cursor < m.cursor) + { + acts.push(Act::Prune); + } + match s.importer { + ImporterPc::Start if s.manifest.is_some() => acts.push(Act::ReadManifest), + ImporterPc::GotManifest(_) => acts.push(Act::ReadPayload), + ImporterPc::CrossCheckFailed | ImporterPc::FetchMissed => acts.push(Act::Retry), + ImporterPc::Imported(a) => { + acts.push(Act::Resume); + // Under Degrade semantics an expired-cursor resume may also + // complete with its resync having FAILED mid-flight (I/O + // error → warn → re-list only). Distinct action: the + // nondeterminism is the scheduler's, not the property's. + if self.resync == ResyncMode::Degrade && !self.resume_ok(s, a) { + acts.push(Act::ResumeResyncDegraded); + } + } + _ => {} + } + } + + fn next_state(&self, s: &St, a: Act) -> Option> { + let mut s = s.clone(); + match a { + Act::Churn => s.head += 1, + Act::DeleteKey => { + s.head += 1; + s.delete_rev = Some(s.head); + } + Act::Compact => s.floor += 1, + Act::Apply(n) => s.applied[n as usize] += 1, + Act::Export(n) => { + let cursor = s.applied[n as usize]; + s.exporters[n as usize] = ExporterPc::Exported(Artifact { + node: n, + cursor, + has_key: Self::key_present_at(&s, cursor), + }); + } + Act::UploadPayload(n) => { + let ExporterPc::Exported(a) = s.exporters[n as usize] else { + return None; + }; + if self.pointer_swap { + // Content-addressed: write-once, no register to clobber. + s.uploaded.insert(a); + } else { + // Atomic LWW overwrite of the shared payload key. + s.payload = Some(a); + } + s.exporters[n as usize] = ExporterPc::PayloadUp(a); + } + Act::Publish(n) => { + let ExporterPc::PayloadUp(a) = s.exporters[n as usize] else { + return None; + }; + if self.pointer_swap { + // THE monotonic guard — the SHARED KERNEL + // (`slipstream::protocol::pointer_publish_allowed`), the + // same function `transport::swap_pointer` executes. The + // LwwPointer mutation bypasses it to prove the checker + // catches a broken guard. + let observed = match s.manifest { + None => slipstream::protocol::PointerState::Absent, + Some(m) => slipstream::protocol::PointerState::Present { + rank: Some(m.cursor as u64), + }, + }; + let allowed = self.mutation == Mutation::LwwPointer + || slipstream::protocol::pointer_publish_allowed( + &observed, + a.cursor as u64, + ); + if allowed { + if let Some(m) = s.manifest + && m.cursor > a.cursor + { + s.regressed = true; + } + s.manifest = Some(a); + } else { + s.refused = true; + } + } else { + // Atomic LWW overwrite — an older round's publish lands. + if let Some(m) = s.manifest + && m.cursor > a.cursor + { + s.regressed = true; + } + s.manifest = Some(a); + } + s.exporters[n as usize] = ExporterPc::Done(a.cursor); + } + Act::Crash(n) => { + // Mid-round crash: local scratch artifact lost, whatever was + // already uploaded stays. The node restarts idle. + s.exporters[n as usize] = ExporterPc::Idle; + } + Act::NextRound(n) => { + s.exporters[n as usize] = ExporterPc::Idle; + } + Act::ReadManifest => { + let m = s.manifest?; + s.importer = ImporterPc::GotManifest(m); + } + Act::ReadPayload => { + let ImporterPc::GotManifest(m) = s.importer else { + return None; + }; + if self.pointer_swap { + // Fetch the payload at the pointer's content address. + // Present unless a prune raced a STALE pointer read (the + // current pointer's target is never pruned — + // `pointer_target_always_fetchable`); a miss is a + // detected NotFound → retry, never wrong data. + if s.uploaded.contains(&m) { + s.importer = ImporterPc::Imported(m); + } else { + s.importer = ImporterPc::FetchMissed; + } + } else { + // The cross-check: embedded manifest (inside the payload + // tar) vs the sibling object, byte equality ⟺ identity. + match s.payload { + Some(p) if p == m => s.importer = ImporterPc::Imported(p), + _ => s.importer = ImporterPc::CrossCheckFailed, + } + } + } + Act::Prune => { + // THE prune guard — the SHARED KERNEL + // (`slipstream::protocol::payload_prunable`), the same + // function `ObjectStoreTransport::prune` executes; the + // strictly-below rule and its dangling-pointer impossibility + // argument live there. The first modeling attempt used + // "everything the pointer doesn't reference" — and the + // checker found the dangling trace, which is how the kernel + // got its rule. PruneAgeOnly resurrects the broken rule to + // prove the checker still catches it. Zero grace + // (`aged_out: true`) is the harshest timing. + let keep = s.manifest?; + if self.mutation == Mutation::PruneAgeOnly { + s.uploaded.retain(|a| *a == keep); + } else { + s.uploaded.retain(|a| { + !slipstream::protocol::payload_prunable( + Some(a.cursor as u64), + keep.cursor as u64, + *a == keep, + true, + ) + }); + } + } + Act::Retry => s.importer = ImporterPc::Start, + Act::Resume => { + let ImporterPc::Imported(a) = s.importer else { + return None; + }; + let status = if self.resume_ok(&s, a) { + // Window intact (shared kernel `resume_window_ok` — the + // same guard `nats.rs` executes): tail replay from the + // embedded cursor. Any delete in the gap has its marker + // retained (delete_rev > a.cursor >= floor), so the + // replay delivers it. + FoldStatus::Synced + } else if self.mutation == Mutation::SilentClamp { + // Expiry detection removed: the resume silently skips + // the gap (NATS's native clamp behavior) — deletes whose + // markers were evicted are lost and the resync never + // triggers, regardless of the resync mode. + Self::relist_only_status(&s, a) + } else if self.resync != ResyncMode::None { + // CursorExpired -> full re-list + a SUCCESSFUL stale-key + // resync: live keys diffed against the fold, vanished + // keys get synthetic deletes. (Under FailStop a failed + // resync fails the watch and changes nothing — this + // action stays enabled for the retry. Under Degrade the + // failed-resync outcome is ResumeResyncDegraded.) + FoldStatus::Synced + } else { + Self::relist_only_status(&s, a) + }; + s.importer = ImporterPc::Resumed(a, status); + } + Act::ResumeResyncDegraded => { + let ImporterPc::Imported(a) = s.importer else { + return None; + }; + if self.resync != ResyncMode::Degrade || self.resume_ok(&s, a) { + return None; + } + // The pre-fix code path: resync I/O failed, one warn line, + // fallback proceeds re-list-only. + s.importer = ImporterPc::Resumed(a, Self::relist_only_status(&s, a)); + } + } + Some(s) + } + + fn properties(&self) -> Vec> { + let mut props: Vec> = Vec::new(); + + if self.pointer_swap { + // ---- The SHIPPED protocol's theorems (transport.rs as of 0.6; + // the empirical twins in tests/multi_export.rs drive these same + // interleavings against the real code). ------------------------- + props.push(Property::::always( + "published cursor never regresses", + |_, s| !s.regressed, + )); + props.push(Property::::always( + "monotone pointer: importer never observes a cursor drop", + |_, s| match (s.importer, s.manifest) { + // Once the pointer holds m, any importer state derived + // from an earlier pointer read has cursor <= m.cursor. + (ImporterPc::GotManifest(g), Some(m)) + | (ImporterPc::Imported(g), Some(m)) + | (ImporterPc::Resumed(g, _), Some(m)) => g.cursor <= m.cursor, + _ => true, + }, + )); + props.push(Property::::always( + "cross-check never fires (torn pair structurally impossible)", + |_, s| s.importer != ImporterPc::CrossCheckFailed, + )); + props.push(Property::::always( + "pointer target always fetchable (write-once before publish)", + |_, s| s.manifest.is_none_or(|m| s.uploaded.contains(&m)), + )); + // Vacuity witness: the monotonic guard is exercised, not just + // present — the slow-exporter interleaving reaches it and is + // refused (the model twin of the clobber hazard, now prevented). + props.push(Property::::sometimes( + "the swap refuses an older publish (clobber attempt occurs and is stopped)", + |_, s| s.refused, + )); + // Prune racing a stale pointer read: the importer's fetch can + // MISS (detected NotFound → retry) but never import wrong data — + // and the miss is reachable, so the prune action is genuinely + // exercised, not vacuously safe. + props.push(Property::::sometimes( + "a prune racing a stale pointer read forces a detected retry", + |_, s| s.importer == ImporterPc::FetchMissed, + )); + props.push(Property::::always( + "a fetch miss only happens on a stale pointer read, never the current one", + |_, s| { + s.importer != ImporterPc::FetchMissed + || matches!(s.manifest, Some(m) if s.uploaded.contains(&m)) + }, + )); + } else { + // ---- The LEGACY two-register layout (pre-0.6): the hazards the + // checker must re-derive. Kept as the machine-checked record of + // WHY the protocol changed — these are the interleavings + // tests/multi_export.rs drives against the real code, where the + // shipped protocol now refuses them. If one becomes unreachable + // the model has drifted from the mechanism and this fails loudly. + props.push(Property::::sometimes( + "HAZARD reachable: published artifact regresses (slow-exporter clobber)", + |_, s| s.regressed, + )); + props.push(Property::::sometimes( + "HAZARD reachable: importer observes a torn pair (detected, bootstrap outage)", + |_, s| s.importer == ImporterPc::CrossCheckFailed, + )); + props.push(Property::::sometimes( + "regression is non-fatal: a post-regression bootstrap still converges", + |_, s| { + s.regressed && matches!(s.importer, ImporterPc::Resumed(_, FoldStatus::Synced)) + }, + )); + } + + // ---- Detection soundness, BOTH protocols: every install is exactly + // one exporter's artifact at that exporter's exported state. With the + // BLAKE3 axiom this is "no silent corruption" — a torn pair can only + // park the importer in CrossCheckFailed, never in Imported. --------- + props.push(Property::::always( + "no mixed import: an installed fold is one exporter's exported state", + |_, s| match s.importer { + ImporterPc::Imported(a) | ImporterPc::Resumed(a, _) => { + a.node < N as u8 + && a.cursor >= 1 + && a.cursor <= s.head + // The artifact's key-set is exactly the bucket state + // at its cursor — imports never Frankenstein. + && a.has_key == s.delete_rev.is_none_or(|d| a.cursor < d) + } + _ => true, + }, + )); + + match self.resync { + ResyncMode::FailStop => { + // ---- THE convergence claim: resync wired with fail-stop + // error semantics (`applied.rs` as it ships): bootstrap NEVER + // silently diverges — over every interleaving of churn, + // deletes, compaction, crashes, racing exporters, and resync + // failures (a failed resync fails the watch; only a + // successful one completes a bootstrap). ------------------- + props.push(Property::::always( + "bootstrap never silently diverges (stale is merely stale)", + |_, s| !matches!(s.importer, ImporterPc::Resumed(_, FoldStatus::StaleKey)), + )); + } + ResyncMode::Degrade => { + // ---- The PRE-FIX code semantics (resync failure → warn → + // re-list only): the convergence theorem is FALSE — silent + // divergence is reachable even with the reader wired. This + // configuration is the machine-checked justification for the + // fail-stop change; it must stay reachable so the model + // remains an honest record of why. + props.push(Property::::sometimes( + "HAZARD reachable: armed resync that degrades on error diverges silently", + |_, s| matches!(s.importer, ImporterPc::Resumed(_, FoldStatus::StaleKey)), + )); + } + ResyncMode::None => { + // ---- No reader wired: divergence is REACHABLE — the resync + // reader is a load-bearing requirement, not an optimization. + // (Holds under the pointer-swap protocol too: the transport + // fix does not remove the resync obligation.) --------------- + props.push(Property::::sometimes( + "HAZARD reachable: silent stale-key divergence without resync", + |_, s| matches!(s.importer, ImporterPc::Resumed(_, FoldStatus::StaleKey)), + )); + } + } + + // Vacuity witness for every always-property above: bootstraps really + // complete in this configuration (Imported and Resumed are reachable, + // so the invariants quantify over live states, not an empty set). + props.push(Property::::sometimes( + "a bootstrap completes and resumes synced", + |_, s| matches!(s.importer, ImporterPc::Resumed(_, FoldStatus::Synced)), + )); + + // Multi-round vacuity witness: a node that already published is back + // in the pipeline (only node n publishes artifacts with node == n, so + // pointer-by-n + n mid-flight means a SECOND round is genuinely + // explored — every theorem above quantifies over repeated rounds). + props.push(Property::::sometimes( + "a publisher runs a second round against its own previous publish", + |_, s| { + s.manifest.is_some_and(|m| { + matches!( + s.exporters[m.node as usize], + ExporterPc::Exported(_) | ExporterPc::PayloadUp(_) + ) + }) + }, + )); + + if self.resync == ResyncMode::FailStop && self.mutation == Mutation::None { + // ---- Terminal liveness, Stateright-native and cycle-proof: + // recompute the enabled-action set inside the invariant — a + // state with NO enabled actions is a maximal execution's end, + // and every such state must hold a COMPLETED, SYNCED bootstrap. + // No run can end with the importer stuck, failed, or diverged; + // retry loops are cycles (never terminal), so they cannot + // satisfy this vacuously. + props.push(Property::::always( + "every maximal run ends with a completed, synced bootstrap", + |m, s| { + let mut acts = Vec::new(); + m.actions(s, &mut acts); + !acts.is_empty() + || matches!(s.importer, ImporterPc::Resumed(_, FoldStatus::Synced)) + }, + )); + } + + props + } +} + +fn run( + model: SnapshotProtocol, + label: &str, +) -> impl Checker> { + let checker = model.checker().spawn_bfs().join(); + println!( + "{label}: {} states, {} unique", + checker.state_count(), + checker.unique_state_count(), + ); + checker +} + +fn check(model: SnapshotProtocol, label: &str) { + run(model, label).assert_properties(); +} + +/// THE SHIPPED CONFIGURATION (pointer-swap transport + fail-stop resync — +/// `transport.rs` and `applied.rs` as of 0.6, executing the SHARED protocol +/// kernels): regression, torn pairs, and dangling pointers are structurally +/// impossible, detection and convergence hold, over every interleaving +/// within bounds. +#[test] +fn shipped_protocol_pointer_swap_failstop_resync() { + check( + SnapshotProtocol::<2>::shipped(ResyncMode::FailStop), + "shipped: pointer-swap + failstop", + ); +} + +/// The shipped transport still requires the resync reader — the pointer-swap +/// fix does not absolve the convergence obligation. +#[test] +fn shipped_protocol_without_resync_still_diverges() { + check( + SnapshotProtocol::<2>::shipped(ResyncMode::None), + "shipped: pointer-swap + no resync", + ); +} + +/// LEGACY two-register transport (pre-0.6) with fail-stop resync: the +/// convergence and detection theorems held, but the clobber-regression and +/// torn-pair hazards are reachable — the machine-checked record of why the +/// transport moved to content-addressed payloads + a monotonic pointer. +#[test] +fn legacy_two_register_transport_has_reachable_hazards() { + check( + SnapshotProtocol::<2>::legacy(ResyncMode::FailStop), + "legacy: two-register + failstop", + ); +} + +/// LEGACY resync semantics (failure degrades to re-list with a warning): +/// silent divergence is reachable even with the reader wired — the +/// machine-checked reason `resync_stale_keys` now fails the watch instead. +#[test] +fn legacy_degrading_resync_diverges() { + check( + SnapshotProtocol::<2>::legacy(ResyncMode::Degrade), + "legacy: degrade-on-error resync", + ); +} + +/// No resync reader wired: silent stale-key divergence is reachable. This +/// pins the resync reader as a correctness requirement for the "stale, never +/// corrupt" claim, independent of the transport protocol. +#[test] +fn no_resync_reader_diverges() { + check( + SnapshotProtocol::<2>::legacy(ResyncMode::None), + "legacy: no resync reader", + ); +} + +// --- Mutation tests: every shared-kernel guard is load-bearing ---------------- +// Each substitutes one deliberately broken guard and asserts the checker +// PRODUCES A COUNTEREXAMPLE for the theorem that guard carries. This proves +// the properties have teeth: a future regression in any kernel cannot pass +// the checker silently. (The unmutated configurations execute the kernels +// themselves, so a kernel regression also fails the main theorems directly.) + +#[test] +fn mutation_lww_pointer_is_caught() { + let mut model = SnapshotProtocol::<2>::shipped(ResyncMode::FailStop); + model.mutation = Mutation::LwwPointer; + let checker = run(model, "mutation: lww pointer"); + assert!( + checker + .discovery("published cursor never regresses") + .is_some(), + "the checker must produce a regression counterexample when the \ + monotonic publish guard is removed" + ); +} + +#[test] +fn mutation_age_only_prune_is_caught() { + let mut model = SnapshotProtocol::<2>::shipped(ResyncMode::FailStop); + model.mutation = Mutation::PruneAgeOnly; + let checker = run(model, "mutation: age-only prune"); + assert!( + checker + .discovery("pointer target always fetchable (write-once before publish)") + .is_some(), + "the checker must produce a dangling-pointer counterexample when \ + prune ignores the strictly-below rule (the original design bug)" + ); +} + +#[test] +fn mutation_silent_clamp_is_caught() { + let mut model = SnapshotProtocol::<2>::shipped(ResyncMode::FailStop); + model.mutation = Mutation::SilentClamp; + let checker = run(model, "mutation: silent clamp"); + assert!( + checker + .discovery("bootstrap never silently diverges (stale is merely stale)") + .is_some(), + "the checker must produce a silent-divergence counterexample when \ + expiry detection is removed (the live NATS clamp bug class)" + ); +} + +// --- Deep configurations for scheduled release runs --------------------------- +// `cargo test --release --test model -- --ignored --nocapture` + +/// More revisions: each level multiplies the state space severalfold without +/// changing the mechanism set — slack against any witness being +/// bound-limited. +#[test] +#[ignore = "deep bounds: run in release"] +fn deep_more_revisions() { + let mut model = SnapshotProtocol::<2>::shipped(ResyncMode::FailStop); + model.max_rev = 5; + check(model, "deep: 2 exporters, rev <= 5"); +} + +/// THREE exporters: the classic check that nothing in the protocol is +/// accidentally pairwise — three-way publish races, two stalled rounds +/// landing after a third's takeover, prune racing two concurrent uploads. +#[test] +#[ignore = "deep bounds: run in release"] +fn deep_three_exporters() { + let model = SnapshotProtocol::<3>::shipped(ResyncMode::FailStop); + check(model, "deep: 3 exporters, rev <= 3"); +} + +/// Three exporters against the legacy layout: the hazards must STILL be +/// reachable at fleet size 3 (model honesty at larger scale). +#[test] +#[ignore = "deep bounds: run in release"] +fn deep_three_exporters_legacy_hazards() { + let model = SnapshotProtocol::<3>::legacy(ResyncMode::FailStop); + check(model, "deep: 3 exporters, legacy two-register"); +} diff --git a/tests/multi_export.rs b/tests/multi_export.rs new file mode 100644 index 0000000..d8816d3 --- /dev/null +++ b/tests/multi_export.rs @@ -0,0 +1,830 @@ +//! Empirical multi-exporter consistency proofs (feature `transport` + an LSM +//! backend), the live twins of the `pointer_swap` model in `tests/model.rs`: +//! real fjall/RocksDB folds driven by live `watch_applied` loops over a +//! throwaway NATS server, genuinely diverged across nodes, driving the exact +//! interleavings the legacy two-register layout failed under — and asserting +//! the shipped content-addressed + monotonic-pointer protocol PREVENTS them. +//! +//! 1. **Slow-exporter clobber → REFUSED.** Node A wins a round and stalls in +//! upload past its lease ttl; node B takes over and publishes a NEWER +//! artifact; A's stale upload then lands LAST. A's payload goes to its own +//! content address (clobbering nothing) and its pointer swap is refused: +//! the published "latest" never regresses, the lease record and the +//! pointer agree, and a bootstrapper gets B's newest state. (Also proven: +//! concurrent exporters' artifacts genuinely differ — each replica +//! exports at its own applied cursor — which is exactly why the pointer +//! must be monotonic rather than last-write-wins.) +//! +//! 2. **Crash between payload and pointer → NOTHING tears.** The crash +//! window that tore the legacy layout (payload landed, manifest didn't) +//! now leaves the old pointer fully consistent: peek and download serve +//! the old artifact throughout; the next round publishes the new one. +//! +//! 3. **Post-compaction, multi-data-file fidelity.** A fold with real LSM +//! structure — flush cycles, overwrites, tombstones, a `settle()` +//! compaction, a fresh tail — exports a multi-SST/multi-table artifact; +//! the source then churns and compacts AGAIN (rewriting/unlinking the +//! files the RocksDB checkpoint hardlinked) before the slow upload ships +//! it. Every hash verifies; the import is byte-exactly the at-export +//! state. +#![cfg(all(feature = "transport", any(feature = "fjall", feature = "rocksdb")))] + +mod common; + +use std::path::Path; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::time::Duration; + +use async_trait::async_trait; +use common::{ManifestPutCrash, TestNats}; +use slipstream::snapshot::{SnapshotError, SnapshotStore}; +use slipstream::{ + ArtifactTransport, BatchConfig, Connection, ExportLease, ExportManifest, ExportRequest, + KvStore, KvUpdate, NatsConnection, NatsConnectionConfig, ObjectStoreTransport, PublishOutcome, + StoreConfig, WatchCursor, WatchScope, run_export_round, watch_applied, +}; +use tempfile::TempDir; +use tokio::sync::{mpsc, oneshot, watch}; +use tokio::time::timeout; + +// --- Shared live-fleet harness (same shape as tests/bootstrap.rs) ------------- + +async fn open_buckets(nats: &TestNats) -> (NatsConnection, Arc, Arc) { + let conn = NatsConnection::new(NatsConnectionConfig { + url: nats.url.clone(), + creds: None, + creds_file: None, + }); + conn.connect().await.expect("connect"); + let routes = conn + .store_with_config(StoreConfig { + name: "routes".into(), + max_bytes: Some(8 * 1024 * 1024), + ..Default::default() + }) + .await + .expect("open routes bucket"); + let leases = conn + .store_with_config(StoreConfig { + name: "leases".into(), + max_bytes: Some(1024 * 1024), + ..Default::default() + }) + .await + .expect("open lease bucket"); + (conn, routes, leases) +} + +struct Node { + exports: mpsc::Sender, + applied: Arc, + shutdown: watch::Sender, + task: tokio::task::JoinHandle>, +} + +fn spawn_node( + bucket: &Arc, + fold: S, + resume: Option, +) -> Node { + let watcher = bucket.watcher().expect("bucket watcher"); + let (ex_tx, ex_rx) = mpsc::channel(1); + let (sd_tx, sd_rx) = watch::channel(false); + let applied = Arc::new(AtomicU64::new(0)); + let applied_w = Arc::clone(&applied); + + let task = tokio::spawn(watch_applied( + watcher, + WatchScope::All, + resume, + None, + Some(fold), + Some(ex_rx), + BatchConfig::default(), + |u: &KvUpdate| match u { + KvUpdate::Put(e) => Some(e.key.clone()), + _ => None, + }, + |_batch: Vec| {}, + move |cur: WatchCursor| { + applied_w.store(cur.as_u64().unwrap_or(0), Ordering::SeqCst); + }, + sd_rx, + )); + + Node { + exports: ex_tx, + applied, + shutdown: sd_tx, + task, + } +} + +async fn wait_applied(node: &Node, at_least: u64) { + timeout(Duration::from_secs(10), async { + loop { + if node.applied.load(Ordering::SeqCst) >= at_least { + return; + } + tokio::time::sleep(Duration::from_millis(20)).await; + } + }) + .await + .unwrap_or_else(|_| { + panic!( + "node never applied rev {at_least} (at {})", + node.applied.load(Ordering::SeqCst) + ) + }); +} + +/// Export through a live node's request channel into `dest`. +async fn export_via(node: &Node, dest: &Path) -> ExportManifest { + let (reply_tx, reply_rx) = oneshot::channel(); + node.exports + .send(ExportRequest { + dest_dir: dest.to_path_buf(), + reply: reply_tx, + }) + .await + .expect("send export request"); + reply_rx.await.expect("reply").expect("export succeeds") +} + +/// The lease's `completed_cursor_hex` rendering of a u64-revision cursor: +/// 8 bytes big-endian, lowercase hex (see `integration.rs`'s +/// `export_lease_complete_publishes_outcome`). +fn cursor_hex(c: &WatchCursor) -> String { + format!("{:016x}", c.as_u64().expect("u64 cursor")) +} + +type ImportFn = fn(&Path, &Path) -> Result<(WatchCursor, S), SnapshotError>; + +// --- Scenario 1: slow-exporter clobber → refused by the monotonic pointer ---- + +/// An [`ArtifactTransport`] whose `upload` parks on a gate — the stand-in for +/// a slow tar/multipart on a stalled node. Signals `reached` when the round +/// arrives at upload (its export is already done by then), then waits for a +/// permit before delegating to the real transport. +struct GatedTransport { + inner: ObjectStoreTransport, + gate: tokio::sync::Semaphore, + reached: std::sync::Mutex>>, +} + +#[async_trait] +impl ArtifactTransport for GatedTransport { + async fn upload( + &self, + key: &str, + artifact_dir: &Path, + ) -> Result { + if let Some(tx) = self.reached.lock().unwrap().take() { + let _ = tx.send(()); + } + let _permit = self.gate.acquire().await.expect("gate never closed"); + self.inner.upload(key, artifact_dir).await + } + async fn manifest(&self, key: &str) -> Result { + self.inner.manifest(key).await + } + async fn download(&self, key: &str, dest_dir: &Path) -> Result { + self.inner.download(key, dest_dir).await + } +} + +/// Two replicas of one fold, genuinely diverged (A's artifact is frozen at the +/// pre-churn cursor while B keeps applying), racing `run_export_round` to the +/// same remote key. A stalls in upload past its ttl; B takes the round over +/// and publishes newer; A's stale upload lands last — and the monotonic +/// pointer swap REFUSES it. The published "latest" never regresses. +async fn slow_exporter_cannot_clobber_newer_artifact( + open: impl Fn(&Path) -> (WatchCursor, S), + import: ImportFn, +) where + S: SnapshotStore + Send + 'static, +{ + let nats = TestNats::start().await; + let (_conn, bucket, leases) = open_buckets(&nats).await; + let writer = bucket.writer().expect("writer"); + let dir = TempDir::new().unwrap(); + + let bucket_dir = TempDir::new().unwrap(); + let fs = + Arc::new(object_store::local::LocalFileSystem::new_with_prefix(bucket_dir.path()).unwrap()); + let transport = ObjectStoreTransport::new(fs.clone(), "artifacts"); + + // Two live replicas of the same fold. + let (_ra, fold_a) = open(&dir.path().join("node-a")); + let (_rb, fold_b) = open(&dir.path().join("node-b")); + let node_a = spawn_node(&bucket, fold_a, None); + let node_b = spawn_node(&bucket, fold_b, None); + + // Deterministic attach for BOTH consumers (KV watches deliver new updates + // only), then the pre-churn history. + let attach_rev = timeout(Duration::from_secs(10), async { + loop { + let v = writer.put("route.seed", b"seed").await.expect("seed"); + tokio::time::sleep(Duration::from_millis(50)).await; + let rev = v.as_u64().expect("nats rev"); + if node_a.applied.load(Ordering::SeqCst) > 0 + && node_b.applied.load(Ordering::SeqCst) > 0 + { + return rev; + } + } + }) + .await + .expect("watches never attached"); + + let mut pre_rev = attach_rev; + for i in 0..10u64 { + pre_rev = writer + .put(&format!("route.pre.{i}"), format!("pre-{i}").as_bytes()) + .await + .expect("put") + .as_u64() + .expect("rev"); + } + wait_applied(&node_a, pre_rev).await; + wait_applied(&node_b, pre_rev).await; + + // Node A wins a round with a short ttl and stalls in upload. The round + // runs the REAL composed path — lease, live export, transport — only the + // upload is parked on the gate. + let (reached_tx, reached_rx) = oneshot::channel(); + let gated = Arc::new(GatedTransport { + inner: ObjectStoreTransport::new(fs.clone(), "artifacts"), + gate: tokio::sync::Semaphore::new(0), + reached: std::sync::Mutex::new(Some(reached_tx)), + }); + let scratch_a = dir.path().join("scratch-a"); + std::fs::create_dir(&scratch_a).unwrap(); + let task_a = { + let leases = Arc::clone(&leases); + let exports = node_a.exports.clone(); + let gated = Arc::clone(&gated); + let scratch = scratch_a.clone(); + tokio::spawn(async move { + let lease = ExportLease::new(leases.as_ref(), "round", "node-a").unwrap(); + run_export_round( + &lease, + Duration::from_secs(2), + &exports, + gated.as_ref(), + "edge/latest", + &scratch, + ) + .await + }) + }; + reached_rx.await.expect("node A reached upload"); + + // Post-export churn: node B's fold moves past the cursor frozen in A's + // artifact — the replicas have genuinely diverged. + let mut final_rev = pre_rev; + for i in 0..8u64 { + final_rev = writer + .put(&format!("route.post.{i}"), format!("post-{i}").as_bytes()) + .await + .expect("put") + .as_u64() + .expect("rev"); + } + wait_applied(&node_b, final_rev).await; + + // Node B takes the round over once A's ttl lapses (polled, not slept). + let lease_b = ExportLease::new(leases.as_ref(), "round", "node-b").unwrap(); + let scratch_b = dir.path().join("scratch-b"); + std::fs::create_dir(&scratch_b).unwrap(); + let manifest_b = timeout(Duration::from_secs(15), async { + loop { + if let Some(m) = run_export_round( + &lease_b, + Duration::from_secs(60), + &node_b.exports, + &transport, + "edge/latest", + &scratch_b, + ) + .await + .expect("node B round") + { + return m; + } + tokio::time::sleep(Duration::from_millis(250)).await; + } + }) + .await + .expect("node B never won the round after A's ttl"); + assert_eq!( + manifest_b.cursor.as_u64(), + Some(final_rev), + "B exported the full post-churn state" + ); + assert_eq!( + transport.manifest("edge/latest").await.unwrap().cursor, + manifest_b.cursor, + "remote holds B's newer artifact before A's stale upload lands" + ); + + // Release A: its stale upload lands LAST — the payload uploads to its + // own content address (clobbering nothing) and the pointer swap REFUSES + // the regression. The round still completes cleanly. + gated.gate.add_permits(1); + let manifest_a = task_a + .await + .unwrap() + .expect("node A round returned an error") + .expect("node A round completes despite overrunning its lease"); + + // Divergence, empirically: concurrent exporters do NOT produce identical + // artifacts — each replica exports at its own applied cursor. This is + // exactly why the pointer must be monotonic rather than last-write-wins. + let (rev_a, rev_b) = ( + manifest_a.cursor.as_u64().expect("rev a"), + manifest_b.cursor.as_u64().expect("rev b"), + ); + assert!( + rev_a < rev_b, + "concurrent exporters produced different artifacts (a={rev_a}, b={rev_b})" + ); + + // PREVENTION (model: `published cursor never regresses`): the remote + // "latest" still holds B's newer artifact — A's stale publish was + // superseded, not last-write-wins. + let remote = transport.manifest("edge/latest").await.unwrap(); + assert_eq!( + remote.cursor, manifest_b.cursor, + "the pointer never regressed: B's newer artifact survived A's stale upload landing last" + ); + + // The fleet-visible lease record and the remote object AGREE (both B) — + // under the legacy two-register layout they disagreed here. + let record = lease_b.current().await.unwrap().expect("lease record"); + assert_eq!(record.holder_id, "node-b", "B owns the round record"); + assert_eq!( + record.completed_cursor_hex.as_deref(), + Some(cursor_hex(&manifest_b.cursor).as_str()), + "the completion record and the published pointer agree on B's cursor" + ); + + // A bootstrapping node C gets B's artifact — the NEWEST published state, + // post-churn keys included — and converges on the live tail. + let dl = dir.path().join("downloaded"); + let got = transport.download("edge/latest", &dl).await.unwrap(); + assert_eq!(got.cursor, manifest_b.cursor); + let dest_c = dir.path().join("node-c"); + let (cursor_c, fold_c) = import(&dl, &dest_c).expect("import published artifact"); + assert_eq!(cursor_c.as_u64(), Some(rev_b)); + assert_eq!( + fold_c.range("route.post.").expect("range").len(), + 8, + "the bootstrapped fold carries the full post-churn state" + ); + + // C is already AT the bucket head (it imported the newest artifact), so + // prove live convergence with a fresh tail written after it resumes. + let node_c = spawn_node(&bucket, fold_c, Some(cursor_c)); + let mut tail_rev = final_rev; + for i in 0..3u64 { + tail_rev = writer + .put(&format!("route.tail.{i}"), format!("tail-{i}").as_bytes()) + .await + .expect("put tail") + .as_u64() + .expect("rev"); + } + wait_applied(&node_c, tail_rev).await; + node_c.shutdown.send(true).unwrap(); + node_c.task.await.unwrap().unwrap(); + + let (final_cursor, fold_c) = open(&dest_c); + assert_eq!(final_cursor.as_u64(), Some(tail_rev)); + let mut fold_state: Vec<(String, Vec)> = fold_c + .range("route.") + .expect("range") + .into_iter() + .map(|e| (e.key, e.value)) + .collect(); + fold_state.sort(); + let mut bucket_state: Vec<(String, Vec)> = bucket + .reader() + .scan("route.") + .await + .expect("scan") + .into_iter() + .map(|e| (e.key, e.value)) + .collect(); + bucket_state.sort(); + assert_eq!( + fold_state, bucket_state, + "the bootstrapped node converged to the bucket" + ); + + node_a.shutdown.send(true).unwrap(); + node_a.task.await.unwrap().unwrap(); + node_b.shutdown.send(true).unwrap(); + node_b.task.await.unwrap().unwrap(); +} + +// --- Scenario 2: crash between payload and pointer swap ----------------------- +// (crash injection: common::ManifestPutCrash) + +/// A healthy old artifact sits at "latest". A newer export's upload crashes +/// after its payload lands but before the pointer swap — and NOTHING tears: +/// the new payload sits at its own content address, the pointer still +/// references the old artifact, and every bootstrap keeps working throughout. +/// The next successful round publishes the new state. +async fn crash_between_payload_and_pointer_keeps_bootstrap_available( + open: impl Fn(&Path) -> (WatchCursor, S), + import: ImportFn, +) where + S: SnapshotStore + Send + 'static, +{ + let nats = TestNats::start().await; + let (_conn, bucket, _leases) = open_buckets(&nats).await; + let writer = bucket.writer().expect("writer"); + let dir = TempDir::new().unwrap(); + + let bucket_dir = TempDir::new().unwrap(); + let fs: Arc = + Arc::new(object_store::local::LocalFileSystem::new_with_prefix(bucket_dir.path()).unwrap()); + // Local FS lacks CAS, so the recovery publish (an Update over the old + // pointer) takes the explicitly opted-in fallback; the crash-window + // semantics under test are store-agnostic and re-proven on real CAS by + // `transport_s3.rs`'s MinIO twin. + let transport = + ObjectStoreTransport::new(fs.clone(), "artifacts").with_non_atomic_pointer_fallback(); + let crash = Arc::new(ManifestPutCrash { + inner: fs.clone(), + armed: AtomicBool::new(false), + }); + let crashing = + ObjectStoreTransport::new(crash.clone(), "artifacts").with_non_atomic_pointer_fallback(); + + // One live node; two real exports at genuinely different cursors. + let (_r, fold) = open(&dir.path().join("node-a")); + let node = spawn_node(&bucket, fold, None); + let attach_rev = timeout(Duration::from_secs(10), async { + loop { + let v = writer.put("route.seed", b"seed").await.expect("seed"); + tokio::time::sleep(Duration::from_millis(50)).await; + if node.applied.load(Ordering::SeqCst) > 0 { + return v.as_u64().expect("rev"); + } + } + }) + .await + .expect("watch never attached"); + + let mut pre_rev = attach_rev; + for i in 0..6u64 { + pre_rev = writer + .put(&format!("route.pre.{i}"), format!("pre-{i}").as_bytes()) + .await + .expect("put") + .as_u64() + .expect("rev"); + } + wait_applied(&node, pre_rev).await; + let artifact_old = dir.path().join("artifact-old"); + let manifest_old = export_via(&node, &artifact_old).await; + + let mut final_rev = pre_rev; + for i in 0..5u64 { + final_rev = writer + .put(&format!("route.post.{i}"), format!("post-{i}").as_bytes()) + .await + .expect("put") + .as_u64() + .expect("rev"); + } + wait_applied(&node, final_rev).await; + let artifact_new = dir.path().join("artifact-new"); + let manifest_new = export_via(&node, &artifact_new).await; + assert!(manifest_old.cursor.as_u64() < manifest_new.cursor.as_u64()); + + // Healthy baseline: the old round completed fully. + assert_eq!( + transport.upload("latest", &artifact_old).await.unwrap(), + PublishOutcome::Published + ); + assert_eq!( + transport.manifest("latest").await.unwrap().cursor, + manifest_old.cursor + ); + + // The new round's upload "crashes" after its payload multipart completes, + // before the pointer swap. + crash.armed.store(true, Ordering::SeqCst); + let err = crashing + .upload("latest", &artifact_new) + .await + .expect_err("injected crash fails the upload"); + assert!( + err.to_string().contains("injected crash"), + "failed for the injected reason: {err}" + ); + + // PREVENTION (model: `cross-check never fires`): no torn state exists. + // The pointer still references the OLD artifact in full — peek and + // download both serve it; bootstrap stayed available straight through + // the crash. (Under the legacy two-register layout this exact crash + // broke every download until the next round.) + assert_eq!( + transport.manifest("latest").await.unwrap().cursor, + manifest_old.cursor, + "the pointer still advertises the old round, consistently" + ); + let dl_during = dir.path().join("dl-during-crash-window"); + let got = transport + .download("latest", &dl_during) + .await + .expect("bootstrap keeps working through the crash window"); + assert_eq!(got.cursor, manifest_old.cursor); + let (cursor_during, fold_during) = + import(&dl_during, &dir.path().join("node-during")).expect("import old artifact"); + assert_eq!(cursor_during, manifest_old.cursor); + assert!( + fold_during.range("route.post.").expect("range").is_empty(), + "the crash window serves the old (consistent) state, not a torn one" + ); + + // The next successful round publishes the new state end-to-end. + assert_eq!( + transport.upload("latest", &artifact_new).await.unwrap(), + PublishOutcome::Published + ); + let dl = dir.path().join("dl-published"); + let got = transport.download("latest", &dl).await.unwrap(); + assert_eq!(got.cursor, manifest_new.cursor); + let (cursor, fold_b) = import(&dl, &dir.path().join("node-b")).expect("import published"); + assert_eq!(cursor, manifest_new.cursor); + assert_eq!( + fold_b.range("route.post.").expect("range").len(), + 5, + "the published artifact carries the post-churn keys" + ); + + node.shutdown.send(true).unwrap(); + node.task.await.unwrap().unwrap(); +} + +// --- Scenario 3: post-compaction, multi-file artifacts ------------------------ + +fn put(key: &str, value: &[u8], rev: u64) -> KvUpdate { + KvUpdate::Put(slipstream::KvEntry { + key: key.to_string(), + value: value.to_vec(), + version: slipstream::VersionToken::from_u64(rev), + }) +} + +fn del(key: &str, rev: u64) -> KvUpdate { + KvUpdate::Delete { + key: key.to_string(), + version: slipstream::VersionToken::from_u64(rev), + } +} + +/// A ~2 KiB value unique to (generation, batch, key) so overwrites really +/// change bytes and post-export churn provably diverges the source. +fn big_value(generation: u32, batch: u64, i: u64) -> Vec { + format!("g{generation}-b{batch}-i{i}-") + .into_bytes() + .into_iter() + .cycle() + .take(2048) + .collect() +} + +fn full_state(fold: &S) -> Vec<(String, Vec)> { + let mut state: Vec<(String, Vec)> = fold + .range("route.") + .expect("range") + .into_iter() + .map(|e| (e.key, e.value)) + .collect(); + state.sort(); + state +} + +/// The realistic-scale fidelity proof: a fold driven through real LSM +/// lifecycle — multiple flush cycles, overwrites, tombstones, an explicit +/// `settle()` compaction, plus a fresh post-compaction tail — exports a +/// MULTI-DATA-FILE artifact. The source then keeps churning and compacts +/// again, which rewrites/unlinks the very files the export staged (RocksDB +/// checkpoints HARDLINK live SSTs into the artifact; this is the stalled +/// node's artifact sitting in scratch while its fold moves on). Only then +/// does the "slow" upload ship it. Proves: every payload hash verifies after +/// the source churn, the import opens, and the imported state is exactly the +/// at-export state — multiple SSTs, compaction, and tombstones included. +async fn compacted_multi_file_artifact_survives_source_churn( + open: impl Fn(&Path) -> (WatchCursor, S), + import: ImportFn, + settle: fn(&S) -> Result<(), SnapshotError>, + is_data_file: fn(&str) -> bool, +) where + S: SnapshotStore + Send + 'static, +{ + let dir = TempDir::new().unwrap(); + let bucket_dir = TempDir::new().unwrap(); + let fs = + Arc::new(object_store::local::LocalFileSystem::new_with_prefix(bucket_dir.path()).unwrap()); + let transport = ObjectStoreTransport::new(fs, "artifacts"); + + let (_r, mut fold) = open(&dir.path().join("source")); + let mut rev = 0u64; + + // Real LSM history: 12 batches × 50 keys over a 100-key space (heavy + // overwrites), a slug of deletes every 4th batch (tombstones), and a + // mid-history settle so part of the history is flushed + compacted. + for b in 0..12u64 { + let mut batch = Vec::new(); + for i in 0..50u64 { + rev += 1; + let key = format!("route.{:03}", (b * 37 + i) % 100); + batch.push(put(&key, &big_value(1, b, i), rev)); + } + if b % 4 == 3 { + for d in 0..10u64 { + rev += 1; + batch.push(del(&format!("route.{:03}", (b + d * 7) % 100), rev)); + } + } + fold.apply(&batch, &WatchCursor::from_u64(rev)) + .expect("apply history batch"); + if b == 5 { + settle(&fold).expect("mid-history settle"); + } + } + // Compact the whole history, then lay a fresh un-compacted tail on top so + // the export's own flush produces a new data file ALONGSIDE the compacted + // one(s) — a genuinely multi-file payload. + settle(&fold).expect("settle: flush + compact the history"); + let mut tail = Vec::new(); + for i in 0..30u64 { + rev += 1; + tail.push(put( + &format!("route.{:03}", i * 3 % 100), + &big_value(2, 99, i), + rev, + )); + } + fold.apply(&tail, &WatchCursor::from_u64(rev)) + .expect("apply post-compaction tail"); + + let artifact = dir.path().join("artifact"); + let manifest = fold.export_to(&artifact).expect("export"); + assert_eq!(manifest.cursor.as_u64(), Some(rev)); + let data_files: Vec<&str> = manifest + .files + .iter() + .filter(|f| is_data_file(&f.path)) + .map(|f| f.path.as_str()) + .collect(); + assert!( + data_files.len() >= 2, + "artifact must carry MULTIPLE data files (compacted history + fresh \ + tail); got {data_files:?} out of {:?}", + manifest.files.iter().map(|f| &f.path).collect::>() + ); + let expected = full_state(&fold); + + // Post-export source churn: overwrite, delete, and compact AGAIN — on + // RocksDB this rewrites/unlinks the very SSTs the artifact hardlinked. + // The artifact must be immune (its links pin the immutable inodes). + for b in 0..6u64 { + let mut batch = Vec::new(); + for i in 0..50u64 { + rev += 1; + batch.push(put( + &format!("route.{:03}", (b * 13 + i) % 100), + &big_value(3, b, i), + rev, + )); + } + rev += 1; + batch.push(del(&format!("route.{:03}", b * 11 % 100), rev)); + fold.apply(&batch, &WatchCursor::from_u64(rev)) + .expect("apply churn batch"); + } + settle(&fold).expect("settle: compact the post-export churn"); + assert_ne!( + full_state(&fold), + expected, + "the churn really diverged the source from the artifact" + ); + + // The "slow" upload ships the artifact only now — after its source files + // were compacted away — then a bootstrap round-trips it. + assert_eq!( + transport + .upload("compacted/latest", &artifact) + .await + .unwrap(), + PublishOutcome::Published + ); + let dl = dir.path().join("downloaded"); + let got = transport.download("compacted/latest", &dl).await.unwrap(); + assert_eq!(got.cursor, manifest.cursor); + let (cursor, imported) = import(&dl, &dir.path().join("imported")) + .expect("every payload hash verifies and the staged copy opens"); + assert_eq!(cursor, manifest.cursor); + assert_eq!( + full_state(&imported), + expected, + "imported state is exactly the at-export state — multiple data files, \ + compaction, and tombstones survived the round trip and source churn" + ); +} + +// --- Backend instantiations ---------------------------------------------------- + +#[cfg(feature = "fjall")] +mod fjall_multi { + use super::*; + use slipstream::{FjallConfig, FjallSnapshot}; + + fn cfg() -> FjallConfig { + FjallConfig { + sync: false, + cache_size_bytes: 64 << 20, + } + } + + #[tokio::test(flavor = "multi_thread")] + async fn fjall_slow_exporter_cannot_clobber_newer_artifact() { + slow_exporter_cannot_clobber_newer_artifact( + |path| FjallSnapshot::open(path, cfg()).expect("open fjall"), + |artifact, dest| FjallSnapshot::import(artifact, dest, cfg()), + ) + .await; + } + + #[tokio::test(flavor = "multi_thread")] + async fn fjall_crash_between_payload_and_pointer_keeps_bootstrap_available() { + crash_between_payload_and_pointer_keeps_bootstrap_available( + |path| FjallSnapshot::open(path, cfg()).expect("open fjall"), + |artifact, dest| FjallSnapshot::import(artifact, dest, cfg()), + ) + .await; + } + + #[tokio::test(flavor = "multi_thread")] + async fn fjall_compacted_multi_file_artifact_survives_source_churn() { + compacted_multi_file_artifact_survives_source_churn( + |path| FjallSnapshot::open(path, cfg()).expect("open fjall"), + |artifact, dest| FjallSnapshot::import(artifact, dest, cfg()), + |fold| fold.settle(), + // fjall/lsm-tree on-disk tables (sorted runs) live at + // `keyspaces//tables/`. + |path| path.contains("/tables/"), + ) + .await; + } +} + +#[cfg(feature = "rocksdb")] +mod rocksdb_multi { + use super::*; + use slipstream::{RocksDbConfig, RocksDbSnapshot}; + + fn cfg() -> RocksDbConfig { + RocksDbConfig { + sync: false, + cache_size_bytes: 64 << 20, + } + } + + #[tokio::test(flavor = "multi_thread")] + async fn rocksdb_slow_exporter_cannot_clobber_newer_artifact() { + slow_exporter_cannot_clobber_newer_artifact( + |path| RocksDbSnapshot::open(path, cfg()).expect("open rocksdb"), + |artifact, dest| RocksDbSnapshot::import(artifact, dest, cfg()), + ) + .await; + } + + #[tokio::test(flavor = "multi_thread")] + async fn rocksdb_crash_between_payload_and_pointer_keeps_bootstrap_available() { + crash_between_payload_and_pointer_keeps_bootstrap_available( + |path| RocksDbSnapshot::open(path, cfg()).expect("open rocksdb"), + |artifact, dest| RocksDbSnapshot::import(artifact, dest, cfg()), + ) + .await; + } + + #[tokio::test(flavor = "multi_thread")] + async fn rocksdb_compacted_multi_file_artifact_survives_source_churn() { + compacted_multi_file_artifact_survives_source_churn( + |path| RocksDbSnapshot::open(path, cfg()).expect("open rocksdb"), + |artifact, dest| RocksDbSnapshot::import(artifact, dest, cfg()), + |fold| fold.settle(), + |path| path.ends_with(".sst"), + ) + .await; + } +} diff --git a/tests/resync.rs b/tests/resync.rs new file mode 100644 index 0000000..13ee286 --- /dev/null +++ b/tests/resync.rs @@ -0,0 +1,476 @@ +//! End-to-end conformance tests for the cursor-expired resync — the leg of +//! the convergence proof (`tests/model.rs`) that rests on live-server +//! behavior nowhere else verified against a real NATS: +//! +//! **The expiry axiom.** The model's `Resume` transition assumes a resume +//! below the retention floor is DETECTED (`KvError::CursorExpired`). In the +//! code that detection is `nats.rs`'s error-string matching on consumer +//! creation — which presumes NATS *errors* on a too-old `ByStartSequence` +//! rather than silently clamping to the first retained sequence. If NATS +//! clamped, the gap (deletes included) would be skipped with no error, no +//! fallback, no resync: silent divergence through a path the model marks +//! Synced. These tests force real per-subject eviction on a throwaway +//! `nats-server` and prove the whole chain end-to-end: eviction → expiry +//! detected → full-watch fallback → stale-key resync → reconciled fold. +//! +//! The proof is self-validating by construction: the deleted key's marker is +//! purged from the stream, so NOTHING in the replay or the re-list can ever +//! deliver its delete — the key disappears from the bootstrapped fold if and +//! only if the expiry was detected and the resync ran. A silent clamp, a +//! missed error string, or a skipped resync each leave the key in the fold +//! and fail the test. +//! +//! The evil twin (`reader: None`) pins the divergence the model proves +//! reachable without the resync: same gap, same fallback, and the deleted +//! key persists — silently — in an otherwise fully converged fold. +//! +//! Generic over the on-disk backends; instantiated for fjall and RocksDB. +#![cfg(any(feature = "fjall", feature = "rocksdb"))] + +mod common; + +use std::path::Path; +use std::sync::Arc; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::time::Duration; + +use common::TestNats; +use slipstream::snapshot::{SnapshotError, SnapshotStore}; +use slipstream::{ + BatchConfig, Connection, ExportRequest, KvReader, KvStore, KvUpdate, NatsConnection, + NatsConnectionConfig, StoreConfig, WatchCursor, WatchScope, watch_applied, +}; +use tempfile::TempDir; +use tokio::sync::{mpsc, oneshot, watch}; +use tokio::time::timeout; + +async fn open_bucket(nats: &TestNats) -> (NatsConnection, Arc) { + let conn = NatsConnection::new(NatsConnectionConfig { + url: nats.url.clone(), + creds: None, + creds_file: None, + }); + conn.connect().await.expect("connect"); + let store = conn + .store_with_config(StoreConfig { + name: "routes".into(), + // History 1 (the default) is load-bearing: every overwrite of a + // subject evicts its older message, which is how the test + // advances the stream's first sequence past the export cursor. + max_bytes: Some(8 * 1024 * 1024), + ..Default::default() + }) + .await + .expect("open bucket"); + (conn, store) +} + +struct Node { + exports: mpsc::Sender, + applied: Arc, + shutdown: watch::Sender, + task: tokio::task::JoinHandle>, +} + +fn spawn_node( + bucket: &Arc, + fold: S, + resume: Option, + reader: Option>, +) -> Node { + let watcher = bucket.watcher().expect("bucket watcher"); + let (ex_tx, ex_rx) = mpsc::channel(1); + let (sd_tx, sd_rx) = watch::channel(false); + let applied = Arc::new(AtomicU64::new(0)); + let applied_w = Arc::clone(&applied); + + let task = tokio::spawn(watch_applied( + watcher, + WatchScope::All, + resume, + reader, + Some(fold), + Some(ex_rx), + BatchConfig::default(), + |u: &KvUpdate| match u { + KvUpdate::Put(e) => Some(e.key.clone()), + _ => None, + }, + |_batch: Vec| {}, + move |cur: WatchCursor| { + applied_w.store(cur.as_u64().unwrap_or(0), Ordering::SeqCst); + }, + sd_rx, + )); + + Node { + exports: ex_tx, + applied, + shutdown: sd_tx, + task, + } +} + +async fn wait_applied(node: &Node, at_least: u64) { + timeout(Duration::from_secs(10), async { + loop { + if node.applied.load(Ordering::SeqCst) >= at_least { + return; + } + tokio::time::sleep(Duration::from_millis(20)).await; + } + }) + .await + .unwrap_or_else(|_| { + panic!( + "node never applied rev {at_least} (at {})", + node.applied.load(Ordering::SeqCst) + ) + }); +} + +type ImportFn = fn(&Path, &Path) -> Result<(WatchCursor, S), SnapshotError>; + +/// Everything both scenarios share: drive a live fold, export at cursor C +/// with `route.gone` present, then — offline — delete `route.gone`, purge its +/// subject so even the delete marker is gone, and overwrite every other +/// subject so the stream's first sequence moves past C. Returns the artifact, +/// its cursor, the final bucket revision, and the bucket handle. +struct ExpiredGap { + _nats: TestNats, + _conn: NatsConnection, + bucket: Arc, + dir: TempDir, + artifact: std::path::PathBuf, + export_rev: u64, + final_rev: u64, +} + +async fn build_expired_gap(open: impl Fn(&Path) -> (WatchCursor, S)) -> ExpiredGap +where + S: SnapshotStore + Send + 'static, +{ + let nats = TestNats::start().await; + let (conn, bucket) = open_bucket(&nats).await; + let writer = bucket.writer().expect("writer"); + let dir = TempDir::new().unwrap(); + + // Node A, live. + let (_r, fold_a) = open(&dir.path().join("node-a")); + let node_a = spawn_node(&bucket, fold_a, None, None); + + // Deterministic attach (KV watches deliver new updates only). + timeout(Duration::from_secs(10), async { + loop { + writer.put("route.seed", b"seed").await.expect("seed"); + tokio::time::sleep(Duration::from_millis(50)).await; + if node_a.applied.load(Ordering::SeqCst) > 0 { + return; + } + } + }) + .await + .expect("node A watch never attached"); + + // The fold state the artifact will carry: route.gone + five keepers. + writer + .put("route.gone", b"present-at-export") + .await + .expect("put gone"); + let mut export_rev = 0; + for i in 0..5u64 { + export_rev = writer + .put(&format!("route.keep.{i}"), format!("v1-{i}").as_bytes()) + .await + .expect("put keep") + .as_u64() + .expect("rev"); + } + wait_applied(&node_a, export_rev).await; + + // Export at cursor C through the live loop; A is done after this. + let artifact = dir.path().join("artifact"); + let (reply_tx, reply_rx) = oneshot::channel(); + node_a + .exports + .send(ExportRequest { + dest_dir: artifact.clone(), + reply: reply_tx, + }) + .await + .expect("send export request"); + let manifest = reply_rx.await.expect("reply").expect("export succeeds"); + let export_rev = manifest.cursor.as_u64().expect("cursor rev"); + node_a.shutdown.send(true).unwrap(); + node_a.task.await.unwrap().unwrap(); + + // --- The offline gap ------------------------------------------------- + // 1. Delete route.gone (leaves a delete marker)... + assert!(writer.delete("route.gone").await.expect("delete gone")); + // 2. ...then PURGE its subject so the marker itself is gone: after this, + // no replay and no re-list can ever deliver the delete. Only the + // resync can reconcile it. + let raw = async_nats::connect(&nats.url).await.expect("raw connect"); + let js = async_nats::jetstream::new(raw); + let mut stream = js.get_stream("KV_routes").await.expect("stream"); + stream + .purge() + .filter("$KV.routes.route.gone") + .await + .expect("purge route.gone subject"); + // 3. Overwrite every subject with messages at or below C (history=1 + // evicts each subject's older revision), pushing first_seq past C. + writer.put("route.seed", b"seed-v2").await.expect("put"); + let mut final_rev = 0; + for i in 0..5u64 { + final_rev = writer + .put(&format!("route.keep.{i}"), format!("v2-{i}").as_bytes()) + .await + .expect("put keep v2") + .as_u64() + .expect("rev"); + } + + // The premise the whole test rests on, asserted explicitly: the stream + // has compacted past the export cursor, so a resume from C MUST hit the + // expiry path (and if NATS silently clamped instead of erroring, the + // missing route.gone delete below would catch it). + let info = stream.info().await.expect("stream info"); + assert!( + info.state.first_sequence > export_rev + 1, + "stream first_seq {} must exceed export cursor {} + 1 — eviction premise", + info.state.first_sequence, + export_rev + ); + + ExpiredGap { + _nats: nats, + _conn: conn, + bucket, + dir, + artifact, + export_rev, + final_rev, + } +} + +/// Import the gap's artifact as node B and run the watch to convergence with +/// the given reader wiring; return B's reopened fold for inspection. +async fn bootstrap_through_gap( + gap: &ExpiredGap, + open: &impl Fn(&Path) -> (WatchCursor, S), + import: ImportFn, + reader: Option>, +) -> S +where + S: SnapshotStore + Send + 'static, +{ + let dest = gap.dir.path().join(if reader.is_some() { + "node-b-resync" + } else { + "node-b-bare" + }); + let (cursor_b, fold_b) = import(&gap.artifact, &dest).expect("import artifact"); + assert_eq!(cursor_b.as_u64(), Some(gap.export_rev)); + assert!( + fold_b.get("route.gone").expect("get").is_some(), + "the artifact carries route.gone — it was live at export time" + ); + + let node_b = spawn_node(&gap.bucket, fold_b, Some(cursor_b), reader); + wait_applied(&node_b, gap.final_rev).await; + node_b.shutdown.send(true).unwrap(); + node_b.task.await.unwrap().unwrap(); + + open(&dest).1 +} + +/// The keepers must equal the bucket in BOTH scenarios — divergence, when it +/// happens, is confined to the unreconciled delete, which is exactly what +/// makes it silent. +fn assert_keepers_converged(fold: &S) { + for i in 0..5u64 { + let e = fold + .get(&format!("route.keep.{i}")) + .expect("get") + .unwrap_or_else(|| panic!("route.keep.{i} missing after bootstrap")); + assert_eq!(e.value, format!("v2-{i}").as_bytes(), "route.keep.{i}"); + } +} + +/// Reader wired (the shipped configuration): the expiry is detected against a +/// REAL nats-server, the fallback runs, and the resync deletes the key whose +/// delete marker no longer exists anywhere in the stream. This is the live +/// verification of the model's `Resume → Synced` transition for the expired +/// path — and of the axiom that NATS errors (rather than silently clamping) +/// on a too-old start sequence. +async fn resync_reconciles_offline_delete( + open: impl Fn(&Path) -> (WatchCursor, S), + import: ImportFn, +) where + S: SnapshotStore + Send + 'static, +{ + let gap = build_expired_gap(&open).await; + let reader = Some(gap.bucket.reader()); + let fold = bootstrap_through_gap(&gap, &open, import, reader).await; + + assert!( + fold.get("route.gone").expect("get").is_none(), + "route.gone must be reconciled away: its delete marker was purged, so \ + only the expiry-detected resync path can have removed it" + ); + assert_keepers_converged(&fold); +} + +/// Reader NOT wired: same gap, same fallback — and the deleted key persists +/// in an otherwise fully converged fold. The live twin of the model's +/// "HAZARD reachable: silent stale-key divergence without resync". +async fn without_reader_stale_key_persists( + open: impl Fn(&Path) -> (WatchCursor, S), + import: ImportFn, +) where + S: SnapshotStore + Send + 'static, +{ + let gap = build_expired_gap(&open).await; + let fold = bootstrap_through_gap(&gap, &open, import, None).await; + + assert!( + fold.get("route.gone").expect("get").is_some(), + "without the resync reader the deleted key persists — the divergence \ + the model proves reachable, pinned against a live server" + ); + assert_keepers_converged(&fold); +} + +/// Pins the live-server behavior that makes proactive expiry detection +/// necessary: an ordered consumer whose `ByStartSequence` falls below the +/// stream's first retained sequence gets NO error — NATS silently delivers +/// from the first available message. This is why `nats.rs` compares the +/// stream's `first_sequence` against the resume point +/// (`check_resume_window`) instead of relying on a consumer-create error, +/// and why the resume below first_seq here MUST surface as +/// `KvError::CursorExpired` from slipstream's own check. +/// +/// If a future nats-server/async-nats starts erroring on the raw seek, the +/// raw-probe half of this test fails and the error-string fallback can be +/// re-evaluated; the slipstream-level half must keep returning +/// `CursorExpired` either way. +#[tokio::test(flavor = "multi_thread")] +async fn nats_silently_clamps_resume_below_first_seq() { + let nats = TestNats::start().await; + let (_conn, bucket) = open_bucket(&nats).await; + let writer = bucket.writer().expect("writer"); + + // Four rounds over two subjects: revs 1..=8, history=1 keeps only the + // last round → first_seq advances to 7. + for round in 0..4u64 { + for k in ["route.a", "route.b"] { + writer + .put(k, format!("{round}").as_bytes()) + .await + .expect("put"); + } + } + let raw = async_nats::connect(&nats.url).await.expect("raw connect"); + let js = async_nats::jetstream::new(raw); + let mut stream = js.get_stream("KV_routes").await.expect("stream"); + let info = stream.info().await.expect("info"); + assert!( + info.state.first_sequence > 3, + "eviction premise: first_seq {} > 3", + info.state.first_sequence + ); + + // RAW async-nats seek below first_seq: no error, delivery starts at the + // first retained message — the silent clamp. + let kv = js.get_key_value("routes").await.expect("kv handle"); + let mut watch = kv + .watch_all_from_revision(3) + .await + .expect("NATS accepts a below-head start sequence without error — the clamp"); + use futures::StreamExt; + let first = timeout(Duration::from_secs(5), watch.next()) + .await + .expect("clamped watch delivers") + .expect("entry") + .expect("entry ok"); + assert!( + first.revision >= info.state.first_sequence, + "delivery starts at the clamped head (rev {}), silently skipping the gap", + first.revision + ); + + // Slipstream's watcher refuses the same resume: proactive expiry. + let watcher = bucket.watcher().expect("watcher"); + let (tx, _rx) = tokio::sync::mpsc::channel(8); + let err = watcher + .watch_all_from(&WatchCursor::from_u64(2), tx) + .await + .expect_err("resume below first_seq must be detected"); + assert!( + matches!(err, slipstream::KvError::CursorExpired), + "expected CursorExpired, got {err:?}" + ); +} + +#[cfg(feature = "fjall")] +mod fjall_resync { + use super::*; + use slipstream::{FjallConfig, FjallSnapshot}; + + fn cfg() -> FjallConfig { + FjallConfig { + sync: false, + cache_size_bytes: 64 << 20, + } + } + + #[tokio::test(flavor = "multi_thread")] + async fn fjall_cursor_expired_resync_reconciles_offline_delete() { + resync_reconciles_offline_delete( + |path| FjallSnapshot::open(path, cfg()).expect("open fjall"), + |artifact, dest| FjallSnapshot::import(artifact, dest, cfg()), + ) + .await; + } + + #[tokio::test(flavor = "multi_thread")] + async fn fjall_cursor_expired_without_reader_keeps_stale_key() { + without_reader_stale_key_persists( + |path| FjallSnapshot::open(path, cfg()).expect("open fjall"), + |artifact, dest| FjallSnapshot::import(artifact, dest, cfg()), + ) + .await; + } +} + +#[cfg(feature = "rocksdb")] +mod rocksdb_resync { + use super::*; + use slipstream::{RocksDbConfig, RocksDbSnapshot}; + + fn cfg() -> RocksDbConfig { + RocksDbConfig { + sync: false, + cache_size_bytes: 64 << 20, + } + } + + #[tokio::test(flavor = "multi_thread")] + async fn rocksdb_cursor_expired_resync_reconciles_offline_delete() { + resync_reconciles_offline_delete( + |path| RocksDbSnapshot::open(path, cfg()).expect("open rocksdb"), + |artifact, dest| RocksDbSnapshot::import(artifact, dest, cfg()), + ) + .await; + } + + #[tokio::test(flavor = "multi_thread")] + async fn rocksdb_cursor_expired_without_reader_keeps_stale_key() { + without_reader_stale_key_persists( + |path| RocksDbSnapshot::open(path, cfg()).expect("open rocksdb"), + |artifact, dest| RocksDbSnapshot::import(artifact, dest, cfg()), + ) + .await; + } +} diff --git a/tests/transport.rs b/tests/transport.rs index 3d44c2b..edb007b 100644 --- a/tests/transport.rs +++ b/tests/transport.rs @@ -17,7 +17,8 @@ use slipstream::snapshot::{SnapshotError, SnapshotStore}; use slipstream::{ AppendLogSnapshot, ArtifactTransport, BatchConfig, Connection, ExportLease, ExportManifest, KvEntry, KvUpdate, KvWriter, NatsConnection, NatsConnectionConfig, ObjectStoreTransport, - StoreConfig, VersionToken, WatchCursor, WatchScope, run_export_round, watch_applied, + PublishOutcome, StoreConfig, VersionToken, WatchCursor, WatchScope, run_export_round, + watch_applied, }; use tempfile::TempDir; use tokio::sync::{mpsc, watch}; @@ -118,34 +119,177 @@ async fn import_remote_bootstraps_a_fold() { ); } -/// A sibling manifest that disagrees with the tar's embedded manifest is -/// rejected at download — the transport is untrusted. +/// A doctored pointer cannot reach the real payload: the payload key is +/// derived from the pointer's bytes (content addressing), so tampering with +/// the pointer dereferences to nothing — rejected at download, nothing lands. #[tokio::test(flavor = "multi_thread")] -async fn download_rejects_manifest_disagreement() { +async fn download_rejects_doctored_pointer() { let (transport, bucket) = local_transport(); let (artifact, _m, dir) = exported_artifact(); - transport.upload("latest", &artifact).await.unwrap(); + assert_eq!( + transport.upload("latest", &artifact).await.unwrap(), + PublishOutcome::Published + ); - // Doctor the sibling manifest object in the "bucket" (cursor 999). - let sibling = bucket + // Doctor the pointer object in the "bucket" (cursor 999): its derived + // payload address now points at an object that does not exist. + let pointer = bucket .path() .join("slipstream-artifacts") .join("latest.manifest.json"); - let raw = std::fs::read(&sibling).unwrap(); + let raw = std::fs::read(&pointer).unwrap(); let mut json: serde_json::Value = serde_json::from_slice(&raw).unwrap(); json["cursor_hex"] = "00000000000003e7".into(); - std::fs::write(&sibling, serde_json::to_vec(&json).unwrap()).unwrap(); + std::fs::write(&pointer, serde_json::to_vec(&json).unwrap()).unwrap(); let dest = dir.path().join("downloaded"); match transport.download("latest", &dest).await { Err(SnapshotError::ArtifactInvalid(msg)) => { - assert!(msg.contains("disagrees"), "{msg}"); + assert!(msg.contains("not found"), "{msg}"); } other => panic!("expected ArtifactInvalid, got {other:?}"), } assert!(!dest.exists(), "nothing lands at the destination"); } +/// The pointer only moves forward: publishing an artifact older than the +/// current pointer is refused with [`PublishOutcome::SupersededByNewer`], +/// and the published manifest is untouched. Runs on `InMemory`, which +/// implements the same conditional-put semantics as S3 — so this exercises +/// the VERIFIED swap path (Create, refusal, If-Match update) without a +/// server; `transport_s3.rs` repeats it against live MinIO. The live-fleet +/// version (a stalled round racing a takeover through `run_export_round`) +/// is `multi_export.rs`'s slow-exporter scenario. +#[tokio::test(flavor = "multi_thread")] +async fn pointer_swap_refuses_regression() { + let transport = ObjectStoreTransport::new( + Arc::new(object_store::memory::InMemory::new()), + "slipstream-artifacts", + ); + let dir = TempDir::new().unwrap(); + + // Two artifacts from one fold at different cursors. + let (_r, mut s) = AppendLogSnapshot::open(&dir.path().join("fold.snap"), u64::MAX).unwrap(); + s.apply(&[put("a", b"1", 3)], &WatchCursor::from_u64(3)) + .unwrap(); + let old = dir.path().join("artifact-old"); + let old_manifest = s.export_to(&old).unwrap(); + s.apply(&[put("b", b"2", 5)], &WatchCursor::from_u64(5)) + .unwrap(); + let new = dir.path().join("artifact-new"); + let new_manifest = s.export_to(&new).unwrap(); + + assert_eq!( + transport.upload("latest", &new).await.unwrap(), + PublishOutcome::Published + ); + match transport.upload("latest", &old).await.unwrap() { + PublishOutcome::SupersededByNewer { remote_cursor } => { + assert_eq!(remote_cursor, new_manifest.cursor); + } + other => panic!("stale publish must be refused, got {other:?}"), + } + assert_eq!( + transport.manifest("latest").await.unwrap().cursor, + new_manifest.cursor, + "the pointer never regressed" + ); + // Equal-or-newer always publishes (LWW among equals is harmless — each + // pointer atomically references its own payload). + assert_eq!( + transport.upload("latest", &new).await.unwrap(), + PublishOutcome::Published + ); + let _ = old_manifest; +} + +/// A non-CAS store (LocalFileSystem) FAILS the pointer swap by default — +/// the degraded last-write-wins publish is outside the verified protocol +/// and must be an explicit opt-in, not a silent downgrade. +#[tokio::test(flavor = "multi_thread")] +async fn non_cas_store_fails_closed_without_opt_in() { + let (transport, _bucket) = local_transport(); + let dir = TempDir::new().unwrap(); + + let (_r, mut s) = AppendLogSnapshot::open(&dir.path().join("fold.snap"), u64::MAX).unwrap(); + s.apply(&[put("a", b"1", 3)], &WatchCursor::from_u64(3)) + .unwrap(); + let first = dir.path().join("artifact-3"); + s.export_to(&first).unwrap(); + s.apply(&[put("b", b"2", 5)], &WatchCursor::from_u64(5)) + .unwrap(); + let second = dir.path().join("artifact-5"); + s.export_to(&second).unwrap(); + + // First publish is create-only — supported everywhere, succeeds. + assert_eq!( + transport.upload("latest", &first).await.unwrap(), + PublishOutcome::Published + ); + // Forward publish needs compare-and-swap — refused on this store with an + // error naming the opt-in, and the pointer is untouched. + match transport.upload("latest", &second).await { + Err(SnapshotError::Backend(msg)) => { + assert!(msg.contains("with_non_atomic_pointer_fallback"), "{msg}"); + } + other => panic!("non-CAS swap must fail closed, got {other:?}"), + } + assert_eq!( + transport.manifest("latest").await.unwrap().cursor.as_u64(), + Some(3), + "the failed swap left the pointer untouched" + ); +} + +/// `prune` collects payloads the pointer no longer references — but never +/// the referenced one, and never inside the grace window. (Local FS store: +/// the second publish takes the explicitly opted-in non-atomic fallback; +/// prune itself is identical on every store and re-verified on MinIO.) +#[tokio::test(flavor = "multi_thread")] +async fn prune_collects_unreferenced_payloads() { + let (transport, bucket) = local_transport(); + let transport = transport.with_non_atomic_pointer_fallback(); + let dir = TempDir::new().unwrap(); + + let (_r, mut s) = AppendLogSnapshot::open(&dir.path().join("fold.snap"), u64::MAX).unwrap(); + s.apply(&[put("a", b"1", 3)], &WatchCursor::from_u64(3)) + .unwrap(); + let old = dir.path().join("artifact-old"); + s.export_to(&old).unwrap(); + s.apply(&[put("b", b"2", 5)], &WatchCursor::from_u64(5)) + .unwrap(); + let new = dir.path().join("artifact-new"); + let new_manifest = s.export_to(&new).unwrap(); + + transport.upload("latest", &old).await.unwrap(); + transport.upload("latest", &new).await.unwrap(); + let payloads = bucket + .path() + .join("slipstream-artifacts") + .join("latest.payloads"); + assert_eq!( + std::fs::read_dir(&payloads).unwrap().count(), + 2, + "both rounds' payloads exist before the prune" + ); + + // Inside the grace window nothing is collected. + assert_eq!( + transport + .prune("latest", Duration::from_secs(3600)) + .await + .unwrap(), + 0 + ); + // Grace elapsed (zero): the unreferenced payload goes, the referenced one + // stays, and the published artifact still round-trips. + assert_eq!(transport.prune("latest", Duration::ZERO).await.unwrap(), 1); + assert_eq!(std::fs::read_dir(&payloads).unwrap().count(), 1); + let downloaded = dir.path().join("downloaded"); + let got = transport.download("latest", &downloaded).await.unwrap(); + assert_eq!(got.cursor, new_manifest.cursor); +} + /// A missing remote object is an ArtifactInvalid, not an opaque backend error. #[tokio::test(flavor = "multi_thread")] async fn missing_remote_artifact_is_artifact_invalid() { @@ -377,7 +521,7 @@ struct FailingTransport; #[async_trait] impl ArtifactTransport for FailingTransport { - async fn upload(&self, _key: &str, _dir: &Path) -> Result<(), SnapshotError> { + async fn upload(&self, _key: &str, _dir: &Path) -> Result { Err(SnapshotError::Backend("injected upload failure".into())) } async fn manifest(&self, _key: &str) -> Result { diff --git a/tests/transport_s3.rs b/tests/transport_s3.rs index fec541f..564c341 100644 --- a/tests/transport_s3.rs +++ b/tests/transport_s3.rs @@ -359,3 +359,141 @@ async fn minio_multipart_upload_round_trips() { "all entries survive the multipart round trip" ); } + +// --- The S3-axiom test for the pointer-swap protocol -------------------------- + +/// Axiom 1 of `tests/model.rs`, verified against a REAL S3 API (MinIO): +/// conditional puts have one winner per slot and the pointer swap is +/// monotonic and crash-safe on this store. +/// +/// Exercises, in order: `PutMode::Create` (first publish), the monotonic +/// refusal (stale publish superseded), `PutMode::Update` / `If-Match` +/// compare-and-swap (forward publish over an existing pointer), the +/// crash-between-payload-and-swap window (pointer stays consistent, +/// bootstrap stays available), recovery, and `prune` (list + delete of +/// unreferenced content-addressed payloads). +#[tokio::test(flavor = "multi_thread")] +async fn minio_pointer_swap_monotonic_and_crash_safe() { + use slipstream::{AppendLogSnapshot, PublishOutcome}; + + let minio = TestMinio::start().await; + let dir = TempDir::new().unwrap(); + + // Four artifacts from one fold at cursors 3 < 5 < 7 < 9. + let (_r, mut s) = AppendLogSnapshot::open(&dir.path().join("fold.snap"), u64::MAX).unwrap(); + let artifact_at = |rev: u64, s: &mut AppendLogSnapshot| { + s.apply( + &[slipstream::KvUpdate::Put(slipstream::KvEntry { + key: format!("route.{rev}"), + value: format!("v{rev}").into_bytes(), + version: slipstream::VersionToken::from_u64(rev), + })], + &WatchCursor::from_u64(rev), + ) + .unwrap(); + let p = dir.path().join(format!("artifact-{rev}")); + s.export_to(&p).unwrap(); + p + }; + let a3 = artifact_at(3, &mut s); + let a5 = artifact_at(5, &mut s); + let a7 = artifact_at(7, &mut s); + let a9 = artifact_at(9, &mut s); + + let transport = minio_transport(&minio); + + // PutMode::Create — first publish wins the empty slot. + assert_eq!( + transport.upload("edge/latest", &a5).await.unwrap(), + PublishOutcome::Published + ); + // Monotonic refusal — a stale publish cannot regress the pointer. + match transport.upload("edge/latest", &a3).await.unwrap() { + PublishOutcome::SupersededByNewer { remote_cursor } => { + assert_eq!(remote_cursor.as_u64(), Some(5)); + } + other => panic!("stale publish must be superseded, got {other:?}"), + } + assert_eq!( + transport + .manifest("edge/latest") + .await + .unwrap() + .cursor + .as_u64(), + Some(5) + ); + // PutMode::Update (If-Match CAS) — forward publish over the existing + // pointer, on the real S3 conditional-put API. + assert_eq!( + transport.upload("edge/latest", &a7).await.unwrap(), + PublishOutcome::Published + ); + assert_eq!( + transport + .manifest("edge/latest") + .await + .unwrap() + .cursor + .as_u64(), + Some(7) + ); + + // Crash between payload and pointer swap, on the real store: the pointer + // stays fully consistent and bootstrap keeps working. + let raw: Arc = { + let mut b = object_store::aws::AmazonS3Builder::new().with_bucket_name(MINIO_BUCKET); + for (k, v) in minio.s3_options() { + b = b.with_config(k.parse().expect("s3 config key"), v); + } + Arc::new(b.build().expect("build raw s3 store")) + }; + let crash = Arc::new(common::ManifestPutCrash { + inner: raw, + armed: std::sync::atomic::AtomicBool::new(true), + }); + let crashing = ObjectStoreTransport::new(crash, "slipstream"); + crashing + .upload("edge/latest", &a9) + .await + .expect_err("injected crash before the pointer swap"); + assert_eq!( + transport + .manifest("edge/latest") + .await + .unwrap() + .cursor + .as_u64(), + Some(7), + "the pointer is untouched by the crashed round" + ); + let dl_during = dir.path().join("dl-during"); + let got = transport + .download("edge/latest", &dl_during) + .await + .expect("bootstrap stays available through the crash window"); + assert_eq!(got.cursor.as_u64(), Some(7)); + + // Recovery: the next round publishes, and prune collects every payload + // the pointer no longer references — on the real list/delete API. + assert_eq!( + transport.upload("edge/latest", &a9).await.unwrap(), + PublishOutcome::Published + ); + let pruned = transport + .prune("edge/latest", Duration::ZERO) + .await + .expect("prune"); + assert_eq!(pruned, 3, "payloads for cursors 3, 5, 7 are unreferenced"); + let dl = dir.path().join("dl-final"); + let got = transport + .download("edge/latest", &dl) + .await + .expect("download after prune"); + assert_eq!(got.cursor.as_u64(), Some(9)); + let (cursor, imported) = + AppendLogSnapshot::import(&dl, &dir.path().join("imported.snap"), u64::MAX) + .expect("import"); + assert_eq!(cursor.as_u64(), Some(9)); + assert_eq!(imported.range("route.").unwrap().len(), 4); +} From 408a585e285646098b94690849432eddbb81657b Mon Sep 17 00:00:00 2001 From: Jared Lunde Date: Thu, 11 Jun 2026 19:39:37 -0700 Subject: [PATCH 2/3] docs: expand ARCHITECTURE.md with transport, protocol kernels, and export pipeline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the sections and content that were missing from the existing doc: - Rewrite summary to describe observable input → output behavior - Add data flow diagrams for artifact bootstrap and export round - Add 9 new Concepts & Terminology entries covering ExportLease, LeaseGuard, ExportManifest, ArtifactTransport, ObjectStoreTransport, PublishOutcome, etc. - Add Machine-Checked Protocol Kernels section documenting the three guards in protocol.rs (pointer_publish_allowed, payload_prunable, resume_window_ok), their Stateright theorem names, and why they are shared between production and model - Add Export Round State Machine with states, transitions, and failure outcomes - Add Export Lease section covering CAS acquisition, embedded TTL rationale, corrupt-lease stealing, and LeaseRecord wire format - Expand transport object layout to show content-addressed payload key derivation - Update Package Structure to include all new source files and test files - Update Failure Modes table with What-Actually-Happens column and export failures - Expand Trust Model to cover the transport layer's verification gates - Add two new Design Decisions: monotonic pointer vs. two-register layout and protocol.rs shared with model - Add BatchConfig, ObjectStoreTransport/ExportLease, and per-backend durability configuration tables Co-Authored-By: Claude Fable 5 --- ARCHITECTURE.md | 302 ++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 269 insertions(+), 33 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index b0f5adb..23e3ab6 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,10 +1,10 @@ # Slipstream Architecture -Trait-based KV abstraction over NATS JetStream: read, write, and watch distributed config with a resumable cursor so services replay only the delta after any restart. +Takes NATS JetStream KV buckets as input and produces a local durable fold — a `SnapshotStore` whose cursor is guaranteed to name only revisions whose `apply` has returned — so edge nodes survive restarts without full replay, and can bootstrap from a content-addressed artifact when the NATS log has compacted past their cursor. ## Data Flow -### Happy path: startup with snapshot +### Happy path: startup with snapshot — no export ``` Disk ──► load(path) ──► replay_log() ──► HashMap + WatchCursor @@ -44,6 +44,52 @@ writer.delete_with_version("k", ver) ──► kv.update(key, []) ──► Revi CAS tombstone (empty-value Put) is how `delete_with_version` works — it writes an empty value via a CAS operation so concurrent writers see a conflict. `get()` and `scan()` filter these out; `entry()` exposes them for CAS callers that need the version. +### Bootstrap via artifact (bounded log, cursor evicted) + +``` +ObjectStore (/.manifest.json) + ↓ download pointer +ExportManifest (cursor, backend, file hashes) + ↓ derive payload key from blake3(manifest)[..8] +ObjectStore (/.payloads/.tar) + ↓ stream to temp dir (verified: embedded manifest == pointer bytes, every file hash re-checked) +Staged artifact dir + ↓ rename into place (atomic; bad artifact never becomes the fold) +SnapshotStore (recovered cursor = manifest.cursor) + ↓ watch_applied(resume=cursor) +NATS JetStream (tail delta from cursor+1) + ↓ apply tail +Converged fold +``` + +### Export round (one winner per fleet, TTL-gated) + +``` +ExportLease::try_acquire(ttl) + ↓ create-only CAS wins; expired/corrupt lease CAS-stolen +watch_applied ExportRequest channel + ↓ pending batch flushed first +store.export_to(artifact_dir) + ↓ backend snapshot + verify-by-reopen (cursor equality gate) +ExportManifest (cursor, backend, file BLAKE3 hashes) + ↓ +ObjectStoreTransport::upload(key, artifact_dir) + ├─ tar data/ → multipart upload (8 MiB chunks, 8 concurrent) + │ key = /.payloads/.tar + └─ swap_pointer() → conditional put at /.manifest.json + ├─ pointer_publish_allowed(current, candidate_rank)? + │ true → CAS put (max 8 retries on concurrent swap) + │ false → SupersededByNewer (slow exporter gives up cleanly) + └─ PublishOutcome::Published | SupersededByNewer + ↓ Published +LeaseGuard::complete(cursor) + ↓ +ObjectStoreTransport::prune(key, grace=4×ttl) + └─ delete payloads: payload_prunable(payload_rank, pointer_rank, is_target, aged_out)? + ↓ +delete local artifact dir (transience enforced — not hoped for) +``` + ### Watch resumption ``` @@ -80,6 +126,15 @@ Every non-`_from` watch is a **state-sync** stream (NATS `DeliverPolicy::LastPer | `watch_applied` | Combinator: batch → apply → *then* advance cursor / fold into `SnapshotStore`; resyncs stale keys on cursor expiry | Not a raw watch; the cursor follows `apply`, not receipt | | `WatchScope` | What `watch_applied` watches: `All`, `Prefix`, or `Prefixes` (multi-filter union) | Not N consumers; `Prefixes` costs one consumer | | `ConnectionCapabilities` | Feature flags for runtime branching (CAS, streaming watch, …) | Not enforced; purely advisory | +| `ExportRequest` | One-shot channel message that asks `watch_applied` to flush + export the current fold | Not a store operation; handled between flushes | +| `ExportLease` | Fleet-wide at-most-one coordinator: CAS key that prevents N nodes exporting the same round | Not a correctness gate; only dedup. Pointer monotonicity is the gate | +| `LeaseGuard` | RAII guard for a won export round: `complete(cursor)` stamps success, `abandon()` frees it early via CAS delete | Not held across restarts; process crash → expiry → next node steals | +| `LeaseRecord` | Value stored in the lease key: holder, acquired/expires timestamps (wall-clock, embedded — no server TTL), optional completed cursor | Expiry is compared by callers, not the store; requires NTP-sane clocks | +| `ExportManifest` | Artifact metadata committed to disk and object store: backend identity, format generation, cursor, per-file BLAKE3 hashes | The manifest bytes are their own content address (blake3[..8]) | +| `ArtifactTransport` | Trait: `upload(key, dir) → ManifestAndOutcome`, `download(key) → ManifestAndDir`, `pointer(key)`, `prune(key, grace)` | Not an object store; adapts `ObjectStore` for the pointer-swap protocol | +| `ObjectStoreTransport` | Concrete `ArtifactTransport` over any `object_store` backend (S3, GCS, Azure, local) | `file://` lacks CAS and FAILS CLOSED unless `with_non_atomic_pointer_fallback()` (dev only) | +| `PublishOutcome` | `Published` (pointer advanced) or `SupersededByNewer` (refused — a newer pointer exists) | Not an error; a slow exporter's normal exit under concurrent rounds | +| `PointerState` | What `swap_pointer` observes before deciding: `Absent`, `Present { rank: Some(n) }`, or `Present { rank: None }` (corrupt) | Used only by `pointer_publish_allowed`; never stored independently | ## Layer Architecture @@ -193,6 +248,88 @@ Every NATS operation is wrapped in `timed()` (30 s). Without it, a CLOSE_WAIT co | FDB | 10-byte versionstamp | `None` | | Unknown | len=0 | `None` | +### Machine-Checked Protocol Kernels (`protocol.rs`) + +Three pure-function guards live in `protocol.rs`. Production code and the Stateright exhaustive model checker (`tests/model.rs`) call the **same functions** — not two hand-synchronized copies. A change to any guard is re-verified against the full bounded state space on the next `cargo test --test model`. Mutation tests prove each guard is load-bearing: substituting a broken variant produces a counterexample. + +**`pointer_publish_allowed(current: &PointerState, candidate_rank: u64) → bool`** + +The monotonic pointer guard. Returns `true` for an open slot, a corrupt pointer, or a candidate at or above the existing cursor — and `false` exactly when the existing pointer is parseable and strictly newer. + +``` +Absent → true (open slot) +Present { rank: None } → true (corrupt pointer is replaceable, not a wedge) +Present { rank: Some(n) } + candidate >= n → true + candidate < n → false (stale publish refused) +``` + +Machine-checked as: _"published cursor never regresses."_ + +**`payload_prunable(payload_rank, pointer_rank, is_pointer_target, aged_out) → bool`** + +The prune guard. A payload is deletable only when all hold: it is not the pointer's target, its rank is parseable AND strictly below the pointer's, and its age has cleared the grace period. + +``` +!is_pointer_target && aged_out && payload_rank.is_some_and(|r| r < pointer_rank) +``` + +Strictly-below (not `<=`) is the structural fix that makes dangling pointers impossible: `pointer_publish_allowed` refuses any candidate strictly below the pointer, and the pointer is monotone — so anything this guard deletes can never be successfully published afterward. The model checker found a dangling-pointer counterexample under an earlier age-only rule; this guard is the fix. + +Machine-checked as: _"pointer target always fetchable"_ under zero-grace pruning. + +**`resume_window_ok(revision: u64, first_sequence: u64) → bool`** + +The cursor-expiry guard. Resume reads `revision + 1` onward; it is sound iff `first_sequence ≤ revision + 1`. NATS does not error on a below-head start — it silently clamps to `first_sequence`, skipping evicted delete markers with no fallback. This check is ours to make. + +``` +first_sequence ≤ revision.saturating_add(1) → sound +first_sequence > revision + 1 → CursorExpired +``` + +Machine-checked as: _"bootstrap never silently diverges."_ Empirically pinned by `tests/resync.rs::nats_silently_clamps_resume_below_first_seq`. + +### Export Round State Machine + +``` +[Idle] ──ExportRequest──► [Acquiring] + │ + try_acquire(ttl) + / \ + Won Lost (another holder) + │ │ + [Exporting] [Idle] + │ + flush pending batch + store.export_to(dir) + verify-by-reopen (cursor eq) + │ + [Uploading] + │ + tar + multipart put + pointer_publish_allowed? + ├─ true → CAS swap (≤8 retries) + └─ false → SupersededByNewer + │ + Published SupersededByNewer + │ │ + [Completing] [Abandoning] + │ │ + complete(cursor) abandon() → CAS delete + prune(grace=4×ttl) + delete local dir + │ + [Idle] +``` + +| State | Entry Condition | On Failure | +| ------------ | ---------------------------------------- | -------------------------------------- | +| Acquiring | `ExportRequest` received by watch task | — | +| Exporting | Lease won (CAS create or takeover) | Abandon lease; delete local dir | +| Uploading | Export + verify-by-reopen succeeded | Abandon lease; delete local dir | +| Completing | `Published` | Prune/complete failures are non-fatal | +| Abandoning | `SupersededByNewer` or upload failed | CAS delete failure is non-fatal | + ## Snapshot Subsystem ### The durable-fold trait @@ -266,6 +403,32 @@ write_update() compact() [blocking: replay → dedup `checkpoint()` writes only a cursor record and calls `BufWriter::flush()` — a `write(2)` into the page cache. This survives a process crash but NOT a power loss. The only `fsync` is in `compact()`. The snapshot is a cache; a lost tail is rebuilt from a NATS scan + watch replay. +### Export Lease (`export_lease.rs`) + +`ExportLease` coordinates fleet-wide at-most-one export rounds using a NATS KV key as a mutex, with expiry embedded in the value (no server TTL machinery required). + +**Acquisition:** +- `create(key, value)` → wins if key absent; exactly one caller fleet-wide succeeds +- If key exists: parse `LeaseRecord` → compare `expires_at` to wall clock + - Expired or unparseable → CAS `update(key, new_record, observed_version)` to steal + - Live → return `None` (another holder is active) + +**`LeaseRecord` wire format (JSON):** +```json +{ "holder_id": "node-42", "acquired_at_unix": 1718000000, "expires_at_unix": 1718000300, + "completed_cursor_hex": null, "completed_at_unix": null } +``` +After upload: `completed_cursor_hex` and `completed_at_unix` are filled in by `complete(cursor)`. + +**`LeaseGuard` RAII discipline:** +- `complete(cursor)` → CAS-rewrite lease with cursor + timestamp (fleet-visible last-export record) +- `abandon()` → CAS-delete the key to free the round early (next trigger elects immediately) +- Drop without completing → lease expires naturally; next node steals after TTL + +**Why embedded TTL over server TTL:** Portable to any `KvWriter`; no version/bucket-flag requirements. The cost is wall-clock comparison — acceptable because a premature steal merely produces a duplicate artifact (last-write-wins on the same key), never corruption. The lease is work-dedup, not a correctness gate. + +**Why corrupt-lease stealing:** One unparseable value would otherwise wedge the fleet at `expires_at` forever. The same rule as `pointer_publish_allowed`'s treatment of corrupt pointers: unknown/corrupt state is replaceable, not a hard stop. + ### Export / Import (replica bootstrap) When the watched bucket is a **bounded** log (size-capped, history evicted), a fold is no longer rebuildable from NATS alone — the folds become the only full replicas. Export/import makes them transferable, which is what lets a new node, a node with a lost/corrupt fold, or a node whose cursor aged out of the log bootstrap at all. @@ -286,7 +449,16 @@ When the watched bucket is a **bounded** log (size-capped, history evicted), a f **Fleet coordination.** `ExportLease` makes exactly one replica perform a given export round: create-only CAS to win, expiry embedded in the value (no server TTL machinery), CAS takeover of expired/corrupt leases, `complete()` publishes the exported cursor on the key (the fleet-visible last-export record), `abandon()` frees a failed round early. Clock skew at worst causes a duplicate export — safe; the lease is dedup, not a correctness gate. -**Transport** (feature `transport`): plain tar of the artifact at `/` via any `object_store` backend, manifest duplicated as a sibling `.manifest.json` uploaded last (remote completeness marker + cheap cursor peek). `run_export_round` composes lease → export → upload → complete → delete-local; per-backend `import_remote` composes download → verify → import. +**Transport** (feature `transport`): `ObjectStoreTransport` adapts any `object_store` backend (S3, GCS, Azure, local). Object layout: + +``` +/.payloads/.tar ← content-addressed payload (write-once) +/.manifest.json ← monotonic pointer (published last) +``` + +The payload key is derived from the manifest bytes, making collisions require hash breakage. The manifest is the single trusted object; the payload key is derived from it at download time — readers never trust a separately-stored pointer to the payload. + +`run_export_round` composes the full at-most-once round: lease → export (via `ExportRequest`) → upload → complete → prune → delete-local. Per-backend `import_remote` composes: download pointer → derive payload key → download + verify → import. ## Connection Lifecycle @@ -364,6 +536,20 @@ fjall has no checkpoint API, so its copy is assembled from parts; rather than tr Per-message TTLs need a new-enough NATS server, a bucket flag, and a backend that supports them; an `expires_at` inside the value works on any `KvWriter` and keeps acquisition/takeover as two plain CAS operations. The cost is wall-clock comparison across nodes — acceptable because a premature steal merely produces a duplicate artifact (last-write-wins on the same key), never corruption: the lease is work-dedup, not a correctness gate. +### Why a content-addressed, write-once payload + monotonic pointer instead of two fixed-key objects? + +The legacy layout wrote the payload at `.payload.tar` and the manifest at `.manifest.json` as independent last-write-wins objects. The model checker proved two failure modes under that layout: (1) a cursor regression when a slow exporter's stale manifest overwrites a faster one's, and (2) a dangling pointer when the manifest advances past a payload that an older concurrent publisher still holds. The shipped layout fixes both structurally: + +- **Content-addressed payload** — the key is `blake3(manifest)[..8].tar`. Two different artifacts can never share a key. Re-uploading the same artifact is an idempotent overwrite. A slow exporter's payload is valid until pruned; it just can't become the pointer. +- **Monotonic pointer via CAS** — `pointer_publish_allowed` refuses any candidate strictly below the current pointer. Combined with CAS (no torn reads between observe and write), the pointer is monotone non-decreasing. Slow exporters always get `SupersededByNewer`, never succeed at regression. +- **Strictly-below prune rule** — `payload_prunable` only deletes ranks strictly below the pointer. Because the pointer is monotone and `pointer_publish_allowed` refuses `< pointer`, a pruned payload can never become reachable again. The model checker found the dangling counterexample under an earlier age-only rule; this is the structural fix. + +Machine-checked as three theorems in `tests/model.rs`: published cursor never regresses, pointer target always fetchable (zero-grace prune), bootstrap never silently diverges. + +### Why extract protocol guards into `protocol.rs` and share them with the model? + +Production code and an exhaustive model checker running the same logic closes the drift gap between proof and implementation. Previously the model had its own inline guard copies; a change to the production guard would not update the model, leaving the proof covering the old variant. Extracting the three guards into `protocol.rs` and importing them from both call sites means every `cargo test --test model` verifies the current production code, not a snapshot. The cost is a hard dependency: `tests/model.rs` imports `pub mod protocol` directly, so the module cannot go private. + ### Why write in sorted key order during compaction? `HashMap` iteration order is random per process. Sorting produces a deterministic byte layout for a given logical state, enabling byte-level snapshot comparison (integrity checksums, test assertions) and making file diffs readable. The O(n log n) sort is negligible relative to the I/O it precedes. @@ -392,6 +578,7 @@ Per-message TTLs need a new-enough NATS server, a bucket flag, and a backend tha - Bucket exists or can be created (at `store()`) - Snapshot CRC per record (at `load()` and `compact()`) - Snapshot magic bytes and format version (at `load()`) +- Resume window soundness — `resume_window_ok(revision, first_sequence)` before attaching (NATS silently clamps; we error explicitly) **What passes through unchecked:** - Key names (no validation; NATS accepts any key) @@ -402,41 +589,65 @@ Per-message TTLs need a new-enough NATS server, a bucket flag, and a backend tha **Why this is acceptable:** Applications own value encoding (JSON, proto, etc.). NATS owns authorization. The store layer is a transport adapter. +**What the transport layer verifies (import path):** +- Embedded manifest in the tar matches the pointer bytes downloaded from object store (content-address cross-check) +- Every payload file's BLAKE3 hash matches the manifest (per-file integrity) +- Recovered cursor of the staged copy equals the manifest cursor (tail-loss detector) +- Backend identity + format generation match the local node's backend + +**What the transport layer does NOT verify:** +- Object store authorization (delegated to `object_store` credentials) +- Pointer freshness (callers decide whether the embedded cursor is recent enough) +- Whether the exporting node's fold was correct (garbage in, garbage out — the integrity guarantees cover transport, not source correctness) + ## Failure Modes -| Failure | Recovery | -| ------------------------------- | -------------------------------------------------------------- | -| `CursorExpired` | `watch_applied` resyncs (synthetic deletes) + falls back to the state-sync re-list; raw callers use `stale_keys()` | -| `WatchError` | Re-subscribe; watch stream dropped (NATS restart, reconnect) | -| `Timeout` on any op | CLOSE_WAIT connection; call `shutdown()` + `connect()` | -| `RevisionMismatch` on CAS | Re-read with `entry()`, resolve conflict, retry | -| `AlreadyExists` on `create()` | Read the live value, decide whether to proceed | -| Snapshot truncated tail | `load()` discards partial record; earlier state preserved | -| Snapshot mid-file CRC mismatch | `SnapshotError::Corrupted`; delete file, do full NATS replay | -| Snapshot wrong format version | `SnapshotError::InvalidFormat`; delete file, full NATS replay | -| `compact()` I/O error | Retry; if persistent, delete file and rebuild from NATS | -| Synadia Cloud stream limit | Raw API path treats as non-fatal; verifies with `get_key_value` | -| Tampered/torn artifact | `ArtifactInvalid` at import (hash/cursor gate); nothing lands at the destination — fetch another artifact | -| Artifact backend/format mismatch | `ArtifactInvalid` before any open; payload format markers re-checked by the engine itself | -| Export under churn (fjall) | Copy retries on GC'd files (×3); verify-by-reopen catches anything torn | -| Export/upload fails mid-round | Lease abandoned, local artifact deleted; next trigger elects a new node | -| Artifact cursor older than the log | `CursorExpired` on resume → full watch fallback; checkpoint more often than log retention | +| Failure | What Actually Happens | Recovery | +| ------------------------------------ | --------------------------------------------------------------------------- | ------------------------------------------------- | +| `CursorExpired` | `watch_applied` lists live keys, diffs fold, applies synthetic deletes, then falls back to state-sync re-list | Automatic; raw callers use `stale_keys()` manually | +| `WatchError` | Watch stream dropped (NATS restart, reconnect) | Re-subscribe | +| `Timeout` on any NATS op | CLOSE_WAIT half-dead TCP parks the `await` without this guard | Call `shutdown()` + `connect()` | +| `RevisionMismatch` on CAS | Concurrent writer won the race | Re-read with `entry()`, resolve, retry | +| `AlreadyExists` on `create()` | Key already present; caller's create was not exclusive | Read live value, decide whether to proceed | +| Snapshot truncated tail | `load()` discards partial final record; earlier records intact | Resume from recovered cursor; tail re-folded | +| Snapshot mid-file CRC mismatch | `SnapshotError::Corrupted` | Delete snapshot, full NATS replay | +| Snapshot wrong format version | `SnapshotError::InvalidFormat` | Delete snapshot, full NATS replay | +| `compact()` I/O error | Writer poisoned; subsequent writes return `Io` | Delete snapshot, rebuild from NATS | +| Synadia Cloud stream limit | Raw API path treats as non-fatal; verifies bucket with `get_key_value` | Non-fatal if bucket exists | +| Crash between payload upload and pointer swap | Old pointer remains fully consistent; payload orphaned | Next export round publishes new pointer; stale payload pruned after grace | +| Slow exporter after newer round published | `pointer_publish_allowed` returns false → `SupersededByNewer` | Lease abandoned, local artifact deleted; payload orphaned until prune | +| Tampered / torn artifact | `ArtifactInvalid` at import (hash + cursor gate); nothing written to destination | Fetch another artifact | +| Artifact backend/format mismatch | `ArtifactInvalid` before any open; engine format markers re-checked internally | Fetch another artifact | +| Export under churn — fjall copy | File GC'd mid-copy; retry ×3; verify-by-reopen catches anything torn | Abandon lease on persistent failure | +| Export/upload fails mid-round | Lease abandoned (CAS delete); local artifact deleted | Next trigger elects a new node | +| Artifact cursor older than NATS log | `CursorExpired` on resume → full watch fallback + stale-key resync | Checkpoint more often than log retention window | +| Corrupt lease value | Treated as expired: CAS-stolen by next acquirer | Non-fatal; one bad write cannot wedge the fleet | +| `prune()` I/O error | Stale payloads linger; warning logged | Retried next export round; correctness unaffected | ## Package Structure -| File | Purpose | -| ----------------- | ------------------------------------------------------------------------------------ | -| `src/kv.rs` | Core traits (`KvReader`, `KvWriter`, `KvWatcher`, `KvTtl`) and types (`KvEntry`, `KvUpdate`, `VersionToken`, `WatchCursor`, `KvError`) | -| `src/stores.rs` | `Connection`, `KvStore`, `StoreConfig`, `StorageType`, `ConnectionCapabilities` | -| `src/nats.rs` | NATS JetStream implementation; bucket creation, scan consumer lifecycle, timeout wrapping, Synadia Cloud workarounds | -| `src/snapshot.rs` | `SnapshotStore` trait; append-only log + `AppendLogSnapshot` (default backend): `SnapshotWriter`, `load()`, `replay_log()`, `compact_to_file()` | -| `src/snapshot_fjall.rs` | `FjallSnapshot`: on-disk `SnapshotStore` backed by fjall (`feature = "fjall"`) | -| `src/snapshot_rocksdb.rs` | `RocksDbSnapshot`: on-disk `SnapshotStore` backed by RocksDB (`feature = "rocksdb"`) | -| `src/snapshot_record.rs` | Shared `[ver_len][version][value]` value-record codec for the LSM backends | -| `src/applied.rs` | `watch_applied` cursor-after-apply combinator, generic over `SnapshotStore`: `WatchScope`, `BatchConfig`, cursor-expired stale-key resync | -| `src/lib.rs` | Re-exports all public types; no logic | -| `benches/` | Criterion benchmarks for snapshot write/checkpoint/load throughput and batch throughput | -| `tests/` | Integration tests (require live NATS) | +| File | What It Does | +| -------------------------- | ------------------------------------------------------------------------------------ | +| `src/kv.rs` | Core traits (`KvReader`, `KvWriter`, `KvWatcher`, `KvTtl`) and types (`KvEntry`, `KvUpdate`, `VersionToken`, `WatchCursor`, `KvError`) | +| `src/stores.rs` | `Connection`, `KvStore`, `StoreConfig`, `StorageType`, `ConnectionCapabilities` | +| `src/nats.rs` | NATS JetStream implementation; bucket creation, scan consumer lifecycle, timeout wrapping, Synadia Cloud workarounds, `check_resume_window` | +| `src/protocol.rs` | Pure-function protocol guards: `pointer_publish_allowed`, `payload_prunable`, `resume_window_ok` — called by both production code and the Stateright model | +| `src/snapshot.rs` | `SnapshotStore` trait; append-only log + `AppendLogSnapshot` (default backend): `SnapshotWriter`, `load()`, `replay_log()`, `compact_to_file()` | +| `src/snapshot_fjall.rs` | `FjallSnapshot`: on-disk `SnapshotStore` backed by fjall (`feature = "fjall"`) | +| `src/snapshot_rocksdb.rs` | `RocksDbSnapshot`: on-disk `SnapshotStore` backed by RocksDB (`feature = "rocksdb"`) | +| `src/snapshot_record.rs` | Shared `[ver_len][version][value]` value-record codec for the LSM backends | +| `src/artifact.rs` | `ExportManifest`, `ArtifactFile`, BLAKE3 integrity; stage-then-rename discipline; backend `export_to` + `import` (append-log, fjall, RocksDB) | +| `src/export_lease.rs` | `ExportLease`, `LeaseGuard`, `LeaseRecord`: fleet-wide at-most-one via embedded-expiry CAS | +| `src/transport.rs` | `ObjectStoreTransport`, `ArtifactTransport` trait, `run_export_round`: monotonic pointer swap, multipart upload, prune, content-addressed keys (`feature = "transport"`) | +| `src/applied.rs` | `watch_applied` cursor-after-apply combinator, generic over `SnapshotStore`: `WatchScope`, `BatchConfig`, cursor-expired stale-key resync, `ExportRequest` handling | +| `src/lib.rs` | Re-exports all public types; no logic | +| `benches/` | Criterion benchmarks: snapshot write/checkpoint/load throughput, batch throughput, ACK subject parsing | +| `tests/transport.rs` | Integration: upload/download/manifest round-trip, pointer swap, prune | +| `tests/transport_s3.rs` | Live MinIO / S3: CAS semantics (create, update, precondition) verified against real object stores | +| `tests/multi_export.rs` | Concurrent exporters, lease contention, fjall + RocksDB backends, multi-file artifacts under churn | +| `tests/resync.rs` | Cursor expiry, stale-key resync, NATS silent-clamp pinning across 3 backends × reader modes | +| `tests/model.rs` | Stateright exhaustive model: proves pointer monotonicity, no dangling pointer, no silent divergence; mutation tests prove each protocol guard is load-bearing | +| `tests/common/mod.rs` | Shared test helpers: ephemeral NATS server, temp dirs, assertion helpers | ## Configuration @@ -466,3 +677,28 @@ Credentials priority: `creds` > `creds_file` > URL-embedded > no auth. | Parameter | Effect | | ------------------- | ------------------------------------------------------------------------------------------- | | `compact_threshold` | Bytes appended since last compaction before `checkpoint()` returns `true`. Typical: 1–10 MB | + +### BatchConfig (watch_applied) + +| Field | Default | Effect | +| -------- | ------- | -------------------------------------------------------------------------- | +| `window` | 10 ms | Max time a batch stays open before flush | +| `max` | 100 | Max updates per batch before early flush | + +### ObjectStoreTransport / ExportLease + +| Setting | Default | Effect | +| ------------------------------------ | ------- | ----------------------------------------------------------------------------------- | +| `prefix` | — | Object-store key namespace; all payloads + pointer live under this prefix | +| `allow_non_atomic_pointer` | false | Opt-in for `file://` stores that lack CAS (dev/test only); FAILS CLOSED by default | +| `ExportLease::ttl` | — | Round period; embedded in `expires_at`; determines prune grace (`4 × ttl`) | + +**Prune grace is `4 × lease_ttl`:** protects payloads a concurrent publisher may still reference (one TTL) plus in-flight bootstrap readers (three more TTLs of headroom). Increase if bootstrap download latency approaches `ttl`. + +### Per-backend durability + +| Backend | Config | Default | Effect | +| ---------------- | ---------- | ---------- | ---------------------------------------------------------------------- | +| `AppendLogSnapshot` | — | page-cache | `checkpoint()` flushes to OS page cache; `compact()` fsyncs the new file | +| `FjallSnapshot` | `sync` | false | `true` → fsync on every `apply` batch; `false` → OS page cache | +| `RocksDbSnapshot` | `sync` | false | `true` → fsync on every `apply` WriteBatch; WAL always on | From 11e509e20c05426c3fd8ab87e86236ddac4e402d Mon Sep 17 00:00:00 2001 From: Jared Lunde Date: Thu, 11 Jun 2026 19:55:41 -0700 Subject: [PATCH 3/3] ci: self-heal cached minimal-profile toolchain before clippy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A cache-restored rustup 1.92 sometimes lacks the clippy/rustfmt components mise.toml requests, fast-failing the clippy step ("'cargo-clippy' is not installed") — a recurrent flake (also 2026-06-04). An idempotent `rustup component add` is ~2s when present and heals the poisoned cache. Co-Authored-By: Claude Fable 5 --- .github/workflows/ci.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index da2d4a6..ddace36 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -12,6 +12,12 @@ jobs: - uses: actions/checkout@v5 - uses: jdx/mise-action@v2 - uses: Swatinem/rust-cache@v2 + # Self-heal a cache-restored minimal-profile toolchain: a cached rustup + # without these components otherwise fails the clippy step with + # "'cargo-clippy' is not installed" (recurrent flake — the components + # requested in mise.toml don't survive every cache round-trip). + # Idempotent and ~2s when the components are already present. + - run: rustup component add clippy rustfmt - run: cargo clippy --all-targets --all-features -- -D warnings # The full feature matrix, or the fjall/rocksdb conformance suites and # the transport tier silently skip (default features exclude them all).