feat: export/import pipeline — artifact codec, per-backend checkpoint, lease, transport, v0.4.0#5
Merged
Merged
Conversation
…s, benches, docs) Found uncommitted in the working tree before the export/import feature work began: the snapshot_record codec split, backend benches, nats.rs changes, and doc updates from the unreleased 0.3.3 rocksdb-backend work. Checkpointed verbatim so feature commits stay separable. The rocksdb backend module itself lands with the export/import commit that extends it (the file mixes both). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…append-log export/import An artifact is a transferable copy of a durable fold: backend files under data/ plus a MANIFEST.json carrying per-file BLAKE3 checksums, backend identity + on-disk format generation, and the watch cursor the payload is exactly consistent with. Export stages, seals (manifest written last, fsynced), and atomically renames — an artifact that exists is complete. Import verifies everything (transport is untrusted), opens the staged copy, gates on cursor equality, and renames — a bad artifact never becomes a fold. Breaking: SnapshotStore gains required cursor() and export_to(); SnapshotError gains ArtifactInvalid; WatchCursor is now PartialEq + Eq. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Export = rocksdb::checkpoint::Checkpoint into the artifact stage (engine flushes the memtable, hardlinks SSTs — consistent by construction), then verify-by-reopen + cursor-equality gate, hash after the verify handle drains background work, seal, rename. Import does not gate on backend_version: RocksDB reads older on-disk formats; its own open is the arbiter. Includes the previously-uncommitted RocksDbSnapshot backend itself (found in-tree from the unreleased 0.3.3 work); the inherent cursor() accessor is replaced by the trait method. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fjall has no checkpoint API, so export is built from parts: persist(SyncAll) (journal complete; &mut self means nothing commits after), best-effort quiesce (rotate memtables, drain flushes/compactions, bounded 10s — uses doc(hidden)-but-pub fjall APIs, compile-checked), copy the DB dir (hardlink immutable tables/blobs with copy fallback, byte-copy journal+metadata+lock, retry ×3 on background-GC races), then verify by reopening the copy — fjall's own recovery is the consistency oracle and cursor equality is a complete tail-loss detector since every apply commits cursor+data in one batch. Hash runs only after the verify handle drops. Inherent cursor() replaced by the trait method. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ckends Round-trip (cursor + state equality), import-then-tail-replay equals the continuous fold, tampered/missing/undeclared payload rejection with nothing landing at the destination, wrong backend / schema / format-generation / doctored-cursor rejection, non-empty-destination refusal (artifact survives), empty-fold round-trip. rocksdb deliberately skips the format-generation gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e watch_applied watch_applied owns its snapshot store by value, so consumers could never export a live fold. It now takes an optional mpsc::Receiver<ExportRequest>; each request is handled between flushes (pending batch flushed first), so the artifact's embedded cursor is exactly the applied cursor. The export runs on a blocking task with the store moved in and taken back — the same offload as the flush path. Export errors reply to the requester only; the watch keeps running. Dropping the sender disarms the arm. Breaking: watch_applied gains the exports parameter (pass None). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A single KV key coordinates the checkpoint loop across every replica of a fold: acquisition is create-only (one winner by construction), the lease lifetime is an expires_at embedded in the value (portable to any KvWriter — no per-message TTL, no server-version or bucket-flag requirements), and an expired/corrupt lease is taken over with a CAS update so the steal race also has one winner. The ttl IS the round period; complete() publishes the exported cursor on the key, making it the fleet-visible last-export record. Clock-skew worst case is a duplicate export — safe, the lease is dedup, not a correctness gate. Integration-tested against a real nats-server: 8-way concurrent acquire has exactly one winner; held lease blocks; expiry allows takeover; completion is fleet-visible. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t_round Wire format: plain tar of the artifact dir (payload files are already lz4/zstd-compressed) at <prefix>/<key>, manifest duplicated as a sibling <key>.manifest.json uploaded LAST (the remote completeness marker, and a cheap cursor peek without downloading the payload). Download cross-checks the embedded manifest against the sibling; the backend import re-verifies every file hash regardless — transport is untrusted. ObjectStoreTransport wraps any object_store backend (S3/GCS/Azure/local-fs; feature 'transport'), multipart-streams uploads memory-bounded, and unpacks downloads stage-then-rename so a torn download never lands at the dest. run_export_round composes the at-most-once round: lease -> export through the watch_applied channel -> upload -> publish completion (only after the upload) -> delete the local artifact (transience enforced). Failures abandon the lease (new LeaseGuard::abandon) so the fleet retries promptly instead of waiting out the ttl. Per-backend import_remote bootstraps a fold from the remote artifact in one call. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ons; MinIO tier tests/bootstrap.rs (fjall + rocksdb): node A folds a real NATS bucket under churn, exports through the watch loop's request channel, node B imports the artifact and resumes from the embedded cursor. The load-bearing assertions are delta-only: B is DELIVERED exactly the post-export tail (counted), the first delivered revision is cursor+1 (no overlap, no gap), and a delete in the tail reaches B — convergence alone would mask a full replay. tests/transport_s3.rs: the same full loop through run_export_round against a real S3 API — a throwaway MinIO per test (mise-installed binary + mc for bucket creation, no Docker; same pattern as TestNats) — plus a ~25 MiB incompressible artifact exercising real multipart upload. tests/common/ holds the shared TestNats/TestMinio guards; mise.toml pins minio (asdf backend; ubi rejects MinIO's extension-less release assets) and mc (aqua). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n; v0.4.0 CI now tests --features fjall,rocksdb,transport (previously default features only, silently skipping the fjall/rocksdb conformance suites on every PR) plus a default-features pass, and clippy covers --all-features. MinIO + mc land on the runner via the existing mise-action — no Docker. ARCHITECTURE.md gains the Export/Import section (artifact anatomy, the cursor-consistency invariant, per-backend mechanics, storage accounting, lease + transport), three Design Decisions entries, and Failure Modes rows. v0.4.0 breaking inventory for downstream bumps: - SnapshotStore gains required cursor() and export_to() - watch_applied gains the exports: Option<mpsc::Receiver<ExportRequest>> parameter (pass None) - SnapshotError gains ArtifactInvalid (breaks exhaustive matches) - inherent FjallSnapshot::cursor()/RocksDbSnapshot::cursor() removed in favor of the trait method (returns owned WatchCursor) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n guide Replace artifact-era latency claims with measured settled-tree data from the 500M-route bench (percentiles, not means): cold gets p50 542us/292us (fjall/rocksdb), absent-key ~400ns via restored bottom-level filters, settle() costs (fjall ~19min full rewrite + 2x disk transient; rocksdb ~40s drain). BACKENDS.md collects the full cross-scale dataset and the backend-selection heuristic in one place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SnapshotStore::{cursor,export_to},MANIFEST.jsonwith per-file BLAKE3 checksums, backend identity/format-generation, and the watch cursor the payload is exactly consistent with). Stage → seal → atomic rename so a present artifact is always complete; import verifies everything before any rename.CheckpointAPI (engine flushes memtable, hardlinks SSTs), verify-by-reopen + cursor-equality gate, hash after verify handle drains background work.persist(SyncAll)→ best-effort quiesce (rotate memtables, drain flushes/compactions, bounded 10s) → hardlink-copy → verify-by-reopen.watch_appliedexport channel —ExportRequestlets consumers export a live fold between flushes; artifact cursor is exactly the applied cursor. Breaking:watch_appliedgains anexports: Option<mpsc::Receiver<ExportRequest>>parameter.ExportLease— create-only KV key coordinates at-most-one export per round fleet-wide; expired leases stolen via CAS;complete()publishes the cursor as the fleet-visible last-export record.ArtifactTransport/ObjectStoreTransport/run_export_round— plain-tar wire format (<prefix>/<key>+ sibling.manifest.jsonuploaded last as completeness marker); multipart-streaming upload; stage-then-rename download;run_export_roundcomposes lease → export → upload → publish → delete local artifact.--features fjall,rocksdb,transport+ default-features +clippy --all-features; MinIO + mc via mise.BACKENDS.mdwith measured benchmark numbers;ARCHITECTURE.mdexport/import section and design decisions.v0.4.0 breaking changes
SnapshotStoregains requiredcursor()andexport_to()watch_appliedgainsexports: Option<mpsc::Receiver<ExportRequest>>NoneSnapshotErrorgainsArtifactInvalidFjallSnapshot::cursor()/RocksDbSnapshot::cursor()removedTest plan
default,fjall,rocksdb,transport,clippy --all-features)🤖 Generated with Claude Code