Skip to content

Commit dba60b3

Browse files
committed
catalog: align tier1 docs with sync compactor path
1 parent 8368f5c commit dba60b3

4 files changed

Lines changed: 36 additions & 20 deletions

File tree

crates/arco-catalog/src/lib.rs

Lines changed: 24 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -13,24 +13,33 @@
1313
//!
1414
//! The catalog uses a two-tier consistency model:
1515
//!
16-
//! - **Tier 1 (Strong Consistency)**: Low-frequency DDL operations (create/drop)
17-
//! use atomic snapshot + manifest writes for immediate consistency
16+
//! - **Tier 1 (Strong Consistency)**: API mutations append tier1 DDL events and
17+
//! then synchronously compact/publish immutable manifest snapshots via
18+
//! `CatalogWriter` + `SyncCompactor`
1819
//! - **Tier 2 (Eventual Consistency)**: High-volume operational facts (lineage events,
1920
//! run metadata) are written to an append-only event log and periodically compacted
2021
//!
22+
//! `Tier1Writer` still exists for low-level/bootstrap flows and tests, but it is no
23+
//! longer the primary API write path.
24+
//!
2125
//! ## Storage Layout
2226
//!
2327
//! All catalog data is scoped to tenant/workspace isolation boundaries using
2428
//! key=value path segments (grep-friendly, self-documenting):
2529
//!
2630
//! ```text
2731
//! tenant={tenant}/workspace={workspace}/
28-
//! ├── manifests/ # Tier 1: Multi-file manifest structure
29-
//! │ ├── root.manifest.json # Root manifest (points to domain manifests)
30-
//! │ ├── catalog.manifest.json # Catalog state (locked writes)
31-
//! │ ├── lineage.manifest.json # Lineage state (locked writes)
32-
//! │ ├── executions.manifest.json # Execution state (compactor writes)
33-
//! │ └── search.manifest.json # Search state (locked writes)
32+
//! ├── manifests/
33+
//! │ ├── root.manifest.json # Stable entrypoint for readers
34+
//! │ ├── catalog.manifest.json # Legacy compatibility mirror
35+
//! │ ├── catalog.pointer.json # Immutable snapshot visibility gate
36+
//! │ ├── catalog/{manifest_id}.json # Immutable catalog snapshots
37+
//! │ ├── lineage.manifest.json # Legacy compatibility mirror
38+
//! │ ├── lineage.pointer.json
39+
//! │ ├── lineage/{manifest_id}.json
40+
//! │ ├── search.manifest.json # Legacy compatibility mirror
41+
//! │ ├── search.pointer.json
42+
//! │ └── search/{manifest_id}.json
3443
//! ├── locks/
3544
//! │ ├── catalog.lock.json # Distributed lock per domain
3645
//! │ ├── lineage.lock.json
@@ -47,21 +56,18 @@
4756
//! ## Example
4857
//!
4958
//! ```rust,ignore
50-
//! use arco_catalog::Tier1Writer;
59+
//! use std::sync::Arc;
60+
//!
61+
//! use arco_catalog::{CatalogWriter, Tier1Compactor};
5162
//! use arco_core::ScopedStorage;
5263
//!
5364
//! // Create workspace-scoped storage
5465
//! let storage = ScopedStorage::new(backend, "acme-corp", "production")?;
66+
//! let compactor = Arc::new(Tier1Compactor::new(storage.clone()));
5567
//!
56-
//! // Initialize catalog manifests (idempotent)
57-
//! let writer = Tier1Writer::new(storage);
58-
//! writer.initialize().await?;
59-
//!
60-
//! // Update catalog with CAS semantics
61-
//! let commit = writer.update(|manifest| {
62-
//! manifest.snapshot_version = 1;
63-
//! Ok(())
64-
//! }).await?;
68+
//! // API writes go through the facade and publish immutable snapshots synchronously.
69+
//! let writer = CatalogWriter::new(storage).with_sync_compactor(compactor);
70+
//! // writer.create_namespace(...).await?;
6571
//! ```
6672
6773
#![forbid(unsafe_code)]

crates/arco-catalog/src/tier1_writer.rs

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -160,6 +160,10 @@ impl Tier1Writer {
160160
///
161161
/// Returns an error if the lock cannot be acquired, manifests are missing, or
162162
/// if the CAS update fails after all retries.
163+
#[deprecated(
164+
since = "0.1.0",
165+
note = "use CatalogWriter + SyncCompactor for API writes, or update_locked for low-level lock-held flows"
166+
)]
163167
pub async fn update<F>(&self, mut update_fn: F) -> Result<CommitRecord>
164168
where
165169
F: FnMut(&mut CatalogDomainManifest) -> Result<()>,
@@ -539,6 +543,7 @@ fn sha256_prefixed(bytes: &[u8]) -> String {
539543
}
540544

541545
#[cfg(test)]
546+
#[allow(deprecated)]
542547
mod tests {
543548
use super::*;
544549
use async_trait::async_trait;

crates/arco-catalog/tests/concurrent_writers.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
//! These tests verify the catalog's distributed locking and CAS mechanisms work
44
//! correctly under contention.
55
6-
#![allow(clippy::unwrap_used, clippy::expect_used)]
6+
#![allow(deprecated, clippy::unwrap_used, clippy::expect_used)]
77

88
use std::sync::Arc;
99
use std::sync::atomic::{AtomicU32, Ordering};

docs/adr/adr-032-immutable-manifest-pointers.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
## Status
44

5-
Proposed
5+
Accepted
66

77
## Context
88

@@ -18,6 +18,11 @@ failover. We need a commit primitive that is:
1818

1919
Arco adopts an immutable manifest snapshot model with a mutable pointer per domain.
2020

21+
Catalog domain writers and the orchestration micro-compactor now use this
22+
snapshot+pointer model for visible state publication. Orchestration orphan
23+
reconciliation/GC for abandoned snapshots and L0 directories remains follow-up
24+
operational work.
25+
2126
### Storage layout
2227

2328
Catalog domains:

0 commit comments

Comments
 (0)