Association-Native Memory Layer for RuVector (Calyx review + ADR-272 tracking)

[ruvnet]

Summary

Tracking issue for ADR-272: Association-Native Memory Layer (Calyx-inspired), plus a technical review of Chris Royse's Calyx white paper and reference engine and how it maps onto RuVector.

• White paper: Calyx: An Association-Native Database and Its Path to Planetary-Scale Grounded Intelligence (ResearchGate pub. 408248277).
• Reference engine: https://github.com/ChrisRoyse/Calyx (Rust, edition 2024, BSL-1.1, pre-1.0).
• Companion paper: The Calculus of Association (ResearchGate pub. 405933676) — frozen embedders as designable measurement instruments, derived-data abundance, teleological constellations.

A first, dependency-free Rust reference implementation (crates/ruvector-calyx) and ADR land in the PR linked below.

---

Review: what Calyx is, and why it matters to RuVector

Calyx's central claim: one input should not collapse into one flattened vector. It should become a constellation — the same object measured through many frozen lenses (semantic, lexical, code, structural/domain, temporal, sensor), each kept as a distinct typed slot, never flattened. Relationships are then derived between slots, grounded against real-world anchors, scored for signal contribution, and gated so the system fails closed instead of answering from "semantic fog".

It organizes this around four verbs — measure, count, differentiate, compose — and eleven subsystems: Aster (LSM storage), Forge (SIMD/CUDA math), Registry (content-addressed lenses), Sextant (fusion search + BM25), Loom (cross-lens associations), Assay (information bits per lens), Lodestar (grounding kernels), Ward (fail-closed guard), Ledger (hash-chained provenance), Anneal (reversible self-optimization), Oracle (grounded prediction). Three trust principles: grounding is mandatory, no flattening, fail closed.

Why it's relevant (not a competitor — a pattern to absorb)

• Validates "memory is the moat." The embedding model is replaceable; the measured association substrate + governance is the product. Same thesis as MetaHarness, pushed to the data layer.
• Composes with Darwin Mode (ADR-266/271). MetaHarness already evolves planners/routers/memory-policy. Calyx says it should also route lenses, not just models — "which lenses inspect this problem, which slots matter, which disagreements are informative, which grounding is required, which model is cheapest once the evidence set exists."
• Maps onto existing RuVector primitives. Per-lens rankings → HNSW/IVF; cross-lens graph → ruvector-graph/ruvector-mincut association edges; grounding/provenance → governance story for Cognitum One.
• Enterprise governance language. Grounded evidence path + model lineage + retrieval provenance + fail-closed behavior — directly applicable to manufacturing, telecom, elder-care, security, legal/finance, coding agents.
• Where we're ahead: Calyx is a pre-1.0 white-paper engine; RuVector already ships HNSW, min-cut, graph memory, Darwin, and edge/RF perception. Where it challenges us: RuVector should grow from a vector database into an association database (multi-slot records, lens manifests, cross-slot graph, anchors, signal density, guard profiles, provenance ledger, panel routing).

Concept → RuVector mapping (implemented in ruvector-calyx)

Calyx	RuVector translation
Constellation	Constellation — one object, many typed slots, never flattened
Lens	LensManifest — content-addressed (name/version/kind/dims)
Cross-term (Loom)	loom::{min,mean}_agreement, top_dissenter (disagreement is signal)
Signal (Assay)	assay::signal_density — MI bits per µs of cost
Fusion (Sextant)	fusion::weighted_rrf — Reciprocal Rank Fusion
Grounding (Lodestar)	Anchor (accepted-answer/passed-test/sensor/citation/reward)
Guard (Ward)	ward::adjudicate → Answer/Refuse(reason) (fail closed)
Provenance (Ledger)	Ledger — hash-chained, replayable
Self-opt (Anneal)	anneal_weights — reversible SA over fusion weights (lens routing)

---

Benchmark result (ADR-272 acceptance — all PASS)

Deterministic calyx-bench vs single-embedding RAG on an adversarial multi-lens corpus (semantic fog within topic; discriminating signal in minority lexical/structural lenses; unanswerable queries test abstention):

Metric	Single-embedding	Calyx multi-lens	Target	Result
Grounded answer accuracy	6.7%	99.2%	≥ +15 pp	+92.5 pp ✓
Recall@10 (answerable)	51.7%	100.0%	≥ +10 pp	+48.3 pp ✓
Unsupported claims	172	0	≥ −50%	−100% ✓
Replayable provenance	n/a	180/180	100%	✓
Anneal (reversible)	—	Δutility +46.2	improves	✓

Assay ranks the cheap lexical lens highest by signal density; Anneal reversibly converges weights onto it, raising accuracy to 100% while dropping the expensive semantic lens — the cost-shifting thesis in miniature.

---

Tracking checklist (ADR-272 follow-ups)

• ADR-272 written (docs/adr/ADR-272-association-native-memory-layer.md)
• crates/ruvector-calyx reference implementation (dependency-free, MIT/Apache-2.0, clean-room)
• Deterministic benchmark + acceptance test (calyx-bench), 20 unit tests, clippy clean
• Back search_lens with ruvector-core HNSW + ruvector-spann partitions
• Map the Loom cross-lens graph onto ruvector-graph/ruvector-mincut association edges (agreement/disagreement searchable)
• Swap FNV provenance hash for BLAKE3 + signed ledger checkpoints
• Wire lens routing into MetaHarness Darwin as an evolvable gene (ADR-266)
• Add a sensor/RF lens (RuView: Wi-Fi CSI, mmWave) for non-text constellations and cross-modal disagreement detection
• Re-validate against a real multi-lens corpus under the ADR-267 SOTA protocol

Note: Calyx's reference engine is BSL-1.1; ruvector-calyx is an independent clean-room implementation of the published architecture pattern and does not copy Calyx source.

[ruvnet]

Update 1 — Adaptive routing + calibration (shipped in #628)

Sequencing chosen deliberately: route → calibrate → stress-test → graph → HNSW. Calibration before speed, because HNSW can hide bad retrieval and calibration tells you where speed/graph work even matters.

What landed (calibrate.rs, routing.rs, calyx-routing-bench):

• Per-query confidence from 4 signals (score margin, cross-lens agreement, grounding/source density, contradiction) → histogram Calibrator (reliability map) → ECE + abstention precision/recall.
• Adaptive router: consult lenses cheapest-first, stop early on calibrated confidence, escalate only when unsure, abstain if the full panel is still not confident. A min_lenses_to_answer gate enforces "don't trust a lone lens." Every decision emits a Witness (lenses consulted, tops, signals, confidence, action, escalation reason, cost/latency).

Result (test split, adaptive vs brute-force — all acceptance targets PASS):

Mode	Grounded acc	Cost µs	Latency µs	ECE	Abstain-P
brute-force (all lenses)	93.8%	15.5	15.5	0.000	88.9%
static (2 cheap lenses)	100.0%	3.5	3.5	0.000	100%
adaptive	100.0%	7.5	7.5	0.000	100%

Targets: accuracy loss −6.2pp (adaptive higher), cost −51.6%, latency −51.6%, ECE 0.000, abstention precision 100%.

Adaptive strictly dominates brute-force (higher accuracy at <½ the cost) because the expensive semantic lens is the low-signal "fog" lens here — always consulting it costs more and can flip a correct answer. Honest nuance the calibration surfaced: a well-chosen static policy is even cheaper, which you can only justify hard-coding after measuring that semantic is low-signal.

Checklist:

• Adaptive lens routing + calibration (route → calibrate)
• Back search_lens with HNSW + ruvector-spann
• Cross-lens Loom graph over ruvector-graph/ruvector-mincut
• BLAKE3 + signed ledger checkpoints
• Lens routing as a MetaHarness Darwin gene (ADR-266)
• Sensor/RF lens (RuView)
• Real-corpus re-validation (ADR-267)

Next per the agreed sequence: harder, realistic eval (stronger baselines, correlated lens noise, ablations) to stress-test these gains before building the graph/min-cut layer.

---

Generated by Claude Code

[ruvnet]

Update 2 — Three novel capabilities (shipped in #628)

Beyond recombining known techniques, three fresher capabilities — one with a theorem behind it. New bin calyx-novel-bench (all acceptance PASS), 31 lib tests, clippy clean.

A. Conformal cross-lens abstention (the defensible one)

Replaces the hand-tuned guard threshold with conformal risk control (Angelopoulos et al. 2022) over the cross-lens agreement score → a distribution-free, finite-sample guarantee that the confident-wrong-answer rate ≤ α. Monte-Carlo (200 splits):

α	threshold	coverage	MC mean risk
hand-tuned (0.45)	0.45	56.3%	11.3% (uncontrolled)
0.10	0.471	51.3%	9.5% ≤ 0.10 ✓
0.05	0.540	49.3%	4.2% ≤ 0.05 ✓
0.01	0.987	38.7%	0.0% ≤ 0.01 ✓

"Fail closed" becomes a dial with a proof. Applying CRC to multi-lens routing/abstention specifically is close to open ground.

B. Disagreement as a query primitive

find_conflicts(lens_a, lens_b) ranks records by cross-lens disagreement (1 − Jaccard of per-lens neighbourhoods) — a query a flat vector DB can't express. Planted 6 conflicts → surfaced 6/6 (100% precision).

C. Learning the router from the ledger

Offline first-visit MC control over exploratory Witness trajectories learns a stop/continue policy. It beats both fixed baselines (utility 0.661 vs static 0.620 / brute −0.057): learns to bail after one lens on likely-unanswerable queries and abstain.

Honesty note: most ingredients are established (multi-vector records, RRF, histogram calibration, conformal prediction, MC control); the contribution is the synthesis over frozen heterogeneous lenses + (A)'s guarantee. Benchmarks are synthetic/illustrative — real-corpus validation (ADR-267) is still the bar for a research claim.

Checklist:

• Adaptive lens routing + calibration
• Conformal cross-lens abstention (distribution-free guarantee)
• Disagreement-as-query primitive
• Ledger-learned routing policy
• Back search_lens with HNSW + ruvector-spann
• Cross-lens Loom graph over ruvector-graph/ruvector-mincut
• BLAKE3 + signed ledger checkpoints
• Real-corpus re-validation (ADR-267) ← the honest next bar

Suggested next: harder, realistic eval on a real multi-lens corpus to pressure-test the conformal exchangeability assumption and the synthetic gains before building the HNSW/graph scale-out.

---

Generated by Claude Code

[ruvnet]

Update 3 — Real-data path (.calyx + CodeSearchNet, shipped in #628)

Answers "how do we use real data for real benchmarks." Key decision: separate lens production from the association layer. Embedding models/BM25 need GPUs+network; the layer we benchmark (fusion, routing, conformal, disagreement) is pure Rust. So real embeddings are computed offline once and serialized — the crate loads precomputed vectors + ground truth with zero deps (ann-benchmarks pattern).

Shipped:

• .calyx v1 binary format + dependency-free std::io loader (corpus.rs): lens manifests, records (slots + grounding anchors), queries with relevance labels. n_relevant==0 = unanswerable (abstaining is correct) → makes conformal risk meaningful. Round-trip tested.
• calyx-real-bench: loads a .calyx (or synthesizes a CodeSearchNet-shaped stand-in and round-trips it), reports MRR@10 / Recall@1,10 / nDCG@10 (single-lens vs fusion) + conformal risk + code↔doc disagreement.
• tools/build_codesearchnet.py: the one model/network step — code lens (joint NL↔code model, e.g. st-codesearch), doc lens (text model, e.g. BGE), dependency-free hashed-token lexical lens; docstrings → NL queries; held-out golds → unanswerable. Emits byte-exact .calyx.

Why CodeSearchNet: code vs docstring are genuinely different views → real cross-lens disagreement (stale/incorrect docstrings), and it's a standard code-search retrieval benchmark with published MRR.

Stand-in results (synthetic, CodeSearchNet-shaped — not a real-data claim; each single lens made individually weak so only fusion pins the function):

System	MRR@10	R@1	nDCG@10
single-lens [code]	0.185	0.039	0.292
single-lens [doc]	0.173	0.039	0.275
single-lens [lexical]	0.204	0.078	0.314
fusion	0.492	0.372	0.555

Fusion +0.288 MRR over best single; conformal risk ≤ α; stale-docstring disagreement 100% precision. Stand-in numbers validate the pipeline — real numbers need the converter run on CodeSearchNet.

Produce real numbers:

pip install datasets sentence-transformers numpy
python crates/ruvector-calyx/tools/build_codesearchnet.py --lang python --n 5000 --out corpus.calyx
cargo run --release -p ruvector-calyx --bin calyx-real-bench -- corpus.calyx

Validity on real runs: official splits (no leakage), bootstrap CIs, conformal exchangeability (cal/test same distribution).

Checklist:

• Real-data path: .calyx format + loader + real-bench + CodeSearchNet converter
• Run converter on real CodeSearchNet → publish real MRR/nDCG/conformal numbers
• Back search_lens with HNSW + ruvector-spann
• Cross-lens Loom graph over ruvector-graph/ruvector-mincut
• BLAKE3 + signed ledger checkpoints

---

Generated by Claude Code