research(nightly): namespace-partitioned multi-agent HNSW memory

[ruvnet]

Summary

• Topic: Namespace-Partitioned Multi-Agent HNSW Memory
• Slug: ns-partitioned-ann
• ADR: docs/adr/ADR-272-ns-partitioned-ann.md
• Crate: crates/ruvector-ns-partition
• Research doc: docs/research/nightly/2026-07-10-ns-partitioned-ann/README.md
• Gist: docs/research/nightly/2026-07-10-ns-partitioned-ann/gist.md

Multi-agent AI systems each need isolated vector memory but must also retrieve knowledge across agent boundaries. This nightly adds a pure-Rust PoC benchmarking three namespace strategies on a 6 000-vector, 8-namespace workload.

Key findings (all measured, no aspirational numbers):

Variant	Single-NS recall	Single-NS latency	Cross-NS recall	Cross-NS latency
GlobalFlat (post-filter)	97.4%	4 390 µs	42.7%	301 µs
Partitioned (per-NS HNSW)	96.3%	202 µs	97.5%	1 446 µs
HierarchicalNS (centroid router)	96.2%	184 µs	52.6%	691 µs

GlobalFlat (the strategy used by Pinecone namespaces, Chroma collections, pgvector filtered search) achieves only 42.7% cross-namespace recall at ef=64 — consistent with ACORN (SIGMOD 2024) measurements of post-filter degradation at ~12.5% selectivity.

Per-namespace HNSW (Partitioned) achieves 97.5% cross-NS recall and is 21.8× faster for single-agent queries than GlobalFlat, with no extra memory overhead (4 779 KB vs 4 988 KB).

What's Included

1. Working Rust PoC (crates/ruvector-ns-partition/, zero external deps):

   • src/hnsw.rs: 240-line minimal HNSW with deterministic LCG, greedy descent, diversity-heuristic pruning
   • src/lib.rs: NamespacedIndex trait + GlobalFlat, Partitioned, HierarchicalNS implementations
   • src/bin/benchmark.rs: deterministic benchmark with brute-force oracle and acceptance gate

2. 7 unit tests — all passing

3. ADR-272 proposing Partitioned as the recommended production strategy with security integration plan (capgated + proof-gate)

4. Research document with 2026 SOTA survey, memory math, failure modes, 10–20 year agent OS thesis

5. SEO gist with public technical article format

Test Plan

• cargo build --release -p ruvector-ns-partition — Finished in 2m 24s
• cargo test -p ruvector-ns-partition — 7 passed, 0 failed
• cargo run --release -p ruvector-ns-partition --bin benchmark — PARTIAL PASS (Partitioned passes all acceptance criteria)
• cargo fmt -p ruvector-ns-partition — clean
• No external dependencies
• All files under 500 lines

Acceptance Gate Result

GlobalFlat      single_recall=97%  cross_recall=43%  lat=4390µs  → FAIL (cross-NS recall below threshold)
Partitioned     single_recall=96%  cross_recall=97%  lat=202µs   → PASS
HierarchicalNS  single_recall=96%  cross_recall=53%  lat=184µs   → FAIL (cross-NS recall with route_k=4/8)

ACCEPTANCE: PARTIAL — Partitioned variant passes all criteria

Ecosystem Connections

• ruvector-capgated (ADR-268): per-namespace capability mask for access control
• ruvector-proof-gate (ADR-227): proof-gated cross-namespace writes
• ruvector-mincut: namespace graph compaction
• mcp-brain: MCP tool surface (memory_ns_search_*)
• ruFlo: on_ns_recall_drop / on_ns_size_exceeds triggers
• rvf: per-namespace portable bundle export

---

This branch should either become a production RuVector capability (NamespacedIndex trait merged into ruvector-core) or a falsified research path with useful evidence.

---

Generated by Claude Code