ADR-057: Replace sql.js with RVF native storage backend

[ruvnet]

Summary

Replace the 18MB sql.js WASM SQLite hard dependency with RuVector Format (RVF) across @claude-flow/shared, @claude-flow/memory, and @claude-flow/embeddings.

ADR: v3/implementation/adrs/ADR-057-rvf-native-storage-backend.md
Branch: feat/adr-057-rvf-storage-backend

Problem

• sql.js is 18MB — the single largest hard dependency in the core install path
• sql.js provides no vector indexing — uses brute-force O(n) cosine similarity
• Adds 35s to Docker cold start and inflates images by ~18MB even after pruning

Proposed Solution

Replace sql.js with RVF (@ruvector/rvf), which provides:

Metric	sql.js	RVF
Package size	18MB WASM	52KB WASM / ~2MB native
Vector search	O(n) brute-force	O(log n) HNSW — 150x-12,500x faster
Quantization	fp32 only	fp16/int8/int4/binary — 2-8x memory savings
Crash safety	Manual export/save	Append-only (no WAL)
Progressive loading	N/A	70% recall on first query

Migration

• Automatic: Transparent migration on first access (legacy .db/.json → .rvf)
• Manual: ruflo migrate run --storage, ruflo migrate rollback --storage
• Backward compatible: Legacy formats permanently readable; .bak files preserved
• 3 formats supported: sql.js .db, better-sqlite3 .db, JSON .json

Implementation Phases

• Phase 1 (Week 1-2): RvfBackend implementing IMemoryBackend interface
• Phase 2 (Week 2-3): RvfEventLog implementing IEventStore in @claude-flow/shared
• Phase 3 (Week 3): RvfEmbeddingCache for @claude-flow/embeddings
• Phase 4 (Week 3-4): CLI migration commands (migrate status/run/rollback/validate --storage)
• Phase 5 (Week 4): Move sql.js to optionalDependencies, update Docker images

Impact

• -18MB from core install size
• 150x-12,500x faster vector search
• 2-8x memory reduction via quantization
• Docker lite image: 324MB → ~306MB

Related

• ADR-053: AgentDB v3 Controller Activation
• ADR-054: RVF-Powered Plugin Marketplace
• ADR-055: AgentDB Controller Bug Remediation
• ADR-056: agentic-flow v3 Integration

[ruvnet]

ADR-057 RVF Storage Backend — Hardening Complete

Branch: feat/adr-057-rvf-clean (640e1ae)
Tests: 193/193 passing (including 64 capability verification tests)
TypeScript: 0 errors across memory, embeddings, and shared packages

What shipped

Security hardening:

• Atomic writes (tmp + rename) across all 3 backends (rvf-backend, embedding-cache, event-log)
• Path validation — reject traversal attempts
• Input validation at system boundaries (embedding service, event log)

Performance fixes:

• Persist lock prevents concurrent flush races
• Stack overflow fix for large dataset serialization (manual loop vs Math.min(...spread))
• timer.unref() prevents process hangs on shutdown
• O(1) eviction lookup in embedding cache (reverse hash→text map)

New: RVF Migration Utility (rvf-migration.ts):

• Bidirectional: JSON ↔ RVF, SQLite → RVF
• Auto-detect format by magic bytes (RVF\0, SQLi, [{)
• Batch processing (default 500), progress callbacks
• RvfMigrator.autoMigrate(source, target) for one-call migration

Backward/forward compatibility:

• Embedding cache format v2 (separate createdAt/accessedAt timestamps)
• v1 files read transparently with automatic upgrade
• PersistentSona auto-persists patterns and trajectories before eviction

New exports from @claude-flow/memory:

• HnswLite, cosineSimilarity, HnswSearchResult
• RvfMigrator, RvfMigrationOptions, RvfMigrationResult

Files changed (9)

File	Change
memory/src/rvf-backend.ts	Atomic writes, persist lock, default namespace, stack overflow fix
memory/src/persistent-sona.ts	Auto-persist patterns/trajectories
memory/src/index.ts	Added HnswLite + RvfMigrator exports
memory/src/rvf-migration.ts	NEW — bidirectional migration utility
embeddings/src/rvf-embedding-cache.ts	Atomic writes, O(1) eviction, format v2
embeddings/src/rvf-embedding-service.ts	Input/dimension validation
shared/src/events/rvf-event-log.ts	Atomic writes, path/input validation
tests/rvf-migration.test.ts	NEW — 15 migration tests
tests/rvf-capability-verify.test.ts	NEW — 64 comprehensive capability tests

Remaining for this issue

• PR to main
• Docker optimization (separate concern, plan drafted)