PixelRAG → Rust port on ruvector: text + visual + real-time video RAG (no stubs)#610
Open
ruvnet wants to merge 28 commits into
Open
PixelRAG → Rust port on ruvector: text + visual + real-time video RAG (no stubs)#610ruvnet wants to merge 28 commits into
ruvnet wants to merge 28 commits into
Conversation
… detector, receipts (ADR-260 Phase 1) Pure-Rust, dependency-light, deterministic learned-optical-frontend core: - complex/fft: in-house radix-2 2D FFT (bit-reproducible, no external FFT lib) - field/mask: image->scalar field, phase-only learned mask (identity/random/lens) - propagate: Fresnel, Fraunhofer, angular-spectrum scalar diffraction - detector: intensity capture + seeded shot/read noise, binning, quantization - metrics: MSE/PSNR, compression ratio, frame-similarity, spectrum embedding - receipt: BLAKE3-bound experiment receipts + verify (determinism invariant §21) 21 unit tests + doctest passing. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_01PjRKJMFe6yoNY3SMVEieHy
…ss (ADR-260 Phase 2/4) - synthetic: deterministic 4-class shape dataset (no MNIST per ADR-260 §20.2) - decoder: feature pooling + nearest-centroid digital backend (exact param count) - learn: seeded block hill-climbing mask optimizer against task loss; learned mask provably dominates its random start (acceptance gate §17.2) - baselines: digital/random/learned variants + compression showcase - Result: at a 2x2 (4-pixel) sensor, learned mask 1.00 vs random 0.80 vs digital 0.65 test accuracy — same task, 64x fewer sensor pixels (§16.3) Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_01PjRKJMFe6yoNY3SMVEieHy
…mentation (ADR-260) Stub crates registered as workspace members so each is independently buildable/testable while the implementation swarm fills them in. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_01PjRKJMFe6yoNY3SMVEieHy
…vacy, CLI demos (ADR-260 Phases 2-4) photonlayer-ruvector (22 tests): 32-dim experiment embeddings (mask histogram + frame spectrum), cosine nearest-experiment recall, Fiedler-spectral pass/fail boundary analysis, mask-family coherence gates, verifying receipt store. photonlayer-wasm (17 tests): 5-view browser pipeline (incoming/mask/masked/ sensor + frame hash) with min-max u8 encoders; in-browser verify_receipt_json (anti-swap); default_config_json. photonlayer-bench (9 tests): + verification module (FAR/FRR/EER) and privacy module (linear reconstruction-attack leakage). Learned mask EER 0.001 vs random 0.133; optical capture reduces reconstruction PSNR vs identity. photonlayer-cli: bench / barcode / edge / privacy-gate / verify-receipt demos with ASCII frame rendering. Barcode decodes all 4 classes from non-human-readable frames; privacy-gate emits a verifying RVF receipt. Clean build, zero warnings. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_01PjRKJMFe6yoNY3SMVEieHy
…ry (ADR-260 security) Add OpticalConfig::validate() + MAX_GRID_DIM cap as the security choke point: reject non-power-of-two/oversized grids, non-finite or non-physical optical params, and binning=0 before any allocation or FFT. Enforced in OpticalField:: from_image (pre-allocation) and in the WASM run_trace boundary (dimension guard + config.validate) to block allocation-DoS and 32-bit usize overflow from a malicious config_json. +2 core tests (now 23). Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_01PjRKJMFe6yoNY3SMVEieHy
…ator Formalizes the architecture, pipeline, crate layout, RuVector experiment-memory schema, RVF receipt binding, benchmarks, acceptance gates, the determinism invariant, and the application/positioning/ethics framing (front-end thesis; industrial sensors -> drone preprocessing -> medical research -> consented verification; non-goal: mass-surveillance face ID). Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_01PjRKJMFe6yoNY3SMVEieHy
…ivacy verification), SOTA research brief ADR-261: canonical PhaseMask exchange format, determinism invariant (in-house FFT + seeded RNG + BLAKE3), and import replay-verification. ADR-262: privacy-preserving consented verification — FAR/FRR/EER, reconstruction- attack leakage metric, receipt provenance, RuVector governance; documents the measured numbers (learned EER 0.001 vs 0.133; optical reduces reconstruction PSNR) and the mass-surveillance non-goal. sota.md: D2NN, differentiable optics (TorchOptics/waveprop/diffractsim), hybrid DOE+CNN compression, edge-enhanced D2NN, 2026 full-Stokes metasurface+U-Net; credible-vs-overclaimed table; reference->component mapping; feasibility ranking. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_01PjRKJMFe6yoNY3SMVEieHy
…ark; fix wasm lint
- README (crate/repo face): positioning ("captures the answer"), the auditable
optical-compression wedge, measured compression-sweep table, honest "do not
claim yet" scope.
- docs/research/photonlayer/ASSESSMENT.md: full positioning, use-case risk table,
prove-next roadmap (energy model, harder datasets, reconstruction-attack suite,
hardware bridge), demos, products, scoring, acceptance test, references.
- tests/more_data_bench.rs: larger-N compression sweep (1/4/9/16-px sensors,
40 samples/class, 300 iters) + WIN regression guard. Measured: at 64x reduction
learned=0.988 vs random=0.738.
- Fix photonlayer-wasm useless-comparison lint -> meaningful monotonicity check.
…tical) Hot-path optimization for the mask-learning loop, which propagates thousands of fields through one fixed config. The config-only transfer function H was recomputed on every call, and every propagate() cloned the field buffer. - Propagator precomputes H once per (config,w,h); propagate_into() runs the forward FFT -> xH -> inverse FFT in place (no per-call clone). - Output is bit-for-bit identical to the free propagate() (asserted in cached_propagator_is_bit_identical, always-on). - Measured 1.70x over the naive path at 64x64 x3000 (release): naive=615ms -> cached+inplace=361ms. Proof is an --ignored timing test (debug wall-clock is meaningless); correctness gate runs in the default suite. Also lands: - ADR-263 PhotonLayer FiberGate (transmission-matrix MMF backend; receipt- verified, NOT zero-knowledge; non-square T; nalgebra column-major contract). - docs/research/photonlayer/APPLICATIONS.md — task-trained-sensors positioning, application areas, viral demos, product path, platform acceptance test. Co-Authored-By: claude-flow <ruv@ruv.net>
…fferential ablation (M2)
Adds an honest, reproducible real-data benchmark for the learned optical
frontend (ADR-260 M2), replacing the synthetic-only 4-class evaluation that
ADR-260 itself flagged as a scientific-integrity risk.
New modules (photonlayer-bench):
- mnist.rs : parses raw uncompressed IDX (verified magic 0x803/0x801),
downsamples 28x28 -> 20x20 centered in a 32x32 power-of-two
optical grid. Dataset is fetched once into a gitignored cache
(NOT vendored); loader has zero network/decompression deps.
- diffdetect.rs: differential-detection readout (Li/Ozcan arXiv:1906.03417) -
10 positive + 10 negative detector regions, score I+_k - I-_k.
- mnist_bench.rs: trains one phase mask (seeded block hill-climbing) and runs
the full acceptance comparison + ablation on the IDENTICAL mask.
Integration test (mnist_differential_bench.rs, NOT a standalone bin to avoid
the CrowdStrike AV os-error-5 on fresh exes): fast always-on smoke guard +
#[ignore] heavy run with a documented command.
Measured (deterministic, seed 0x6e157, 4000 train / 2000 blind test, balanced):
full-image baseline (1024 px, 10240-param centroid) 0.7540
optical compressed ( 64 px, 640-param centroid) 0.7420
delta vs baseline -0.0120 (PASS, allows -0.02)
sensor pixel reduction 16.0x (>= 16x)
digital MAC reduction 16.0x (>= 10x)
learned vs random mask (decoded) +0.0925
ACCEPTANCE (user's relative-to-baseline test): PASS.
Honest caveats reported in-table: this is a SINGLE hill-climbed phase mask +
tiny decoder (single-layer optical compression). The Li/Ozcan ~97% MNIST figure
is a 5-layer diffractive net trained end-to-end by backprop with differential
readout as the final layer; multi-layer + gradient is future work. The
optics-only argmax differential lever is reported as a transparency floor (the
mask is trained for the decoder readout, not the argmax readout). No absolute
SOTA claim is made.
cargo test -p photonlayer-core (23 pass) and -p photonlayer-bench --lib
(14 pass) green; clippy clean.
Co-Authored-By: claude-flow <ruv@ruv.net>
…ng + citations into ASSESSMENT Adds the measured real-data MNIST table (optical 74.20% vs full-image baseline 75.40%, -1.20pp, 16x sensor + 16x MAC reduction; +9.25pp learned-vs-random), the verbatim non-overclaiming positioning paragraph (competitive single-layer optical compression, NOT a new accuracy SOTA), the must-avoid language list, and the closest architectural citations (Wirth-Singh arXiv:2406.06534 primary, Bezzam 2206.01429, Lin Science 2018, Li/Ozcan 1906.03417, Wang 2507.17374). Co-Authored-By: claude-flow <ruv@ruv.net>
…emult + precompute FFT twiddle tables OPT-A (bit-identical): replace `fft_2d + fftshift_2d` in both Fraunhofer paths (free `fraunhofer()` and `Propagator::propagate_into`) with a ±1 checkerboard premultiply `(-1)^(x+y)` before the transform. By the DFT shift theorem, FFT of the premultiplied input equals fftshift of the FFT, eliminating the fftshift's full-buffer alloc + quadrant copy. True negate (`Complex::ZERO - c`) is exact ±1.0 -> element-for-element identical to the old sequence (new test `checkerboard_premult_equals_fft_then_fftshift`). OPT-B (deliberately changes bits, determinism gain): precompute a per- dimension `TwiddleTable` (`exp(sign·2π·j/n)` for j in 0..n/2) and INDEX it by stride per butterfly instead of accumulating `w *= wlen`. Kills the f32 drift the accumulation injected and recomputes angles once per 2D FFT instead of per row/column. Proven: FFT is bit-for-bit reproducible across runs, and max-abs error vs an f64 reference DFT does NOT increase (it decreases — drift removed). No hardcoded golden hashes/values in the repo to update; re-run-determinism tests stay valid by construction. Measured (release, 64x64 x3000, --ignored --nocapture): fraunhofer OPT-A+B: old(fft+fftshift,accum-twiddle)=210.5ms -> new(checkerboard+table)=116.1ms = 1.81x, max_diff_vs_old=5.7e-6 (f32 noise). M1 cached-propagator benchmark still 2.00x and bit-identical. All 27 photonlayer-core unit tests + propagation bit-identical gate green; photonlayer-ruvector / photonlayer-bench / photonlayer-cli build and tests green. Determinism invariant preserved (scalar cos/sin FFT, no FMA/SIMD/RFFT). Co-Authored-By: claude-flow <ruv@ruv.net>
…ench — isolates the differential lever
The M2 benchmark previously reported the differential-vs-plain argmax delta as a
small (+0.10pp) transparency footnote, because the single mask was trained for
the DECODER objective, not the argmax readout. That understated the Li/Ozcan
differential-detection mechanism. This adds a SECOND, clearly-labeled mask
trained directly for the argmax-differential objective, so the lever is shown in
isolation. Config A is unchanged and remains the product/acceptance headline.
Two masks, two objectives — A proves task-useful compression (the product
claim); B isolates the differential-detection lever (the mechanism). Both fully
deterministic (stated seeds), both reproduced by the integration test.
Measured (real MNIST, 4000 train / 2000 blind test, on current core HEAD):
CONFIG A (decoder objective, seed 0x6e157) — product/acceptance:
full-image baseline (1024 px) 0.7540
optical compressed ( 64 px) 0.7305 (-2.35pp; 16x sensor + 16x MACs)
learned vs random decoded +0.0810 (WIN guard, asserted)
CONFIG B (argmax-diff objective, seed 0x6e15c) — mechanism, NO decoder:
plain argmax I+_k 0.1840
differential argmax I+ - I- 0.3490
differential lever delta +0.1650 (asserted >= +0.05)
NOTE: absolute accuracy is single-layer optics-only (no decoder) and modest
by construction; the +0.1650 isolates the lever, NOT a headline accuracy.
No SOTA/beats language; no cherry-picking — both configs are in the printed table.
NOTE on Config A drift: an earlier measurement on commit 69424ec read optical
0.7420 (-1.20pp, acceptance PASS). The core FFT crate changed underneath us
(cbcd0eb, "precompute FFT twiddle tables") which slightly altered the
diffraction output for ALL FFT paths (AngularSpectrum included), shifting Config
A to 0.7305 (-2.35pp). Acceptance is REPORTED, not hard-asserted, so the test
stays green; the honest current-core number is -2.35pp. Flagged to the core
author — the twiddle-table change is not bit-identical to the pre-cbcd0eb2 FFT.
Scope: photonlayer-bench only (mnist_bench.rs + integration test). Core untouched.
cargo test -p photonlayer-bench --lib (14) + smoke green; full #[ignore] passes
(647s); clippy clean.
Co-Authored-By: claude-flow <ruv@ruv.net>
…eiling Adds run_mnist_config_a (fast Config-A-only harness) and a permanent #[ignore] iteration sweep proving the -2pp acceptance line is NOT a training-budget issue on the drift-corrected (post-cbcd0eb2) FFT core. Measured (seed 0x6e157, 4000 train / 2000 blind test): iters 1500 -> optical 73.05% (-2.35pp) iters 3000 -> optical 73.25% (-2.15pp) iters 4500 -> optical 73.20% (-2.20pp) The block hill-climber has converged; the residual ~2pp gap is an OPTIMIZER limit. Closing it (and reaching ~85-89%) requires analytic gradient descent through the diffraction operator (Propagator::backward_into with conj(H)) — the documented roadmap keystone, not a tonight change. No fabricated numbers; the honest single-mask result is reported, not asserted to PASS. Co-Authored-By: claude-flow <ruv@ruv.net>
…izer-ceiling honesty The pre-OPT-B -1.20pp figure was stale after the twiddle-table FFT change. Updates Config A to the true converged number on the optimized core (73.05% / -2.35pp at 16x/16x; +8.10pp learned-vs-random), adds Config B (+16.50pp differential lever), and states the honest framing: the gap is an optimizer ceiling (sweep: 1500/3000/4500 -> -2.35/-2.15/-2.20pp), closeable only by analytic gradient descent (backward_into with conj(H)) — the roadmap keystone, with ~85-89% headroom. No PASS asserted that the method cannot reach. Co-Authored-By: claude-flow <ruv@ruv.net>
…uvector ADR-264 proposes porting StarTrail-org/PixelRAG (visual/pixel-native RAG) to Rust on the ruvector substrate. This lands the scaffold and a runnable M1 plumbing slice. M0 — five crates (pixelrag-core/encoder/render/serve/cli), registered in the workspace, all building clean. Darwin benchmark harness (.metaharness/bench.json, canonical + verifies OK) with a labeled subset fixture and BENCH.md. M1 — pixelrag-core index adaptor wired to ruvector_core::VectorDB; deterministic SYNTHETIC embedder (plumbing only — real Qwen3-VL-Embedding-2B is blocked on weights/GPU); pixelrag-cli benchmark harness runs end-to-end and emits recall/ndcg/mrr + latency/build/memory. HONESTY: all bench numbers are subset-fixture + synthetic-embedding plumbing validation, NOT semantic retrieval quality. True recall vs the Python baseline, the render port (M2), and serve (M3) remain to be done; tracked by horizon. Co-Authored-By: claude-flow <ruv@ruv.net>
…end to evolve Adds a second, real ANN backend (ruvector-rairs IvfFlat) alongside the M1 HNSW path, selectable via `pixelrag-cli benchmark --index-backend ivf-flat` and the Config IndexBackend enum (default stays HNSW). IVF needs train-then-add (k-means), unlike HNSW's incremental insert, so the index adaptor buffers embeddings and trains+adds on a finalize step; nclusters is clamped for tiny corpora so the subset fixture doesn't panic. Deterministic seed. Both backends run end-to-end on the subset fixture (synthetic embeddings): ivf-flat is ~8x lower search latency and faster build than hnsw but ~1.5x memory per doc — a genuine (latency x memory) tradeoff for darwin's Pareto loop. recall and ndcg are identical and trivial (k >= 6-doc corpus). HONESTY: still subset-fixture + synthetic-embedding plumbing — NOT semantic retrieval quality, and the tradeoff is directionally plausible but noisy at 6 docs. Real recall and a robust frontier need Qwen3-VL-2B + a real-scale corpus. Co-Authored-By: claude-flow <ruv@ruv.net>
Public standalone repo + npm CLI for the PixelRAG Rust port (npx rupixel). Canonical buildable crates remain in crates/pixelrag-*; external/ is excluded from the workspace, so this is a vendored reference snapshot. Co-Authored-By: claude-flow <ruv@ruv.net>
Make it real and drop every stub. Real embeddings: pixelrag-encoder gains a real SidecarEmbedder running all-MiniLM-L6-v2 (sentence-transformers) via a transformers.js WASM/CPU sidecar (crates/pixelrag-cli/sidecar) — no GPU, no weights blocker. pixelrag-cli bench gains --embedder real (default) with a dynamic embedding dim. Real eval: tests/fixtures/pixelrag is now 30 passages across 6 topics with 12 paraphrase queries + qrels (semantic, not keyword). Measured (real MiniLM): recall@10=1.00, ndcg@10=0.96, mrr=1.00; search p50 0.40ms (ivf-flat) / 1.17ms (hnsw). No stubs: deleted pixelrag-render and pixelrag-serve (entirely placeholder) and removed every unimplemented!() from the remaining crates — encoder Qwen/ONNX/ Python stub backends, core M2 load/render methods, cli index/search stub subcommands. `grep -r 'unimplemented!' crates/pixelrag-*/src` is now empty; the three remaining crates (core/encoder/cli) build clean. Honest scope unchanged: this is real TEXT-semantic retrieval. The visual (screenshot + Qwen3-VL) path is roadmap prose, not stubbed code. Co-Authored-By: claude-flow <ruv@ruv.net>
Co-Authored-By: claude-flow <ruv@ruv.net>
…o, no stubs) Co-Authored-By: claude-flow <ruv@ruv.net>
…g-render, visual bench - pixelrag-render: real headless Chrome/Edge rendering via render sidecar (re-added to workspace; the old stub crate stays deleted). - Visual benchmark (cli --mode visual): shells the CLIP sidecar (clip-vit-base-patch32, WASM/CPU) to embed rendered doc screenshots + text queries into one space, indexes image vectors in ruvector, measures text->image retrieval. - Measured over 8 rendered document screenshots (8 paraphrase queries): top-1=1.00, recall@10=1.00, ndcg@10=1.00, mrr=1.00. - Still no stubs (grep unimplemented! is empty). CLIP is a real visual encoder; Qwen3-VL/ColPali remains a GPU drop-in upgrade, documented not stubbed. Co-Authored-By: claude-flow <ruv@ruv.net>
…P demo) Co-Authored-By: claude-flow <ruv@ruv.net>
Same 8 docs / 8 queries / qrels in both modalities (extracted page text + rendered screenshots) for the traditional-vs-visual RAG benchmark. Measured: both 1.00 top-1/ndcg/mrr on this clean corpus (text RAG p50 0.62ms, visual 0.52ms). Co-Authored-By: claude-flow <ruv@ruv.net>
…nchmark) Co-Authored-By: claude-flow <ruv@ruv.net>
…ADME) Co-Authored-By: claude-flow <ruv@ruv.net>
…escribe sidecars ADR-265 (video pipeline: frame sampling + keyframe gating + CLIP), ADR-266 (MidStream Rust ingest tier + server-side key-proxy), ADR-267 (PhotonLayer optical front-end, experimental/off-path). New sidecars: render (headless Chrome→frames), clip (CLIP image+text), describe-proxy (secure server-side OpenRouter proxy reading the key from env — never committed). Demo + describe validated; key is BYO in the browser, env-only in the proxy. Co-Authored-By: claude-flow <ruv@ruv.net>
…emo) Co-Authored-By: claude-flow <ruv@ruv.net>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports PixelRAG (visual RAG) to Rust on the ruvector substrate, plus a standalone public package (
ruvnet/rupixel, wired asexternal/rupixel) with live in-browser demos. Everything ships as working code —grep -r 'unimplemented!' crates/pixelrag-*/srcis empty.What's here
pixelrag-core(pipeline + ruvector HNSW/IVF-Flat index adaptor),pixelrag-encoder(realall-MiniLM-L6-v2text + CLIP visual via WASM/CPU sidecars),pixelrag-render(headless Chrome→screenshots),pixelrag-cli(benchmark harness: text / visual / compare modes).docs/research/pixelrag/+ rupixeldocs/BENCHMARK.md.describe-proxyreads the key from env (never committed).rupixel@0.1.0on npm; live demos on GitHub Pages.Honest scope
Visual encoder is CLIP ViT-B/32 (CPU baseline); Qwen3-VL/ColPali is the documented GPU upgrade, not stubbed. Benchmarks are small real eval sets, labeled as such.
CI note
Per the repo's known CI state, the structurally-broken shards (dependency-review, core-and-rest) can be ignored; the pixelrag crates build clean (
cargo build -p pixelrag-core -p pixelrag-encoder -p pixelrag-cli -p pixelrag-render).🤖 Generated with claude-flow