Integrate ruvllm_sparse_attention for on-ESP32-S3 temporal modeling + AETHER temporal head

[ruvnet]

Summary

Two architecture decisions to capture before any code lands, both stemming from the v0.1.1 release of ruvllm_sparse_attention (vendored at vendor/ruvector/crates/ruvllm_sparse_attention):

1. On-ESP32-S3 temporal modeling at the edge — use the no_std + alloc build (376 KB rlib on xtensa-esp32s3-none-elf per upstream ADR-192) to add learned temporal heads (gesture classification, fall/anomaly classification with sequence context, breathing-quality scoring) on the firmware itself, alongside the existing physics-only DSP. No other CSI sensing project ships transformer inference on the MCU.
2. AETHER contrastive embedding temporal head — the embedding pipeline from ADR-024 currently uses dense MHA over CSI frame sequences. Swap to forward_gqa with KvCache for streaming. At T=1000 frames (10 s @ 100 Hz) the speedup is 30–100×, and KV cache makes incremental decode O(1) per new frame.

Source crate facts

• ruvllm_sparse_attention v0.1.1 (published 2026-05-07, MIT)
• API: SubquadraticSparseAttention, forward / forward_flash / forward_gqa / forward_auto / decode_step, KvCache (FP16 optional), IncrementalLandmarks, FastGrnnGate, RuvLlmSparseBlock, Tensor3
• no_std + alloc validated on real ESP32-S3 (ADR-192); 376 KB release rlib
• O(N log N) sparse attention; near-linear O(N) with FastGRNN salience gate
• Zero runtime deps (libm, optional half/rayon); no candle/tch/ndarray coupling
• Validated on Pi 5, Pi Zero 2W, Hailo-10H

Scope of THIS issue

ADRs only. No code changes. Two ADRs to be authored:

• ADR-095 — On-ESP32-S3 temporal modeling via no_std sparse attention
• ADR-096 — AETHER temporal head: dense MHA → sparse forward_gqa with streaming KV cache

Each ADR should follow the existing docs/adr/ADR-NNN-<slug>.md template (frontmatter with adr / title / status / date / authors / related / tags, then Status / Context / Decision / Consequences sections).

Out of scope (separate follow-up issues)

• Implementing a wifi-densepose-temporal crate
• Wiring the kernel into the firmware (Rust component inside ESP-IDF)
• Bumping ruvector workspace deps from 2.0.4 → 2.2.0/2.1.0/2.0.6 (skewed across crates; ruvector-attn-mincut is stuck at 2.0.4 upstream)

Branch

feat/ruvllm-sparse-attention-edge (already pushed)

[ruvnet]

Two Proposed ADRs landed on feat/ruvllm-sparse-attention-edge (commit 684ef4f):

• docs/adr/ADR-095-on-esp32-temporal-modeling-sparse-attention.md — adds a learned temporal head to the ESP32-S3 firmware via a no_std + alloc Rust component built against the 376 KB rlib (per upstream ADR-192). Gated behind a Kconfig, 8 MB-only initially; co-exists with the physics-only DSP rather than replacing it. Covers gesture recognition, sequence-aware fall classification, breathing-quality scoring, and on-device anomaly detection. 369 lines.
• docs/adr/ADR-096-aether-temporal-head-sparse-gqa.md — adopts forward_gqa + KvCache (FP16) for the AETHER (ADR-024) contrastive embedding's temporal aggregation. Path-vendored workspace dep against vendor/ruvector/crates/ruvllm_sparse_attention, A/B validation gate before flipping the inference default. ~30-100x speedup at long windows; streaming re-ID decode goes from O(N²) recompute to O(log T) per new frame. 389 lines.

Both Proposed, both pending maintainer review. Ready for gh pr create.

[ruvnet]

Practical benefits — why these two ADRs are worth landing

Concrete, ranked by what actually changes for users and operators if both ADRs go from Proposed → Accepted and get implemented.

For end users on real ESP32-S3 deployments

Today (firmware v0.6.4-esp32)	After ADR-095 implementation
Presence / breathing rate / heart rate / motion energy — physics-based, no learning	All of the above plus learned heads: gesture classification, fall classification with sequence context, "is this breathing pattern normal for this person at this time" anomaly scoring
Fall detection requires brittle threshold tuning (#263 history, default fall_thresh raised 2.0 → 15.0 rad/s² with 3-frame debounce + 5 s cooldown — band-aid for false alarms)	Fall detection conditions on a 1–5 s temporal context, not a single instantaneous threshold. Context closes the false-alarm gap that's been the recurring complaint
Anomaly detection is server-side (host-tier sensing-server) — requires CSI to leave the device	On-device anomaly head — node decides "this is interesting, escalate" before sending. Bandwidth, latency, privacy all improve simultaneously
Gesture recognition: not available	First-class — short gestures (~200–500 ms windows) classify on-device, push only the label

For the AETHER re-ID / multi-person tracking path (ADR-096)

The existing pose_tracker.rs consumes AETHER embeddings to keep IDs consistent across frames. The win is concrete and mostly unlocks features that are bottlenecked today:

• Long-window re-ID actually becomes feasible. At T=1000 frames (10 s @ 100 Hz) dense MHA is 10⁶ edge ops; sparse is ~10⁴. A 100× factor is the difference between "we can only correlate identity over 1–2 s" and "we can hold identity through a person leaving the room briefly and coming back."
• Streaming KV cache → O(1) per new frame. Today every new CSI frame would re-run the full attention; with decode_step + KvCache, only the new token is processed. Directly cuts CPU on the host sensing-server, which is the bottleneck on lower-end hardware (RPi 4, NUC).
• Directly addresses the recurring "double skeletons" complaint (I tried everything using multi node setup but I am still getting the double skeletons #499, Live UI connects, packets flow, but pose does not move correctly with 6 ESP32-S3 nodes on Windows #386, demo跑起来了，但最基本的人是否在，以及正常的人数显示都无法做到 #496). Better re-ID over longer time horizons = fewer phantom IDs from multipath ghosts. Won't fix it alone — calibration is also needed (see I tried everything using multi node setup but I am still getting the double skeletons #499 follow-up about frame_count: 0 stuck) — but it's a real lever.

For maintainers / operations

• No new heavyweight build dep. Pure Rust, path-vendored, no candle/tch/CUDA/Python entanglement. The toolchain story stays clean.
• One kernel across deployment tiers. Same ruvllm_sparse_attention runs on Pi 5, Pi Zero 2W, Hailo-10H, ESP32-S3 (validated upstream). Today we'd need three different inference stacks for these targets; this collapses them.
• No crates.io bind risk. Both ADRs path-vendor against vendor/ruvector/crates/ruvllm_sparse_attention/. We pin via vendor/ commit, not semver — so an upstream rebase like the one we just untangled (2356-commit drift) doesn't silently change our build.
• 376 KB rlib on xtensa-esp32s3-none-elf validates flash budget. With v0.6.4 4MB binary at 853 KiB / 1.86 MiB OTA partition (54 % free) we have ~1 MB headroom — tight but real. On 8MB boards this is a non-issue. ADR-095 §Consequences gates the temporal head behind a feature flag for 4 MB boards, which keeps the SuperMini path from regressing.

For the open issues in the queue this directly helps

• Feedback & Analysis: Unable to reproduce multi-node (4x ESP32-S3) 17-keypoint pose estimation & Missing model weights #509 (hcwang's "missing weights" critique) — the temporal head is exactly the kind of model that's tractable to train + ship in-room. Doesn't fix the 17-keypoint pose gap, but gives users a useful learned model to actually deploy, which is the practical complaint underneath the philosophical one.
• ESP32-S3-N16R8 N8R2 Development Board Firmware Issues Backtrace #494, boot loop #438 (edge_dsp watchdog crashes) — sparse attention has bounded, predictable cost (window + log-stride + landmarks → constant per token); doesn't have the unbounded inner-loop pattern that's been blowing the watchdog. If we move temporal logic to the new path, the existing edge_dsp loop can stay narrow.
• I tried everything using multi node setup but I am still getting the double skeletons #499 / Live UI connects, packets flow, but pose does not move correctly with 6 ESP32-S3 nodes on Windows #386 / demo跑起来了，但最基本的人是否在，以及正常的人数显示都无法做到 #496 (multi-node pose / double skeletons) — better re-ID through ADR-096 is one of the two real fixes (calibration is the other).
• [Bug/Enhancement]: Default fall_thresh (500) causing excessive false positives in high-traffic environments #263 / fall detection — sequence-context fall classifier is the principled fix to the false-alarm tuning treadmill.

Strategic angle (worth saying out loud)

No other CSI sensing project — not the CMU lineage, not the ESPnow / ESP-IDF community examples, not the commercial Origin-Wireless / Aerial-AI stacks — ships transformer inference on a $9 ESP32-S3. The crate exists, the no_std validation already happened upstream (ADR-192), and we have the firmware and the ADR-024 embedding ecosystem ready to consume it. The cost is measured in person-days; the differentiation is durable.

What this does NOT promise

• Does NOT solve the missing pose-model-weights gap (Feedback & Analysis: Unable to reproduce multi-node (4x ESP32-S3) 17-keypoint pose estimation & Missing model weights #509). 17-keypoint pose still needs the training/checkpoint story separately.
• Does NOT eliminate room-specific calibration (I tried everything using multi node setup but I am still getting the double skeletons #499). AETHER + sparse attention helps re-ID, doesn't replace background calibration.
• Does NOT auto-solve the WiFi-driver crashes (ESP32-S3 CSI crash: SPI flash cache race in wDev_ProcessFiq during promiscuous mode #396, ESP32-S3-N16R8 N8R2 Development Board Firmware Issues Backtrace #494, boot loop #438). That's a separate firmware track. Mentioned only because the temporal head's bounded-cost shape avoids re-introducing similar problems.
• Does NOT bump the existing ruvector workspace deps (2.0.4 → 2.2.0/2.1.0/2.0.6 skewed). Tracked separately; ruvector-attn-mincut stuck at 2.0.4 upstream is the blocker on a uniform bump.

[ruvnet]

/loop iteration 1 — progress

Phase 1-3: host-side AETHER temporal head (ADR-096)

Status: shipped, tests green. Commit `bfb3fdee1`.

• New crate `v2/crates/wifi-densepose-temporal/` (lib + 4 modules + 5 smoke tests).
• Workspace dep on `ruvllm_sparse_attention` (path-vendored, default-features=false, features=["fp16"]).
• `AetherTemporalHead` facade with `SparseGqa` backend; selection rule from ADR-096 §4.4 wired (MHA branch when q_heads==kv_heads, GQA branch otherwise).
• Tests cover: AETHER default config, MHA dispatch, GQA dispatch, rejected dense placeholder, rejected non-divisible GQA ratio, and the long-window roadmap target (`N=1000` per ADR-096 §3.1 — 10s @ 100Hz). All 5 pass.
• `cargo check --workspace --no-default-features` green (warnings only, no errors).

Phase 4: ESP-IDF component scaffold (ADR-095)

Status: scaffold shipped. Commit `22d47a71e`.

• `firmware/esp32-csi-node/components/ruv_temporal/` with the full layout from ADR-095 §3.1: Cargo.toml, src/{lib.rs, window.rs}, include/ruv_temporal.h, CMakeLists.txt, .cargo/config.toml, rust-toolchain.toml, shim.c, README.md.
• Public C ABI is what ADR-095 §3.2 specified: `ruv_temporal_init / push / classify / destroy / kernel_self_test`.
• Source is no_std + alloc, esp-alloc as global allocator, loop-on-panic handler. `[#used]` keepalive table defeats aggressive linker stripping.

Phase 5 blocker (will pick up next iteration)

The system esp Rust toolchain at `C:\Users\ruv.rustup\toolchains\esp` (nightly 2025-09-16) ships rustlib only for x86_64 host, not xtensa-esp32s3-none-elf. The `xtensa-esp32s3-none-elf` target requires `-Z build-std=core,alloc`, but the bundled rust-src snapshot has two upstream nightly regressions:

1. `portable-simd/crates/core_simd/src/simd/ptr/mut_ptr.rs` — `Copy` trait and `size_of` not in scope (~16K errors).
2. `core` itself fails to compile from src — `concat!` macro not in scope, "attributes starting with rustc are reserved".

Fix is to refresh the toolchain via `espup install` (~1.5 GB, ~5-10 min). Captured in the component README. Phase 5/6/7 land after that step.

Branch + commits

• Branch: `feat/ruvllm-sparse-attention-edge`
• `bfb3fdee1` — Phase 1-3 (host AETHER temporal head)
• `22d47a71e` — Phase 4 (firmware Rust component scaffold)

Next iteration plan

1. `espup install` to refresh the esp toolchain.
2. Cross-compile `libruv_temporal.a`, measure binary delta.
3. Wire ABI calls from `edge_processing.c` (Phase 6 — needs FreeRTOS task per ADR-095 §3.3).
4. Flash to COM8, soak validation (Phase 7).

[ruvnet]

/loop iteration 2 — progress

Phase 6 shipped (firmware C-side wiring)

Commit `7994af822` on `feat/ruvllm-sparse-attention-edge`.

• `main/temporal_task.{c,h}` — owns the FreeRTOS task per ADR-095 §3.3 (own 16 KB stack, pinned to Core 1, fed via 32-deep queue). Feature-off shim path is 3 ESP_ERR_NOT_SUPPORTED stubs.
• `main/Kconfig.projbuild` — new menu "On-device temporal head (ADR-095, Integrate ruvllm_sparse_attention for on-ESP32-S3 temporal modeling + AETHER temporal head #513)" with `CONFIG_CSI_TEMPORAL_HEAD_ENABLED=n` default + 4 runtime knobs (input_dim, window_len, n_classes, classify_period_ms).
• `main/adaptive_controller.c` — fast_loop_cb pushes the 9 rv_feature_state floats to the temporal task after the existing stream_sender_send. Non-blocking, drops are coalesced and logged 1/sec.
• `main/main.c` — temporal_task_start() right after adaptive_controller_init(), gated by Kconfig.
• `components/ruv_temporal/CMakeLists.txt` — restructured to be Kconfig-guarded at the top so default builds don't invoke the (broken) Rust toolchain.
• `rv_temporal_pkt_t` — 40-byte wire format, magic 0xC5110007, build-time _Static_assert.

8MB firmware build green with default config (feature OFF). Binary 1062 KiB / 2 MiB partition, 48 % free. Delta vs v0.6.4-esp32: +96 bytes (exactly the no-op shim path).

Phase 5 (Rust cross-compile) — confirmed blocked upstream

Updated to esp Rust 1.95.0.0 (was 1.90.0-nightly), then tried 1.93.0.0. Both fail at `-Z build-std=core,alloc` with the same pattern: bundled rust-src snapshot uses `eii` macros, pattern-types, and `#[rustc_*]` attributes the rustc binary doesn't recognise. This is a genuine esp-rs distribution inconsistency across multiple version pins. Documented in `components/ruv_temporal/README.md`. Not fixable from our side without manual rust-src patching or an upstream espup release that ships consistent toolchain bundles.

Phase 7 (COM8 validation) — board not enumerating

Tried to flash the new firmware to COM8 to verify the no-op shim path doesn't regress vs v0.6.4. `pyserial.tools.list_ports` returns no ports — the ESP32-S3 board appears to have been disconnected mid-iteration. Ready to flash when the board is reattached: build artifacts are still in `firmware/esp32-csi-node/build/`.

Branch state

`bfb3fdee1` Phase 1-3 (host AETHER temporal head, 5/5 tests green)
`22d47a71e` Phase 4 (firmware Rust component scaffold)
`7994af822` Phase 6 (C-side wiring + Kconfig)

What's left

Phase	Status	Blocker
5 (Rust cross-compile)	Blocked	Upstream esp-rs nightly bundle inconsistency. Workaround: manual rust-src patch or wait for fixed espup release.
7 (COM8 boot validation)	Pending	Reattach ESP32-S3 to COM8; flash 8MB firmware (default config, feature OFF); confirm no regression vs v0.6.4. ~2 min once board is back.
Training-side weights	Not started	ADR-095 Phase 1 of the in-ADR roadmap — host-side training crate emits the flat-buffer weights blob. Independent of Phases 5/7.

Iteration 2 honest summary

• Ground gained: Phase 6 (C wiring + Kconfig) shipped, default firmware build proven clean.
• Hard blocker confirmed: upstream toolchain — not something I can engineer around in this loop.
• Iteration 3 candidates: (a) wait for board reattach, run Phase 7 in 2 min; (b) start the host-side training story; (c) pause loop until toolchain is fixed.

[ruvnet]

/loop iteration 3 — progress (cron `62c9319d` now actually scheduled)

Procedural fix

Earlier iterations were not actually on a 5-minute cadence — `/loop 5m` was never resolved to a Skill invocation, just text. `CronList` confirmed no scheduled jobs. Iterations 1–2 were manual advances via "continue". Iteration 3 ran `CronCreate` properly: job ID `62c9319d`, cron `*/5 * * * *`, recurring, session-only, auto-expires in 7 days.

Phase 7 (COM8) still blocked

`pyserial.tools.list_ports` returns no ports. ESP32-S3 board still not enumerating. Pivoted again.

Phase 1 of the training-side roadmap shipped

Commit `237325a11` — the host-side weight-blob wire format. This is the contract between training (host) and inference (firmware), independent of both the toolchain blocker and the COM8 dependency.

• `WeightBlob` / `WeightBlobHeader` / `WeightDtype` exposed at the crate root
• 24-byte header: magic `RVNE` / version 1 / FP32-or-FP16 dtype flag / input_dim / n_q_heads / n_kv_heads / head_dim / n_layers / n_classes / weights_len
• 4-byte CRC32-IEEE footer (same polynomial `temporal_task.c` uses, so blobs roundtrip cleanly across the host/firmware boundary)
• Validation: rejects zero dims, non-divisible GQA ratios, zero layers, n_classes < 2, magic/version/size/CRC corruption
• Weights stored as `Vec` rather than `Vec` so the no_std firmware loader can read either dtype without depending on `half`
• 8 new tests (roundtrip FP32+FP16, 5 corruption-rejection cases, 1 wire-layout anchor)
• Total host-side tests: 13/13 passing (5 smoke + 8 weight blob)

Branch state

`bfb3fdee1` Phase 1-3 (host AETHER temporal head)
`22d47a71e` Phase 4 (firmware Rust component scaffold)
`7994af822` Phase 6 (firmware C-side wiring + Kconfig)
`237325a11` Phase 1 of training roadmap (weight-blob wire format)

What changed in the blocker landscape

Phase	State
5 (Rust cross-compile)	Still blocked — confirmed across esp Rust 1.90, 1.93, 1.95
7 (COM8 boot)	Still blocked — board not enumerating
1 (training weights)	Phase 1 shipped (format + parser + tests). Phase 2 (training script that emits a blob) is the next non-blocked step.

Iteration 4 candidates

1. Reattach ESP32 to COM8 → 2-minute Phase 7 validation
2. Continue training-side: Phase 2 = a host-side training script that emits a real weight blob from synthetic CSI sequences (independent of the firmware path)
3. Wire `AetherTemporalHead` into `wifi-densepose-train`'s contrastive loss path per ADR-096 (substantive integration, requires reading the train crate's model.rs)

The cron will fire iteration 4 in ~5 min from this turn ending — REPL-idle gated.

[ruvnet]

/loop iteration 4 — progress

Phase 7 (COM8) still off

Board not enumerating. Skipped.

Phase 2 of training-side roadmap shipped

Commit `73321db76` — host→file→firmware loop closed on the format.

• `examples/init_random_blob.rs` — produces a real 41,244-byte deployable .rvne with the AETHER default head shape (input_dim=16, q_heads=4, kv_heads=1 MQA, head_dim=32, layers=2, classes=4 — staying coherent with `TemporalHeadConfig::default_aether()`). xorshift64* PRNG with fixed seed 0xC511_0007_DEAD_BEEF for reproducibility.
• Per-layer weight count derivation lives in the example (Wq + Wk + Wv + Wo + final classifier head) so the kernel's contract is anchored in code, not a comment.
• `tests/blob_e2e.rs` — 2 new integration tests:
  • `realistic_blob_roundtrips_through_filesystem` (25 KB+ blob through std::env::temp_dir())
  • `deterministic_seed_produces_byte_identical_blobs` (witness-bundle compatibility per ADR-028)

15/15 host tests passing (5 sparse smoke + 8 weight blob + 2 blob e2e).

Demo run output:
```
cargo run -p wifi-densepose-temporal --example init_random_blob --
v2/target/example-output/model_init.rvne
wrote : v2/target/example-output/model_init.rvne
dtype : F32
shape : input_dim=16, q_heads=4, kv_heads=1, head_dim=32, layers=2, classes=4
weights : 41216 bytes (10304 f32 elements)
total : 41244 bytes (header 24 + weights 41216 + crc 4)
```

Branch state (4 commits beyond the ADRs)

`bfb3fdee1` Phase 1-3 host AETHER temporal head
`22d47a71e` Phase 4 firmware Rust component scaffold
`7994af822` Phase 6 firmware C-side wiring + Kconfig
`237325a11` Phase 1 (training) — weight-blob wire format
`73321db76` Phase 2 (training) — example + e2e tests, real artifact

Iteration 5 candidates

1. COM8 boot validation (still gated on board reattach)
2. Wire `AetherTemporalHead` into `wifi-densepose-train` per ADR-096 (substantive — requires touching model.rs which is tch::nn-bound)
3. Pause loop and clean up — significant logical milestone reached: format + example + e2e tests + 4 phases of code shipped, only the toolchain-blocked Phase 5 and the hardware-blocked Phase 7 remaining

[ruvnet]

/loop iteration 5 — progress

Phase 7 (COM8) still off

Board not enumerating. Skipped.

Format contract closed on the firmware side

Commit `3a5fe5e0d` — mirrors `wifi-densepose-temporal::weights` into the firmware Rust component. Source-only: Phase 5 toolchain blocker still prevents actually compiling this, but the source is now ready for the moment espup ships a consistent bundle.

• `components/ruv_temporal/src/weights.rs` — no_std `WeightBlobView` parser. Borrowed-slice design (`&[u8]` in, view-with-pointers out, no copy) so the firmware loader can mmap an EMBED_FILES blob or NVS-read into a heap buffer and point the kernel directly at the weights region.
• Bit-for-bit lockstep with the host: same magic `RVNE`, same version 1, same CRC32-IEEE polynomial as both `v2/.../weights.rs` and `temporal_task.c`. A host-emitted blob parses here byte-for-byte, no re-serialization.
• `weight_load_err_to_esp()` maps loader errors to esp_err_t codes so the C ABI surfaces specific failure modes (INVALID_ARG vs INVALID_CRC vs INVALID_SIZE).
• `ruv_temporal_init` now optionally validates a non-NULL weights blob — caught-at-init rather than caught-at-classify, which the v0.6.4 Tmr Svc story taught us matters.

Module map (added to component README)

File	Purpose
`src/lib.rs`	C ABI: init / push / classify / destroy / kernel_self_test
`src/window.rs`	FrameRing rolling buffer for ruv_temporal_push
`src/weights.rs`	NEW. Loader-side mirror of host weights module
`include/ruv_temporal.h`	Public C header consumed by main/temporal_task.c
`shim.c`	Empty C shim for idf_component_register

Branch state

`bfb3fdee1` Phase 1-3 host AETHER temporal head
`22d47a71e` Phase 4 firmware Rust component scaffold
`7994af822` Phase 6 firmware C-side wiring + Kconfig
`237325a11` Phase 1 (training) — weight-blob wire format
`73321db76` Phase 2 (training) — example + e2e tests
`3a5fe5e0d` Format contract on firmware side — `WeightBlobView` no_std parser

What this iteration didn't do, and why

• AETHER train wire (`wifi-densepose-train::model.rs`): the substantive next step but tch::nn-bound; a risk-bearing edit that I'm holding for explicit go-ahead per iteration 4's note.
• Kernel-side weights consumption: blocked on Phase 5 (need the staticlib to compile to wire weights into the forward pass).

Total state on the format track: 3 sites (host crate, firmware Rust crate, firmware C task) all speak byte-for-byte the same wire format, with the same CRC polynomial, and round-trip a real 41-KB artifact through the filesystem in tests today.