Feedback & Analysis: Unable to reproduce multi-node (4x ESP32-S3) 17-keypoint pose estimation & Missing model weights

[hcwang]

Hi, thanks for sharing this ambitious and fascinating project.

I have been trying to set up a local environment using 4x ESP32-S3 boards to accurately detect the number of people, their positions, and estimate 17-keypoint joint poses. Despite trying various configurations and testing multiple methods, I haven't been successful. Because there is currently a lack of comprehensive documentation on how to achieve this specific multi-node setup, I decided to dive deep into the source code, architecture, and existing issues.

Based on my technical analysis, I’d like to share my findings and ask a few questions regarding the current state of the project.

1. The "Real" Infrastructure vs. The "Mocked" AI Layer

From my code review, the project currently feels more like a Proof of Concept (PoC) scaffold rather than a fully functional pose-estimation system:

The Real Part (Hardware & Data Pipeline): The infrastructure layer is genuine. The ESP32-S3 firmware correctly utilizes ESP-IDF APIs to extract CSI (Channel State Information) subcarrier amplitude/phase data and transmits the binary frames via UDP. The Rust backend and WebSocket transport layers are also solidly implemented.
The Missing Part (AI Inference): The core AI layer seems incomplete. While the network architecture for DensePoseHead is defined in the code, there are no pre-trained weights (.pth or .onnx files) available in the repository.
2. Hardware Limitations (ESP32 vs. Research NICs)

The concept of "DensePose From WiFi" is scientifically valid (e.g., CMU's pioneering research). However, bridging the gap to ESP32s seems highly challenging:

CMU's research heavily relies on multi-antenna commercial NICs (like Intel 5300 or Atheros with 3x3 MIMO) which provide rich spatial resolution.
ESP32s only have 1x1 SISO antennas. While a 4-node mesh could theoretically compensate for this, achieving the microsecond/nanosecond-level clock synchronization required for accurate 3D spatial field reconstruction and triangulation across discrete ESP32s is notoriously difficult.
3. Why the 17-keypoint pose estimation cannot be reproduced

I noticed the same behavior mentioned in Issue #506. The reason the skeletons sometimes animate even when hardware is disconnected (or fail to animate properly when connected) is due to:

Simulation Fallback: The backend/UI relies heavily on a "simulation mode". When the system lacks real UDP data (or real inference outputs), it renders pre-generated skeletal animations to populate the dashboard.
Hardcoded Logic: When real data is received, since there are no actual neural network weights loaded, older code paths (like pose_service.py) seem to rely on simplistic if-else thresholds on signal norms (e.g., if feature_norm > 2.0 return "walking") instead of actually computing the 17 COCO keypoint coordinates via tensor operations.
Multi-node Fusion: The logic to fuse tensors from 4 distinct nodes into a single coherent pose prediction appears to be mostly at the schema/structural level and lacks trained implementations.
My Questions

I really love the vision of RuView and the idea of edge-based CSI sensing. To help contributors and users align their expectations:

Is the 17-keypoint pose estimation using multiple ESP32s genuinely achievable with the current state of this codebase?
Are there plans to release the pre-trained weights for the neural network, or is the intention for users to collect their own ground-truth data and train from scratch? If the latter, could you provide a pipeline/documentation on how to train a model for the multi-ESP32 setup?
Thank you for your time and for open-sourcing this project! Looking forward to your insights.

[stryngs]

506 got closed out. If you start asking hard questions I bet yours gets closed out too =)

[ruvnet]

Thanks for the careful, well-grounded analysis — your read of the codebase is largely accurate, and it's a useful framing for everyone hitting the same gap.

Honest answers to your three questions:

1. Is 17-keypoint pose with multiple ESP32-S3s genuinely achievable in the current state of the codebase?

For the infrastructure path, yes — CSI extraction, multi-node TDM, the cross-viewpoint fusion math (v2/crates/wifi-densepose-ruvector/src/viewpoint/), the AETHER re-ID embeddings (ADR-024), and the pose tracker scaffolding (v2/crates/wifi-densepose-signal/src/ruvsense/pose_tracker.rs) are all real code with real tests, not just types.

For end-to-end inference returning calibrated 17-keypoint COCO coordinates with non-zero confidences, no — not in any deployment that doesn't bring its own trained model. That part of your analysis is correct.

2. Are there plans to release pre-trained weights?

Short answer: not as a single bundled drop, because the model is room-coupled. The bigger blocker is the cross-environment generalization gap that's the entire subject of ADR-027 (MERIDIAN). A model trained on a 4×ESP32-S3 layout in one room with one geometry / one furniture configuration / one set of multipath signatures does not transfer cleanly to a different room without recalibration — that's the unsolved hard problem in CSI sensing today, not a thing we have a quiet weights file for. CMU's results work in the room they were trained in. Same for the MM-Fi / Wi-Pose datasets.

What's realistic to ship is a starter checkpoint plus a documented fine-tune-in-your-room pipeline. The training crate (v2/crates/wifi-densepose-train) and ADR-015 (MM-Fi + Wi-Pose datasets) and ADR-016 (RuVector training pipeline integration) describe that pipeline; ground-truth capture against a webcam-driven Mediapipe / MMPose teacher is the intended workflow. None of that is in the public release as a checkpoint yet.

3. What works today without trained weights?

• Presence (presence_score)
• Motion energy (motion_score)
• Breathing rate (respiration_bpm) — physics-based, not learned
• Heart rate (heartbeat_bpm) — same
• Multi-person tracking IDs and per-person bounding zones (zone: "tracked") — geometric, not learned
• Anomaly / fall detection — also physics-based

These are reliable on real ESP32-S3 hardware today. What you do NOT get without training is calibrated 17-keypoint COCO coordinates — those will return shape-correct skeletons with confidence: 0 (see #510 for the exact JSON shape this produces).

On the SISO vs. MIMO point — your analysis is right that ESP32-S3 is 1×1 SISO and CMU used 3×3. The project's bet is that 4 spatially diverse 1×1 nodes with TDM + cross-viewpoint fusion + AETHER re-ID can recover enough angular information for room-scale tracking, which is plausible for presence/motion/biometrics and is exactly what the multi-node fusion math is computing. For full 17-keypoint pose against research-grade ground truth, that bet is more speculative and unproven outside trained rooms.

I'd rather say this clearly than have the dashboard's animated skeletons read as a working pose estimator that just needs config tweaking. It doesn't, today.

If the presence/motion/biometrics path is enough for what you're doing, I think the project earns its keep. If you specifically need 17-keypoint pose, the realistic next step is the training pipeline, not a config flag.

---

Generated by Claude Code