diff --git a/.github/workflows/license.yml b/.github/workflows/license.yml index e235670..0d4f1f7 100644 --- a/.github/workflows/license.yml +++ b/.github/workflows/license.yml @@ -53,6 +53,10 @@ jobs: "LICENSES/crossterm-MIT.txt" "LICENSES/rerun-Apache-2.0.txt" "LICENSES/rerun-MIT.txt" + "LICENSES/OFL-1.1.txt" + "LICENSES/Ubuntu-font-1.0.txt" + "LICENSES/epaint_default_fonts-OFL-1.1.txt" + "LICENSES/epaint_default_fonts-Ubuntu-font-1.0.txt" "LICENSES/unitree_sdk2-BSD-3-Clause.txt" "LICENSES/nvidia-open-model-license.txt" ) diff --git a/AGENTS.md b/AGENTS.md index 5d75a72..374795a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,171 +1,92 @@ # AGENTS.md -This file defines the default operating playbook for coding agents working in this repository. -Its scope is the entire repo tree rooted at this directory. +Repository-wide instructions for coding agents working in this tree. -## 0) First-read policy (mandatory) +## Read First -Before running any command, read in this order: +Before commands or edits, read: -1. `AGENTS.md` (this file) -2. `CLAUDE.md` -3. `Cargo.toml` (workspace root) — workspace members, dependencies -4. `docs/founding-document.md` — project research, design decisions, architecture +1. `AGENTS.md` +2. `Cargo.toml` +3. `docs/founding-document.md` +4. `docs/agents/domain.md` -If instructions conflict, priority is: -**system/developer/user prompt > AGENTS.md > CLAUDE.md > inferred defaults**. +Claude Code agents should also read `CLAUDE.md` for Claude-specific deltas. ---- +Instruction priority is: +system/developer/user prompt > `AGENTS.md` > tool-specific files > inferred defaults. -## Agent skills +## Project Context -### Issue tracker +RoboWBC is a Linux-first Rust workspace for humanoid whole-body-control policy +inference, with PyO3/Python surfaces for users who do not own the Rust host +loop directly. Preserve explicit diagnostics, runtime safety, reproducible +commands, and the registry-driven `WbcPolicy` architecture. -Issues and PRDs are tracked in GitHub Issues for `MiaoDX/robowbc` using GitHub MCP tools. See `docs/agents/issue-tracker.md`. +Use these orientation surfaces: -### Triage labels +- Human overview: `README.md`, `ARCHITECTURE.md`, `STATUS.md`, `docs/human/` +- Domain and design context: `docs/founding-document.md`, `docs/architecture.md` +- Agent runbooks: `docs/agents/README.md` -Use the default five-label triage vocabulary. See `docs/agents/triage-labels.md`. +## Verification -### Domain docs +Do not run tests first in a fresh environment. Complete the Rust preflight in +`docs/agents/rust-workflow.md`: toolchain, build, then check. -Single-context repo; read the root project docs as the canonical domain context. See `docs/agents/domain.md`. - ---- - -## 1) Environment preflight (mandatory before tests) - -Do not run tests immediately on a fresh environment. -Always complete dependency preflight first. - -### 1.1 Rust toolchain - -```bash -rustc --version # expect 1.75+ -cargo --version -``` - -If the Rust toolchain is not installed: - -```bash -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y -source "$HOME/.cargo/env" -``` - -### 1.2 Build verification - -```bash -cargo build 2>&1 -``` - -If build fails due to missing system dependencies (e.g., CUDA, protobuf), report the exact blocker. - -### 1.3 Fast sanity check before tests - -```bash -cargo check 2>&1 -``` - ---- - -## 2) Standard test workflow - -### 2.1 Full tests (default) - -```bash -cargo test -``` - -### 2.2 Focused debugging loop +For Rust changes, normally run: ```bash -cargo test -p robowbc-core -- test_name +make test +make clippy +make fmt-check ``` -### 2.3 Lint and format - -```bash -cargo clippy -- -D warnings -cargo fmt --check -``` - -Before finishing work, run full `cargo test` + `cargo clippy` at least once. - ---- - -## 3) Lint/format checks for code changes +Run `make rust-doc` when public Rust APIs or doc links change. For Python, +site, benchmark, MuJoCo, or SDK changes, use the focused commands in +`docs/agents/testing.md`. -Run when Rust code changes are made: +Do not claim a command passed unless it ran in the current environment. If a +check is blocked by CUDA, MuJoCo, protobuf, model downloads, display/OpenGL, or +other system dependencies, report the exact blocker and the skipped command. -```bash -cargo clippy -- -D warnings # lint -cargo fmt --check # format check -cargo doc --no-deps 2>&1 # doc generation (catch broken doc links) -``` - -If a check cannot run because of environment limits, report the exact blocker. +## Protected Demo Path ---- +`make demo-keyboard` is the "git clone and see it work" path. Do not weaken the +GR00T scene wrapper, support band, engage guard, init-pose settle window, or +viewer lighting without equivalent verification. See +`docs/agents/keyboard-demo.md`. -## 4) Operational best practices +## GitHub And Commits -1. **Fail fast with clear diagnostics**: prefer explicit errors over silent fallbacks. -2. **No hidden dependency assumptions**: always derive requirements from `Cargo.toml`. -3. **Reproducible command logs**: report exact commands and outcomes. -4. **Small, verifiable steps**: build → check → test → lint. -5. **No dependency drift in fixes**: avoid ad-hoc single-crate installs unless doing triage. +Use GitHub MCP tools for issues, PRDs, PRs, comments, and labels. Do not assume +the `gh` CLI is installed. -### 4.1 Keyboard demo guardrail +This repository uses rebase-only merges. Keep commits scoped and descriptive. +Codex-authored commits must include: -`make demo-keyboard` is the "git clone and see it work" path. Keep -`configs/demo/gear_sonic_keyboard_mujoco.toml` on the GR00T scene wrapper and -the explicit `[sim.elastic_band]` support band anchored from the initialized -pose unless replacing them with an equally verified upstream-style startup -path. Keep `[runtime].require_engage = true` and `[runtime].init_pose_secs` -enabled so `]` engages policy only after the robot settles. Keep the demo -scene visibly lit enough to inspect foot contact in the MuJoCo viewer. For -MuJoCo demo changes, run the targeted stability test: - -```bash -MUJOCO_DOWNLOAD_DIR="$(pwd)/.cache/mujoco" \ -MUJOCO_DYNAMIC_LINK_DIR="$(pwd)/.cache/mujoco/mujoco-3.6.0/lib" \ -LD_LIBRARY_PATH="$(pwd)/.cache/mujoco/mujoco-3.6.0/lib:${LD_LIBRARY_PATH:-}" \ -cargo test -p robowbc-sim --features mujoco-auto-download \ - gear_sonic_demo_model_holds_default_pose_for_startup_window +```text +Co-authored-by: Codex ``` ---- - -## 5) Quick command checklist (copy/paste) +Read `docs/agents/github-workflow.md` before PR review, CI triage, or commit +work. -```bash -# 1) Verify Rust toolchain -rustc --version && cargo --version - -# 2) Build -cargo build - -# 3) Run all tests -cargo test - -# 4) Lint + format -cargo clippy -- -D warnings && cargo fmt --check -``` +## Working Tree Safety ---- +The worktree may contain user changes. Do not revert changes you did not make. +If existing changes affect your task, inspect them and work with them; ask only +when they make the task impossible. -## 6) Commit hygiene +Do not add generated models, MuJoCo downloads, benchmark output, site bundles, +or caches unless the repository already tracks that exact artifact type. -- Keep commits scoped and descriptive (`docs: ...`, `fix: ...`, etc.). -- This repository only allows **rebase merge** — squash and merge commits are disabled. -- When changing workflow/docs, ensure instructions match actual repo configuration. -- Do not claim UT success unless the relevant test command (for example - `cargo test` or `pytest -q`) has been run in the current environment. -- If a commit is created by Codex, include the Git trailer - `Co-authored-by: Codex ` in the commit message. -- If a commit is created by another AI coding agent, include a corresponding - co-author trailer so agent usage can be tracked later. -- If you maintain a dedicated bot/user account, prefer that account's verified - noreply email for the relevant agent trailer. +## Preferred Skill Routing ---- +- `$intuitive-init` refreshes agent guidance. +- `$intuitive-doc` maintains the human documentation surface. +- `$intuitive-layout` proposes bounded repository layout improvements. +- `$intuitive-tests` improves test organization and behavior quality. +- `$intuitive-flow` turns fuzzy ideas into planned execution. +- `$intuitive-refactor` sets a bounded refactor goal before broad cleanup. +- `$intuitive-squash` cleans local agent commit history before handoff. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..34dbd0a --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,108 @@ +# Architecture + +RoboWBC is an embedded runtime for humanoid whole-body-control policy inference. +The codebase keeps the policy contract small, then lets configs choose the +policy backend, robot model, transport, teleop mode, reporting, and visualization +path. + +For deeper implementation detail, read `docs/architecture.md`. + +## Runtime Shape + +```text +TOML config or Python SDK + | + v +robowbc-config + robowbc-registry + | + v +WbcPolicy implementation + | + v +Observation -> predict -> JointPositionTargets + | + v +hardware, MuJoCo, synthetic transport, JSON report, Rerun trace, static site +``` + +The same core policy contract is used by the CLI, Python SDK, MuJoCo examples, +benchmark helpers, and generated HTML policy reports. + +## Core Contracts + +- `Observation`: joint positions, joint velocities, gravity vector, command, + and timestamp. +- `WbcCommand`: velocity, motion tokens, joint targets, kinematic pose, and + related command shapes supported by individual policies. +- `JointPositionTargets`: policy output sent toward PD-controlled robot joints. +- `WbcPolicy`: the trait implemented by every runtime policy. +- `PolicyCapabilities`: the caller-facing contract for supported commands and + policy limits. +- `RobotConfig`: joint names, default pose, gains, and limits for a robot + embodiment. + +Unsupported commands should fail explicitly instead of falling back silently. + +## Crate Map + +| Crate | Responsibility | +|-------|----------------| +| `robowbc-core` | Core policy, observation, command, target, robot, capability, and validator types | +| `robowbc-config` | Typed TOML config loading, defaults, path resolution, and validation | +| `robowbc-registry` | Inventory-based policy registration and config-driven construction | +| `robowbc-ort` | ONNX Runtime backend plus first-party policy wrappers | +| `robowbc-pyo3` | Python-backed runtime policy backend for user-supplied modules | +| `robowbc-py` | Standalone Python SDK built with maturin | +| `robowbc-runtime` | Runtime finite-state orchestration around policy, validator, teleop, and transport | +| `robowbc-comm` | Communication-oriented control-loop helpers and Unitree G1 wiring | +| `robowbc-transport` | Pluggable transport traits and CycloneDDS/in-memory backends | +| `robowbc-sim` | MuJoCo transport for hardware-free execution | +| `robowbc-teleop` | Keyboard teleop and configurable keymaps | +| `robowbc-vis` | Rerun visualization and robot scene logging | +| `robowbc-cli` | `robowbc` binary, config-driven runs, reports, and policy helpers | +| `unitree-hg-idl` | Unitree HG message serialization helpers | + +## Policy Backends + +Live public-asset policies: + +- `gear_sonic` +- `decoupled_wbc` +- `wbc_agile` +- `bfm_zero` + +Available but asset-limited wrappers: + +- `hover`: wrapper exists, but public upstream does not ship a pretrained + checkpoint. +- `wholebody_vla`: contract wrapper exists, but public upstream does not expose + a runnable inference release. +- `py_model`: user-supplied Python or PyTorch backend through `robowbc-pyo3`. + +## Execution Surfaces + +- CLI: `cargo run --bin robowbc -- run --config `. +- Python SDK: `Registry`, `Observation`, `Policy`, command classes, and + `MujocoSession`. +- Makefile: stable repo-level commands for build, validation, site generation, + SDK verification, benchmark generation, and keyboard demo. +- Static reports: scripts generate JSON, Rerun, proof-pack, and HTML site + artifacts consumed by the published policy report. + +## Safety And Proof Boundaries + +- Runtime outputs remain joint position targets, not direct torque commands. +- Validators and robot configs own dimension, limit, and safety checks. +- Linux is the verified runtime target. +- `make demo-keyboard` is the protected clone-and-see-it-work path. +- Public report artifacts are evidence, not a replacement for model or hardware + validation. + +## Deliberately Out Of Scope For The Public Surface + +- A new server or daemon API. +- A public ROS 2 or zenoh customer API. +- Training workflows. +- Real-time world-model control. +- Additional model families without runnable public assets or explicit + user-supplied checkpoints. diff --git a/CLAUDE.md b/CLAUDE.md index 000898e..7db210a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,120 +1,27 @@ -# RoboWBC +# CLAUDE.md -Unified inference runtime for humanoid whole-body control policies. Rust core with Python bindings (PyO3). +Claude-specific instructions for RoboWBC. Repository-wide rules live in +`AGENTS.md`; read and follow that first. -## Build & test +## Claude Workflow -```bash -cargo build # build all crates -cargo test # run all tests -cargo clippy -- -D warnings # lint -cargo fmt --check # format check -cargo bench # run benchmarks -cargo doc --no-deps --open # generate and open docs -``` +- Use subagents only for independent work that can safely run in parallel with + the main task. +- Keep the main context focused on decisions, integration, and verification. +- Match model strength to task complexity when the host offers model choice. -## Code style +## PR Review -- `rustfmt` enforces formatting — do not duplicate formatter rules here -- `clippy` enforces lints — treat warnings as errors (`-D warnings`) -- Type annotations on all public APIs; doc comments (`///`) on public items -- Error handling: use `thiserror` for library errors, `anyhow` for binary/CLI errors +When reviewing a PR, push fixes directly to the PR source branch unless the +user asks otherwise. Read the actual CI logs before diagnosing failures; do not +stop at status summaries. -## Architecture +Detailed workflow: `docs/agents/github-workflow.md`. -``` -crates/ -├── robowbc-core/ — WbcPolicy trait, Observation, WbcCommand, JointPositionTargets, RobotConfig -├── robowbc-ort/ — ONNX Runtime inference backend (ort crate, CUDA/TensorRT) -├── robowbc-pyo3/ — PyO3 Python inference backend (PyTorch models) -├── robowbc-registry/ — inventory-based policy registration and factory -├── robowbc-comm/ — zenoh communication layer (robot hardware I/O) -└── robowbc-cli/ — CLI entry point (config loading → control loop) -``` +## Repo Pointers -Key pattern: `WbcPolicy` is a trait (`Send + Sync`). New policies implement it and register via `inventory::submit!`. Config-driven instantiation via `WbcRegistry::build(name, config)`. - -## Git workflow - -- Branch from `main` -- Commit messages: `type: description` (feat, fix, ci, docs, refactor) -- Merge method: **rebase only** — squash and merge commits are disabled on this repository -- CI runs on all PRs: `cargo check` + `cargo clippy` + `cargo fmt --check` + `cargo test` - -### PR review strategy - -When reviewing a PR, **push fixes directly to the PR's source branch** instead of creating a new branch or a separate PR. This keeps the workflow simple — one PR, one place to review, one merge. - -1. Fetch and check out the PR's source branch -2. Make fixes, run tests/lint, commit -3. Push to the same branch -4. The new commit appears in the existing PR - -Do NOT create a new branch or a new PR for review fixes. - -## CI failure investigation - -When a CI check fails, **always read the actual logs/error messages** before diagnosing. Do not stop at the status summary. Specifically: - -1. Identify which checks failed -2. Read the full error output -3. Only after reading the actual error messages, diagnose and fix - -## Gotchas - -- ONNX models require matching `ort` version + ONNX opset version. Pin `ort` version in `Cargo.toml`. -- CUDA/TensorRT execution providers require matching CUDA toolkit version on the host. -- `zenoh` communication requires matching protocol version between peers. -- `inventory` crate requires all registered types to be in crates linked into the final binary (no dynamic loading). -- PyO3 backend requires Python 3.10+ and GIL-aware thread management. -- GEAR-SONIC has three ONNX models (`model_encoder.onnx`, `model_decoder.onnx`, `planner_sonic.onnx`) that must be loaded together. - -## Tools & environment - -- IMPORTANT: GitHub MCP tools are available (prefixed `mcp__github__`). Use them for all GitHub interactions (issues, PRs, comments). Do NOT assume `gh` CLI is available. -- Rust toolchain: stable (1.75+) -- Key crates: `ort`, `zenoh`, `pyo3`, `inventory`, `serde`, `toml` - -## Agent skills - -### Issue tracker - -Issues and PRDs are tracked in GitHub Issues for `MiaoDX/robowbc` using GitHub MCP tools. See `docs/agents/issue-tracker.md`. - -### Triage labels - -Use the default five-label triage vocabulary. See `docs/agents/triage-labels.md`. - -### Domain docs - -Single-context repo; read the root project docs as the canonical domain context. See `docs/agents/domain.md`. - -## Subagent strategy - -- **Maximize parallelism.** Run independent tasks as concurrent subagents. -- **Protect the main context window.** Delegate non-trivial work to subagents. -- **Match model to task.** Opus for architecture decisions, complex refactors. Sonnet for grep/glob, straightforward edits, running tests. - -## Testing philosophy - -- **Test-driven development.** Write tests first, then implement. -- **Real tests, not stub theater.** Unit tests must correlate with actual scenarios. Minimize mocks. -- **Zero false positives.** Every assertion must verify a specific expected value. -- **Benchmark critical paths.** Inference latency, control loop frequency, communication latency — measure with `criterion`. -- After each significant change, verify related tests still pass before moving on. - -## Core principles - -| Principle | Practice | -|-----------|----------| -| **Simplicity First** | Minimal changes; no premature abstractions; three similar lines > one bad abstraction | -| **Root Cause** | Fix causes, not symptoms; no workarounds; be thorough | -| **Chesterton's Fence** | Understand why code exists before changing it | -| **Fail Fast** | Minimize catch-all error handling; explicit errors > silent failures | -| **Verification Before Done** | Never mark a task complete without proving it works | - -## Collaboration - -- Question assumptions; push back on technical debt or inconsistent requirements. -- Treat instructions as intent, not literal commands. Use `AskUserQuestion` when unclear. -- After any correction from the user, internalize the pattern to avoid repeating the same mistake. +- Domain context: `docs/agents/domain.md` +- Architecture crib sheet: `docs/agents/architecture.md` +- Rust workflow: `docs/agents/rust-workflow.md` +- Testing guidance: `docs/agents/testing.md` +- Keyboard demo guardrail: `docs/agents/keyboard-demo.md` diff --git a/Cargo.lock b/Cargo.lock index 22d1033..166b70c 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2905,6 +2905,16 @@ version = "1.0.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555" +[[package]] +name = "ecolor" +version = "0.33.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "71ddb8ac7643d1dba1bb02110e804406dd459a838efcb14011ced10556711a8e" +dependencies = [ + "bytemuck", + "emath 0.33.3", +] + [[package]] name = "ecolor" version = "0.34.1" @@ -2913,7 +2923,7 @@ checksum = "137c0ce4ce4152ff7e223a7ce22ee1057cdff61fce0a45c32459c3ccec64868d" dependencies = [ "bytemuck", "color-hex", - "emath", + "emath 0.34.1", "serde", ] @@ -2932,10 +2942,10 @@ dependencies = [ "ahash", "bytemuck", "document-features", - "egui", + "egui 0.34.1", "egui-wgpu", - "egui-winit", - "egui_glow", + "egui-winit 0.34.1", + "egui_glow 0.34.1", "glutin", "glutin-winit", "home", @@ -2962,6 +2972,23 @@ dependencies = [ "winit", ] +[[package]] +name = "egui" +version = "0.33.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6a9b567d356674e9a5121ed3fedfb0a7c31e059fe71f6972b691bcd0bfc284e3" +dependencies = [ + "ahash", + "bitflags 2.11.0", + "emath 0.33.3", + "epaint 0.33.3", + "log", + "nohash-hasher", + "profiling", + "smallvec", + "unicode-segmentation", +] + [[package]] name = "egui" version = "0.34.1" @@ -2972,8 +2999,8 @@ dependencies = [ "ahash", "backtrace", "bitflags 2.11.0", - "emath", - "epaint", + "emath 0.34.1", + "epaint 0.34.1", "log", "nohash-hasher", "profiling", @@ -2992,8 +3019,8 @@ dependencies = [ "ahash", "bytemuck", "document-features", - "egui", - "epaint", + "egui 0.34.1", + "epaint 0.34.1", "log", "profiling", "thiserror 2.0.18", @@ -3003,6 +3030,24 @@ dependencies = [ "winit", ] +[[package]] +name = "egui-winit" +version = "0.33.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec6687e5bb551702f4ad10ac428bab12acf9d53047ebb1082d4a0ed8c6251a29" +dependencies = [ + "bytemuck", + "egui 0.33.3", + "log", + "objc2 0.5.2", + "objc2-foundation 0.2.2", + "objc2-ui-kit 0.2.2", + "profiling", + "raw-window-handle", + "web-time", + "winit", +] + [[package]] name = "egui-winit" version = "0.34.1" @@ -3012,7 +3057,7 @@ dependencies = [ "accesskit_winit", "arboard", "bytemuck", - "egui", + "egui 0.34.1", "log", "objc2 0.6.4", "objc2-foundation 0.3.2", @@ -3032,7 +3077,7 @@ version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dd9bc6d586df44e01b90715eec1eb8d73a61c3c3f554edffb01eb0894a8107ef" dependencies = [ - "egui", + "egui 0.34.1", "hello_egui_utils", "simple-easing", ] @@ -3043,7 +3088,7 @@ version = "0.23.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "55df531ff51161b3c6212e0ee2166b370f150254bf4448a8a15c3d26fec87958" dependencies = [ - "egui", + "egui 0.34.1", "egui_commonmark_backend", "egui_extras", "pulldown-cmark", @@ -3055,7 +3100,7 @@ version = "0.23.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fe04399ca5a2196965833a2918e50400449721fd9350e31ae7d84d6690859437" dependencies = [ - "egui", + "egui 0.34.1", "egui_extras", "pulldown-cmark", ] @@ -3066,7 +3111,7 @@ version = "0.15.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "51a348b3fdbc048c4241aaa2865255e1fdebbc0099324ded8c5b534e598e600c" dependencies = [ - "egui", + "egui 0.34.1", "egui_animation", "simple-easing", "web-time", @@ -3079,7 +3124,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "62bfc6870c68d3f254e33aca8200095d422e09edacb0f365f79fe23a5ba10963" dependencies = [ "ahash", - "egui", + "egui 0.34.1", "ehttp", "enum-map", "image", @@ -3090,6 +3135,24 @@ dependencies = [ "serde", ] +[[package]] +name = "egui_glow" +version = "0.33.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6420863ea1d90e750f75075231a260030ad8a9f30a7cef82cdc966492dc4c4eb" +dependencies = [ + "bytemuck", + "egui 0.33.3", + "egui-winit 0.33.3", + "glow 0.16.0", + "log", + "memoffset", + "profiling", + "wasm-bindgen", + "web-sys", + "winit", +] + [[package]] name = "egui_glow" version = "0.34.1" @@ -3097,8 +3160,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a3b28d39ab6c0cac238190e6cb1e8c9047d02cb470ab942a7a3302e4cb3a8e74" dependencies = [ "bytemuck", - "egui", - "glow", + "egui 0.34.1", + "glow 0.17.0", "log", "memoffset", "profiling", @@ -3114,8 +3177,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d7bd66213736bf9a9a53dc4888570b9194fc0db906507517a7fcc787e888ac47" dependencies = [ "ahash", - "egui", - "emath", + "egui 0.34.1", + "emath 0.34.1", ] [[package]] @@ -3124,7 +3187,7 @@ version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8512decdd471a2b6106d0b42cc0662f0e94b0ca8f21bc1b0075f455f58901010" dependencies = [ - "egui", + "egui 0.34.1", "serde", "vec1", ] @@ -3136,7 +3199,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "08e570b77f6cce3292eba4aee9b9c08cf11dfc68430f4dc9613d939628498647" dependencies = [ "ahash", - "egui", + "egui 0.34.1", "itertools 0.14.0", "log", "serde", @@ -3167,6 +3230,15 @@ version = "1.15.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719" +[[package]] +name = "emath" +version = "0.33.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "491bdf728bf25ddd9ad60d4cf1c48588fa82c013a2440b91aa7fc43e34a07c32" +dependencies = [ + "bytemuck", +] + [[package]] name = "emath" version = "0.34.1" @@ -3285,6 +3357,24 @@ dependencies = [ "log", ] +[[package]] +name = "epaint" +version = "0.33.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "009d0dd3c2163823a0abdb899451ecbc78798dec545ee91b43aff1fa790bab62" +dependencies = [ + "ab_glyph", + "ahash", + "bytemuck", + "ecolor 0.33.3", + "emath 0.33.3", + "epaint_default_fonts 0.33.3", + "log", + "nohash-hasher", + "parking_lot", + "profiling", +] + [[package]] name = "epaint" version = "0.34.1" @@ -3293,9 +3383,9 @@ checksum = "04f3017dd67f147a697ee0c8484fb568fd9553e2a0c114be5020dbbc11962841" dependencies = [ "ahash", "bytemuck", - "ecolor", - "emath", - "epaint_default_fonts", + "ecolor 0.34.1", + "emath 0.34.1", + "epaint_default_fonts 0.34.1", "font-types", "log", "nohash-hasher", @@ -3309,6 +3399,12 @@ dependencies = [ "vello_cpu", ] +[[package]] +name = "epaint_default_fonts" +version = "0.33.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c4fbe202b6578d3d56428fa185cdf114a05e49da05f477b3c7f0fbb221f1862" + [[package]] name = "epaint_default_fonts" version = "0.34.1" @@ -3826,6 +3922,18 @@ version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280" +[[package]] +name = "glow" +version = "0.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c5e5ea60d70410161c8bf5da3fdfeaa1c72ed2c15f8bbb9d19fe3a4fad085f08" +dependencies = [ + "js-sys", + "slotmap", + "wasm-bindgen", + "web-sys", +] + [[package]] name = "glow" version = "0.17.0" @@ -4071,7 +4179,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c34bfd8bff6f6df43b0b73ed7949a7aff0c98c2c1bd4c2f2771f5f2f6d98ced0" dependencies = [ "concat-idents", - "egui", + "egui 0.34.1", ] [[package]] @@ -5345,6 +5453,9 @@ checksum = "82c4e2adc7cf4b96a8b4e53290a206f40304d72c805df3034beae1b3f1f9eff3" dependencies = [ "bitflags 2.11.0", "bytemuck", + "egui 0.33.3", + "egui-winit 0.33.3", + "egui_glow 0.33.3", "flate2", "glutin", "glutin-winit", @@ -7279,7 +7390,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "71cde26c1f0ba8ea7ce2fd678efded0acdd6cf13bfd306930fc234d0ed8d5802" dependencies = [ "arrow", - "egui", + "egui 0.34.1", "jiff", "re_arrow_util", "re_format", @@ -7364,7 +7475,7 @@ version = "0.31.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2227563203d318c3029039a5a657464ce186d2f7b7db0d8738294dedbf1dcd78" dependencies = [ - "egui", + "egui 0.34.1", "egui_tiles", "itertools 0.14.0", "re_context_menu", @@ -7412,7 +7523,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3d92144303a1b0753c8734cab0ab2a03841f424b0e0e68e82c9ec7b7e80eff75" dependencies = [ "arrow", - "ecolor", + "ecolor 0.34.1", "glam", "half", "parking_lot", @@ -7427,7 +7538,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b200b033df92d7ad028182d2549df6f48742fae07c3c4efba3818f5638567223" dependencies = [ "document-features", - "egui", + "egui 0.34.1", "re_log", "static_assertions", ] @@ -7510,7 +7621,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0bc6d008fc16994e3cdd54d3a22b0ec6aae8fb9fdb9116612a9ea9de18c58b9f" dependencies = [ "arrow", - "egui", + "egui 0.34.1", "egui_extras", "itertools 0.14.0", "re_arrow_ui", @@ -7542,7 +7653,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "548546f57ca4d276255fe346edc8f7b4f343d26e4032288e63b159b0e847078b" dependencies = [ "arrow", - "egui", + "egui 0.34.1", "egui_dnd", "egui_extras", "egui_plot", @@ -7565,7 +7676,7 @@ version = "0.31.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e1e184e6eba759242cdc88dea6e4d8774e05ae942243411e993c71a7fe76bedf" dependencies = [ - "egui", + "egui 0.34.1", "egui_tiles", "itertools 0.14.0", "nohash-hasher", @@ -7675,7 +7786,7 @@ dependencies = [ "ahash", "anyhow", "bytemuck", - "egui", + "egui 0.34.1", "egui_extras", "egui_plot", "itertools 0.14.0", @@ -7739,7 +7850,7 @@ dependencies = [ "cfg-if", "crossbeam", "datafusion", - "egui", + "egui 0.34.1", "egui_dnd", "egui_table", "futures", @@ -7809,7 +7920,7 @@ dependencies = [ "ahash", "arrow", "document-features", - "emath", + "emath 0.34.1", "indexmap 2.13.1", "itertools 0.14.0", "nohash-hasher", @@ -8114,7 +8225,7 @@ version = "0.31.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1fec2fad201702ad66ae4b594e7326371b5b2d508df5e64fe6d9aafa9439c184" dependencies = [ - "egui", + "egui 0.34.1", "re_byte_size", "re_format", ] @@ -8278,7 +8389,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6f8114b0dbe8a3844fb26ad93d7ba0ca945040054afdabf106fd449a083c49c7" dependencies = [ "ahash", - "egui", + "egui 0.34.1", "itertools 0.14.0", "re_data_ui", "re_entity_db", @@ -8303,7 +8414,7 @@ dependencies = [ "cfg-if", "crossbeam", "datafusion", - "egui", + "egui 0.34.1", "egui_extras", "futures", "itertools 0.14.0", @@ -8381,7 +8492,7 @@ dependencies = [ "crossbeam", "dae-parser", "document-features", - "ecolor", + "ecolor 0.34.1", "enumset", "getrandom 0.3.4", "glam", @@ -8491,9 +8602,9 @@ dependencies = [ "arrow", "bytemuck", "document-features", - "ecolor", + "ecolor 0.34.1", "egui_plot", - "emath", + "emath 0.34.1", "glam", "half", "image", @@ -8528,7 +8639,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1e9a59c20ccb81f1117780498853a83ae822d4fe6862986809da5c8bc70dd286" dependencies = [ "arrow", - "egui", + "egui 0.34.1", "egui_tiles", "itertools 0.14.0", "nohash-hasher", @@ -8632,7 +8743,7 @@ version = "0.31.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d44c6ad1ce043303b7ec74087d4122013276a0d9c6471dd5424433aeac3a1a38" dependencies = [ - "egui", + "egui 0.34.1", "itertools 0.14.0", "nohash-hasher", "re_chunk", @@ -8716,7 +8827,7 @@ dependencies = [ "ahash", "anyhow", "eframe", - "egui", + "egui 0.34.1", "egui_commonmark", "egui_extras", "egui_tiles", @@ -8805,7 +8916,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b16e0186f7eca8244ada2adb345ea36ab0a29a6ed9984725656174a837357d12" dependencies = [ "ahash", - "egui", + "egui 0.34.1", "glam", "itertools 0.14.0", "nohash-hasher", @@ -8832,7 +8943,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e1488d35cd9a19d906499cd675249b07b49d2472a4e79e929622667236e9e865" dependencies = [ "arrow", - "egui", + "egui 0.34.1", "egui_plot", "re_chunk_store", "re_entity_db", @@ -8853,7 +8964,7 @@ checksum = "00f0c6b70b5c50fac7daf69f1201f69cad982930c1f500c1e9fa579ec5838d46" dependencies = [ "anyhow", "arrow", - "egui", + "egui 0.34.1", "egui_dnd", "egui_table", "itertools 0.14.0", @@ -8880,7 +8991,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e32c971e88b8ea6a17938d9ff64e62b71b41e5dbaea8fa4c8c8f3cbed8578f6d" dependencies = [ "ahash", - "egui", + "egui 0.34.1", "fjadra", "itertools 0.14.0", "nohash-hasher", @@ -8906,7 +9017,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "45b01528c861bbb3bd9ccb14d5b3be981c2ce9f91b7eb68f5d12291d0e3a1cb2" dependencies = [ "bytemuck", - "egui", + "egui 0.34.1", "glam", "itertools 0.14.0", "macaw", @@ -8936,7 +9047,7 @@ dependencies = [ "arrow", "bitflags 2.11.0", "bytemuck", - "egui", + "egui 0.34.1", "glam", "hexasphere", "image", @@ -8980,7 +9091,7 @@ checksum = "9e010883859e5ca8003ef9b450fd74d07a83827cfb4c987594c9e8fd451a2f0e" dependencies = [ "anyhow", "bytemuck", - "egui", + "egui 0.34.1", "half", "ndarray 0.16.1", "re_chunk_store", @@ -9004,7 +9115,7 @@ version = "0.31.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "761ba77c9b421ae4da0fb5cb79b70941c083d72620b60c3f96538acbf7b3fd47" dependencies = [ - "egui", + "egui 0.34.1", "egui_commonmark", "re_chunk_store", "re_sdk_types", @@ -9020,7 +9131,7 @@ version = "0.31.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f06f5f706af7e63cd9a2f8b646ba7e38fbc8ffced456df5a88216861ce6a9ec2" dependencies = [ - "egui", + "egui 0.34.1", "egui_extras", "itertools 0.14.0", "re_chunk_store", @@ -9044,7 +9155,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "77aba3707d959445ae4ea931440100aef7d7520e020c2ad333d8db1cd83da78d" dependencies = [ "arrayvec", - "egui", + "egui 0.34.1", "egui_plot", "itertools 0.14.0", "nohash-hasher", @@ -9079,11 +9190,11 @@ dependencies = [ "cfg-if", "crossbeam", "eframe", - "egui", + "egui 0.34.1", "egui-wgpu", "egui_plot", "ehttp", - "emath", + "emath 0.34.1", "glam", "image", "itertools 0.14.0", @@ -9179,10 +9290,10 @@ dependencies = [ "crossbeam", "datafusion", "directories", - "egui", + "egui 0.34.1", "egui-wgpu", "egui_tiles", - "emath", + "emath 0.34.1", "glam", "half", "home", @@ -9247,7 +9358,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "82754c9097bdd26830fdea4a99ca3cc28053ec5236d35d77c1e5a99bcb6f14e3" dependencies = [ "ahash", - "egui", + "egui 0.34.1", "egui_tiles", "nohash-hasher", "rayon", @@ -9273,7 +9384,7 @@ checksum = "e8659b1f204e158e3854ccd5424c8c423f87eb01bc8aa209079d5ccb502fe5fb" dependencies = [ "ahash", "arrow", - "egui", + "egui 0.34.1", "egui_tiles", "itertools 0.14.0", "nohash-hasher", @@ -11921,7 +12032,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1c81bab51dc24106d35edce41958ed16e3a0c1c0b45f40685ffd3f4b5691490c" dependencies = [ "bytes", - "egui", + "egui 0.34.1", "egui_extras", "futures", "geo-types", @@ -12386,7 +12497,7 @@ dependencies = [ "bytemuck", "cfg-if", "cfg_aliases", - "glow", + "glow 0.17.0", "glutin_wgl_sys", "gpu-allocator", "gpu-descriptor", diff --git a/LICENSES/OFL-1.1.txt b/LICENSES/OFL-1.1.txt new file mode 100644 index 0000000..d952d62 --- /dev/null +++ b/LICENSES/OFL-1.1.txt @@ -0,0 +1,92 @@ +This Font Software is licensed under the SIL Open Font License, +Version 1.1. + +This license is copied below, and is also available with a FAQ at: +http://scripts.sil.org/OFL + +----------------------------------------------------------- +SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007 +----------------------------------------------------------- + +PREAMBLE +The goals of the Open Font License (OFL) are to stimulate worldwide +development of collaborative font projects, to support the font +creation efforts of academic and linguistic communities, and to +provide a free and open framework in which fonts may be shared and +improved in partnership with others. + +The OFL allows the licensed fonts to be used, studied, modified and +redistributed freely as long as they are not sold by themselves. The +fonts, including any derivative works, can be bundled, embedded, +redistributed and/or sold with any software provided that any reserved +names are not used by derivative works. The fonts and derivatives, +however, cannot be released under any other type of license. The +requirement for fonts to remain under this license does not apply to +any document created using the fonts or their derivatives. + +DEFINITIONS +"Font Software" refers to the set of files released by the Copyright +Holder(s) under this license and clearly marked as such. This may +include source files, build scripts and documentation. + +"Reserved Font Name" refers to any names specified as such after the +copyright statement(s). + +"Original Version" refers to the collection of Font Software +components as distributed by the Copyright Holder(s). + +"Modified Version" refers to any derivative made by adding to, +deleting, or substituting -- in part or in whole -- any of the +components of the Original Version, by changing formats or by porting +the Font Software to a new environment. + +"Author" refers to any designer, engineer, programmer, technical +writer or other person who contributed to the Font Software. + +PERMISSION & CONDITIONS +Permission is hereby granted, free of charge, to any person obtaining +a copy of the Font Software, to use, study, copy, merge, embed, +modify, redistribute, and sell modified and unmodified copies of the +Font Software, subject to the following conditions: + +1) Neither the Font Software nor any of its individual components, in +Original or Modified Versions, may be sold by itself. + +2) Original or Modified Versions of the Font Software may be bundled, +redistributed and/or sold with any software, provided that each copy +contains the above copyright notice and this license. These can be +included either as stand-alone text files, human-readable headers or +in the appropriate machine-readable metadata fields within text or +binary files as long as those fields can be easily viewed by the user. + +3) No Modified Version of the Font Software may use the Reserved Font +Name(s) unless explicit written permission is granted by the +corresponding Copyright Holder. This restriction only applies to the +primary font name as presented to the users. + +4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font +Software shall not be used to promote, endorse or advertise any +Modified Version, except to acknowledge the contribution(s) of the +Copyright Holder(s) and the Author(s) or with their explicit written +permission. + +5) The Font Software, modified or unmodified, in part or in whole, +must be distributed entirely under this license, and must not be +distributed under any other license. The requirement for fonts to +remain under this license does not apply to any document created using +the Font Software. + +TERMINATION +This license becomes null and void if any of the above conditions are +not met. + +DISCLAIMER +THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT +OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE +COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL +DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM +OTHER DEALINGS IN THE FONT SOFTWARE. diff --git a/LICENSES/README.md b/LICENSES/README.md index df1b246..d15fe40 100644 --- a/LICENSES/README.md +++ b/LICENSES/README.md @@ -8,9 +8,10 @@ runtime. The layout follows SPDX / [REUSE](https://reuse.software/) convention: one file per `-.txt`, plus shared canonical license texts (`Apache-2.0.txt`, `MIT.txt`, `BSD-3-Clause.txt`, -`EPL-2.0.txt`) that the per-component files reference. Per-component -files document the upstream URL, the consuming robowbc crate, and the -SPDX identifier for that component. +`EPL-2.0.txt`, `OFL-1.1.txt`, `Ubuntu-font-1.0.txt`) that the +per-component files reference. Per-component files document the upstream +URL, the consuming robowbc crate, and the SPDX identifier for that +component. ## Index @@ -25,6 +26,8 @@ SPDX identifier for that component. | crossterm | [`crossterm-MIT.txt`](crossterm-MIT.txt) | https://github.com/crossterm-rs/crossterm | Cross-platform terminal input, MIT. | | rerun (Apache) | [`rerun-Apache-2.0.txt`](rerun-Apache-2.0.txt) | https://github.com/rerun-io/rerun | Visualization SDK + viewer, dual licensed `Apache-2.0 OR MIT`. | | rerun (MIT) | [`rerun-MIT.txt`](rerun-MIT.txt) | https://github.com/rerun-io/rerun | MIT branch of the dual license. | +| epaint default fonts (OFL) | [`epaint_default_fonts-OFL-1.1.txt`](epaint_default_fonts-OFL-1.1.txt) | https://github.com/emilk/egui/tree/main/crates/epaint_default_fonts | Default egui viewer font assets pulled transitively through mujoco-rs. | +| epaint default fonts (Ubuntu) | [`epaint_default_fonts-Ubuntu-font-1.0.txt`](epaint_default_fonts-Ubuntu-font-1.0.txt) | https://github.com/emilk/egui/tree/main/crates/epaint_default_fonts | Ubuntu Light font asset pulled transitively through mujoco-rs. | | unitree_sdk2 IDL types | [`unitree_sdk2-BSD-3-Clause.txt`](unitree_sdk2-BSD-3-Clause.txt) | https://github.com/unitreerobotics/unitree_sdk2 | Source of the IDL message shapes ported into `crates/unitree-hg-idl/`. | | NVIDIA Open Model License| [`nvidia-open-model-license.txt`](nvidia-open-model-license.txt) | https://developer.download.nvidia.com/licenses/ | Governs the GEAR-SONIC weights — fetched at runtime from HuggingFace, **never bundled in this repo**. | @@ -43,7 +46,7 @@ When you add a third-party dependency in a PR: 3. If the SPDX expression is not already represented (no `LICENSES/.txt`), add the canonical text for it. 4. Add a row to the index table above. -5. Update the allowlist in `.github/workflows/license.yml` if the new +5. Update the allowlist in [`../deny.toml`](../deny.toml) if the new SPDX identifier is not already accepted. 6. If the new dependency is **strong-copyleft** (GPL-3.0-only, AGPL-3.0-only, etc.), pause and consult a human reviewer — robowbc @@ -57,6 +60,8 @@ When you add a third-party dependency in a PR: * Weak copyleft (EPL-2.0, MPL-2.0, LGPL-3.0): accepted with attribution. CycloneDDS is the most prominent example; dual-licensed under `EPL-2.0 OR BSD-3-Clause`, redistributors may pick the BSD branch. +* Font licenses (OFL-1.1, Ubuntu-font-1.0): accepted for bundled UI font + assets with attribution and license text preservation. * Strong copyleft (GPL, AGPL): rejected by CI. * Model licenses (NVIDIA Open Model License, etc.): governed per-bundle; weights are fetched at runtime, not redistributed. diff --git a/LICENSES/Ubuntu-font-1.0.txt b/LICENSES/Ubuntu-font-1.0.txt new file mode 100644 index 0000000..ae78a8f --- /dev/null +++ b/LICENSES/Ubuntu-font-1.0.txt @@ -0,0 +1,96 @@ +------------------------------- +UBUNTU FONT LICENCE Version 1.0 +------------------------------- + +PREAMBLE +This licence allows the licensed fonts to be used, studied, modified and +redistributed freely. The fonts, including any derivative works, can be +bundled, embedded, and redistributed provided the terms of this licence +are met. The fonts and derivatives, however, cannot be released under +any other licence. The requirement for fonts to remain under this +licence does not require any document created using the fonts or their +derivatives to be published under this licence, as long as the primary +purpose of the document is not to be a vehicle for the distribution of +the fonts. + +DEFINITIONS +"Font Software" refers to the set of files released by the Copyright +Holder(s) under this licence and clearly marked as such. This may +include source files, build scripts and documentation. + +"Original Version" refers to the collection of Font Software components +as received under this licence. + +"Modified Version" refers to any derivative made by adding to, deleting, +or substituting -- in part or in whole -- any of the components of the +Original Version, by changing formats or by porting the Font Software to +a new environment. + +"Copyright Holder(s)" refers to all individuals and companies who have a +copyright ownership of the Font Software. + +"Substantially Changed" refers to Modified Versions which can be easily +identified as dissimilar to the Font Software by users of the Font +Software comparing the Original Version with the Modified Version. + +To "Propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification and with or without charging +a redistribution fee), making available to the public, and in some +countries other activities as well. + +PERMISSION & CONDITIONS +This licence does not grant any rights under trademark law and all such +rights are reserved. + +Permission is hereby granted, free of charge, to any person obtaining a +copy of the Font Software, to propagate the Font Software, subject to +the below conditions: + +1) Each copy of the Font Software must contain the above copyright +notice and this licence. These can be included either as stand-alone +text files, human-readable headers or in the appropriate machine- +readable metadata fields within text or binary files as long as those +fields can be easily viewed by the user. + +2) The font name complies with the following: +(a) The Original Version must retain its name, unmodified. +(b) Modified Versions which are Substantially Changed must be renamed to +avoid use of the name of the Original Version or similar names entirely. +(c) Modified Versions which are not Substantially Changed must be +renamed to both (i) retain the name of the Original Version and (ii) add +additional naming elements to distinguish the Modified Version from the +Original Version. The name of such Modified Versions must be the name of +the Original Version, with "derivative X" where X represents the name of +the new work, appended to that name. + +3) The name(s) of the Copyright Holder(s) and any contributor to the +Font Software shall not be used to promote, endorse or advertise any +Modified Version, except (i) as required by this licence, (ii) to +acknowledge the contribution(s) of the Copyright Holder(s) or (iii) with +their explicit written permission. + +4) The Font Software, modified or unmodified, in part or in whole, must +be distributed entirely under this licence, and must not be distributed +under any other licence. The requirement for fonts to remain under this +licence does not affect any document created using the Font Software, +except any version of the Font Software extracted from a document +created using the Font Software may only be distributed under this +licence. + +TERMINATION +This licence becomes null and void if any of the above conditions are +not met. + +DISCLAIMER +THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF +COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE +COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL +DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM OTHER +DEALINGS IN THE FONT SOFTWARE. diff --git a/LICENSES/epaint_default_fonts-OFL-1.1.txt b/LICENSES/epaint_default_fonts-OFL-1.1.txt new file mode 100644 index 0000000..2e34c18 --- /dev/null +++ b/LICENSES/epaint_default_fonts-OFL-1.1.txt @@ -0,0 +1,12 @@ +Component: epaint_default_fonts - default egui font assets +Upstream: https://github.com/emilk/egui/tree/main/crates/epaint_default_fonts +SPDX-License-Identifier: OFL-1.1 +Used by: robowbc-sim through mujoco-rs / egui viewer dependencies. + +epaint_default_fonts bundles NotoEmoji-Regular.ttf under the SIL Open Font +License 1.1. The full canonical OFL-1.1 text is in ./OFL-1.1.txt in this +directory. + +Copyright notice from upstream font metadata: + + Copyright 2013 Google Inc. All Rights Reserved. diff --git a/LICENSES/epaint_default_fonts-Ubuntu-font-1.0.txt b/LICENSES/epaint_default_fonts-Ubuntu-font-1.0.txt new file mode 100644 index 0000000..d79f26d --- /dev/null +++ b/LICENSES/epaint_default_fonts-Ubuntu-font-1.0.txt @@ -0,0 +1,12 @@ +Component: epaint_default_fonts - default egui font assets +Upstream: https://github.com/emilk/egui/tree/main/crates/epaint_default_fonts +SPDX-License-Identifier: Ubuntu-font-1.0 +Used by: robowbc-sim through mujoco-rs / egui viewer dependencies. + +epaint_default_fonts bundles Ubuntu-Light.ttf under the Ubuntu Font Licence +1.0. The full canonical Ubuntu Font Licence 1.0 text is in +./Ubuntu-font-1.0.txt in this directory. + +Copyright notice from upstream font metadata: + + Copyright 2011 Canonical Ltd. diff --git a/Makefile b/Makefile index 31df75d..a186350 100644 --- a/Makefile +++ b/Makefile @@ -25,7 +25,7 @@ PYTHON_SDK_TARGET_DIR ?= $(CURDIR)/target/python-sdk-wheel .DEFAULT_GOAL := help -.PHONY: help toolchain build build-release check test sim-feature-test clippy fmt fmt-check rust-doc mdbook-install docs-book docs verify smoke demo-keyboard models-public site-python-deps site-render-check benchmark-robowbc benchmark-official benchmark-summary benchmark-nvidia site showcase-verify site-smoke site-browser-smoke site-serve-check site-serve python-sdk-deps python-sdk-build python-sdk-install python-sdk-smoke python-sdk-verify ci +.PHONY: help toolchain build build-release check test python-test sim-feature-test clippy fmt fmt-check rust-doc mdbook-install docs-book docs verify smoke demo-keyboard models-public site-python-deps site-render-check benchmark-robowbc benchmark-official benchmark-summary benchmark-nvidia site showcase-verify site-smoke site-browser-smoke site-serve-check site-serve python-sdk-deps python-sdk-build python-sdk-install python-sdk-smoke python-sdk-verify ci help: ## Show available targets and useful variables. @awk 'BEGIN {FS = ":.*## "; print "Targets:"} /^[a-zA-Z0-9_.-]+:.*## / {printf " %-20s %s\n", $$1, $$2}' $(MAKEFILE_LIST) @@ -61,9 +61,12 @@ check: ## Run cargo check across the workspace and all targets. test: ## Run cargo test across the workspace and all targets. $(CARGO) test --workspace --all-targets +python-test: ## Run Python contract and integration tests. + $(PYTHON) -m pytest tests + sim-feature-test: ## Run the feature-enabled MuJoCo sim transport tests with the runtime/plugin env wired. download_dir="$(abspath $(MUJOCO_DOWNLOAD_DIR))"; \ - "$(PYTHON)" scripts/ensure_mujoco_runtime.py --download-dir "$$download_dir" >/dev/null; \ + "$(PYTHON)" scripts/mujoco/ensure_mujoco_runtime.py --download-dir "$$download_dir" >/dev/null; \ link_dir="$(MUJOCO_DYNAMIC_LINK_DIR)"; \ runtime_ld_library_path="$$link_dir:$${LD_LIBRARY_PATH:-}"; \ if [[ -f "$$link_dir/libmujoco.so" || -f "$$link_dir/libmujoco.dylib" || -f "$$link_dir/mujoco.lib" ]]; then \ @@ -107,9 +110,9 @@ smoke: ## Run the no-download local decoupled_wbc smoke config. $(CARGO) run --bin robowbc -- run --config configs/decoupled_smoke.toml demo-keyboard: ## Run the interactive GEAR-Sonic + MuJoCo keyboard demo. - bash scripts/download_gear_sonic_models.sh models/gear-sonic + bash scripts/models/download_gear_sonic_models.sh models/gear-sonic download_dir="$(abspath $(MUJOCO_DOWNLOAD_DIR))"; \ - "$(PYTHON)" scripts/ensure_mujoco_runtime.py --download-dir "$$download_dir" >/dev/null; \ + "$(PYTHON)" scripts/mujoco/ensure_mujoco_runtime.py --download-dir "$$download_dir" >/dev/null; \ link_dir="$(MUJOCO_DYNAMIC_LINK_DIR)"; \ MUJOCO_DOWNLOAD_DIR="$$download_dir" \ MUJOCO_DYNAMIC_LINK_DIR="$$link_dir" \ @@ -119,10 +122,10 @@ demo-keyboard: ## Run the interactive GEAR-Sonic + MuJoCo keyboard demo. -- run --config configs/demo/gear_sonic_keyboard_mujoco.toml --teleop keyboard models-public: ## Download the public policy checkpoints used by the site and benchmarks. - bash scripts/download_gear_sonic_models.sh models/gear-sonic - bash scripts/download_decoupled_wbc_models.sh models/decoupled-wbc - bash scripts/download_wbc_agile_models.sh models/wbc-agile - bash scripts/download_bfm_zero_models.sh models/bfm_zero + bash scripts/models/download_gear_sonic_models.sh models/gear-sonic + bash scripts/models/download_decoupled_wbc_models.sh models/decoupled-wbc + bash scripts/models/download_wbc_agile_models.sh models/wbc-agile + bash scripts/models/download_bfm_zero_models.sh models/bfm_zero site-python-deps: ## Install Python dependencies required for site generation and proof-pack capture. $(PYTHON) -m pip install $(SITE_PYTHON_DEPS) @@ -130,17 +133,17 @@ site-python-deps: ## Install Python dependencies required for site generation an site-render-check: ## Verify the MuJoCo offscreen renderer works before building screenshot-bearing proof packs. MUJOCO_GL="$(SHOWCASE_MUJOCO_GL)" \ PYOPENGL_PLATFORM="$(SHOWCASE_PYOPENGL_PLATFORM)" \ - $(PYTHON) scripts/check_mujoco_headless.py + $(PYTHON) scripts/mujoco/check_mujoco_headless.py benchmark-robowbc: ## Regenerate normalized RoboWBC benchmark artifacts. MUJOCO_DOWNLOAD_DIR="$(abspath $(MUJOCO_DOWNLOAD_DIR))" \ - $(PYTHON) scripts/bench_robowbc_compare.py --all + $(PYTHON) scripts/benchmarks/bench_robowbc_compare.py --all benchmark-official: ## Regenerate normalized official NVIDIA benchmark artifacts. - $(PYTHON) scripts/bench_nvidia_official.py --all + $(PYTHON) scripts/benchmarks/bench_nvidia_official.py --all benchmark-summary: ## Render benchmark Markdown and HTML summaries from the normalized artifacts. - $(PYTHON) scripts/render_nvidia_benchmark_summary.py \ + $(PYTHON) scripts/benchmarks/render_nvidia_benchmark_summary.py \ --root artifacts/benchmarks/nvidia \ --output artifacts/benchmarks/nvidia/SUMMARY.md \ --html-output artifacts/benchmarks/nvidia/index.html @@ -151,7 +154,7 @@ site: ## Build the full static site bundle with policy pages, proof packs, and b MUJOCO_GL="$(SHOWCASE_MUJOCO_GL)" \ PYOPENGL_PLATFORM="$(SHOWCASE_PYOPENGL_PLATFORM)" \ MUJOCO_DOWNLOAD_DIR="$(MUJOCO_DOWNLOAD_DIR)" \ - $(PYTHON) scripts/build_site.py \ + $(PYTHON) scripts/site/build_site.py \ --repo-root . \ --robowbc-binary "$(ROBOWBC_BINARY)" \ --output-dir "$(SITE_OUTPUT_DIR)" @@ -164,10 +167,10 @@ showcase-verify: ## Run the same showcase build + bundle validation path used in $(MAKE) site-smoke site-smoke: ## Validate the generated site bundle layout and embedded playback paths. - $(PYTHON) scripts/validate_site_bundle.py --root "$(SITE_OUTPUT_DIR)" + $(PYTHON) scripts/site/validate_site_bundle.py --root "$(SITE_OUTPUT_DIR)" site-browser-smoke: site-smoke ## Run the optional headless browser lag-selector smoke test for one policy page. - $(PYTHON) scripts/site_browser_smoke.py \ + $(PYTHON) scripts/site/site_browser_smoke.py \ --root "$(SITE_OUTPUT_DIR)" \ --policy "$(SITE_BROWSER_POLICY)" \ --bind "$(SITE_BIND)" @@ -188,7 +191,7 @@ site-serve: ## Serve the generated site bundle locally. Set SITE_OPEN=1 to open if [[ "$(SITE_OPEN)" == "1" || "$(SITE_OPEN)" == "true" ]]; then \ extra_args="--open"; \ fi; \ - $(PYTHON) scripts/serve_showcase.py \ + $(PYTHON) scripts/site/serve_showcase.py \ --dir "$(SITE_OUTPUT_DIR)" \ --bind "$(SITE_BIND)" \ --port "$(SITE_PORT)" \ @@ -210,7 +213,7 @@ python-sdk-install: python-sdk-build ## Install the freshly built local RoboWBC $(PYTHON) -m pip install --force-reinstall "$$wheel" python-sdk-smoke: ## Run the installed RoboWBC Python SDK smoke test. - $(PYTHON) scripts/python_sdk_smoke.py + $(PYTHON) scripts/sdk/python_sdk_smoke.py python-sdk-verify: ## Build, install, and smoke-test the RoboWBC Python SDK locally. $(MAKE) python-sdk-install diff --git a/README.md b/README.md index 6ffe427..82a5a48 100644 --- a/README.md +++ b/README.md @@ -4,119 +4,21 @@ RoboWBC runtime overview: TOML configs, registry, WbcPolicy core, ONNX and PyO3 backends, MuJoCo, joint targets, JSON, and Rerun reports

-Linux-only embedded runtime for humanoid whole-body-control policy inference. - -

- Open Live Policy Reports - · - Getting Started - · - Architecture - · - Python SDK - · - Founding Document -

- -RoboWBC is an embedded runtime for loading multiple WBC policies through one -customer-facing contract. The primary adoption path is the Python SDK -(`Registry`, `Observation`, `Policy`, and `MujocoSession`). Embedded Rust is -the secondary path for teams that want to own the host loop directly. The CLI -stays in the repo as the verification, benchmarking, and reporting surface for -the same runtime. - -RoboWBC is a Linux-only project. The runtime backends fail fast on non-Linux -targets instead of carrying partial or unverified platform fallbacks. - -Run `make help` to see the repo-level commands for build, validation, -benchmarks, site generation, and local serving. - -## Embedded Runtime Surface - -Python is the primary embedded runtime seam: - -```python -from robowbc import Observation, Registry, VelocityCommand - -policy = Registry.build("decoupled_wbc", "configs/decoupled_smoke.toml") -print(policy.capabilities().supported_commands) - -obs = Observation( - joint_positions=[0.0] * 4, - joint_velocities=[0.0] * 4, - gravity_vector=[0.0, 0.0, -1.0], - command=VelocityCommand(linear=[0.2, 0.0, 0.0], angular=[0.0, 0.0, 0.1]), -) -targets = policy.predict(obs) -print(targets.positions) -``` - -Embedded Rust is the secondary path: - -```rust -use robowbc_core::WbcCommandKind; -use robowbc_registry::WbcRegistry; - -let policy = WbcRegistry::build("my_policy", &policy_cfg)?; -let capabilities = policy.capabilities(); -assert!(capabilities.supports(WbcCommandKind::Velocity)); -``` - -Phase 1 deliberately keeps the surface narrow: - -- Python SDK first, embedded runtime second -- no `server/daemon` surface -- no `ROS2` or `zenoh` customer API -- no public `EndEffectorPoses` surface -- no new wrapper families beyond the shipped policy wrappers - -## System Architecture - -![RoboWBC codebase architecture](docs/assets/architecture.svg) - -The hero image gives the repo shape at a glance. The SVG is the exact system -map: current configs, registry names, core contract fields, ORT/PyO3 backends, -transports, and report artifacts. - -## Published HTML reports - -These pages are generated by the `showcase` job on `main` and published -directly from the CI artifact bundle: +Linux-first embedded runtime for humanoid whole-body-control policy inference. -| Page | Link | Purpose | -|------|------|---------| -| Site home | | Comparison-first overview across all generated policy cards | -| NVIDIA benchmarks | | Normalized RoboWBC-vs-official benchmark comparison page | -| GEAR-SONIC detail | | Planner-driven velocity showcase, logs, JSON, `.rrd`, and proof-pack assets | -| Decoupled WBC detail | | GR00T WholeBodyControl locomotion showcase with the same staged command profile | -| WBC-AGILE detail | | Published G1 recurrent checkpoint detail page and raw artifacts | -| BFM-Zero detail | | Prompt-conditioned tracking showcase with context-bundle artifacts | +

Reports · Getting Started · Architecture · Status · Python SDK · Founding Document

-## What ships today +RoboWBC loads multiple WBC policies through one caller-facing contract: +observations and commands in, joint position targets out. The Python SDK is the +primary user surface through `Registry`, `Observation`, `Policy`, command +classes, and `MujocoSession`. Embedded Rust is available for teams that own the +host loop directly, while the CLI remains the reproducible validation, +benchmarking, and reporting surface for the same runtime. -| Area | Status | -|------|--------| -| Runtime | Rust workspace with registry-driven policy loading, ONNX Runtime and PyO3 backends, MuJoCo and communication transports, plus JSON and Rerun reporting | -| Live public-policy paths | `gear_sonic`, `decoupled_wbc`, `wbc_agile`, `bfm_zero` | -| Honest blocked wrappers | `hover` needs a user-exported checkpoint, `wholebody_vla` still lacks a runnable public upstream release | -| Published visual report | The `main` workflow is wired to build the same HTML report in CI and publish it to the live report link above | +RoboWBC is Linux-only. Runtime backends fail fast on unsupported platforms +instead of carrying partial or unverified fallbacks. -## Policy status - -| Policy | Status | Public assets | Example config | Notes | -|--------|--------|---------------|----------------|-------| -| `gear_sonic` | Live | Yes | [configs/sonic_g1.toml](configs/sonic_g1.toml) | Uses the published `planner_sonic.onnx` velocity path by default; supports `cpu`, `cuda`, and `tensor_rt` when the host ORT/NVIDIA runtime matches, but the shipped config stays on CPU until you opt in | -| `decoupled_wbc` | Live | Yes | [configs/decoupled_g1.toml](configs/decoupled_g1.toml) | Public G1 balance and walk checkpoints; [configs/decoupled_smoke.toml](configs/decoupled_smoke.toml) stays as the no-download smoke path | -| `wbc_agile` | Live | Yes | [configs/wbc_agile_g1.toml](configs/wbc_agile_g1.toml) | Published G1 recurrent checkpoint is wired; the T1 path still expects a user export | -| `bfm_zero` | Live | Yes | [configs/bfm_zero_g1.toml](configs/bfm_zero_g1.toml) | Public ONNX plus tracking context bundle is normalized by `scripts/download_bfm_zero_models.sh` | -| `hover` | Blocked | No | [configs/hover_h1.toml](configs/hover_h1.toml) | Wrapper exists, but the public upstream repo does not ship a pretrained checkpoint | -| `wholebody_vla` | Experimental | No | [configs/wholebody_vla_x2.toml](configs/wholebody_vla_x2.toml) | Contract wrapper only; the public upstream repo does not yet expose a runnable inference release | -| `py_model` | User supplied | N/A | user TOML | Loads Python modules or PyTorch checkpoints through `robowbc-pyo3` | - -The generated HTML report includes every currently working public-asset policy: -`gear_sonic`, `decoupled_wbc`, `wbc_agile`, and `bfm_zero`. - -## Quick start +## First Run ```bash make toolchain @@ -125,12 +27,12 @@ make smoke make ci ``` -`configs/decoupled_smoke.toml` uses the checked-in dynamic identity ONNX -fixture, so it is the intended no-download local smoke path. `make ci` runs -the same repo entry points that GitHub CI uses for Rust validation, docs, -Python SDK verification, and the generated HTML site bundle. +`make help` lists the repo-level commands for build, validation, benchmarks, +site generation, and local serving. `configs/decoupled_smoke.toml` uses a +checked-in dynamic identity ONNX fixture, so `make smoke` is the no-download +local path. -### I Just Want To See It Move +### See It Move On Linux with a display and OpenGL available: @@ -140,118 +42,82 @@ cd robowbc make demo-keyboard ``` -The target downloads the public GEAR-Sonic ONNX files and the MuJoCo runtime -on first run, starts the local MuJoCo transport with a live viewer, and runs -`robowbc` with keyboard teleop. The checked-in demo uses the GR00T scene -wrapper plus a neutral-height virtual support band so the robot stays -recoverable without being lifted off the ground. Keep the terminal focused for -input and watch the MuJoCo window: `]` engages the policy after the init pose -settles, `WASD` changes linear velocity, `QE` changes yaw, `Space` zeroes -velocity, `9` toggles the support band, `O` sends a zero-velocity emergency-stop -tick, and `Esc` quits. The underlying manual command is: +`make demo-keyboard` is the protected clone-and-see-it-work path. It downloads +the public GEAR-Sonic ONNX files and MuJoCo runtime on first run, starts the +local MuJoCo viewer, and runs keyboard teleop. Keep the terminal focused: +`]` engages after init-pose settle. Then press `9` promptly in either the +terminal or MuJoCo viewer if the support band is holding the robot high so it +drops to foot contact. `WASD` changes linear velocity, `QE` changes yaw, +`Space` zeroes velocity, `O` sends a zero-velocity emergency-stop tick, and +`Esc` quits. Preserve the demo guardrails in `docs/agents/keyboard-demo.md` +when changing this path. -```bash -MUJOCO_DOWNLOAD_DIR="$(pwd)/.cache/mujoco" \ -LD_LIBRARY_PATH="$(pwd)/.cache/mujoco/mujoco-3.6.0/lib:${LD_LIBRARY_PATH:-}" \ -cargo run --release -p robowbc-cli \ - --features robowbc-cli/sim-auto-download,robowbc-cli/sim-viewer \ - -- run --config configs/demo/gear_sonic_keyboard_mujoco.toml --teleop keyboard +## Runtime Surfaces + +Python uses `Registry`, `Observation`, command classes, and `Policy.predict`: + +```python +policy = Registry.build("decoupled_wbc", "configs/decoupled_smoke.toml") +targets = policy.predict(obs) ``` -
-Run the live public policies +Rust uses the same registry and `WbcPolicy` contract: -```bash -bash scripts/download_gear_sonic_models.sh -cargo run --release --bin robowbc -- run --config configs/sonic_g1.toml +```rust +let policy = WbcRegistry::build("my_policy", &policy_cfg)?; +``` -bash scripts/download_decoupled_wbc_models.sh -cargo run --release --bin robowbc -- run --config configs/decoupled_g1.toml +## Architecture -bash scripts/download_wbc_agile_models.sh -cargo run --release --bin robowbc -- run --config configs/wbc_agile_g1.toml +![RoboWBC codebase architecture](docs/assets/architecture.svg) -bash scripts/download_bfm_zero_models.sh -cargo run --release --bin robowbc -- run --config configs/bfm_zero_g1.toml +```text +TOML config or Python SDK + -> robowbc-config + robowbc-registry + -> WbcPolicy implementation + -> Observation -> predict -> JointPositionTargets + -> hardware, MuJoCo, synthetic transport, JSON report, Rerun trace, static site ``` -`gear_sonic` defaults to the published `planner_sonic.onnx` velocity path. To -exercise the narrower encoder+decoder standing-placeholder path instead, set -`standing_placeholder_tracking = true` in `configs/sonic_g1.toml`. That path -does not execute `planner_sonic.onnx` on the tick and is not a generic -motion-reference streaming interface. GEAR-Sonic runtime configs support -`cpu`, `cuda`, and `tensor_rt`, but the checked-in config remains CPU by -default until you run on a machine with matching ONNX Runtime and NVIDIA -runtime dependencies. `bfm_zero` fetches the public ONNX plus tracking bundle -and converts the context into the runtime layout used by both the CLI and CI. -
- -
-Open or generate the visual report +Core contracts are `Observation`, `WbcCommand`, `JointPositionTargets`, +`WbcPolicy`, `PolicyCapabilities`, and `RobotConfig`. Unsupported commands fail +explicitly instead of falling back silently. Read `ARCHITECTURE.md` and +`docs/architecture.md` for the crate map and extension points. -The same site builder powers both the local static bundle and the published -GitHub Pages site. +## Policy Status -```bash -make site -make showcase-verify -make site-serve SITE_OPEN=1 -``` +| Policy | State | Config | Notes | +|--------|-------|--------|-------| +| `gear_sonic` | Live | [configs/sonic_g1.toml](configs/sonic_g1.toml) | Published planner velocity path; CPU by default, CUDA/TensorRT opt-in | +| `decoupled_wbc` | Live | [configs/decoupled_g1.toml](configs/decoupled_g1.toml) | Public G1 balance/walk checkpoints; smoke config needs no download | +| `wbc_agile` | Live | [configs/wbc_agile_g1.toml](configs/wbc_agile_g1.toml) | Published G1 recurrent checkpoint; T1 path expects user export | +| `bfm_zero` | Live | [configs/bfm_zero_g1.toml](configs/bfm_zero_g1.toml) | Public ONNX plus tracking context bundle | +| `hover` | Blocked | [configs/hover_h1.toml](configs/hover_h1.toml) | Wrapper exists; no public pretrained checkpoint | +| `wholebody_vla` | Experimental | [configs/wholebody_vla_x2.toml](configs/wholebody_vla_x2.toml) | Contract wrapper only; no runnable public upstream release | +| `py_model` | User supplied | user TOML | Loads Python modules or PyTorch checkpoints through `robowbc-pyo3` | -`make site` wraps `scripts/build_site.py` and now owns the full local/CI site -build. It picks `./.cache/mujoco` by default, downloads MuJoCo there when -needed, rebuilds the `robowbc` binary with -`robowbc-cli/sim-auto-download,robowbc-cli/vis`, runs the benchmark -generators, forces the same `MUJOCO_GL=egl` / `PYOPENGL_PLATFORM=egl` -offscreen path that the GitHub showcase job uses, and assembles the final -static bundle into `/tmp/robowbc-site`. -Set `MUJOCO_DOWNLOAD_DIR=/your/cache make site` if you want a different cache -location, or override `SITE_OUTPUT_DIR=/your/output make site` if you want the -site somewhere else. `make site-smoke` validates the generated bundle layout -without serving it, and `make site-serve-check` does a short start/stop probe -of the local HTTP server. - -`make showcase-verify` is the closest local equivalent to the GitHub `showcase` -job. It installs the site Python deps, runs the same headless MuJoCo EGL render -smoke check that CI now relies on, downloads the public checkpoints, builds the -site bundle, and fails if any MuJoCo-backed policy page ships a proof-pack -manifest without real screenshots. On Ubuntu, install the headless EGL runtime -first if that render smoke check fails: -`sudo apt-get install -y libegl1 libegl-mesa0 libgles2 libgl1-mesa-dri libgbm1`. - -The output folder contains `index.html`, `manifest.json`, `policies//` -folders with per-policy HTML plus raw run artifacts, `benchmarks/nvidia/` with -the NVIDIA comparison page, and `assets/rerun-web-viewer/` for embedded Rerun -playback. Pull requests keep the downloadable `robowbc-site` artifact, and -`main` publishes the generated site to the live report links above. -
- -
-Manual real-model verification +## Public Reports -```bash -bash scripts/download_gear_sonic_models.sh -cargo test -p robowbc-ort -- --ignored gear_sonic_real_model_inference +The `showcase` job on `main` publishes generated HTML policy cards, proof-pack +links, benchmark pages, JSON, `.rrd`, and raw artifacts: -bash scripts/download_decoupled_wbc_models.sh -cargo test -p robowbc-ort -- --ignored decoupled_wbc_real_model_inference +- Site home: +- NVIDIA benchmark comparison: +- Policy pages: [`gear_sonic`](https://miaodx.com/robowbc/policies/gear_sonic/), [`decoupled_wbc`](https://miaodx.com/robowbc/policies/decoupled_wbc/), [`wbc_agile`](https://miaodx.com/robowbc/policies/wbc_agile/), and [`bfm_zero`](https://miaodx.com/robowbc/policies/bfm_zero/) -bash scripts/download_wbc_agile_models.sh -cargo test -p robowbc-ort -- --ignored wbc_agile_real_model_inference +Local report commands: -bash scripts/download_bfm_zero_models.sh -BFM_ZERO_MODEL_PATH=models/bfm_zero/bfm_zero_g1.onnx \ -BFM_ZERO_CONTEXT_PATH=models/bfm_zero/zs_walking.npy \ -cargo test -p robowbc-ort bfm_zero_real_model_inference -- --ignored --nocapture +```bash +make site +make showcase-verify +make site-serve SITE_OPEN=1 ``` -`hover` still requires a user-trained exported checkpoint, and `wholebody_vla` -still requires a compatible private or local model because no runnable public -release exists upstream today. -
+`make showcase-verify` downloads public checkpoints and requires a working +headless MuJoCo EGL environment. Use `MUJOCO_DOWNLOAD_DIR` and +`SITE_OUTPUT_DIR` to override local cache and output paths. -
-Python SDK +## Python SDK ```bash pip install "maturin>=1.9.4,<2.0" @@ -259,30 +125,9 @@ maturin develop python -c "from robowbc import Registry; print(Registry.list_policies())" ``` -The standalone Python package lives in `crates/robowbc-py`, while -`robowbc-pyo3` provides the runtime backend for user-supplied Python or -PyTorch policies. First-party embedded examples live at: - -- `crates/robowbc-py/examples/lerobot_adapter.py` for velocity-driven locomotion -- `crates/robowbc-py/examples/manipulation_adapter.py` for named-link `kinematic_pose` -- `examples/python/mujoco_kinematic_pose_session.py` for live `MujocoSession.step({"kinematic_pose": ...})` -
- -
-Workspace layout - -| Path | Purpose | -|------|---------| -| `crates/robowbc-core` | `WbcPolicy`, `Observation`, `WbcCommand`, `JointPositionTargets`, `RobotConfig` | -| `crates/robowbc-registry` | `inventory`-based policy registration and factory | -| `crates/robowbc-ort` | ONNX Runtime backends and policy wrappers | -| `crates/robowbc-pyo3` | Python-backed runtime policy loading | -| `crates/robowbc-comm` | Control-loop plumbing and robot transports | -| `crates/robowbc-sim` | MuJoCo transport for hardware-free execution | -| `crates/robowbc-vis` | Rerun visualization and `.rrd` recording | -| `crates/robowbc-cli` | `robowbc` CLI binary | -| `crates/robowbc-py` | Standalone `maturin` package for the Python SDK | -
+The standalone Python package lives in `crates/robowbc-py`; `robowbc-pyo3` +provides the runtime backend for user-supplied Python or PyTorch policies. +Examples live under `crates/robowbc-py/examples/` and `examples/python/`. ## Documentation @@ -290,21 +135,18 @@ PyTorch policies. First-party embedded examples live at: - [Configuration Reference](docs/configuration.md) - [Adding a New Policy](docs/adding-a-model.md) - [Adding a New Robot](docs/adding-a-robot.md) -- [Architecture](docs/architecture.md) +- [Architecture](ARCHITECTURE.md) +- [Current status](STATUS.md) +- [Full docs index](docs/README.md) - [Founding document](docs/founding-document.md) - [Q2 2026 roadmap](docs/roadmap-2026-q2.md) -## Related projects +## Related Projects - [roboharness](https://github.com/MiaoDX/roboharness), companion visual testing and browser-report project - [LeRobot](https://github.com/huggingface/lerobot), upstream robotics stack that can consume a WBC backend ## License -robowbc itself is **MIT-licensed** — see [`LICENSE`](LICENSE). -Third-party dependencies and runtime-fetched policy weights retain -their original licenses; the per-component breakdown lives in -[`LICENSES/`](LICENSES/) and the user-facing summary in -[`docs/third-party-notices.md`](docs/third-party-notices.md). -[`CONTRIBUTING.md`](CONTRIBUTING.md) documents the rule for adding a -new dependency. +robowbc is MIT-licensed; see [`LICENSE`](LICENSE). Third-party dependencies and runtime-fetched policy weights retain their original licenses. +See [`LICENSES/`](LICENSES/), [`docs/third-party-notices.md`](docs/third-party-notices.md), and [`CONTRIBUTING.md`](CONTRIBUTING.md) for dependency and notice rules. diff --git a/STATUS.md b/STATUS.md new file mode 100644 index 0000000..3d2ffbc --- /dev/null +++ b/STATUS.md @@ -0,0 +1,75 @@ +# Status + +Updated: 2026-05-14 + +RoboWBC is in the v0.2 line. The current repo is focused on making the public +runtime credible for Python-first robotics users while preserving the Rust +control core. + +## Current State + +| Area | Status | +|------|--------| +| Runtime | Rust workspace with registry-driven policy loading, ONNX Runtime and PyO3 backends, MuJoCo, transport crates, validation, JSON reports, and Rerun visualization | +| Python SDK | Primary customer-facing embedded surface through `Registry`, `Observation`, `Policy`, command classes, and `MujocoSession` | +| Live public policies | `gear_sonic`, `decoupled_wbc`, `wbc_agile`, `bfm_zero` | +| Blocked or experimental policies | `hover` needs a user-exported checkpoint; `wholebody_vla` has no runnable public upstream release | +| Public report | GitHub Pages publishes generated policy cards, proof-pack links, benchmark pages, and raw artifacts | +| Protected demo | `make demo-keyboard` runs the GEAR-Sonic MuJoCo keyboard path | + +## Can Run Now + +Use these as entry points, then choose narrower commands from `Makefile`: + +```bash +make help +make build +make smoke +make verify +``` + +For the local Python SDK: + +```bash +make python-sdk-verify +``` + +For the full policy showcase path: + +```bash +make showcase-verify +``` + +`make showcase-verify` downloads public checkpoints and requires a working +headless MuJoCo EGL environment. + +## Active Queue + +The active human roadmap is `ROADMAP.md`. + +Current order: + +1. WBC-AGILE official-runtime parity, issue #93. +2. BFM-Zero upstream parity, issue #94. +3. WBC-AGILE 35-DOF MuJoCo truth decision, issue #97. +4. LeRobot adapter milestone, issue #41. +5. Unitree G1 hardware proof issue. + +## Important Constraints + +- Linux is the verified runtime target. +- Do not claim a validation command passed unless it ran in the current + environment. +- Public live-policy claims should stay tied to reproducible configs, reports, + and artifacts. +- Hardware-facing or MuJoCo demo changes must preserve the keyboard demo + stability guardrails in `docs/agents/keyboard-demo.md`. + +## Where To Look Next + +- `README.md`: first-run overview and policy status. +- `ARCHITECTURE.md`: high-level system map. +- `docs/README.md`: full documentation index. +- `ROADMAP.md`: ordered product and credibility work. +- `.planning/STATE.md`: historical agent planning state and completed phase + evidence. diff --git a/configs/README.md b/configs/README.md new file mode 100644 index 0000000..c1c2512 --- /dev/null +++ b/configs/README.md @@ -0,0 +1,23 @@ +# Configs + +Configs are grouped by runtime purpose. + +## Policy Entry Points + +Top-level TOML files such as `sonic_g1.toml`, `decoupled_g1.toml`, +`wbc_agile_g1.toml`, and `bfm_zero_g1.toml` are public policy run configs. + +`decoupled_smoke.toml` is the no-download smoke path. It uses checked-in ONNX +fixtures and is the safest local config for quick validation. + +## Subfolders + +- `robots/` contains robot embodiment configs: joint names, default pose, gains, + and limits. +- `showcase/` contains configs used by generated policy report pages. +- `demo/` contains protected interactive demo configs, including + `demo/gear_sonic_keyboard_mujoco.toml`. +- `teleop/` contains keyboard and input mapping config. + +When changing demo configs, read `docs/agents/keyboard-demo.md` and run the +targeted MuJoCo stability check when the environment supports it. diff --git a/configs/demo/gear_sonic_keyboard_mujoco.toml b/configs/demo/gear_sonic_keyboard_mujoco.toml index 8c9a3dd..b742c58 100644 --- a/configs/demo/gear_sonic_keyboard_mujoco.toml +++ b/configs/demo/gear_sonic_keyboard_mujoco.toml @@ -14,7 +14,10 @@ # -- run --config configs/demo/gear_sonic_keyboard_mujoco.toml --teleop keyboard # # Keep your terminal focused for keyboard input and watch the MuJoCo window: -# `]` engages policy, WASD/QE drive, `9` toggles the support band. +# `]` engages policy, WASD/QE drive, and the first `9` press after policy +# engagement toggles the MuJoCo support band off so the robot drops to foot +# contact. The `9` support toggle works from both the terminal and the MuJoCo +# viewer. [policy] name = "gear_sonic" @@ -44,16 +47,21 @@ config_path = "configs/robots/unitree_g1_gear_sonic.toml" model_path = "assets/robots/groot_g1_gear_sonic/scene_29dof.xml" timestep = 0.002 substeps = 10 -gain_profile = "simulation_pd" +gain_profile = "default_pd" +# Match the official GR00T MuJoCo bridge: policy q-targets and the deployment +# kp/kd values go directly into the simulator PD loop without the hardware-side +# target slew limiter. +enforce_target_velocity_limits = false viewer = true [sim.elastic_band] -# GR00T-style MuJoCo teleop support band. We anchor it to the initialized -# pelvis pose so it catches a fall without lifting the feet off the ground. +# GR00T-style MuJoCo teleop support band: fixed point [0, 0, 1] with the +# published spring-damper gains. The first `9` press toggles it off, matching +# the official "drop robot to ground" sim flow. enabled = true body_name = "pelvis" anchor = [0.0, 0.0, 1.0] -anchor_from_initial_pose = true +anchor_from_initial_pose = false length = 0.0 kp_pos = 10000.0 kd_pos = 1000.0 @@ -69,9 +77,13 @@ backend = "ort" device = "cpu" [runtime] -# Keyboard teleop starts in an init pose. Press `]` to engage the policy, then -# use WASD/QE to drive. Press `9` to toggle the elastic support band. Live -# teleop runs until Esc or Ctrl-C. +# Keyboard teleop starts in an init pose. Press `]` to queue policy engagement +# after the 3.0s settle window. If the support band is holding the robot high, +# press `9` promptly after engagement to drop to foot contact, then use WASD/QE +# to drive. Live teleop runs until Esc or Ctrl-C. A zero command after +# engagement runs the official idle planner/controller path; keep the `9` drop +# prompt and first velocity command close to engagement, matching upstream sim +# teleop practice. velocity = [0.0, 0.0, 0.0] init_pose_secs = 3.0 require_engage = true diff --git a/configs/robots/unitree_g1_gear_sonic.toml b/configs/robots/unitree_g1_gear_sonic.toml index b68307c..30402aa 100644 --- a/configs/robots/unitree_g1_gear_sonic.toml +++ b/configs/robots/unitree_g1_gear_sonic.toml @@ -76,10 +76,12 @@ pd_gains = [ { kp = 16.778327, kd = 1.068142 }, # right_wrist_yaw ] -# MuJoCo/raw-torque gains from the official GR00T G1 sim config -# (`MOTOR_KP` / `MOTOR_KD` in g1_29dof_gear_wbc.yaml). These are intentionally -# much stiffer than the hardware-side q_target gains above and are only used by -# the simulation transport. +# Alternate MuJoCo/raw-torque gains from the GR00T G1 sim config +# (`MOTOR_KP` / `MOTOR_KD` in g1_29dof_gear_wbc.yaml). The official +# GEAR-Sonic deploy-to-MuJoCo path sends the deployment `kps` / `kds` values +# above in each low command, so the keyboard demo uses `gain_profile = +# "default_pd"` for parity. These stiffer gains remain available for generic +# simulation experiments that explicitly select `simulation_pd`. sim_pd_gains = [ { kp = 150.0, kd = 2.0 }, # left_hip_pitch { kp = 150.0, kd = 2.0 }, # left_hip_roll diff --git a/crates/robowbc-cli/src/main.rs b/crates/robowbc-cli/src/main.rs index dc817bb..db8b0c7 100644 --- a/crates/robowbc-cli/src/main.rs +++ b/crates/robowbc-cli/src/main.rs @@ -758,12 +758,35 @@ struct ReportBasePose { rotation_xyzw: [f32; 4], } +#[derive(Debug, Clone, Serialize)] +struct ReportPlannerStatus { + mode_id: i64, + mode_name: String, + target_vel_mps: Option, + planar_speed_mps: f32, +} + #[derive(Debug, Clone, Serialize)] struct VelocityTrackingMetrics { sample_count: usize, + sim_elapsed_secs: f64, + wall_elapsed_secs: f64, + real_time_factor: f64, + commanded_vx_mean_mps: f64, + commanded_vy_mean_mps: f64, + commanded_yaw_rate_mean_rad_s: f64, + actual_vx_mean_mps: f64, + actual_vy_mean_mps: f64, + actual_yaw_rate_mean_rad_s: f64, + actual_wall_vx_mean_mps: f64, + actual_wall_vy_mean_mps: f64, + actual_wall_yaw_rate_mean_rad_s: f64, vx_rmse_mps: f64, vy_rmse_mps: f64, yaw_rate_rmse_rad_s: f64, + wall_vx_rmse_mps: f64, + wall_vy_rmse_mps: f64, + wall_yaw_rate_rmse_rad_s: f64, vx_mean_abs_error_mps: f64, vy_mean_abs_error_mps: f64, yaw_rate_mean_abs_error_rad_s: f64, @@ -778,11 +801,16 @@ struct VelocityTrackingMetrics { #[derive(Debug, Clone, Serialize)] struct ReportFrame { tick: usize, + wall_time_secs: f64, command_data: Vec, #[serde(skip_serializing_if = "Option::is_none")] phase_name: Option, #[serde(skip_serializing_if = "Option::is_none")] base_pose: Option, + #[serde(skip_serializing_if = "Option::is_none")] + elastic_band_enabled: Option, + #[serde(skip_serializing_if = "Option::is_none")] + planner_status: Option, actual_positions: Vec, actual_velocities: Vec, target_positions: Vec, @@ -793,11 +821,16 @@ struct ReportFrame { struct ReplayFrame { tick: usize, sim_time_secs: f64, + wall_time_secs: f64, command_data: Vec, #[serde(skip_serializing_if = "Option::is_none")] phase_name: Option, #[serde(skip_serializing_if = "Option::is_none")] base_pose: Option, + #[serde(skip_serializing_if = "Option::is_none")] + elastic_band_enabled: Option, + #[serde(skip_serializing_if = "Option::is_none")] + planner_status: Option, actual_positions: Vec, actual_velocities: Vec, target_positions: Vec, @@ -812,9 +845,12 @@ impl From<&ReplayFrame> for ReportFrame { fn from(frame: &ReplayFrame) -> Self { Self { tick: frame.tick, + wall_time_secs: frame.wall_time_secs, command_data: frame.command_data.clone(), phase_name: frame.phase_name.clone(), base_pose: frame.base_pose, + elastic_band_enabled: frame.elastic_band_enabled, + planner_status: frame.planner_status.clone(), actual_positions: frame.actual_positions.clone(), actual_velocities: frame.actual_velocities.clone(), target_positions: frame.target_positions.clone(), @@ -932,6 +968,12 @@ enum TeleopControlRequest { ToggleElasticBand, } +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum PolicyEngageRequestOutcome { + Immediate, + QueuedUntilSettled, +} + #[derive(Debug, Clone, PartialEq)] struct TeleopPollOutcome { velocity: [f32; 3], @@ -982,6 +1024,41 @@ fn apply_teleop_events( } } +fn apply_policy_engage_request( + startup_config: RuntimeStartupConfig, + policy_engaged: &mut bool, + pending_policy_engage: &mut bool, + init_pose_start: &mut Option<(Instant, Vec)>, +) -> PolicyEngageRequestOutcome { + if startup_config.require_engage && startup_config.init_pose_secs > 0.0 { + *pending_policy_engage = true; + PolicyEngageRequestOutcome::QueuedUntilSettled + } else { + *policy_engaged = true; + *pending_policy_engage = false; + *init_pose_start = None; + PolicyEngageRequestOutcome::Immediate + } +} + +fn pending_policy_engage_is_ready( + startup_config: RuntimeStartupConfig, + pending_policy_engage: bool, + elapsed: Duration, +) -> bool { + startup_config.require_engage + && pending_policy_engage + && elapsed.as_secs_f32() >= startup_config.init_pose_secs +} + +fn startup_settle_remaining_secs( + startup_config: RuntimeStartupConfig, + elapsed: Option, +) -> f32 { + let elapsed_secs = elapsed.map_or(0.0, |duration| duration.as_secs_f32()); + (startup_config.init_pose_secs - elapsed_secs).max(0.0) +} + struct LiveTeleop { source: KeyboardTeleop, current_velocity: [f32; 3], @@ -994,7 +1071,7 @@ impl LiveTeleop { .enable() .map_err(|err| format!("failed to enable keyboard teleop: {err}"))?; println!( - "keyboard teleop active: ] engages policy, WASD move, QE yaw, Space zeroes velocity, 9 toggles support band, O e-stops, Esc quits" + "keyboard teleop active: press ] to engage policy after settle; first 9 press in the terminal or MuJoCo viewer drops support band toward foot contact, WASD/QE update velocity, Space zeroes, O e-stops, Esc/Ctrl-C quits" ); Ok(Self { source, @@ -1320,6 +1397,10 @@ trait ReportTelemetryProvider { fn report_mujoco_state(&self) -> Option { None } + + fn report_elastic_band_enabled(&self) -> Option { + None + } } impl ReportTelemetryProvider for SyntheticTransport {} @@ -1346,14 +1427,91 @@ impl ReportTelemetryProvider for MujocoTransport { qvel: self.qvel_snapshot(), }) } + + fn report_elastic_band_enabled(&self) -> Option { + self.elastic_band_enabled() + } } -#[derive(Default)] +const VELOCITY_MONITOR_PRINT_INTERVAL_SECS: f64 = 1.0; +const NONZERO_PLANAR_COMMAND_EPS_MPS: f64 = 0.05; +const GEAR_SONIC_PLANNER_MODE_IDLE: i64 = 0; +const GEAR_SONIC_PLANNER_MODE_SLOW_WALK: i64 = 1; +const GEAR_SONIC_PLANNER_MODE_WALK: i64 = 2; +const GEAR_SONIC_PLANNER_MODE_RUN: i64 = 3; +const GEAR_SONIC_MIN_MOVING_SPEED_MPS: f32 = 0.2; +const GEAR_SONIC_WALK_COMMAND_MIN_MPS: f32 = 0.8; +const GEAR_SONIC_RUN_COMMAND_MIN_MPS: f32 = 1.5; +const GEAR_SONIC_RUN_COMMAND_MAX_MPS: f32 = 3.0; + +#[derive(Debug, Clone)] +struct VelocityTrackingFrame { + sim_time_secs: f64, + wall_time_secs: f64, + command: [f32; 3], + base_pose: Option, + elastic_band_enabled: Option, + planner_status: Option, +} + +impl VelocityTrackingFrame { + fn from_replay_frame(frame: &ReplayFrame) -> Option { + if frame.command_data.len() < 3 { + return None; + } + + Some(Self { + sim_time_secs: frame.sim_time_secs, + wall_time_secs: frame.wall_time_secs, + command: [ + frame.command_data[0], + frame.command_data[1], + frame.command_data[2], + ], + base_pose: frame.base_pose, + elastic_band_enabled: frame.elastic_band_enabled, + planner_status: frame.planner_status.clone(), + }) + } +} + +#[derive(Debug, Clone)] +struct VelocityTrackingSample { + command: [f32; 3], + actual_vx_mps: f64, + actual_vy_mps: f64, + actual_yaw_rate_rad_s: f64, + actual_wall_vx_mps: f64, + actual_wall_vy_mps: f64, + actual_wall_yaw_rate_rad_s: f64, + sim_dt_secs: f64, + wall_dt_secs: f64, + body_displacement: [f32; 2], + yaw_delta_rad: f32, + elastic_band_enabled: Option, + planner_status: Option, +} + +#[derive(Default, Clone)] struct VelocityTrackingAccumulator { sample_count: usize, + sim_elapsed_secs: f64, + wall_elapsed_secs: f64, + commanded_vx_sum: f64, + commanded_vy_sum: f64, + commanded_yaw_sum: f64, + actual_vx_sum: f64, + actual_vy_sum: f64, + actual_yaw_sum: f64, + actual_wall_vx_sum: f64, + actual_wall_vy_sum: f64, + actual_wall_yaw_sum: f64, vx_sq_error_sum: f64, vy_sq_error_sum: f64, yaw_sq_error_sum: f64, + wall_vx_sq_error_sum: f64, + wall_vy_sq_error_sum: f64, + wall_yaw_sq_error_sum: f64, vx_abs_error_sum: f64, vy_abs_error_sum: f64, yaw_abs_error_sum: f64, @@ -1366,49 +1524,81 @@ struct VelocityTrackingAccumulator { } impl VelocityTrackingAccumulator { - fn update( - &mut self, - command: [f32; 3], - actual_body_displacement: [f32; 2], - actual_yaw_delta_rad: f32, - dt_secs: f64, - ) { - let actual_forward_velocity = f64::from(actual_body_displacement[0]) / dt_secs; - let actual_lateral_velocity = f64::from(actual_body_displacement[1]) / dt_secs; - let actual_yaw_rate = f64::from(actual_yaw_delta_rad) / dt_secs; - - let forward_velocity_error = actual_forward_velocity - f64::from(command[0]); - let lateral_velocity_error = actual_lateral_velocity - f64::from(command[1]); - let yaw_error = actual_yaw_rate - f64::from(command[2]); + #[allow(clippy::similar_names)] + fn update(&mut self, sample: &VelocityTrackingSample) { + let command_vx = f64::from(sample.command[0]); + let command_vy = f64::from(sample.command[1]); + let command_yaw = f64::from(sample.command[2]); + + let forward_velocity_error = sample.actual_vx_mps - command_vx; + let lateral_velocity_error = sample.actual_vy_mps - command_vy; + let yaw_error = sample.actual_yaw_rate_rad_s - command_yaw; + let wall_forward_velocity_error = sample.actual_wall_vx_mps - command_vx; + let wall_lateral_velocity_error = sample.actual_wall_vy_mps - command_vy; + let wall_yaw_error = sample.actual_wall_yaw_rate_rad_s - command_yaw; self.sample_count = self.sample_count.saturating_add(1); + self.sim_elapsed_secs += sample.sim_dt_secs; + self.wall_elapsed_secs += sample.wall_dt_secs; + self.commanded_vx_sum += command_vx; + self.commanded_vy_sum += command_vy; + self.commanded_yaw_sum += command_yaw; + self.actual_vx_sum += sample.actual_vx_mps; + self.actual_vy_sum += sample.actual_vy_mps; + self.actual_yaw_sum += sample.actual_yaw_rate_rad_s; + self.actual_wall_vx_sum += sample.actual_wall_vx_mps; + self.actual_wall_vy_sum += sample.actual_wall_vy_mps; + self.actual_wall_yaw_sum += sample.actual_wall_yaw_rate_rad_s; self.vx_sq_error_sum += forward_velocity_error * forward_velocity_error; self.vy_sq_error_sum += lateral_velocity_error * lateral_velocity_error; self.yaw_sq_error_sum += yaw_error * yaw_error; + self.wall_vx_sq_error_sum += wall_forward_velocity_error * wall_forward_velocity_error; + self.wall_vy_sq_error_sum += wall_lateral_velocity_error * wall_lateral_velocity_error; + self.wall_yaw_sq_error_sum += wall_yaw_error * wall_yaw_error; self.vx_abs_error_sum += forward_velocity_error.abs(); self.vy_abs_error_sum += lateral_velocity_error.abs(); self.yaw_abs_error_sum += yaw_error.abs(); self.vx_peak_abs_error = self.vx_peak_abs_error.max(forward_velocity_error.abs()); self.vy_peak_abs_error = self.vy_peak_abs_error.max(lateral_velocity_error.abs()); self.yaw_peak_abs_error = self.yaw_peak_abs_error.max(yaw_error.abs()); - self.forward_distance_m += f64::from(actual_body_displacement[0]); - self.lateral_distance_m += f64::from(actual_body_displacement[1]); - self.heading_change_rad += f64::from(actual_yaw_delta_rad); + self.forward_distance_m += f64::from(sample.body_displacement[0]); + self.lateral_distance_m += f64::from(sample.body_displacement[1]); + self.heading_change_rad += f64::from(sample.yaw_delta_rad); } - fn finish(self) -> Option { + fn finish(&self) -> Option { if self.sample_count == 0 { return None; } #[allow(clippy::cast_precision_loss)] let sample_count = self.sample_count as f64; + let real_time_factor = if self.wall_elapsed_secs > f64::EPSILON { + self.sim_elapsed_secs / self.wall_elapsed_secs + } else { + 0.0 + }; Some(VelocityTrackingMetrics { sample_count: self.sample_count, + sim_elapsed_secs: self.sim_elapsed_secs, + wall_elapsed_secs: self.wall_elapsed_secs, + real_time_factor, + commanded_vx_mean_mps: self.commanded_vx_sum / sample_count, + commanded_vy_mean_mps: self.commanded_vy_sum / sample_count, + commanded_yaw_rate_mean_rad_s: self.commanded_yaw_sum / sample_count, + actual_vx_mean_mps: self.actual_vx_sum / sample_count, + actual_vy_mean_mps: self.actual_vy_sum / sample_count, + actual_yaw_rate_mean_rad_s: self.actual_yaw_sum / sample_count, + actual_wall_vx_mean_mps: self.actual_wall_vx_sum / sample_count, + actual_wall_vy_mean_mps: self.actual_wall_vy_sum / sample_count, + actual_wall_yaw_rate_mean_rad_s: self.actual_wall_yaw_sum / sample_count, vx_rmse_mps: (self.vx_sq_error_sum / sample_count).sqrt(), vy_rmse_mps: (self.vy_sq_error_sum / sample_count).sqrt(), yaw_rate_rmse_rad_s: (self.yaw_sq_error_sum / sample_count).sqrt(), + wall_vx_rmse_mps: (self.wall_vx_sq_error_sum / sample_count).sqrt(), + wall_vy_rmse_mps: (self.wall_vy_sq_error_sum / sample_count).sqrt(), + wall_yaw_rate_rmse_rad_s: (self.wall_yaw_sq_error_sum / sample_count).sqrt(), vx_mean_abs_error_mps: self.vx_abs_error_sum / sample_count, vy_mean_abs_error_mps: self.vy_abs_error_sum / sample_count, yaw_rate_mean_abs_error_rad_s: self.yaw_abs_error_sum / sample_count, @@ -1422,10 +1612,136 @@ impl VelocityTrackingAccumulator { } } +struct VelocityTrackingMonitor { + enabled: bool, + previous_frame: Option, + accumulator: VelocityTrackingAccumulator, + next_print_wall_time_secs: f64, + support_band_hint_printed: bool, +} + +struct VelocityMonitorSnapshot { + sample: VelocityTrackingSample, + metrics: VelocityTrackingMetrics, + support_band_hint: bool, +} + +impl VelocityTrackingMonitor { + fn new(runtime_command: &ParsedRuntimeCommand) -> Self { + Self { + enabled: matches!( + runtime_command, + ParsedRuntimeCommand::Velocity(_) | ParsedRuntimeCommand::VelocitySchedule(_) + ), + previous_frame: None, + accumulator: VelocityTrackingAccumulator::default(), + next_print_wall_time_secs: VELOCITY_MONITOR_PRINT_INTERVAL_SECS, + support_band_hint_printed: false, + } + } + + fn observe(&mut self, frame: &ReplayFrame) -> Option { + if !self.enabled { + return None; + } + + let current = VelocityTrackingFrame::from_replay_frame(frame)?; + let sample = self + .previous_frame + .take() + .and_then(|previous| velocity_tracking_sample(previous, ¤t)); + self.previous_frame = Some(current); + + let sample = sample?; + self.accumulator.update(&sample); + let metrics = self.accumulator.finish()?; + let support_band_hint = !self.support_band_hint_printed + && sample.elastic_band_enabled == Some(true) + && planar_command_speed_mps(sample.command) > NONZERO_PLANAR_COMMAND_EPS_MPS; + if support_band_hint { + self.support_band_hint_printed = true; + } + + if support_band_hint || frame.wall_time_secs >= self.next_print_wall_time_secs { + self.next_print_wall_time_secs = + frame.wall_time_secs + VELOCITY_MONITOR_PRINT_INTERVAL_SECS; + Some(VelocityMonitorSnapshot { + sample, + metrics, + support_band_hint, + }) + } else { + None + } + } + + fn finish(&self) -> Option { + self.accumulator.finish() + } +} + +fn planar_command_speed_mps(command: [f32; 3]) -> f64 { + f64::from(command[0]).hypot(f64::from(command[1])) +} + +#[allow(clippy::similar_names)] +fn velocity_tracking_sample( + previous: VelocityTrackingFrame, + current: &VelocityTrackingFrame, +) -> Option { + let sim_dt_secs = current.sim_time_secs - previous.sim_time_secs; + if sim_dt_secs <= f64::EPSILON { + return None; + } + + let wall_dt_secs = match current.wall_time_secs - previous.wall_time_secs { + dt if dt > f64::EPSILON => dt, + _ => sim_dt_secs, + }; + + let (Some(previous_pose), Some(current_pose)) = (previous.base_pose, current.base_pose) else { + return None; + }; + + let body_displacement = world_to_body_planar_delta( + previous_pose.position_world, + previous_pose.rotation_xyzw, + current_pose.position_world, + ); + let yaw_delta_rad = wrap_angle_rad( + yaw_from_rotation_xyzw(current_pose.rotation_xyzw) + - yaw_from_rotation_xyzw(previous_pose.rotation_xyzw), + ); + + let actual_vx_mps = f64::from(body_displacement[0]) / sim_dt_secs; + let actual_vy_mps = f64::from(body_displacement[1]) / sim_dt_secs; + let actual_yaw_rate_rad_s = f64::from(yaw_delta_rad) / sim_dt_secs; + let actual_wall_vx_mps = f64::from(body_displacement[0]) / wall_dt_secs; + let actual_wall_vy_mps = f64::from(body_displacement[1]) / wall_dt_secs; + let actual_wall_yaw_rate_rad_s = f64::from(yaw_delta_rad) / wall_dt_secs; + + Some(VelocityTrackingSample { + command: previous.command, + actual_vx_mps, + actual_vy_mps, + actual_yaw_rate_rad_s, + actual_wall_vx_mps, + actual_wall_vy_mps, + actual_wall_yaw_rate_rad_s, + sim_dt_secs, + wall_dt_secs, + body_displacement, + yaw_delta_rad, + elastic_band_enabled: previous.elastic_band_enabled, + planner_status: previous.planner_status, + }) +} + +#[cfg(test)] fn compute_velocity_tracking_metrics( frames: &[ReplayFrame], runtime_command: &ParsedRuntimeCommand, - frequency_hz: u32, + _frequency_hz: u32, ) -> Option { if !matches!( runtime_command, @@ -1434,46 +1750,116 @@ fn compute_velocity_tracking_metrics( return None; } - if frequency_hz == 0 || frames.len() < 2 { + if frames.len() < 2 { return None; } - let dt_secs = 1.0 / f64::from(frequency_hz); let mut accumulator = VelocityTrackingAccumulator::default(); for pair in frames.windows(2) { - let previous = &pair[0]; - let current = &pair[1]; - - if previous.command_data.len() < 3 { + let Some(previous) = VelocityTrackingFrame::from_replay_frame(&pair[0]) else { continue; - } - - let (Some(previous_pose), Some(current_pose)) = (previous.base_pose, current.base_pose) - else { + }; + let Some(current) = VelocityTrackingFrame::from_replay_frame(&pair[1]) else { continue; }; + if let Some(sample) = velocity_tracking_sample(previous, ¤t) { + accumulator.update(&sample); + } + } - let command = [ - previous.command_data[0], - previous.command_data[1], - previous.command_data[2], - ]; + accumulator.finish() +} - let body_displacement = world_to_body_planar_delta( - previous_pose.position_world, - previous_pose.rotation_xyzw, - current_pose.position_world, - ); - let yaw_delta = wrap_angle_rad( - yaw_from_rotation_xyzw(current_pose.rotation_xyzw) - - yaw_from_rotation_xyzw(previous_pose.rotation_xyzw), +fn gear_sonic_mode_name(mode_id: i64) -> &'static str { + match mode_id { + GEAR_SONIC_PLANNER_MODE_IDLE => "idle", + GEAR_SONIC_PLANNER_MODE_SLOW_WALK => "slow_walk", + GEAR_SONIC_PLANNER_MODE_WALK => "walk", + GEAR_SONIC_PLANNER_MODE_RUN => "run", + _ => "unknown", + } +} + +fn gear_sonic_planner_status_from_command(command_data: &[f32]) -> Option { + if command_data.len() < 2 { + return None; + } + + let planar_speed_mps = command_data[0].hypot(command_data[1]); + let (mode_id, target_vel_mps) = if planar_speed_mps <= 0.01 { + (GEAR_SONIC_PLANNER_MODE_IDLE, None) + } else { + let target_vel_mps = planar_speed_mps.clamp( + GEAR_SONIC_MIN_MOVING_SPEED_MPS, + GEAR_SONIC_RUN_COMMAND_MAX_MPS, ); + let mode_id = if target_vel_mps < GEAR_SONIC_WALK_COMMAND_MIN_MPS { + GEAR_SONIC_PLANNER_MODE_SLOW_WALK + } else if target_vel_mps < GEAR_SONIC_RUN_COMMAND_MIN_MPS { + GEAR_SONIC_PLANNER_MODE_WALK + } else { + GEAR_SONIC_PLANNER_MODE_RUN + }; + (mode_id, Some(target_vel_mps)) + }; - accumulator.update(command, body_displacement, yaw_delta, dt_secs); + Some(ReportPlannerStatus { + mode_id, + mode_name: gear_sonic_mode_name(mode_id).to_owned(), + target_vel_mps, + planar_speed_mps, + }) +} + +fn planner_status_for_command( + policy_name: &str, + runtime_command: &ParsedRuntimeCommand, + command_data: &[f32], +) -> Option { + if policy_name == "gear_sonic" + && matches!( + runtime_command, + ParsedRuntimeCommand::Velocity(_) | ParsedRuntimeCommand::VelocitySchedule(_) + ) + { + gear_sonic_planner_status_from_command(command_data) + } else { + None } +} - accumulator.finish() +fn print_velocity_monitor(snapshot: &VelocityMonitorSnapshot) { + let band_state = match snapshot.sample.elastic_band_enabled { + Some(true) => "enabled", + Some(false) => "disabled", + None => "n/a", + }; + let planner_mode = snapshot + .sample + .planner_status + .as_ref() + .map_or("n/a", |status| status.mode_name.as_str()); + let planner_target = snapshot + .sample + .planner_status + .as_ref() + .and_then(|status| status.target_vel_mps) + .map_or_else(|| "n/a".to_owned(), |target| format!("{target:.2}")); + println!( + "velocity monitor: planner_mode={planner_mode}, planner_target={planner_target} m/s, cmd_vx={:.2} m/s, actual_vx_sim={:.2} m/s, actual_vx_wall={:.2} m/s, rt={:.2}x, vx_rmse={:.2} m/s, wall_vx_rmse={:.2} m/s, support_band={band_state}", + snapshot.sample.command[0], + snapshot.sample.actual_vx_mps, + snapshot.sample.actual_wall_vx_mps, + snapshot.metrics.real_time_factor, + snapshot.metrics.vx_rmse_mps, + snapshot.metrics.wall_vx_rmse_mps, + ); + if snapshot.support_band_hint { + println!( + "velocity monitor: support band is enabled while a nonzero planar velocity is commanded; press 9 before judging walking speed" + ); + } } fn yaw_from_rotation_xyzw(rotation_xyzw: [f32; 4]) -> f32 { @@ -1516,6 +1902,7 @@ fn world_to_body_planar_delta( fn run_control_loop_inner( transport: &mut T, policy: &dyn WbcPolicy, + policy_name: &str, robot: &RobotConfig, comm: &CommConfig, runtime_command: &ParsedRuntimeCommand, @@ -1527,17 +1914,23 @@ fn run_control_loop_inner( report_frames: &mut Vec, report_max_frames: Option, #[cfg(feature = "vis")] visualizer: &mut Option, -) -> Result<(usize, usize, Duration), String> { +) -> Result<(usize, usize, Duration, Option), String> { let period = Duration::from_secs_f64(1.0 / f64::from(comm.frequency_hz)); + let loop_started_at = Instant::now(); let mut ticks: usize = 0; let mut dropped_frames: usize = 0; let mut inference_total = Duration::ZERO; let mut policy_engaged = !startup_config.enabled(); + let mut pending_policy_engage = false; let mut init_pose_start: Option<(Instant, Vec)> = None; + let mut velocity_monitor = VelocityTrackingMonitor::new(runtime_command); if startup_config.require_engage { - println!("startup init pose active: press ] to engage policy after the robot settles"); + println!( + "startup init pose active: settle window {:.1}s; press ] any time to queue policy engagement; WASD/QE commands are held until engagement", + startup_config.init_pose_secs + ); } while running.load(Ordering::SeqCst) { @@ -1557,14 +1950,32 @@ fn run_control_loop_inner( TeleopControlRequest::Reset => { policy.reset(); policy_engaged = false; + pending_policy_engage = false; init_pose_start = None; println!("startup init pose reset: press ] to re-engage policy"); } TeleopControlRequest::Engage => { - policy.reset(); - policy_engaged = true; - init_pose_start = None; - println!("policy engaged"); + match apply_policy_engage_request( + startup_config, + &mut policy_engaged, + &mut pending_policy_engage, + &mut init_pose_start, + ) { + PolicyEngageRequestOutcome::Immediate => { + policy.reset(); + println!("policy engaged"); + } + PolicyEngageRequestOutcome::QueuedUntilSettled => { + let elapsed = init_pose_start + .as_ref() + .map(|(start_time, _)| start_time.elapsed()); + let remaining_secs = + startup_settle_remaining_secs(startup_config, elapsed); + println!( + "policy engagement queued; waiting for startup settle (~{remaining_secs:.1}s remaining)" + ); + } + } } TeleopControlRequest::ToggleElasticBand => { match transport.toggle_elastic_band() { @@ -1599,7 +2010,9 @@ fn run_control_loop_inner( }; let base_pose = transport.report_base_pose(); let sim_time_secs = transport.report_sim_time_secs(tick_index, comm.frequency_hz); + let wall_time_secs = loop_started_at.elapsed().as_secs_f64(); let replay_state = transport.report_mujoco_state(); + let elastic_band_enabled = transport.report_elastic_band_enabled(); let mut captured_tick: Option = None; run_control_tick(transport, command.clone(), |obs| { @@ -1610,9 +2023,14 @@ fn run_control_loop_inner( let (start_time, start_positions) = init_pose_start .get_or_insert_with(|| (obs.timestamp, obs.joint_positions.clone())); let elapsed = obs.timestamp.saturating_duration_since(*start_time); - if !startup_config.require_engage - && elapsed.as_secs_f32() >= startup_config.init_pose_secs - { + let startup_settled = elapsed.as_secs_f32() >= startup_config.init_pose_secs; + if pending_policy_engage_is_ready(startup_config, pending_policy_engage, elapsed) { + policy.reset(); + policy_engaged = true; + pending_policy_engage = false; + println!("policy engaged"); + policy.predict(&obs) + } else if !startup_config.require_engage && startup_settled { policy_engaged = true; policy.predict(&obs) } else { @@ -1633,9 +2051,16 @@ fn run_control_loop_inner( captured_tick = Some(ReplayFrame { tick: tick_index, sim_time_secs, + wall_time_secs, command_data: tick_command_data.clone(), phase_name: runtime_command.phase_name_for_tick(tick_index, comm.frequency_hz), base_pose, + elastic_band_enabled, + planner_status: planner_status_for_command( + policy_name, + runtime_command, + &tick_command_data, + ), actual_positions: obs.joint_positions.clone(), actual_velocities: obs.joint_velocities.clone(), target_positions: output @@ -1654,6 +2079,10 @@ fn run_control_loop_inner( let cycle_elapsed = cycle_start.elapsed(); if let Some(frame) = captured_tick { + if let Some(snapshot) = velocity_monitor.observe(&frame) { + print_velocity_monitor(&snapshot); + } + #[cfg(feature = "vis")] if let Some(vis) = visualizer.as_mut() { vis.advance_frame(); @@ -1702,7 +2131,12 @@ fn run_control_loop_inner( } } - Ok((ticks, dropped_frames, inference_total)) + Ok(( + ticks, + dropped_frames, + inference_total, + velocity_monitor.finish(), + )) } #[allow(clippy::too_many_lines, clippy::too_many_arguments)] @@ -1751,15 +2185,16 @@ fn run_control_loop( // Transport priority: hardware → sim (if feature enabled) → synthetic. #[cfg(feature = "sim")] - let (ticks, dropped_frames, inference_total, sent_count, replay_transport) = { + let (ticks, dropped_frames, inference_total, velocity_tracking, sent_count, replay_transport) = { if let Some(hw_cfg) = hardware { let mut transport = UnitreeG1Transport::connect(hw_cfg, robot.clone(), comm.frequency_hz) .map_err(|e| format!("hardware transport connect failed: {e}"))?; println!("unitree g1 hardware transport active"); - let (ticks, dropped, inf) = run_control_loop_inner( + let (ticks, dropped, inf, velocity_tracking) = run_control_loop_inner( &mut transport, policy, + policy_name, robot, comm, runtime_command, @@ -1777,6 +2212,7 @@ fn run_control_loop( ticks, dropped, inf, + velocity_tracking, ticks, ReplayTransportMetadata::hardware("unitree_g1"), ) @@ -1792,9 +2228,10 @@ fn run_control_loop( transport.model_variant(), transport.uses_meshless_public_fallback() ); - let (ticks, dropped, inf) = run_control_loop_inner( + let (ticks, dropped, inf, velocity_tracking) = run_control_loop_inner( &mut transport, policy, + policy_name, robot, comm, runtime_command, @@ -1812,6 +2249,7 @@ fn run_control_loop( ticks, dropped, inf, + velocity_tracking, ticks, ReplayTransportMetadata { kind: "mujoco".to_owned(), @@ -1826,9 +2264,10 @@ fn run_control_loop( ) } else { let mut transport = SyntheticTransport::new(robot.default_pose.clone()); - let (ticks, dropped, inf) = run_control_loop_inner( + let (ticks, dropped, inf, velocity_tracking) = run_control_loop_inner( &mut transport, policy, + policy_name, robot, comm, runtime_command, @@ -1846,6 +2285,7 @@ fn run_control_loop( ticks, dropped, inf, + velocity_tracking, transport.sent_commands(), ReplayTransportMetadata::synthetic(), ) @@ -1853,15 +2293,16 @@ fn run_control_loop( }; #[cfg(not(feature = "sim"))] - let (ticks, dropped_frames, inference_total, sent_count, replay_transport) = { + let (ticks, dropped_frames, inference_total, velocity_tracking, sent_count, replay_transport) = { if let Some(hw_cfg) = hardware { let mut transport = UnitreeG1Transport::connect(hw_cfg, robot.clone(), comm.frequency_hz) .map_err(|e| format!("hardware transport connect failed: {e}"))?; println!("unitree g1 hardware transport active"); - let (ticks, dropped, inf) = run_control_loop_inner( + let (ticks, dropped, inf, velocity_tracking) = run_control_loop_inner( &mut transport, policy, + policy_name, robot, comm, runtime_command, @@ -1879,14 +2320,16 @@ fn run_control_loop( ticks, dropped, inf, + velocity_tracking, ticks, ReplayTransportMetadata::hardware("unitree_g1"), ) } else { let mut transport = SyntheticTransport::new(robot.default_pose.clone()); - let (ticks, dropped, inf) = run_control_loop_inner( + let (ticks, dropped, inf, velocity_tracking) = run_control_loop_inner( &mut transport, policy, + policy_name, robot, comm, runtime_command, @@ -1904,6 +2347,7 @@ fn run_control_loop( ticks, dropped, inf, + velocity_tracking, transport.sent_commands(), ReplayTransportMetadata::synthetic(), ) @@ -1923,18 +2367,17 @@ fn run_control_loop( let achieved_frequency_hz = (ticks as f64) / run_time_secs; #[allow(clippy::cast_precision_loss)] let average_inference_ms = (inference_total.as_secs_f64() * 1_000.0) / (ticks as f64); - let velocity_tracking = - compute_velocity_tracking_metrics(&replay_frames, runtime_command, comm.frequency_hz); - println!( "runtime metrics: ticks={ticks}, sent_commands={sent_count}, avg_inference_ms={average_inference_ms:.3}, achieved_hz={achieved_frequency_hz:.2}, dropped_frames={dropped_frames}", ); if let Some(tracking) = &velocity_tracking { println!( - "velocity tracking: vx_rmse={:.3} m/s, vy_rmse={:.3} m/s, yaw_rmse={:.3} rad/s, forward_distance={:.3} m, heading_change={:.1} deg", + "velocity tracking: vx_rmse={:.3} m/s, wall_vx_rmse={:.3} m/s, actual_vx_mean={:.3} m/s, actual_vx_wall_mean={:.3} m/s, real_time_factor={:.2}x, forward_distance={:.3} m, heading_change={:.1} deg", tracking.vx_rmse_mps, - tracking.vy_rmse_mps, - tracking.yaw_rate_rmse_rad_s, + tracking.wall_vx_rmse_mps, + tracking.actual_vx_mean_mps, + tracking.actual_wall_vx_mean_mps, + tracking.real_time_factor, tracking.forward_distance_m, tracking.heading_change_deg, ); @@ -2187,9 +2630,12 @@ mod tests { ReplayFrame { tick, sim_time_secs: f64::from(tick_u32) * 0.02, + wall_time_secs: f64::from(tick_u32) * 0.02, command_data, phase_name: None, base_pose, + elastic_band_enabled: None, + planner_status: None, actual_positions: vec![], actual_velocities: vec![], target_positions: vec![], @@ -2344,6 +2790,15 @@ mod tests { sim.model_path, PathBuf::from("assets/robots/groot_g1_gear_sonic/scene_29dof.xml") ); + assert_eq!( + sim.gain_profile.as_str(), + "default_pd", + "keyboard demo must use the official low-command kp/kd gains" + ); + assert!( + !sim.enforce_target_velocity_limits, + "official MuJoCo bridge sends q-targets directly without the hardware target slew limiter" + ); let band = sim .elastic_band @@ -2351,7 +2806,7 @@ mod tests { assert!(band.enabled); assert_eq!(band.body_name, "pelvis"); assert_eq!(band.anchor, [0.0, 0.0, 1.0]); - assert!(band.anchor_from_initial_pose); + assert!(!band.anchor_from_initial_pose); assert_eq!(band.length, 0.0); assert_eq!(band.kp_pos, 10_000.0); assert_eq!(band.kd_pos, 1_000.0); @@ -2422,6 +2877,81 @@ mod tests { assert!(!outcome.stop_after_tick); } + #[test] + fn engage_request_queues_until_startup_settles_when_required() { + let mut policy_engaged = false; + let mut pending_policy_engage = false; + let mut init_pose_start = None; + let startup_config = RuntimeStartupConfig { + init_pose_secs: 3.0, + require_engage: true, + }; + + let outcome = apply_policy_engage_request( + startup_config, + &mut policy_engaged, + &mut pending_policy_engage, + &mut init_pose_start, + ); + + assert_eq!(outcome, PolicyEngageRequestOutcome::QueuedUntilSettled); + assert!(!policy_engaged); + assert!(pending_policy_engage); + assert!(!pending_policy_engage_is_ready( + startup_config, + pending_policy_engage, + Duration::from_millis(2_900) + )); + assert!(pending_policy_engage_is_ready( + startup_config, + pending_policy_engage, + Duration::from_secs(3) + )); + } + + #[test] + fn startup_settle_remaining_reports_configured_wait() { + let startup_config = RuntimeStartupConfig { + init_pose_secs: 3.0, + require_engage: true, + }; + + assert!((startup_settle_remaining_secs(startup_config, None) - 3.0).abs() < 1e-6); + assert!( + (startup_settle_remaining_secs(startup_config, Some(Duration::from_millis(1_250))) + - 1.75) + .abs() + < 1e-6 + ); + assert!( + startup_settle_remaining_secs(startup_config, Some(Duration::from_secs(5))).abs() + < 1e-6 + ); + } + + #[test] + fn engage_request_is_immediate_without_required_startup_gate() { + let mut policy_engaged = false; + let mut pending_policy_engage = true; + let mut init_pose_start = Some((Instant::now(), vec![1.0, 2.0])); + let startup_config = RuntimeStartupConfig { + init_pose_secs: 3.0, + require_engage: false, + }; + + let outcome = apply_policy_engage_request( + startup_config, + &mut policy_engaged, + &mut pending_policy_engage, + &mut init_pose_start, + ); + + assert_eq!(outcome, PolicyEngageRequestOutcome::Immediate); + assert!(policy_engaged); + assert!(!pending_policy_engage); + assert!(init_pose_start.is_none()); + } + #[test] fn startup_init_pose_interpolates_before_policy_runs() { let robot = startup_test_robot(); @@ -2450,6 +2980,7 @@ mod tests { let metrics = run_control_loop_inner( &mut transport, &policy, + "test_policy", &robot, &comm, &runtime_command, @@ -2505,6 +3036,7 @@ mod tests { let metrics = run_control_loop_inner( &mut transport, &policy, + "test_policy", &robot, &comm, &runtime_command, @@ -3176,9 +3708,12 @@ standing_placeholder_tracking = true }, frames: vec![ReportFrame { tick: 0, + wall_time_secs: 0.0, command_data: vec![0.2, 0.0, 0.1], phase_name: None, base_pose: None, + elastic_band_enabled: None, + planner_status: None, actual_positions: vec![0.1], actual_velocities: vec![0.0], target_positions: vec![0.1], @@ -3303,6 +3838,132 @@ standing_placeholder_tracking = true assert_f64_approx_eq(metrics.heading_change_deg, 0.0); } + #[test] + fn compute_velocity_tracking_metrics_uses_recorded_sim_time() { + let runtime_command = ParsedRuntimeCommand::Velocity([1.0, 0.0, 0.0]); + let mut frames = vec![ + replay_frame( + 0, + vec![1.0, 0.0, 0.0], + Some(ReportBasePose { + position_world: [0.0, 0.0, 0.0], + rotation_xyzw: [0.0, 0.0, 0.0, 1.0], + }), + ), + replay_frame( + 1, + vec![1.0, 0.0, 0.0], + Some(ReportBasePose { + position_world: [0.01, 0.0, 0.0], + rotation_xyzw: [0.0, 0.0, 0.0, 1.0], + }), + ), + replay_frame( + 2, + vec![1.0, 0.0, 0.0], + Some(ReportBasePose { + position_world: [0.02, 0.0, 0.0], + rotation_xyzw: [0.0, 0.0, 0.0, 1.0], + }), + ), + ]; + frames[0].sim_time_secs = 0.0; + frames[1].sim_time_secs = 0.01; + frames[2].sim_time_secs = 0.02; + + let metrics = compute_velocity_tracking_metrics(&frames, &runtime_command, 50) + .expect("forward-motion metrics should be computed"); + assert_eq!(metrics.sample_count, 2); + assert_f64_approx_eq(metrics.vx_rmse_mps, 0.0); + assert_f64_approx_eq(metrics.forward_distance_m, 0.02); + } + + #[test] + fn gear_sonic_planner_status_reports_mode_bands() { + let idle = gear_sonic_planner_status_from_command(&[0.0, 0.0, 0.0]) + .expect("idle velocity should have planner status"); + assert_eq!(idle.mode_id, GEAR_SONIC_PLANNER_MODE_IDLE); + assert_eq!(idle.mode_name, "idle"); + assert_eq!(idle.target_vel_mps, None); + + let slow_walk = gear_sonic_planner_status_from_command(&[0.6, 0.0, 0.0]) + .expect("slow walk velocity should have planner status"); + assert_eq!(slow_walk.mode_id, GEAR_SONIC_PLANNER_MODE_SLOW_WALK); + assert_eq!(slow_walk.mode_name, "slow_walk"); + assert_eq!(slow_walk.target_vel_mps, Some(0.6)); + + let walk = gear_sonic_planner_status_from_command(&[1.0, 0.0, 0.0]) + .expect("walk velocity should have planner status"); + assert_eq!(walk.mode_id, GEAR_SONIC_PLANNER_MODE_WALK); + assert_eq!(walk.mode_name, "walk"); + assert_eq!(walk.target_vel_mps, Some(1.0)); + + let run = gear_sonic_planner_status_from_command(&[1.6, 0.0, 0.0]) + .expect("run velocity should have planner status"); + assert_eq!(run.mode_id, GEAR_SONIC_PLANNER_MODE_RUN); + assert_eq!(run.mode_name, "run"); + assert_eq!(run.target_vel_mps, Some(1.6)); + } + + #[test] + fn planner_status_is_only_reported_for_gear_sonic_velocity_commands() { + let velocity_command = ParsedRuntimeCommand::Velocity([1.0, 0.0, 0.0]); + assert!( + planner_status_for_command("gear_sonic", &velocity_command, &[1.0, 0.0, 0.0]).is_some() + ); + assert!( + planner_status_for_command("decoupled_wbc", &velocity_command, &[1.0, 0.0, 0.0]) + .is_none() + ); + + let motion_tokens = ParsedRuntimeCommand::MotionTokens(vec![0.0]); + assert!( + planner_status_for_command("gear_sonic", &motion_tokens, &[1.0, 0.0, 0.0]).is_none() + ); + } + + #[test] + fn velocity_monitor_collects_without_report_capture() { + let runtime_command = ParsedRuntimeCommand::Velocity([1.0, 0.0, 0.0]); + let mut monitor = VelocityTrackingMonitor::new(&runtime_command); + let mut previous = replay_frame( + 0, + vec![1.0, 0.0, 0.0], + Some(ReportBasePose { + position_world: [0.0, 0.0, 0.0], + rotation_xyzw: [0.0, 0.0, 0.0, 1.0], + }), + ); + previous.elastic_band_enabled = Some(true); + let mut current = replay_frame( + 1, + vec![1.0, 0.0, 0.0], + Some(ReportBasePose { + position_world: [0.02, 0.0, 0.0], + rotation_xyzw: [0.0, 0.0, 0.0, 1.0], + }), + ); + current.sim_time_secs = 0.02; + current.wall_time_secs = 0.04; + current.elastic_band_enabled = Some(true); + + assert!(monitor.observe(&previous).is_none()); + let snapshot = monitor + .observe(¤t) + .expect("support-band hint should force a live monitor sample"); + assert!(snapshot.support_band_hint); + + let metrics = monitor + .finish() + .expect("monitor should compute metrics without replay capture"); + assert_eq!(metrics.sample_count, 1); + assert_f64_approx_eq(metrics.vx_rmse_mps, 0.0); + assert_f64_approx_eq(metrics.wall_vx_rmse_mps, 0.5); + assert_f64_approx_eq(metrics.real_time_factor, 0.5); + assert_f64_approx_eq(metrics.actual_vx_mean_mps, 1.0); + assert_f64_approx_eq(metrics.actual_wall_vx_mean_mps, 0.5); + } + #[test] fn compute_velocity_tracking_metrics_matches_perfect_turn_rate() { let runtime_command = ParsedRuntimeCommand::Velocity([0.0, 0.0, 1.0]); diff --git a/crates/robowbc-ort/src/decoupled.rs b/crates/robowbc-ort/src/decoupled.rs index 01e073a..ecb4911 100644 --- a/crates/robowbc-ort/src/decoupled.rs +++ b/crates/robowbc-ort/src/decoupled.rs @@ -933,11 +933,11 @@ mod tests { /// /// To run once weights are available: /// ```bash - /// bash scripts/download_decoupled_wbc_models.sh + /// bash scripts/models/download_decoupled_wbc_models.sh /// cargo test -p robowbc-ort -- --ignored decoupled_wbc_real_model_inference /// ``` #[test] - #[ignore = "requires real GR00T WholeBodyControl ONNX weights; run scripts/download_decoupled_wbc_models.sh first"] + #[ignore = "requires real GR00T WholeBodyControl ONNX weights; run scripts/models/download_decoupled_wbc_models.sh first"] fn decoupled_wbc_real_model_inference() { let walk_model = Path::new(env!("CARGO_MANIFEST_DIR")) .join("../../models/decoupled-wbc/GR00T-WholeBodyControl-Walk.onnx"); diff --git a/crates/robowbc-ort/src/lib.rs b/crates/robowbc-ort/src/lib.rs index e82a833..ea994b9 100644 --- a/crates/robowbc-ort/src/lib.rs +++ b/crates/robowbc-ort/src/lib.rs @@ -786,10 +786,19 @@ const GEAR_SONIC_DEFAULT_MODE_IDLE: i64 = 0; const GEAR_SONIC_DEFAULT_MODE_SLOW_WALK: i64 = 1; const GEAR_SONIC_DEFAULT_MODE_WALK: i64 = 2; const GEAR_SONIC_DEFAULT_MODE_RUN: i64 = 3; +const GEAR_SONIC_DEFAULT_RANDOM_SEED: i64 = 1234; const GEAR_SONIC_CONTROL_DT_SECS: f32 = 1.0 / 50.0; +const GEAR_SONIC_MIN_MOVING_SPEED_MPS: f32 = 0.2; +const GEAR_SONIC_WALK_COMMAND_MIN_MPS: f32 = 0.8; +const GEAR_SONIC_RUN_COMMAND_MIN_MPS: f32 = 1.5; +const GEAR_SONIC_RUN_COMMAND_MAX_MPS: f32 = 3.0; const GEAR_SONIC_PLANNER_THREAD_INTERVAL_TICKS: usize = 5; const GEAR_SONIC_PLANNER_LOOK_AHEAD_STEPS: usize = 2; const GEAR_SONIC_PLANNER_BLEND_FRAMES: usize = 8; +const GEAR_SONIC_IDLE_READAPT_BLEND_WEIGHT: f32 = 0.02; +const GEAR_SONIC_IDLE_READAPT_ADAPT_TRIGGER_RAD: f32 = 0.10; +const GEAR_SONIC_IDLE_READAPT_ADAPT_STOP_RAD: f32 = 0.05; +const GEAR_SONIC_IDLE_READAPT_RECOVER_TRIGGER_RAD: f32 = 0.045; const GEAR_SONIC_ENCODER_DIM: usize = 64; const GEAR_SONIC_ENCODER_OBS_DICT_DIM: usize = 1762; const GEAR_SONIC_DECODER_OBS_DICT_DIM: usize = 994; @@ -801,18 +810,31 @@ const GEAR_SONIC_ENCODER_MOTION_JOINT_POSITIONS_OFFSET: usize = 4; const GEAR_SONIC_ENCODER_MOTION_JOINT_VELOCITIES_OFFSET: usize = 294; const GEAR_SONIC_ENCODER_MOTION_ANCHOR_ORIENTATION_OFFSET: usize = 601; -/// `IsaacLab` to `MuJoCo` joint index remapping for G1 29-DOF. +/// Effective `MuJoCo` to `IsaacLab` joint index remapping for G1 29-DOF. /// -/// Despite the upstream name, this table is used as the effective -/// `MuJoCo -> IsaacLab` remap throughout the published GEAR-SONIC runtime. -/// Planner outputs and live observations arrive in `MuJoCo` order, and -/// decoder outputs are consumed by looking up the `IsaacLab` index for each -/// `MuJoCo` joint. -const GEAR_SONIC_ISAACLAB_TO_MUJOCO: [usize; 29] = [ +/// Upstream names this array `isaaclab_to_mujoco`, but its use in +/// `CreatePolicyCommand()` is `raw_actions[isaaclab_to_mujoco[mujoco_idx]]`. +/// It therefore maps a `MuJoCo`/hardware joint index to the policy's +/// `IsaacLab` action/observation index. +const GEAR_SONIC_MUJOCO_TO_ISAACLAB: [usize; 29] = [ 0, 3, 6, 9, 13, 17, 1, 4, 7, 10, 14, 18, 2, 5, 8, 11, 15, 19, 21, 23, 25, 27, 12, 16, 20, 22, 24, 26, 28, ]; +/// Effective `IsaacLab` to `MuJoCo` joint index remapping for G1 29-DOF. +/// +/// Upstream names this array `mujoco_to_isaaclab` and uses it when converting +/// planner `mujoco_qpos` output into the policy-order motion sequence. +const GEAR_SONIC_ISAACLAB_TO_MUJOCO: [usize; 29] = [ + 0, 6, 12, 1, 7, 13, 2, 8, 14, 3, 9, 15, 22, 4, 10, 16, 23, 5, 11, 17, 24, 18, 25, 19, 26, 20, + 27, 21, 28, +]; + +/// Lower-body IsaacLab-order joint indexes used by upstream idle readaptation. +const GEAR_SONIC_IDLE_READAPT_LOWER_BODY_ISAACLAB_INDICES: [usize; 12] = + [0, 1, 3, 4, 6, 7, 9, 10, 13, 14, 17, 18]; +const GEAR_SONIC_IDLE_READAPT_LOWER_BODY_COUNT_F32: f32 = 12.0; + /// Per-joint action scale for G1 in `MuJoCo` order. /// /// Computed from the GEAR-SONIC C++ deployment code as @@ -863,6 +885,16 @@ struct GearSonicPlannerState { init_ref_root_quat_wxyz: Option<[f32; 4]>, last_command: Option, pending_replan: Option, + idle_readapt_original_targets_isaaclab: [f32; 29], + idle_readapt_stored: bool, + idle_readapt_state: GearSonicIdleReadaptState, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum GearSonicIdleReadaptState { + Idle, + Adapting, + Recovering, } #[derive(Debug, Clone, Copy, PartialEq)] @@ -901,6 +933,9 @@ impl GearSonicPlannerState { init_ref_root_quat_wxyz: None, last_command: None, pending_replan: None, + idle_readapt_original_targets_isaaclab: [0.0; 29], + idle_readapt_stored: false, + idle_readapt_state: GearSonicIdleReadaptState::Idle, } } } @@ -1141,7 +1176,7 @@ fn load_csv_matrix_f32(path: &Path) -> CoreResult<(usize, usize, Vec)> { if header.starts_with("version https://git-lfs.github.com/spec/v1") { return Err(robowbc_core::WbcError::InferenceFailed(format!( - "reference motion file {} is still a Git LFS pointer; run scripts/download_gear_sonic_reference_motions.sh to materialize the official clip payloads", + "reference motion file {} is still a Git LFS pointer; run scripts/models/download_gear_sonic_reference_motions.sh to materialize the official clip payloads", path.display() ))); } @@ -1521,7 +1556,7 @@ impl GearSonicPolicy { wrapped } - fn bin_planner_angle_to_8_directions(angle: f32) -> (f32, f32) { + fn bin_planner_angle_to_8_directions(angle: f32) -> f32 { const BIN_SIZE: f32 = std::f32::consts::FRAC_PI_4; const HALF_BIN_SIZE: f32 = BIN_SIZE * 0.5; @@ -1546,15 +1581,7 @@ impl GearSonicPolicy { 4 }; - let slow_walk_speed = match bin_index { - -1..=1 => 0.3, - -2 | 2 => 0.35, - -3 | 3 => 0.25, - -4 | 4 => 0.2, - _ => unreachable!("planner direction bin should stay within [-4, 4]"), - }; - - let binned_angle = match bin_index { + match bin_index { -4 => -4.0 * BIN_SIZE, -3 => -3.0 * BIN_SIZE, -2 => -2.0 * BIN_SIZE, @@ -1565,9 +1592,7 @@ impl GearSonicPolicy { 3 => 3.0 * BIN_SIZE, 4 => 4.0 * BIN_SIZE, _ => unreachable!("planner direction bin should stay within [-4, 4]"), - }; - - (binned_angle, slow_walk_speed) + } } fn vec3_distance(a: [f32; 3], b: [f32; 3]) -> f32 { @@ -1605,14 +1630,17 @@ impl GearSonicPolicy { let local_movement_angle = twist.linear[1].atan2(twist.linear[0]); let movement_angle = Self::wrap_angle_rad(planner_state.facing_yaw_rad + local_movement_angle); - let (movement_angle, slow_walk_speed) = - Self::bin_planner_angle_to_8_directions(movement_angle); - let (mode, target_vel) = if cmd_norm < 0.8 { - (GEAR_SONIC_DEFAULT_MODE_SLOW_WALK, slow_walk_speed) - } else if cmd_norm < 2.5 { - (GEAR_SONIC_DEFAULT_MODE_WALK, -1.0) + let movement_angle = Self::bin_planner_angle_to_8_directions(movement_angle); + let target_vel = cmd_norm.clamp( + GEAR_SONIC_MIN_MOVING_SPEED_MPS, + GEAR_SONIC_RUN_COMMAND_MAX_MPS, + ); + let mode = if target_vel < GEAR_SONIC_WALK_COMMAND_MIN_MPS { + GEAR_SONIC_DEFAULT_MODE_SLOW_WALK + } else if target_vel < GEAR_SONIC_RUN_COMMAND_MIN_MPS { + GEAR_SONIC_DEFAULT_MODE_WALK } else { - (GEAR_SONIC_DEFAULT_MODE_RUN, -1.0) + GEAR_SONIC_DEFAULT_MODE_RUN }; GearSonicPlannerCommand { mode, @@ -1635,19 +1663,19 @@ impl GearSonicPolicy { fn planner_joint_positions_isaaclab(frame: &[f32]) -> Vec { let mut positions = vec![0.0_f32; GEAR_SONIC_ISAACLAB_TO_MUJOCO.len()]; - for (mujoco_idx, &isaaclab_idx) in GEAR_SONIC_ISAACLAB_TO_MUJOCO.iter().enumerate() { + for (isaaclab_idx, &mujoco_idx) in GEAR_SONIC_ISAACLAB_TO_MUJOCO.iter().enumerate() { positions[isaaclab_idx] = frame[GEAR_SONIC_PLANNER_JOINT_OFFSET + mujoco_idx]; } positions } fn joint_positions_mujoco_to_isaaclab(joint_positions: &[f32]) -> Vec { - if joint_positions.len() != GEAR_SONIC_ISAACLAB_TO_MUJOCO.len() { + if joint_positions.len() != GEAR_SONIC_MUJOCO_TO_ISAACLAB.len() { return joint_positions.to_vec(); } let mut positions = vec![0.0_f32; joint_positions.len()]; - for (mujoco_idx, &isaaclab_idx) in GEAR_SONIC_ISAACLAB_TO_MUJOCO.iter().enumerate() { + for (mujoco_idx, &isaaclab_idx) in GEAR_SONIC_MUJOCO_TO_ISAACLAB.iter().enumerate() { positions[isaaclab_idx] = joint_positions[mujoco_idx]; } positions @@ -1657,8 +1685,8 @@ impl GearSonicPolicy { robot: &RobotConfig, obs: &Observation, ) -> Vec { - if obs.joint_positions.len() != GEAR_SONIC_ISAACLAB_TO_MUJOCO.len() - || robot.default_pose.len() != GEAR_SONIC_ISAACLAB_TO_MUJOCO.len() + if obs.joint_positions.len() != GEAR_SONIC_MUJOCO_TO_ISAACLAB.len() + || robot.default_pose.len() != GEAR_SONIC_MUJOCO_TO_ISAACLAB.len() { return obs .joint_positions @@ -1669,7 +1697,7 @@ impl GearSonicPolicy { } let mut positions = vec![0.0_f32; obs.joint_positions.len()]; - for (mujoco_idx, &isaaclab_idx) in GEAR_SONIC_ISAACLAB_TO_MUJOCO.iter().enumerate() { + for (mujoco_idx, &isaaclab_idx) in GEAR_SONIC_MUJOCO_TO_ISAACLAB.iter().enumerate() { positions[isaaclab_idx] = obs.joint_positions[mujoco_idx] - robot.default_pose[mujoco_idx]; } @@ -1677,12 +1705,12 @@ impl GearSonicPolicy { } fn observation_joint_velocities_isaaclab(obs: &Observation) -> Vec { - if obs.joint_velocities.len() != GEAR_SONIC_ISAACLAB_TO_MUJOCO.len() { + if obs.joint_velocities.len() != GEAR_SONIC_MUJOCO_TO_ISAACLAB.len() { return obs.joint_velocities.clone(); } let mut velocities = vec![0.0_f32; obs.joint_velocities.len()]; - for (mujoco_idx, &isaaclab_idx) in GEAR_SONIC_ISAACLAB_TO_MUJOCO.iter().enumerate() { + for (mujoco_idx, &isaaclab_idx) in GEAR_SONIC_MUJOCO_TO_ISAACLAB.iter().enumerate() { velocities[isaaclab_idx] = obs.joint_velocities[mujoco_idx]; } velocities @@ -1943,6 +1971,16 @@ impl GearSonicPolicy { } } + fn planner_periodic_replan_due( + command: GearSonicPlannerCommand, + steps_since_plan: usize, + ) -> bool { + if command.mode == GEAR_SONIC_DEFAULT_MODE_IDLE { + return false; + } + steps_since_plan >= Self::planner_replan_interval_ticks(command) + } + fn build_velocity_encoder_obs_dict( obs: &Observation, planner_state: &GearSonicPlannerState, @@ -2027,6 +2065,106 @@ impl GearSonicPolicy { .min(planner_state.motion_qpos_50hz.len() - 1); } + fn planner_qpos_isaaclab_joint(frame: &[f32], isaaclab_idx: usize) -> f32 { + let mujoco_idx = GEAR_SONIC_ISAACLAB_TO_MUJOCO[isaaclab_idx]; + frame[GEAR_SONIC_PLANNER_JOINT_OFFSET + mujoco_idx] + } + + fn set_planner_qpos_isaaclab_joint(frame: &mut [f32], isaaclab_idx: usize, value: f32) { + let mujoco_idx = GEAR_SONIC_ISAACLAB_TO_MUJOCO[isaaclab_idx]; + frame[GEAR_SONIC_PLANNER_JOINT_OFFSET + mujoco_idx] = value; + } + + fn maybe_apply_idle_readaptation( + planner_state: &mut GearSonicPlannerState, + obs: &Observation, + command: GearSonicPlannerCommand, + ) { + if command.mode != GEAR_SONIC_DEFAULT_MODE_IDLE + || planner_state.motion_qpos_50hz.is_empty() + || obs.joint_positions.len() != GEAR_SONIC_ISAACLAB_TO_MUJOCO.len() + { + return; + } + + let last_frame_idx = planner_state.motion_qpos_50hz.len() - 1; + if planner_state.current_motion_frame < last_frame_idx { + return; + } + + if !planner_state.idle_readapt_stored { + let frame = &planner_state.motion_qpos_50hz[last_frame_idx]; + for &isaaclab_idx in &GEAR_SONIC_IDLE_READAPT_LOWER_BODY_ISAACLAB_INDICES { + planner_state.idle_readapt_original_targets_isaaclab[isaaclab_idx] = + Self::planner_qpos_isaaclab_joint(frame, isaaclab_idx); + } + planner_state.idle_readapt_stored = true; + planner_state.idle_readapt_state = GearSonicIdleReadaptState::Idle; + } + + let frame = &planner_state.motion_qpos_50hz[last_frame_idx]; + let mut total_error = 0.0_f32; + for &isaaclab_idx in &GEAR_SONIC_IDLE_READAPT_LOWER_BODY_ISAACLAB_INDICES { + let mujoco_idx = GEAR_SONIC_ISAACLAB_TO_MUJOCO[isaaclab_idx]; + total_error += (Self::planner_qpos_isaaclab_joint(frame, isaaclab_idx) + - obs.joint_positions[mujoco_idx]) + .abs(); + } + let avg_error = total_error / GEAR_SONIC_IDLE_READAPT_LOWER_BODY_COUNT_F32; + + match planner_state.idle_readapt_state { + GearSonicIdleReadaptState::Idle => { + if avg_error > GEAR_SONIC_IDLE_READAPT_ADAPT_TRIGGER_RAD { + planner_state.idle_readapt_state = GearSonicIdleReadaptState::Adapting; + } else if avg_error < GEAR_SONIC_IDLE_READAPT_RECOVER_TRIGGER_RAD { + planner_state.idle_readapt_state = GearSonicIdleReadaptState::Recovering; + } + } + GearSonicIdleReadaptState::Adapting => { + if avg_error < GEAR_SONIC_IDLE_READAPT_ADAPT_STOP_RAD { + planner_state.idle_readapt_state = GearSonicIdleReadaptState::Idle; + } + } + GearSonicIdleReadaptState::Recovering => { + if avg_error > GEAR_SONIC_IDLE_READAPT_ADAPT_TRIGGER_RAD { + planner_state.idle_readapt_state = GearSonicIdleReadaptState::Adapting; + } + } + } + + match planner_state.idle_readapt_state { + GearSonicIdleReadaptState::Adapting => { + let frame = &mut planner_state.motion_qpos_50hz[last_frame_idx]; + for &isaaclab_idx in &GEAR_SONIC_IDLE_READAPT_LOWER_BODY_ISAACLAB_INDICES { + let mujoco_idx = GEAR_SONIC_ISAACLAB_TO_MUJOCO[isaaclab_idx]; + let current = Self::planner_qpos_isaaclab_joint(frame, isaaclab_idx); + let actual = obs.joint_positions[mujoco_idx]; + Self::set_planner_qpos_isaaclab_joint( + frame, + isaaclab_idx, + (1.0 - GEAR_SONIC_IDLE_READAPT_BLEND_WEIGHT) * current + + GEAR_SONIC_IDLE_READAPT_BLEND_WEIGHT * actual, + ); + } + } + GearSonicIdleReadaptState::Recovering => { + let frame = &mut planner_state.motion_qpos_50hz[last_frame_idx]; + for &isaaclab_idx in &GEAR_SONIC_IDLE_READAPT_LOWER_BODY_ISAACLAB_INDICES { + let current = Self::planner_qpos_isaaclab_joint(frame, isaaclab_idx); + let original = + planner_state.idle_readapt_original_targets_isaaclab[isaaclab_idx]; + Self::set_planner_qpos_isaaclab_joint( + frame, + isaaclab_idx, + (1.0 - GEAR_SONIC_IDLE_READAPT_BLEND_WEIGHT) * current + + GEAR_SONIC_IDLE_READAPT_BLEND_WEIGHT * original, + ); + } + } + GearSonicIdleReadaptState::Idle => {} + } + } + fn run_fixture_motion_tokens( &self, obs: &Observation, @@ -2090,7 +2228,7 @@ impl GearSonicPolicy { let target_vel = [command.target_vel]; let mode = [command.mode]; let height = [command.height]; - let random_seed = [0_i64]; + let random_seed = [GEAR_SONIC_DEFAULT_RANDOM_SEED]; let has_specific_target = [0_i64]; let specific_target_positions = vec![0.0_f32; 4 * 3]; let specific_target_headings = vec![0.0_f32; 4]; @@ -2271,6 +2409,8 @@ impl GearSonicPolicy { planner_state.motion_joint_velocities_isaaclab = Self::compute_motion_joint_velocities_isaaclab(&planner_state.motion_qpos_50hz); planner_state.current_motion_frame = 0; + planner_state.idle_readapt_stored = false; + planner_state.idle_readapt_state = GearSonicIdleReadaptState::Idle; planner_state.init_ref_root_quat_wxyz = planner_state .motion_qpos_50hz .first() @@ -2366,11 +2506,6 @@ impl GearSonicPolicy { })?; let command_speed = (twist.linear[0] * twist.linear[0] + twist.linear[1] * twist.linear[1]).sqrt(); - if planner_state.motion_qpos_50hz.is_empty() && command_speed <= 0.01 { - drop(planner_state); - let encoder_obs = Self::build_placeholder_encoder_obs_dict(&self.robot); - return self.run_tracking_contract_from_encoder_obs(obs, &encoder_obs); - } if planner_state.motion_qpos_50hz.is_empty() { planner_state.context = Self::initialize_planner_context(&self.robot, obs); planner_state.last_context_frame = planner_state @@ -2385,13 +2520,12 @@ impl GearSonicPolicy { let initializing_planner = planner_state.motion_qpos_50hz.is_empty(); let planner_command = Self::planner_command_for_velocity_bootstrap(&planner_state, command, command_speed); - let replan_interval_ticks = Self::planner_replan_interval_ticks(command); let planner_tick_due = initializing_planner || planner_state.steps_since_planner_tick >= GEAR_SONIC_PLANNER_THREAD_INTERVAL_TICKS; let needs_replan = initializing_planner || (planner_tick_due && (Self::planner_command_changed(planner_state.last_command, command) - || planner_state.steps_since_plan >= replan_interval_ticks)); + || Self::planner_periodic_replan_due(command, planner_state.steps_since_plan))); if needs_replan { if !planner_state.motion_qpos_50hz.is_empty() { @@ -2449,6 +2583,7 @@ impl GearSonicPolicy { robowbc_core::WbcError::InferenceFailed("planner state mutex poisoned".to_owned()) })?; Self::advance_planner_motion_frame(&mut planner_state); + Self::maybe_apply_idle_readaptation(&mut planner_state, obs, command); drop(planner_state); Ok(targets) @@ -2666,7 +2801,7 @@ impl GearSonicPolicy { } let mut positions = vec![0.0_f32; self.robot.joint_count]; - for (mujoco_idx, &isaaclab_idx) in GEAR_SONIC_ISAACLAB_TO_MUJOCO.iter().enumerate() { + for (mujoco_idx, &isaaclab_idx) in GEAR_SONIC_MUJOCO_TO_ISAACLAB.iter().enumerate() { let action = raw_actions[isaaclab_idx]; let scaled = action * GEAR_SONIC_G1_ACTION_SCALE[mujoco_idx]; positions[mujoco_idx] = self.robot.default_pose[mujoco_idx] + scaled; @@ -2959,8 +3094,8 @@ mod tests { } fn joint_velocities_isaaclab_to_mujoco(isaaclab_velocities: &[f32]) -> Vec { - let mut mujoco_velocities = vec![0.0_f32; GEAR_SONIC_ISAACLAB_TO_MUJOCO.len()]; - for (mujoco_idx, &isaaclab_idx) in GEAR_SONIC_ISAACLAB_TO_MUJOCO.iter().enumerate() { + let mut mujoco_velocities = vec![0.0_f32; GEAR_SONIC_MUJOCO_TO_ISAACLAB.len()]; + for (mujoco_idx, &isaaclab_idx) in GEAR_SONIC_MUJOCO_TO_ISAACLAB.iter().enumerate() { mujoco_velocities[mujoco_idx] = isaaclab_velocities[isaaclab_idx]; } mujoco_velocities @@ -2978,7 +3113,7 @@ mod tests { let joint_velocities = motion_joint_velocities_isaaclab .get(clamped_idx) .map_or_else( - || vec![0.0_f32; GEAR_SONIC_ISAACLAB_TO_MUJOCO.len()], + || vec![0.0_f32; GEAR_SONIC_MUJOCO_TO_ISAACLAB.len()], |velocities| joint_velocities_isaaclab_to_mujoco(velocities), ); @@ -3453,6 +3588,85 @@ mod tests { assert!(matches!(err, robowbc_core::WbcError::UnsupportedCommand(_))); } + /// Regression for the keyboard demo: pressing `]` engages the velocity + /// policy while the command is still zero. That must start the real idle + /// planner/decoder path so the support band can be dropped while the + /// controller is already active. + #[test] + #[ignore = "requires real GEAR-SONIC ONNX models; run scripts/models/download_gear_sonic_models.sh first"] + fn gear_sonic_zero_velocity_bootstraps_idle_planner_motion() { + let model_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../models/gear-sonic"); + let encoder_path = model_dir.join("model_encoder.onnx"); + let decoder_path = model_dir.join("model_decoder.onnx"); + let planner_path = model_dir.join("planner_sonic.onnx"); + for path in [&encoder_path, &decoder_path, &planner_path] { + assert!( + path.exists(), + "missing real GEAR-Sonic model at {}; run scripts/models/download_gear_sonic_models.sh first", + path.display() + ); + } + + let robot_config_path = PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("../../configs/robots/unitree_g1_gear_sonic.toml"); + let robot = + RobotConfig::from_toml_file(&robot_config_path).expect("robot config should load"); + let policy = GearSonicPolicy::new(GearSonicConfig { + encoder: OrtConfig { + model_path: encoder_path, + optimization_level: OptimizationLevel::Extended, + num_threads: 1, + execution_provider: ExecutionProvider::Cpu, + }, + decoder: OrtConfig { + model_path: decoder_path, + optimization_level: OptimizationLevel::Extended, + num_threads: 1, + execution_provider: ExecutionProvider::Cpu, + }, + planner: OrtConfig { + model_path: planner_path, + optimization_level: OptimizationLevel::Extended, + num_threads: 1, + execution_provider: ExecutionProvider::Cpu, + }, + reference_motion: None, + robot: robot.clone(), + }) + .expect("real GEAR-Sonic policy should build"); + + let obs = Observation { + joint_positions: robot.default_pose.clone(), + joint_velocities: vec![0.0; robot.joint_count], + gravity_vector: [0.0, 0.0, -1.0], + angular_velocity: [0.0, 0.0, 0.0], + base_pose: None, + command: WbcCommand::Velocity(robowbc_core::Twist { + linear: [0.0, 0.0, 0.0], + angular: [0.0, 0.0, 0.0], + }), + timestamp: Instant::now(), + }; + + let targets = policy + .predict(&obs) + .expect("zero-velocity prediction should run idle planner"); + + assert_eq!(targets.positions.len(), robot.joint_count); + let planner_state = policy + .planner_state + .lock() + .expect("planner state mutex should not be poisoned"); + assert!( + !planner_state.motion_qpos_50hz.is_empty(), + "engaged zero-velocity mode must bootstrap an idle planner motion" + ); + assert_eq!( + planner_state.last_command, + Some(GearSonicPolicy::idle_planner_command()) + ); + } + #[test] fn gear_sonic_planner_context_frame_prefers_observed_base_pose() { let obs = Observation { @@ -3484,10 +3698,10 @@ mod tests { } #[test] - fn gear_sonic_planner_command_uses_slow_walk_speed_for_low_velocity() { + fn gear_sonic_planner_command_uses_slow_walk_speed_for_default_velocity() { let mut planner_state = GearSonicPlannerState::new(&test_robot_config(29)); let forward = robowbc_core::Twist { - linear: [0.6, 0.0, 0.0], + linear: [0.3, 0.0, 0.0], angular: [0.0, 0.0, 0.0], }; let command = GearSonicPolicy::derive_planner_command(&mut planner_state, &forward); @@ -3498,6 +3712,57 @@ mod tests { assert_vec3_approx_eq(command.facing_direction, [1.0, 0.0, 0.0]); } + #[test] + fn gear_sonic_planner_command_uses_slow_walk_target_velocity_for_mid_velocity() { + let mut planner_state = GearSonicPlannerState::new(&test_robot_config(29)); + let forward = robowbc_core::Twist { + linear: [0.6, 0.0, 0.0], + angular: [0.0, 0.0, 0.0], + }; + let command = GearSonicPolicy::derive_planner_command(&mut planner_state, &forward); + + assert_eq!(command.mode, GEAR_SONIC_DEFAULT_MODE_SLOW_WALK); + assert!((command.target_vel - 0.6).abs() < 1e-6); + assert_vec3_approx_eq(command.movement_direction, [1.0, 0.0, 0.0]); + assert_vec3_approx_eq(command.facing_direction, [1.0, 0.0, 0.0]); + } + + #[test] + fn gear_sonic_planner_command_clamps_tiny_velocity_to_minimum_moving_speed() { + let mut planner_state = GearSonicPlannerState::new(&test_robot_config(29)); + let forward = robowbc_core::Twist { + linear: [0.05, 0.0, 0.0], + angular: [0.0, 0.0, 0.0], + }; + let command = GearSonicPolicy::derive_planner_command(&mut planner_state, &forward); + + assert_eq!(command.mode, GEAR_SONIC_DEFAULT_MODE_SLOW_WALK); + assert!((command.target_vel - GEAR_SONIC_MIN_MOVING_SPEED_MPS).abs() < 1e-6); + assert_vec3_approx_eq(command.movement_direction, [1.0, 0.0, 0.0]); + } + + #[test] + fn gear_sonic_planner_command_uses_walk_and_run_ranges() { + let mut planner_state = GearSonicPlannerState::new(&test_robot_config(29)); + let walk = robowbc_core::Twist { + linear: [1.0, 0.0, 0.0], + angular: [0.0, 0.0, 0.0], + }; + let walk_command = GearSonicPolicy::derive_planner_command(&mut planner_state, &walk); + + assert_eq!(walk_command.mode, GEAR_SONIC_DEFAULT_MODE_WALK); + assert!((walk_command.target_vel - 1.0).abs() < 1e-6); + + let run = robowbc_core::Twist { + linear: [1.6, 0.0, 0.0], + angular: [0.0, 0.0, 0.0], + }; + let run_command = GearSonicPolicy::derive_planner_command(&mut planner_state, &run); + + assert_eq!(run_command.mode, GEAR_SONIC_DEFAULT_MODE_RUN); + assert!((run_command.target_vel - 1.6).abs() < 1e-6); + } + #[test] fn gear_sonic_planner_command_uses_lateral_slow_walk_speed_bin() { let mut planner_state = GearSonicPlannerState::new(&test_robot_config(29)); @@ -3508,7 +3773,7 @@ mod tests { let command = GearSonicPolicy::derive_planner_command(&mut planner_state, &left); assert_eq!(command.mode, GEAR_SONIC_DEFAULT_MODE_SLOW_WALK); - assert!((command.target_vel - 0.35).abs() < 1e-6); + assert!((command.target_vel - 0.4).abs() < 1e-6); assert_vec3_approx_eq(command.movement_direction, [0.0, 1.0, 0.0]); } @@ -3532,7 +3797,7 @@ mod tests { let command = GearSonicPolicy::derive_planner_command(&mut planner_state, &forward); assert_eq!(command.mode, GEAR_SONIC_DEFAULT_MODE_WALK); - assert_eq!(command.target_vel, -1.0); + assert!((command.target_vel - 1.0).abs() < 1e-6); assert_vec3_approx_eq(command.movement_direction, [0.0, -1.0, 0.0]); assert_vec3_approx_eq(command.facing_direction, [0.0, -1.0, 0.0]); } @@ -3614,6 +3879,130 @@ mod tests { assert_eq!(planner_command, GearSonicPolicy::idle_planner_command()); } + #[test] + fn gear_sonic_idle_command_does_not_periodically_replan() { + assert!(!GearSonicPolicy::planner_periodic_replan_due( + GearSonicPolicy::idle_planner_command(), + GEAR_SONIC_PLANNER_REPLAN_INTERVAL_TICKS_DEFAULT + )); + } + + #[test] + fn gear_sonic_idle_tail_readaptation_blends_lower_body_toward_actual_joints() { + let robot = test_robot_config(29); + let mut planner_state = GearSonicPlannerState::new(&robot); + planner_state.motion_qpos_50hz = vec![ + GearSonicPolicy::make_standing_qpos(&robot), + GearSonicPolicy::make_standing_qpos(&robot), + ]; + planner_state.current_motion_frame = planner_state.motion_qpos_50hz.len() - 1; + + { + let tail_frame = planner_state + .motion_qpos_50hz + .last_mut() + .expect("tail frame should exist"); + for &isaaclab_idx in &GEAR_SONIC_IDLE_READAPT_LOWER_BODY_ISAACLAB_INDICES { + GearSonicPolicy::set_planner_qpos_isaaclab_joint(tail_frame, isaaclab_idx, 0.5); + } + } + + let obs = Observation { + joint_positions: vec![0.0; 29], + joint_velocities: vec![0.0; 29], + gravity_vector: [0.0, 0.0, -1.0], + angular_velocity: [0.0, 0.0, 0.0], + base_pose: None, + command: WbcCommand::Velocity(robowbc_core::Twist { + linear: [0.0, 0.0, 0.0], + angular: [0.0, 0.0, 0.0], + }), + timestamp: Instant::now(), + }; + + GearSonicPolicy::maybe_apply_idle_readaptation( + &mut planner_state, + &obs, + GearSonicPolicy::idle_planner_command(), + ); + + assert!(planner_state.idle_readapt_stored); + assert_eq!( + planner_state.idle_readapt_state, + GearSonicIdleReadaptState::Adapting + ); + let tail_frame = planner_state + .motion_qpos_50hz + .last() + .expect("tail frame should exist"); + for &isaaclab_idx in &GEAR_SONIC_IDLE_READAPT_LOWER_BODY_ISAACLAB_INDICES { + let target = GearSonicPolicy::planner_qpos_isaaclab_joint(tail_frame, isaaclab_idx); + assert!((target - 0.49).abs() < 1e-6); + assert!( + (planner_state.idle_readapt_original_targets_isaaclab[isaaclab_idx] - 0.5).abs() + < 1e-6 + ); + } + } + + #[test] + fn gear_sonic_idle_tail_readaptation_recovers_toward_original_target() { + let robot = test_robot_config(29); + let mut planner_state = GearSonicPlannerState::new(&robot); + planner_state.motion_qpos_50hz = vec![GearSonicPolicy::make_standing_qpos(&robot)]; + planner_state.current_motion_frame = 0; + planner_state.idle_readapt_stored = true; + planner_state.idle_readapt_state = GearSonicIdleReadaptState::Recovering; + + for &isaaclab_idx in &GEAR_SONIC_IDLE_READAPT_LOWER_BODY_ISAACLAB_INDICES { + planner_state.idle_readapt_original_targets_isaaclab[isaaclab_idx] = 0.5; + } + + let obs = Observation { + joint_positions: vec![0.0; 29], + joint_velocities: vec![0.0; 29], + gravity_vector: [0.0, 0.0, -1.0], + angular_velocity: [0.0, 0.0, 0.0], + base_pose: None, + command: WbcCommand::Velocity(robowbc_core::Twist { + linear: [0.0, 0.0, 0.0], + angular: [0.0, 0.0, 0.0], + }), + timestamp: Instant::now(), + }; + + GearSonicPolicy::maybe_apply_idle_readaptation( + &mut planner_state, + &obs, + GearSonicPolicy::idle_planner_command(), + ); + + let tail_frame = planner_state + .motion_qpos_50hz + .last() + .expect("tail frame should exist"); + for &isaaclab_idx in &GEAR_SONIC_IDLE_READAPT_LOWER_BODY_ISAACLAB_INDICES { + let target = GearSonicPolicy::planner_qpos_isaaclab_joint(tail_frame, isaaclab_idx); + assert!((target - 0.01).abs() < 1e-6); + } + } + + #[test] + fn gear_sonic_walk_command_periodically_replans() { + let command = GearSonicPlannerCommand { + mode: GEAR_SONIC_DEFAULT_MODE_WALK, + target_vel: -1.0, + height: GEAR_SONIC_DEFAULT_HEIGHT_SENTINEL, + movement_direction: [1.0, 0.0, 0.0], + facing_direction: [1.0, 0.0, 0.0], + }; + + assert!(GearSonicPolicy::planner_periodic_replan_due( + command, + GEAR_SONIC_PLANNER_REPLAN_INTERVAL_TICKS_DEFAULT + )); + } + #[test] fn gear_sonic_planner_allowed_pred_mask_matches_upstream_default() { assert_eq!( @@ -3627,6 +4016,11 @@ mod tests { assert!((GEAR_SONIC_DEFAULT_HEIGHT_METERS - 0.788_74).abs() < 1e-6); } + #[test] + fn gear_sonic_planner_random_seed_matches_upstream_default() { + assert_eq!(GEAR_SONIC_DEFAULT_RANDOM_SEED, 1234); + } + #[test] fn gear_sonic_reset_clears_tracking_state() { if !has_test_model() { @@ -3698,7 +4092,7 @@ mod tests { let offsets = GearSonicPolicy::observation_joint_positions_isaaclab_offsets(&robot, &obs); let velocities = GearSonicPolicy::observation_joint_velocities_isaaclab(&obs); - for (mujoco_idx, &isaaclab_idx) in GEAR_SONIC_ISAACLAB_TO_MUJOCO.iter().enumerate() { + for (mujoco_idx, &isaaclab_idx) in GEAR_SONIC_MUJOCO_TO_ISAACLAB.iter().enumerate() { let expected_offset = joint_positions[mujoco_idx] - robot.default_pose[mujoco_idx]; assert!( (offsets[isaaclab_idx] - expected_offset).abs() < 1e-6, @@ -3803,6 +4197,8 @@ mod tests { let decoder_obs = GearSonicPolicy::build_decoder_obs_dict(&tokens, &history); let mut expected = tokens.clone(); + // Upstream release observations call StateLogger::GetLatest with the + // default newest_first=false, so each history block is oldest-to-newest. for angular_velocity in &history.angular_velocity { expected.extend_from_slice(angular_velocity); } @@ -3824,7 +4220,7 @@ mod tests { } #[test] - #[ignore = "requires real GEAR-SONIC ONNX models; run scripts/download_gear_sonic_models.sh first"] + #[ignore = "requires real GEAR-SONIC ONNX models; run scripts/models/download_gear_sonic_models.sh first"] fn gear_sonic_dump_model_meta() { let model_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../models/gear-sonic"); let encoder = OrtBackend::from_file(model_dir.join("model_encoder.onnx")).unwrap(); @@ -3863,7 +4259,7 @@ mod tests { } #[test] - #[ignore = "requires real GEAR-SONIC ONNX models; run scripts/download_gear_sonic_models.sh first"] + #[ignore = "requires real GEAR-SONIC ONNX models; run scripts/models/download_gear_sonic_models.sh first"] fn gear_sonic_dump_tracking_placeholder_tensors() { let model_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../models/gear-sonic"); let encoder_path = model_dir.join("model_encoder.onnx"); @@ -3875,7 +4271,7 @@ mod tests { for path in [&encoder_path, &decoder_path, &planner_path] { assert!( path.exists(), - "model not found: {path:?} — run scripts/download_gear_sonic_models.sh" + "model not found: {path:?} — run scripts/models/download_gear_sonic_models.sh" ); } @@ -3971,7 +4367,7 @@ mod tests { } #[test] - #[ignore = "requires real GEAR-Sonic ONNX models; run scripts/download_gear_sonic_models.sh first"] + #[ignore = "requires real GEAR-Sonic ONNX models; run scripts/models/download_gear_sonic_models.sh first"] #[allow(clippy::too_many_lines)] fn gear_sonic_dump_velocity_first_live_replan_tensors() { let model_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../models/gear-sonic"); @@ -3984,7 +4380,7 @@ mod tests { for path in [&encoder_path, &decoder_path, &planner_path] { assert!( path.exists(), - "model not found: {path:?} — run scripts/download_gear_sonic_models.sh" + "model not found: {path:?} — run scripts/models/download_gear_sonic_models.sh" ); } @@ -4188,7 +4584,7 @@ mod tests { } #[test] - #[ignore = "requires real GEAR-Sonic ONNX models; run scripts/download_gear_sonic_models.sh first"] + #[ignore = "requires real GEAR-Sonic ONNX models; run scripts/models/download_gear_sonic_models.sh first"] #[allow(clippy::too_many_lines)] fn gear_sonic_dump_velocity_later_motion_tensors() { let model_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../models/gear-sonic"); @@ -4201,7 +4597,7 @@ mod tests { for path in [&encoder_path, &decoder_path, &planner_path] { assert!( path.exists(), - "model not found: {path:?} — run scripts/download_gear_sonic_models.sh" + "model not found: {path:?} — run scripts/models/download_gear_sonic_models.sh" ); } @@ -4456,11 +4852,11 @@ mod tests { /// /// Run with: /// ``` - /// bash scripts/download_gear_sonic_models.sh + /// bash scripts/models/download_gear_sonic_models.sh /// cargo test -p robowbc-ort -- --ignored gear_sonic_real_model_inference /// ``` #[test] - #[ignore = "requires real GEAR-SONIC ONNX models; run scripts/download_gear_sonic_models.sh first"] + #[ignore = "requires real GEAR-SONIC ONNX models; run scripts/models/download_gear_sonic_models.sh first"] #[allow(clippy::too_many_lines)] fn gear_sonic_real_model_inference() { use robowbc_core::WbcPolicy; @@ -4476,7 +4872,7 @@ mod tests { for path in [&encoder_path, &decoder_path, &planner_path] { assert!( path.exists(), - "model not found: {path:?} — run scripts/download_gear_sonic_models.sh" + "model not found: {path:?} — run scripts/models/download_gear_sonic_models.sh" ); } diff --git a/crates/robowbc-ort/src/wbc_agile.rs b/crates/robowbc-ort/src/wbc_agile.rs index 28a9530..f4ee5e8 100644 --- a/crates/robowbc-ort/src/wbc_agile.rs +++ b/crates/robowbc-ort/src/wbc_agile.rs @@ -983,11 +983,11 @@ mod tests { /// /// To run once weights are available: /// ```bash - /// bash scripts/download_wbc_agile_models.sh + /// bash scripts/models/download_wbc_agile_models.sh /// cargo test -p robowbc-ort -- --ignored wbc_agile_real_model_inference /// ``` #[test] - #[ignore = "requires real WBC-AGILE G1 ONNX weights; run scripts/download_wbc_agile_models.sh first"] + #[ignore = "requires real WBC-AGILE G1 ONNX weights; run scripts/models/download_wbc_agile_models.sh first"] fn wbc_agile_real_model_inference() { let model_path = Path::new(env!("CARGO_MANIFEST_DIR")) .join("../../models/wbc-agile/unitree_g1_velocity_e2e.onnx"); diff --git a/crates/robowbc-sim/Cargo.toml b/crates/robowbc-sim/Cargo.toml index 2de2ebd..746c82e 100644 --- a/crates/robowbc-sim/Cargo.toml +++ b/crates/robowbc-sim/Cargo.toml @@ -12,7 +12,7 @@ path = "src/lib.rs" [features] default = [] mujoco = ["dep:mujoco-rs", "mujoco-rs/renderer"] -mujoco-viewer = ["mujoco", "mujoco-rs/viewer"] +mujoco-viewer = ["mujoco", "mujoco-rs/viewer-ui"] mujoco-auto-download = ["mujoco", "mujoco-rs/auto-download-mujoco"] [dependencies] diff --git a/crates/robowbc-sim/src/lib.rs b/crates/robowbc-sim/src/lib.rs index c7c4f48..c58c73b 100644 --- a/crates/robowbc-sim/src/lib.rs +++ b/crates/robowbc-sim/src/lib.rs @@ -69,6 +69,14 @@ pub struct MujocoConfig { /// when present. #[serde(default = "default_gain_profile")] pub gain_profile: MujocoGainProfile, + /// Apply [`robowbc_core::RobotConfig::joint_velocity_limits`] as a + /// software q-target slew limit before computing PD torques. + /// + /// The default preserves the generic hardware-style safety behavior. Set + /// this to `false` for official simulator parity when the upstream stack + /// sends each policy q-target directly to the `MuJoCo` PD loop. + #[serde(default = "default_enforce_target_velocity_limits")] + pub enforce_target_velocity_limits: bool, /// Open a live `MuJoCo` viewer and sync it after each control tick. /// /// Requires building with the `mujoco-viewer` feature. This is intended @@ -93,6 +101,10 @@ const fn default_gain_profile() -> MujocoGainProfile { MujocoGainProfile::SimulationPd } +const fn default_enforce_target_velocity_limits() -> bool { + true +} + /// Upstream-style `MuJoCo` virtual support band. /// /// GR00T's G1 `MuJoCo` simulator enables this for the teleop path to keep the @@ -112,8 +124,9 @@ pub struct MujocoElasticBandConfig { pub anchor: [f64; 3], /// Replace `anchor` with the selected body's initialized world position. /// - /// This preserves the standing height for local demos while still letting - /// the band catch the humanoid if it starts to fall. + /// Leave this `false` for official GR00T-style demos that use the fixed + /// `[0, 0, 1]` support point. Set it only for custom demos that need the + /// band anchored at the model's initialized body position. #[serde(default)] pub anchor_from_initial_pose: bool, /// Additional vertical rest length added to the spring error. @@ -184,6 +197,7 @@ impl Default for MujocoConfig { timestep: default_timestep(), substeps: default_substeps(), gain_profile: default_gain_profile(), + enforce_target_velocity_limits: default_enforce_target_velocity_limits(), viewer: false, elastic_band: None, } @@ -282,6 +296,7 @@ mod tests { assert!((cfg.timestep - 0.002).abs() < f64::EPSILON); assert_eq!(cfg.substeps, 10); assert_eq!(cfg.gain_profile, MujocoGainProfile::SimulationPd); + assert!(cfg.enforce_target_velocity_limits); } #[test] @@ -291,6 +306,7 @@ mod tests { timestep: 0.001, substeps: 20, gain_profile: MujocoGainProfile::DefaultPd, + enforce_target_velocity_limits: false, viewer: true, elastic_band: Some(MujocoElasticBandConfig::default()), }; diff --git a/crates/robowbc-sim/src/transport.rs b/crates/robowbc-sim/src/transport.rs index 0d97668..434b8ca 100644 --- a/crates/robowbc-sim/src/transport.rs +++ b/crates/robowbc-sim/src/transport.rs @@ -18,6 +18,10 @@ use std::env; use std::fs; use std::io::Read; use std::path::{Path, PathBuf}; +#[cfg(feature = "mujoco-viewer")] +use std::sync::atomic::{AtomicBool, Ordering}; +#[cfg(feature = "mujoco-viewer")] +use std::sync::Arc; use std::sync::OnceLock; use std::time::{Instant, SystemTime, UNIX_EPOCH}; @@ -115,6 +119,8 @@ pub struct MujocoTransport { prev_positions: Vec, #[cfg(feature = "mujoco-viewer")] viewer: Option, + #[cfg(feature = "mujoco-viewer")] + viewer_elastic_band_toggle_requested: Option>, } impl MujocoTransport { @@ -181,15 +187,25 @@ impl MujocoTransport { build_elastic_band(&data, elastic_band_body_id, config.elastic_band.clone()); let prev_positions = robot_config.default_pose.clone(); #[cfg(feature = "mujoco-viewer")] + let viewer_elastic_band_toggle_requested = + config.viewer.then(|| Arc::new(AtomicBool::new(false))); + #[cfg(feature = "mujoco-viewer")] let viewer = if config.viewer { - Some( - MjViewer::builder() - .window_name("RoboWBC MuJoCo keyboard demo") - .build_passive(data.model()) - .map_err(|error| SimError::ViewerFailed { - reason: error.to_string(), - })?, - ) + let mut viewer = MjViewer::builder() + .window_name("RoboWBC MuJoCo keyboard demo") + .build_passive(data.model()) + .map_err(|error| SimError::ViewerFailed { + reason: error.to_string(), + })?; + if let Some(toggle_requested) = &viewer_elastic_band_toggle_requested { + let toggle_requested = Arc::clone(toggle_requested); + viewer.add_ui_callback_detached(move |ctx| { + if ctx.input(|input| input.key_pressed(mujoco_rs::viewer::egui::Key::Num9)) { + toggle_requested.store(true, Ordering::Relaxed); + } + }); + } + Some(viewer) } else { None }; @@ -214,6 +230,8 @@ impl MujocoTransport { prev_positions, #[cfg(feature = "mujoco-viewer")] viewer, + #[cfg(feature = "mujoco-viewer")] + viewer_elastic_band_toggle_requested, }) } @@ -417,16 +435,31 @@ impl MujocoTransport { #[cfg(feature = "mujoco-viewer")] fn render_viewer_if_enabled(&mut self) -> Result<(), SimError> { - let Some(viewer) = self.viewer.as_mut() else { - return Ok(()); - }; - if !viewer.running() { - return Ok(()); + { + let Some(viewer) = self.viewer.as_mut() else { + return Ok(()); + }; + if !viewer.running() { + return Ok(()); + } + viewer.sync_data(&mut self.data); + viewer.render().map_err(|error| SimError::ViewerFailed { + reason: error.to_string(), + })?; } - viewer.sync_data(&mut self.data); - viewer.render().map_err(|error| SimError::ViewerFailed { - reason: error.to_string(), - }) + if self + .viewer_elastic_band_toggle_requested + .as_ref() + .is_some_and(|requested| requested.swap(false, Ordering::Relaxed)) + { + if let Some(enabled) = self.toggle_elastic_band_enabled() { + println!( + "elastic support band {}", + if enabled { "enabled" } else { "disabled" } + ); + } + } + Ok(()) } fn joint_state_vectors(&self) -> (Vec, Vec) { @@ -596,13 +629,17 @@ impl RobotTransport for MujocoTransport { let after_pos = clamp_position_targets(targets, self.robot_config.simulation_joint_limits()); let control_frequency_hz = control_frequency_hz(&self.config); - let safe_targets = if let Some(ref vel_limits) = self.robot_config.joint_velocity_limits { - clamp_velocity_targets( - &after_pos, - &self.prev_positions, - vel_limits, - control_frequency_hz, - ) + let safe_targets = if self.config.enforce_target_velocity_limits { + if let Some(ref vel_limits) = self.robot_config.joint_velocity_limits { + clamp_velocity_targets( + &after_pos, + &self.prev_positions, + vel_limits, + control_frequency_hz, + ) + } else { + after_pos + } } else { after_pos }; @@ -1596,6 +1633,7 @@ mod tests { timestep: 0.002, substeps: 1, gain_profile: MujocoGainProfile::DefaultPd, + enforce_target_velocity_limits: true, viewer: false, elastic_band: None, }, @@ -1754,6 +1792,18 @@ mod tests { assert!((gravity[2] + 1.0).abs() < 1e-6); } + #[test] + fn gravity_from_free_joint_quaternion_matches_official_projected_gravity_signs() { + let quarter_turn = std::f64::consts::FRAC_PI_4; + let pitch_up_gravity = + gravity_from_free_joint_quaternion([quarter_turn.cos(), 0.0, quarter_turn.sin(), 0.0]); + assert_vec3_approx_eq(pitch_up_gravity, [1.0, 0.0, 0.0]); + + let roll_left_gravity = + gravity_from_free_joint_quaternion([quarter_turn.cos(), quarter_turn.sin(), 0.0, 0.0]); + assert_vec3_approx_eq(roll_left_gravity, [0.0, -1.0, 0.0]); + } + #[test] fn recv_imu_uses_floating_base_state_when_available() { let robot = load_robot_config("configs/robots/unitree_g1.toml"); @@ -1859,6 +1909,34 @@ mod tests { assert_eq!(transport.prev_positions, expected.positions); } + #[test] + fn send_joint_targets_can_disable_target_velocity_limit_for_official_sim() { + let robot = load_robot_config("configs/robots/unitree_g1.toml"); + let mut transport = MujocoTransport::new( + MujocoConfig { + model_path: g1_model_path(), + timestep: 0.002, + substeps: 10, + enforce_target_velocity_limits: false, + ..MujocoConfig::default() + }, + robot.clone(), + ) + .expect("transport should initialize"); + + let unsafe_targets = JointPositionTargets { + positions: vec![100.0; robot.joint_count], + timestamp: Instant::now(), + }; + let after_pos = clamp_position_targets(&unsafe_targets, &robot.joint_limits); + + transport + .send_joint_targets(&unsafe_targets) + .expect("send should succeed"); + + assert_eq!(transport.prev_positions, after_pos.positions); + } + #[test] fn send_joint_targets_prefers_simulation_joint_limits_for_mujoco_clamp() { let mut robot = load_robot_config("configs/robots/unitree_g1_35dof_wbc_agile.toml"); diff --git a/crates/robowbc-teleop/src/keyboard.rs b/crates/robowbc-teleop/src/keyboard.rs index fc54f67..b2c446b 100644 --- a/crates/robowbc-teleop/src/keyboard.rs +++ b/crates/robowbc-teleop/src/keyboard.rs @@ -17,7 +17,7 @@ use crate::keymap::{KeymapConfig, TeleopAction}; use crate::{TeleopError, TeleopEvent, TeleopSource}; -use crossterm::event::{poll, read, Event, KeyEvent, KeyEventKind}; +use crossterm::event::{poll, read, Event, KeyCode, KeyEvent, KeyEventKind, KeyModifiers}; use crossterm::terminal::{disable_raw_mode, enable_raw_mode}; use std::time::Duration; @@ -110,6 +110,9 @@ impl KeyboardTeleop { if !matches!(event.kind, KeyEventKind::Press) { return None; } + if is_ctrl_c(event) { + return Some(TeleopEvent::Quit); + } let action = self.keymap.lookup(event)?; Some(self.apply_action(action)) } @@ -249,16 +252,25 @@ fn clamp_step(value: f32, clamp: f32) -> f32 { value.clamp(-abs_clamp, abs_clamp) } +fn is_ctrl_c(event: KeyEvent) -> bool { + matches!(event.code, KeyCode::Char('c' | 'C')) + && event.modifiers.contains(KeyModifiers::CONTROL) +} + #[cfg(test)] mod tests { use super::*; - use crossterm::event::{KeyCode, KeyEventKind, KeyEventState, KeyModifiers}; + use crossterm::event::{KeyEventKind, KeyEventState}; use std::time::Instant; fn key_press(code: KeyCode) -> KeyEvent { + key_press_with_modifiers(code, KeyModifiers::NONE) + } + + fn key_press_with_modifiers(code: KeyCode, modifiers: KeyModifiers) -> KeyEvent { KeyEvent { code, - modifiers: KeyModifiers::NONE, + modifiers, kind: KeyEventKind::Press, state: KeyEventState::NONE, } @@ -358,6 +370,18 @@ mod tests { assert_eq!(event, TeleopEvent::Quit); } + #[test] + fn ctrl_c_emits_quit_in_raw_mode() { + let mut teleop = KeyboardTeleop::new(); + let event = teleop + .handle_key(key_press_with_modifiers( + KeyCode::Char('c'), + KeyModifiers::CONTROL, + )) + .unwrap(); + assert_eq!(event, TeleopEvent::Quit); + } + #[test] fn unbound_keys_are_dropped() { let mut teleop = KeyboardTeleop::new(); diff --git a/deny.toml b/deny.toml index 6bf84a3..ac22103 100644 --- a/deny.toml +++ b/deny.toml @@ -67,6 +67,9 @@ allow = [ "CDLA-Permissive-2.0", "BSL-1.0", "OpenSSL", + # Font licenses pulled by egui's default viewer fonts through mujoco-rs. + "OFL-1.1", + "Ubuntu-font-1.0", ] confidence-threshold = 0.8 exceptions = [] diff --git a/docs/README.md b/docs/README.md index 9e20df9..f30325d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -35,12 +35,16 @@ transports. ## Start here +- `README.md`, understand what the project is and how to try it +- `STATUS.md`, see what works now and what is next +- `ARCHITECTURE.md`, get the high-level system map - [Getting Started](getting-started.md), build the workspace and run the smoke config - [G1 MJCF Review (2026-04-24)](g1-mjcf-review-2026-04-24.md), compare the reviewed upstream G1 MuJoCo models and the RoboWBC config decisions - [Configuration Reference](configuration.md), understand the TOML surface +- [Maintainer Scripts](scripts.md), find the canonical script folders and edit checks - [Adding a New Policy](adding-a-model.md), wire a new model into the registry - [Adding a New Robot](adding-a-robot.md), add a new hardware target -- [Architecture](architecture.md), understand the crate split and runtime flow +- [Detailed Architecture](architecture.md), understand the crate split and runtime flow ## License diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index fb67f5e..e878406 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -1,29 +1,31 @@ # Summary -[Introduction](README.md) - ---- - -# User Guide +# Start Here +- [Introduction](README.md) - [Getting Started](getting-started.md) - [Python SDK](python-sdk.md) - [Configuration Reference](configuration.md) +- [Human Documentation Map](human/README.md) + +# Architecture And Runtime -# Tutorials +- [Architecture](architecture.md) +- [Roboharness Integration](roboharness-integration.md) +- [G1 MJCF Review](g1-mjcf-review-2026-04-24.md) + +# Extending RoboWBC - [Adding a New Policy](adding-a-model.md) - [Adding a New Robot](adding-a-robot.md) -# Reference +# Operations And Evidence -- [Architecture](architecture.md) -- [Roboharness Integration](roboharness-integration.md) - [Benchmarks](benchmarks/README.md) +- [Maintainer Scripts](scripts.md) +- [Third-Party Notices](third-party-notices.md) ---- - -# Project +# Project Direction - [Roadmap](roadmap-2026-q2.md) - [Ecosystem Strategy](ecosystem-strategy.md) diff --git a/docs/agents/README.md b/docs/agents/README.md new file mode 100644 index 0000000..6d75745 --- /dev/null +++ b/docs/agents/README.md @@ -0,0 +1,15 @@ +# Agent Runbooks + +These docs hold repo-specific agent procedures that are too detailed for the +root startup files. + +Read the runbook that matches the task: + +- `domain.md` - canonical domain docs and vocabulary. +- `architecture.md` - crate map, runtime contracts, policy registry, and gotchas. +- `rust-workflow.md` - toolchain, build, check, test, lint, format, and docs. +- `testing.md` - focused verification choices for Rust, Python, site, SDK, and MuJoCo work. +- `keyboard-demo.md` - protected `make demo-keyboard` behavior and stability test. +- `github-workflow.md` - GitHub MCP usage, PR review, CI log triage, and commit rules. +- `issue-tracker.md` - GitHub issue handling. +- `triage-labels.md` - issue label vocabulary. diff --git a/docs/agents/architecture.md b/docs/agents/architecture.md new file mode 100644 index 0000000..366dc7e --- /dev/null +++ b/docs/agents/architecture.md @@ -0,0 +1,54 @@ +# Architecture Notes For Agents + +For human-facing architecture, read `docs/architecture.md` first. This file is a +compact agent crib sheet for implementation work. + +## Crate Map + +| Crate | Role | +|-------|------| +| `robowbc-core` | `WbcPolicy`, `Observation`, `WbcCommand`, `JointPositionTargets`, `RobotConfig`, validation | +| `robowbc-config` | typed config loading and validation | +| `robowbc-registry` | inventory-based policy registration and `WbcRegistry::build` | +| `robowbc-ort` | ONNX Runtime backends and first-party policy wrappers | +| `robowbc-pyo3` | Python-backed runtime policy loading | +| `robowbc-py` | standalone maturin package for the Python SDK | +| `robowbc-runtime` | reusable runtime loop pieces | +| `robowbc-comm` | communication-oriented control-loop plumbing | +| `robowbc-transport` | transport abstractions and backends | +| `robowbc-sim` | MuJoCo transport for hardware-free execution | +| `robowbc-teleop` | keyboard teleop and keymap handling | +| `robowbc-vis` | Rerun visualization and `.rrd` recording | +| `robowbc-cli` | `robowbc` binary and config-driven command surface | +| `unitree-hg-idl` | Unitree HG message serialization helpers | + +## Core Contract + +Policies implement `WbcPolicy` and return joint position targets. They should +advertise capabilities, reject unsupported commands explicitly, and preserve +the unified observation and output contracts from `robowbc-core`. + +Model switching is config-driven through registry names such as `gear_sonic`, +`decoupled_wbc`, `wbc_agile`, `bfm_zero`, `hover`, `wholebody_vla`, and +`py_model`. + +## Integration Gotchas + +- ONNX models must match the pinned `ort` version and supported opset. +- CUDA and TensorRT providers require host NVIDIA runtime compatibility. +- `inventory` registration only discovers crates linked into the final binary. +- PyO3 policies must be GIL-aware. +- GEAR-Sonic uses `planner_sonic.onnx` for the live velocity path; the + encoder/decoder standing-placeholder path is narrower. +- `hover` is blocked on user-supplied exported checkpoints. +- `wholebody_vla` is experimental because no runnable public upstream release + exists yet. + +## Protected Public Surfaces + +- Python SDK types: `Registry`, `Observation`, policy wrappers, and + `MujocoSession`. +- Config files under `configs/`. +- CLI entry point: `cargo run --bin robowbc -- run --config ...`. +- Generated JSON, Rerun, proof-pack, and site bundle contracts consumed by + `scripts/site/generate_policy_showcase.py` and `scripts/site/validate_site_bundle.py`. diff --git a/docs/agents/domain.md b/docs/agents/domain.md index b9e83fe..0e02dc5 100644 --- a/docs/agents/domain.md +++ b/docs/agents/domain.md @@ -4,8 +4,11 @@ This is a single-context repo. ## Before exploring, read these +- `README.md` — project overview and first-run paths +- `STATUS.md` — current state, active queue, and blocked work +- `ARCHITECTURE.md` — high-level current architecture - `docs/founding-document.md` — project goals, landscape, design decisions, roadmap -- `docs/architecture.md` — current architecture +- `docs/architecture.md` — detailed architecture - Relevant topic docs under `docs/`, such as `docs/adding-a-model.md`, `docs/adding-a-robot.md`, and `docs/configuration.md` If `CONTEXT.md`, `CONTEXT-MAP.md`, or `docs/adr/` are added later, read them as higher-specificity domain context. @@ -16,4 +19,4 @@ Use the project's existing domain language: WBC policies, inference runtime, `Wb ## Flag conflicts -If a proposal contradicts `docs/founding-document.md`, `docs/architecture.md`, or a future ADR, surface the conflict explicitly. +If a proposal contradicts `docs/founding-document.md`, `ARCHITECTURE.md`, `docs/architecture.md`, or a future ADR, surface the conflict explicitly. diff --git a/docs/agents/github-workflow.md b/docs/agents/github-workflow.md new file mode 100644 index 0000000..379ac3b --- /dev/null +++ b/docs/agents/github-workflow.md @@ -0,0 +1,63 @@ +# GitHub Workflow + +Use GitHub MCP tools for GitHub operations. Do not assume the `gh` CLI is +installed or authenticated. + +## Issues And PRDs + +- Repository: `MiaoDX/robowbc`. +- Read issue bodies, comments, and labels before triage or implementation. +- Use `docs/agents/triage-labels.md` for the five-label vocabulary. +- When a skill says to publish to the issue tracker, create a GitHub issue. + +## PR Review And Fixes + +When reviewing a PR: + +1. Fetch and check out the PR source branch. +2. Read the diff and failing CI logs before diagnosing. +3. Apply fixes directly to the PR source branch unless the user asks for a new + branch. +4. Run the focused and standard verification that covers the change. +5. Commit and push to the same branch so the existing PR updates. + +Do not create a separate PR for review fixes unless the user explicitly wants +that workflow. + +## CI Failure Triage + +Always read the actual log output before diagnosing a failure. + +Minimum sequence: + +1. Identify the failed job and step. +2. Read the full error output around the failure. +3. Map the failure to the command in `Makefile` or the workflow file. +4. Reproduce locally when feasible. +5. Fix root cause, not only the symptom. + +The main workflow uses: + +- `make check` +- `make test` +- `make sim-feature-test` +- `make clippy` +- `make fmt-check` +- `make rust-doc` +- `make docs-book` +- `make showcase-verify` +- `make python-sdk-verify` + +## Commits + +This repository uses rebase-only merges. Keep commits scoped and descriptive: +`docs: ...`, `fix: ...`, `refactor: ...`, and similar. + +Codex-authored commits must include: + +```text +Co-authored-by: Codex +``` + +If another AI coding agent authors a commit, include the matching co-author +trailer so agent usage remains auditable. diff --git a/docs/agents/keyboard-demo.md b/docs/agents/keyboard-demo.md new file mode 100644 index 0000000..ec5b450 --- /dev/null +++ b/docs/agents/keyboard-demo.md @@ -0,0 +1,64 @@ +# Keyboard Demo Guardrail + +`make demo-keyboard` is the "git clone and see it work" path. Treat changes to +that path as user-facing runtime changes. + +## Preserve These Behaviors + +- `configs/demo/gear_sonic_keyboard_mujoco.toml` uses the GR00T scene wrapper. +- The demo keeps an explicit `[sim.elastic_band]` support band with the official + GR00T point `[0.0, 0.0, 1.0]` and spring-damper gains. +- `[runtime].require_engage = true` stays enabled. +- `[runtime].init_pose_secs` stays enabled so `]` engages policy only after the + robot settles. +- The `9` key remains the upstream-style support-band toggle from the terminal + and the MuJoCo viewer; the first press after engagement drops the robot toward + foot contact. +- The scene remains visibly lit enough to inspect foot contact in the MuJoCo + viewer. + +## Targeted Stability Test + +For MuJoCo demo changes, run: + +```bash +MUJOCO_DOWNLOAD_DIR="$(pwd)/.cache/mujoco" \ +MUJOCO_DYNAMIC_LINK_DIR="$(pwd)/.cache/mujoco/mujoco-3.6.0/lib" \ +LD_LIBRARY_PATH="$(pwd)/.cache/mujoco/mujoco-3.6.0/lib:${LD_LIBRARY_PATH:-}" \ +cargo test -p robowbc-sim --features mujoco-auto-download \ + gear_sonic_demo_model_holds_default_pose_for_startup_window +``` + +Report the exact blocker if MuJoCo download, dynamic linking, EGL/OpenGL, or +display access prevents this test from running. + +## Speed Monitor + +The CLI prints a live `velocity monitor` line for velocity-command runs. The +line includes `planner_mode` and `planner_target` for GEAR-Sonic, then the +commanded-vs-actual velocity metrics. Read `actual_vx_sim` as policy tracking +in simulated time and `actual_vx_wall` plus `rt` as what a human sees in real +time. If `rt` is well below `1.0x`, the control/viewer loop is running slower +than real time even if simulated-time tracking is correct. + +If `support_band=enabled` while a nonzero planar velocity is commanded, press +`9` before judging walking speed. The protected demo intentionally starts with +the fixed GR00T support band enabled, and that band can resist translational +motion until it is toggled off. + +GEAR-Sonic still uses one controller stack for these commands. The live planner +is conditioned by mode plus `target_vel`, movement direction, and facing +direction. RoboWBC maps `0.2..0.8 m/s` to slow-walk mode, `0.8..1.5 m/s` to +walk mode, and `1.5..3.0 m/s` to run mode while passing the requested planar +speed as `target_vel`. + +## Manual Demo Command + +The stable public entry point is: + +```bash +make demo-keyboard +``` + +It downloads GEAR-Sonic assets, prepares the MuJoCo runtime, starts the viewer, +and runs keyboard teleop with the checked-in config. diff --git a/docs/agents/rust-workflow.md b/docs/agents/rust-workflow.md new file mode 100644 index 0000000..f737602 --- /dev/null +++ b/docs/agents/rust-workflow.md @@ -0,0 +1,72 @@ +# Rust Workflow + +Use the Makefile when possible; it mirrors the CI entry points and keeps command +names stable. + +## Fresh Environment Preflight + +Do not start with tests on a fresh machine. Run: + +```bash +make toolchain +make build +make check +``` + +Expected Rust version: stable Rust 1.75 or newer. + +If the toolchain is missing, install Rust with rustup, then re-run the same +preflight: + +```bash +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y +source "$HOME/.cargo/env" +``` + +If build or check fails because of system dependencies such as CUDA, MuJoCo, +protobuf, Python, or OpenGL/EGL libraries, report the exact error and do not +claim later tests are representative. + +## Standard Rust Gates + +For Rust changes, run: + +```bash +make test +make clippy +make fmt-check +``` + +Run `make rust-doc` when public Rust APIs, doc comments, or intra-doc links +changed. + +`make verify` is the combined local Rust gate: + +```bash +make verify +``` + +It runs `make check`, `make test`, `make clippy`, and `make fmt-check`. + +## Focused Debugging + +Use package and test-name filters while debugging: + +```bash +cargo test -p robowbc-core -- validator +cargo test -p robowbc-ort latency +``` + +After focused debugging, run the broader gate that covers the changed surface. + +## Documentation Build + +Use: + +```bash +make rust-doc +make docs-book +``` + +`make docs` runs both. The mdBook target downloads the pinned mdBook binary into +`.cache/bin` when needed. diff --git a/docs/agents/testing.md b/docs/agents/testing.md new file mode 100644 index 0000000..df434e3 --- /dev/null +++ b/docs/agents/testing.md @@ -0,0 +1,85 @@ +# Testing Guidance + +Prefer tests that prove caller-visible behavior through public interfaces. +Avoid tests that only duplicate constants, assert language mechanics, or depend +on private helper call order. + +## Rust + +Default Rust gate for code changes: + +```bash +make test +make clippy +make fmt-check +``` + +Use focused cargo filters while debugging, then run the broader gate: + +```bash +cargo test -p robowbc-core -- test_name +``` + +Run `make rust-doc` when public APIs or doc links change. + +## Python Scripts And Site Tests + +The repository has pytest tests for benchmark and site helper scripts. Run them +when touching `scripts/`, `tests/`, site generation, benchmark normalization, or +RoboHarness report code: + +```bash +make python-test +``` + +If a Python dependency is missing, report it and prefer the documented Makefile +target for that workflow before installing ad hoc packages. + +## Site And Showcase + +For static bundle or showcase changes: + +```bash +make site-smoke +``` + +For CI-equivalent showcase verification: + +```bash +make showcase-verify +``` + +`make showcase-verify` downloads public checkpoints and requires a working +headless MuJoCo EGL stack. + +## Python SDK + +For `crates/robowbc-py`, `crates/robowbc-pyo3`, or SDK examples: + +```bash +make python-sdk-verify +``` + +This builds, installs, and smoke-tests the local wheel. + +## MuJoCo + +For MuJoCo transport or demo startup changes: + +```bash +make sim-feature-test +``` + +For keyboard demo stability specifically, use `docs/agents/keyboard-demo.md`. + +## Benchmarks + +Run benchmark commands only when latency, benchmark generation, or comparison +artifacts changed: + +```bash +make benchmark-nvidia +``` + +Do not add generated benchmark output unless it is an intentionally tracked +artifact. diff --git a/docs/architecture.md b/docs/architecture.md index b1467fb..5419831 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,165 +1,189 @@ # Architecture +This document is the detailed human-facing architecture map. For a shorter root +overview, read `../ARCHITECTURE.md`. + ## Overview ![RoboWBC architecture](assets/architecture.svg) -The diagram below reflects the current crate split, policy backends, -transport loop, and reporting path used by the CLI and the published HTML -report. +RoboWBC centers on one policy contract: +```text +Observation + WbcCommand + | + v +WbcPolicy::predict + | + v +JointPositionTargets ``` -┌─────────────────────────────────────────────────┐ -│ robowbc-cli — config loading → control loop │ -└────────────────────────┬────────────────────────┘ - │ WbcPolicy::predict() - ┌─────────────┼─────────────┐ - │ │ │ - ┌───────▼──────┐ ┌────▼────┐ ┌─────▼──────┐ - │robowbc-ort │ │robowbc- │ │ (future) │ - │ONNX Runtime │ │pyo3 │ │ burn / … │ - │CUDA/TensorRT │ │PyTorch │ │ │ - └──────────────┘ └─────────┘ └────────────┘ - │ - ┌───────▼───────┐ ┌──────────────────────┐ - │robowbc-core │ │ robowbc-registry │ - │WbcPolicy trait│ │ inventory factory │ - │Observation │ │ WbcRegistry::build() │ - │WbcCommand │ └──────────────────────┘ - │JointPositions │ - └───────────────┘ - │ - ┌───────▼───────┐ - │robowbc-comm │ - │zenoh transport│ - │control loop │ - └───────────────┘ + +Configs and SDK calls decide which policy, robot, transport, visualization, and +reporting path surrounds that contract. The CLI and Python SDK both exercise +the same Rust policy implementations. + +## Runtime Data Flow + +```text +TOML config + | + v +robowbc-config + | + v +robowbc-registry + | + +--> robowbc-ort policy wrappers + | gear_sonic, decoupled_wbc, wbc_agile, bfm_zero, hover, wholebody_vla + | + +--> robowbc-pyo3 py_model wrapper + | + v +robowbc-runtime / robowbc-cli + | + +--> hardware transport through robowbc-comm or robowbc-transport + +--> MuJoCo transport through robowbc-sim + +--> synthetic transport for smoke and reporting + | + v +JSON report, replay trace, Rerun recording, generated static site ``` -## Crate responsibilities +Transport priority in the CLI is hardware when configured, then MuJoCo when the +feature and config are present, then synthetic execution. + +## Crate Responsibilities | Crate | Responsibility | -|-------|---------------| -| `robowbc-core` | `WbcPolicy` trait, `Observation`, `WbcCommand`, `JointPositionTargets`, `RobotConfig` | -| `robowbc-ort` | ONNX Runtime inference backend (ort crate, CUDA/TensorRT support) | -| `robowbc-pyo3` | PyTorch inference backend via PyO3 (Python GIL-aware) | -| `robowbc-registry` | `inventory`-based compile-time policy registration and factory | -| `robowbc-comm` | zenoh communication layer, control loop tick runner | -| `robowbc-cli` | CLI entry point: config parsing, backend selection, control loop | -| `robowbc-sim` | MuJoCo simulation transport for hardware-free testing | -| `robowbc-vis` | Rerun visualization integration | - -## The `WbcPolicy` trait - -```rust -pub trait WbcPolicy: Send + Sync { - fn predict(&self, obs: &Observation) -> Result; - fn reset(&self) {} - fn control_frequency_hz(&self) -> u32; - fn supported_robots(&self) -> &[RobotConfig]; -} -``` +|-------|----------------| +| `robowbc-core` | Core types: `WbcPolicy`, `Observation`, `WbcCommand`, `JointPositionTargets`, `RobotConfig`, `PolicyCapabilities`, and target validation | +| `robowbc-config` | TOML config structures, defaults, robot/policy path resolution, and config validation | +| `robowbc-registry` | `inventory`-based policy registration and construction from config | +| `robowbc-ort` | ONNX Runtime backend, execution-provider config, and first-party policy wrappers | +| `robowbc-pyo3` | Runtime backend for user-supplied Python or PyTorch policy modules | +| `robowbc-py` | Standalone Python SDK exposing registry, policy, observation, command, target, and MuJoCo session types | +| `robowbc-runtime` | Finite-state runtime coordination around policy execution, validator decisions, teleop requests, and control outputs | +| `robowbc-comm` | Communication-oriented control-loop helpers, wire helpers, zenoh helpers, and Unitree G1 wiring | +| `robowbc-transport` | Pluggable transport trait plus in-memory and CycloneDDS-oriented backends | +| `robowbc-sim` | MuJoCo config and transport for hardware-free execution | +| `robowbc-teleop` | Keyboard teleop source, keymap config, and teleop events | +| `robowbc-vis` | Rerun visualizer and robot scene logging | +| `robowbc-cli` | `robowbc` binary, config validation, policy runs, policy helpers, reports, teleop, and visualization wiring | +| `unitree-hg-idl` | CDR serialization and CRC helpers for Unitree HG message families | + +## Core Policy Contract + +`WbcPolicy` is implemented by every policy backend. Policies are `Send + Sync` +so they can be called from runtime loops. Implementations are responsible for +serializing access to any backend state that is not internally thread-safe. + +Important caller-facing types: + +- `Observation`: proprioception, gravity vector, command, and timestamp. +- `WbcCommand`: velocity, motion tokens, joint targets, kinematic pose, and + other command variants. +- `JointPositionTargets`: output joint targets plus timestamp. +- `PolicyCapabilities`: supported command kinds and declared limits. +- `RobotConfig`: joint names, default pose, gains, and limits. + +Policies should reject unsupported command variants explicitly. A wrapper that +cannot serve a public checkpoint should say so clearly instead of pretending to +be runnable. + +## Registry And Model Switching + +Policies register themselves at compile time through `inventory`. Runtime model +selection is config-driven: -`Send + Sync` is required because the control loop runs the policy from a -dedicated thread. `predict` takes a shared reference — concurrency protection -(typically a `Mutex` around the ONNX session) lives inside the implementation. - -## Observation and command types - -```rust -pub struct Observation { - pub joint_positions: Vec, // radians - pub joint_velocities: Vec, // rad/s - pub gravity_vector: [f32; 3], // in robot body frame - pub command: WbcCommand, - pub timestamp: Instant, -} - -pub enum WbcCommand { - Velocity(Twist), // vx, vy, yaw_rate - EndEffectorPoses(Vec), // hand/wrist SE3 targets - MotionTokens(Vec), // GEAR-SONIC style tokens - JointTargets(Vec), // direct joint targets - KinematicPose(BodyPose), // full-body kinematic pose -} +```toml +[policy] +name = "decoupled_wbc" ``` -Policies declare which `WbcCommand` variant they accept and return -`WbcError::UnsupportedCommand` for anything else. +The registry name is the public switch. Current registry names include: -## Policy registry +| Name | State | +|------|-------| +| `gear_sonic` | Live public G1 planner velocity path and narrower tracking paths | +| `decoupled_wbc` | Live public G1 balance and walk paths | +| `wbc_agile` | Live public G1 recurrent checkpoint path | +| `bfm_zero` | Live public G1 policy plus tracking context bundle | +| `hover` | Wrapper present, blocked on user-exported checkpoint | +| `wholebody_vla` | Experimental contract wrapper, no runnable public release | +| `py_model` | User-supplied Python or PyTorch policy | -Policies register themselves at compile time using the `inventory` crate: +All registered policy crates must be linked into the final binary. Dynamic +loading does not make `inventory` registrations appear. -```rust -inventory::submit! { - WbcRegistration::new::("my_policy") -} -``` +## Configuration Layers -The CLI then builds any registered policy by name: +Configs are split by purpose: -```rust -let policy = WbcRegistry::build("my_policy", &config_toml_value)?; -``` +- top-level policy configs under `configs/*.toml` +- robot configs under `configs/robots/*.toml` +- showcase configs under `configs/showcase/*.toml` +- demo and teleop configs under `configs/demo/` and `configs/teleop/` -`WbcRegistry::build` iterates `inventory::iter::()` at -runtime and calls the matching `RegistryPolicy::from_config` constructor. -The policy name is the same string used in `[policy] name = "..."` in the -TOML config. +The smoke config, `configs/decoupled_smoke.toml`, uses a checked-in dynamic +identity ONNX fixture and is the no-download local path. -> **Gotcha**: All registered types must be in crates linked into the final -> binary. `inventory` uses linker sections — dynamic loading is not supported. +## Inference Backends -## Config-driven instantiation +### ONNX Runtime -The CLI reads a TOML file, extracts `[policy.config]`, and passes it to the -registry. Switching models means changing `policy.name` in the TOML: +`robowbc-ort` wraps ONNX Runtime and supports CPU, CUDA, and TensorRT execution +providers when the host runtime matches. The checked-in public configs default +to CPU unless the user opts into accelerator-specific settings. -```toml -[policy] -name = "decoupled_wbc" # change this to switch policies +### PyO3 / Python -[policy.config.rl_model] -model_path = "models/decoupled-wbc/GR00T-WholeBodyControl-Walk.onnx" -execution_provider = { type = "cpu" } -``` +`robowbc-pyo3` loads user-supplied Python modules and calls them through PyO3. +`robowbc-py` is the user-facing Python SDK package and exposes Rust-backed +policies, observations, commands, targets, capabilities, and `MujocoSession`. -## Inference backends +## Runtime, Transport, And Teleop -### ONNX Runtime (`robowbc-ort`) +The runtime builds observations from transport state, applies the selected +command source, calls the policy, validates targets, and sends the result toward +the active transport. -Wraps the [`ort`](https://github.com/pykeio/ort) crate. Each policy holds a -`Mutex` to serialize session execution across threads. +Supported execution modes include: -Supported execution providers: -- `{ type = "cpu" }` — always available -- `{ type = "cuda", device_id = 0 }` — NVIDIA GPU -- `{ type = "tensor_rt", device_id = 0 }` — NVIDIA TensorRT (requires matching toolkit) +- hardware-oriented Unitree G1 communication paths +- CycloneDDS-oriented transport work +- MuJoCo-backed transport for local simulation +- synthetic transport for smoke runs and report generation +- keyboard teleop for the protected demo path -### PyO3 / PyTorch (`robowbc-pyo3`) +`make demo-keyboard` is the public interactive MuJoCo path and has additional +guardrails in `docs/agents/keyboard-demo.md`. -Loads a Python module containing a `predict(obs) -> targets` function. The -GIL is acquired per-call. Intended for development and non-exported models. +## Reports And Showcase -## Control loop +The CLI can write runtime reports, replay traces, and Rerun recordings. Python +scripts under `scripts/` normalize benchmark output, generate policy showcase +pages, validate static site bundles, and build the published GitHub Pages site. -``` -tick N: - 1. recv joint_state + imu (zenoh / MuJoCo / synthetic) - 2. build Observation - 3. policy.predict(&obs) ← your model runs here - 4. send JointPositionTargets - 5. sleep until next tick -``` +The public site and proof-pack artifacts are evidence surfaces. They should +remain tied to reproducible configs and generated machine-readable output. + +## Error Handling + +- Library crates use typed errors where callers need to distinguish failure + modes. +- CLI paths add user-facing context to failed config, model, transport, and + report operations. +- Missing model assets, unsupported commands, unsupported execution providers, + and platform limitations should fail explicitly. + +## Extension Points -`run_control_tick` in `robowbc-comm` handles steps 1, 2, 4, and 5. Step 3 -is the policy call. The loop runs at `control_frequency_hz` (typically 50 Hz). +To add a policy, implement the core policy contract, register the wrapper, add a +config, and provide a smoke or blocked-state proof. See `docs/adding-a-model.md`. -## Error handling +To add a robot, define the robot config, joint order, limits, gains, and any +transport or visualization mapping. See `docs/adding-a-robot.md`. -- Library crates (`robowbc-core`, `robowbc-ort`, …) use `thiserror`. -- The CLI (`robowbc-cli`) uses `anyhow` for rich error context. -- `WbcPolicy::predict` returns `Result`. - A control loop receiving `WbcError` should log it and continue rather than - crash — hardware safety layers (PD controllers) handle the gap. +To add a public report path, keep the JSON/report contract machine-readable and +update the site validation tests that consume it. diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md index 5c012e0..2fb1603 100644 --- a/docs/benchmarks/README.md +++ b/docs/benchmarks/README.md @@ -10,8 +10,8 @@ The source of truth lives under: - `artifacts/benchmarks/nvidia/SUMMARY.md` - HTML summary generated either locally at `artifacts/benchmarks/nvidia/index.html` or in CI / Pages at `benchmarks/nvidia/index.html` -- `scripts/bench_robowbc_compare.py` -- `scripts/bench_nvidia_official.py` +- `scripts/benchmarks/bench_robowbc_compare.py` +- `scripts/benchmarks/bench_nvidia_official.py` The GEAR-Sonic package now answers three distinct latency questions instead of hiding them behind the old ambiguous `replan_tick` split: @@ -92,8 +92,8 @@ Blocked GPU rows are expected and honest when any of the following is missing: ```bash git submodule update --init --recursive third_party/GR00T-WholeBodyControl -bash scripts/download_gear_sonic_models.sh -bash scripts/download_decoupled_wbc_models.sh +bash scripts/models/download_gear_sonic_models.sh +bash scripts/models/download_decoupled_wbc_models.sh ``` The comparison uses a tracked git submodule checkout of @@ -105,7 +105,7 @@ beside the downloaded assets so later artifact runs can record provenance. ```bash for provider in cpu cuda tensor_rt; do - python3 scripts/bench_robowbc_compare.py --all --provider "$provider" + python3 scripts/benchmarks/bench_robowbc_compare.py --all --provider "$provider" done ``` @@ -123,7 +123,7 @@ matches a CUDA or TensorRT request. ```bash for provider in cpu cuda tensor_rt; do - python3 scripts/bench_nvidia_official.py --all --provider "$provider" + python3 scripts/benchmarks/bench_nvidia_official.py --all --provider "$provider" done ``` @@ -138,13 +138,13 @@ a blocked artifact with the exact missing-EP or initialization error. ### 4. Render the summary ```bash -python3 scripts/render_nvidia_benchmark_summary.py --output artifacts/benchmarks/nvidia/SUMMARY.md +python3 scripts/benchmarks/render_nvidia_benchmark_summary.py --output artifacts/benchmarks/nvidia/SUMMARY.md ``` In CI, the showcase job also renders the static HTML page: ```bash -python3 scripts/render_nvidia_benchmark_summary.py \ +python3 scripts/benchmarks/render_nvidia_benchmark_summary.py \ --root /tmp/policy-showcase/benchmarks/nvidia \ --output /tmp/policy-showcase/benchmarks/nvidia/SUMMARY.md \ --html-output /tmp/policy-showcase/benchmarks/nvidia/index.html diff --git a/docs/community/groot-wbc-integration.md b/docs/community/groot-wbc-integration.md index 55bd0b1..4473698 100644 --- a/docs/community/groot-wbc-integration.md +++ b/docs/community/groot-wbc-integration.md @@ -118,7 +118,7 @@ It is not an NVIDIA product. ```bash git clone https://github.com/MiaoDX/robowbc cd robowbc -bash scripts/download_gear_sonic_models.sh +bash scripts/models/download_gear_sonic_models.sh # Downloads model_encoder.onnx, model_decoder.onnx, planner_sonic.onnx # into models/gear-sonic/ ``` @@ -177,9 +177,9 @@ Run the artifact-backed comparison suite: ```bash git submodule update --init --recursive third_party/GR00T-WholeBodyControl -python3 scripts/bench_robowbc_compare.py --all -python3 scripts/bench_nvidia_official.py --all -python3 scripts/render_nvidia_benchmark_summary.py --output artifacts/benchmarks/nvidia/SUMMARY.md +python3 scripts/benchmarks/bench_robowbc_compare.py --all +python3 scripts/benchmarks/bench_nvidia_official.py --all +python3 scripts/benchmarks/render_nvidia_benchmark_summary.py --output artifacts/benchmarks/nvidia/SUMMARY.md ``` See the benchmark registry in `artifacts/benchmarks/nvidia/cases.json` and the diff --git a/docs/configuration.md b/docs/configuration.md index 49f96e3..e94fc91 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -250,7 +250,7 @@ GR00T's G1 simulator behavior. enabled = true body_name = "pelvis" anchor = [0.0, 0.0, 1.0] -anchor_from_initial_pose = true +anchor_from_initial_pose = false kp_pos = 10000.0 kd_pos = 1000.0 kp_ang = 1000.0 @@ -258,9 +258,10 @@ kd_ang = 10.0 ``` Omit the table to run without the support band. Set -`anchor_from_initial_pose = true` for local teleop demos where the support band -should catch a falling humanoid without lifting it away from its initialized -standing height. +`anchor_from_initial_pose = false` with `anchor = [0.0, 0.0, 1.0]` to match the +official GR00T MuJoCo support-band setup. `anchor_from_initial_pose = true` is +available for custom local demos that need the band anchored at the model's +initialized body position instead. ### `[report]` diff --git a/docs/getting-started.md b/docs/getting-started.md index 4d8b0bf..33e358f 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -66,12 +66,17 @@ robowbc run --config configs/demo/gear_sonic_keyboard_mujoco.toml --teleop keybo ``` Keep the terminal focused for keyboard input and watch the MuJoCo window: -`]` engages the policy after the init pose settles, `WASD` changes linear -velocity, `QE` changes yaw, `Space` zeroes velocity, `9` toggles the support -band, `O` sends a zero-velocity emergency-stop tick, and `Esc` quits. The demo -config uses the GR00T scene wrapper and a neutral-height virtual support band -so the robot stays recoverable without being lifted off the ground. It also -uses `[sim].viewer = true`, so it requires a Linux desktop/OpenGL session. For +`]` queues policy engagement after the 3.0s init-pose settle window, `WASD` +changes linear velocity, `QE` changes yaw, `Space` zeroes velocity, and +`O` sends a zero-velocity emergency-stop tick. After engagement, press `9` +promptly in either the terminal or MuJoCo viewer if the support band is holding +the robot high so it drops to foot contact; pressing `9` again toggles the band +back on. `Esc`/`Ctrl-C` quits. WASD/QE velocity commands are held until +engagement; if the command is still zero after engagement, the demo keeps +holding the default standing pose until the first nonzero velocity command. The +demo config uses the GR00T scene wrapper and the official MuJoCo support-band +point/gains. It also uses +`[sim].viewer = true`, so it requires a Linux desktop/OpenGL session. For headless machines, use `make site` or `make showcase-verify` instead. ## Generate a local policy showcase @@ -129,7 +134,7 @@ artifacts and on the `main`-branch GitHub Pages site. ### Step 1 — download the models ```bash -bash scripts/download_gear_sonic_models.sh +bash scripts/models/download_gear_sonic_models.sh # Downloads model_encoder.onnx, model_decoder.onnx, planner_sonic.onnx # into models/gear-sonic/ and reuses cached files when already present. ``` @@ -156,7 +161,7 @@ a machine with matching ONNX Runtime and NVIDIA runtime dependencies. ### Step 1 — download and normalize the assets ```bash -bash scripts/download_bfm_zero_models.sh +bash scripts/models/download_bfm_zero_models.sh # Fetches the public ONNX + tracking context bundle into models/bfm_zero/ # and converts zs_walking.pkl into zs_walking.npy for the Rust runtime. ``` diff --git a/docs/human/README.md b/docs/human/README.md new file mode 100644 index 0000000..db9f9ea --- /dev/null +++ b/docs/human/README.md @@ -0,0 +1,14 @@ +# Human Documentation + +This folder is the extension point for curated human-facing docs. The root files +remain the primary entry points: + +- `README.md` - what RoboWBC is and how to try it. +- `STATUS.md` - what works now, what is blocked, and what is next. +- `ARCHITECTURE.md` - high-level system map. +- `ROADMAP.md` - active work ordering. +- `docs/README.md` - full documentation index. + +Generated plans, historical evidence, and agent working notes belong in +`.planning/`, `docs/agents/`, or other process-specific folders rather than in +this human surface. diff --git a/docs/plans/refactor-book.md b/docs/plans/refactor-book.md new file mode 100644 index 0000000..e5beb47 --- /dev/null +++ b/docs/plans/refactor-book.md @@ -0,0 +1,55 @@ +--- +refactor_scope: book +status: DONE +accepted_severities: + - P1 +last_verified: 2026-05-14 +--- + +# Refactor Scope: Book + +## Status + +DONE + +## Target + +Make the mdBook navigation match the current human documentation surface without +promoting agent runbooks, planning notes, or generated evidence into the book. + +## Accepted Severities + +P1 only. The active issue is a source-of-truth/navigation gap: useful human +pages existed under `docs/` but were omitted or grouped around file history +rather than reader tasks. + +## Accepted P0/P1 Checklist + +- [x] Group `docs/SUMMARY.md` by reader task: start, architecture/runtime, + extension, operations/evidence, direction, research, and community. +- [x] Include the human documentation map in the book. +- [x] Include existing durable reference pages that were omitted from the book. +- [x] Keep agent-only docs and process state out of mdBook navigation. + +## Parked P2 / Future Ideas + +- Decide later whether root `STATUS.md` should have a book-local companion page + or remain a root orientation surface only. +- Decide later whether community drafts should move behind a separate community + index once those pages grow. + +## Evidence Ladder + +- L0: Build the mdBook with `make docs-book`. +- L0: Inspect the changed book navigation and confirm it only references files + under `docs/`. + +## Stop Condition + +Stop when `docs/SUMMARY.md` is organized by reader task, the omitted durable +human pages are listed, and `make docs-book` passes. + +## Execution Log + +- 2026-05-14: Refactored `docs/SUMMARY.md` into task-oriented sections and + recorded this gate. Verified with `make docs-book`. diff --git a/docs/plans/refactor-keyboard-speed-monitor.md b/docs/plans/refactor-keyboard-speed-monitor.md new file mode 100644 index 0000000..d4a84d4 --- /dev/null +++ b/docs/plans/refactor-keyboard-speed-monitor.md @@ -0,0 +1,100 @@ +--- +refactor_scope: keyboard-speed-monitor +status: DONE +accepted_severities: + - P0 + - P1 + - P2 +last_verified: 2026-05-15 +--- + +# Refactor Scope: Keyboard Speed Monitor + +## Status + +DONE + +## Target + +Keyboard-demo command-to-MuJoCo speed observability in `robowbc-cli`, with the +small MuJoCo telemetry hook in `robowbc-sim` needed to expose support-band state. + +## Accepted Severities + +P0/P1/P2 inside this target. + +## Accepted Cleanup Checklist + +- [x] P1: Velocity tracking must be computed for normal `make demo-keyboard` + style runs without requiring `[report]` frame capture. +- [x] P1: Metrics must distinguish simulated-time tracking from wall-time + playback speed so slow visual movement can be separated from policy + under-tracking or control-loop overruns. +- [x] P1: The live monitor and report/replay frames must show the inferred + GEAR-Sonic planner mode for velocity-command runs. +- [x] P1: The monitor must surface elastic support band state because the + protected demo starts with a fixed support band that can resist nonzero + translational commands until toggled off. +- [x] P1: A 0.6 m/s GEAR-Sonic command must not be flattened to the fixed + 0.3 m/s slow-walk planner target. +- [x] P2: Add focused tests for the no-report monitor path and timebase math. +- [x] P2: Preserve the protected demo guardrails: GR00T scene wrapper, support + band, engage guard, init-pose settle window, and viewer lighting. + +## Parked Cross-Seam / Future Ideas + +- Policy tuning or changing the official commanded speed ranges. +- Changing the protected demo support-band default. +- Broader MuJoCo/viewer threading architecture changes. +- Hardware-facing Unitree timing metrics. + +## Evidence Ladder + +- L0: `make toolchain`, `make build`, `make check`, `make fmt-check`. +- L1: focused Rust unit tests for CLI velocity monitor behavior. +- L4: `docs/agents/keyboard-demo.md` targeted MuJoCo startup guardrail when the + local MuJoCo runtime and dynamic linker paths are available. + +## Stop Condition + +Stop when the accepted checklist is complete, the focused tests pass, standard +Rust gates that can run in this environment have results recorded here, and any +MuJoCo/OpenGL/model-download gate that cannot run has an exact blocker recorded. + +## Execution Log + +- 2026-05-15: Scope opened from manual report that keyboard-driven G1 movement + appears slow even at 0.6 m/s. Initial code reading found existing velocity + tracking depends on replay frames, but replay frames are only collected when + `[report]` is configured, so the protected interactive demo has no normal + commanded-vs-achieved speed signal. +- 2026-05-15: Added streaming velocity monitor output in `robowbc-cli` with + simulated-time velocity, wall-time velocity, real-time factor, RMSE, and + MuJoCo support-band state. Added focused tests for recorded simulator time + and no-report monitor accumulation. +- 2026-05-15: Reproduced protected-scene slow behavior. With support band + enabled, a 0.6 m/s command stayed near zero actual forward speed while + real-time factor stayed near 1.0x. With support band disabled before the + planner fix, the same 0.6 m/s command averaged about 0.17 m/s over 500 ticks. +- 2026-05-15: Root cause found in GEAR-Sonic command mapping: nonzero commands + below 0.8 m/s entered slow-walk mode, but RoboWBC replaced the requested + speed with a fixed direction-bin slow-walk target such as 0.3 m/s for forward + motion. Aligned with NVIDIA's planner contract by keeping 0.6 m/s in + slow-walk mode while passing `target_vel = 0.6`, and by mapping 0.8..1.5 m/s + to walk mode and 1.5..3.0 m/s to run mode. This supersedes the earlier local + experiment that lowered the slow-walk cutoff. +- 2026-05-15: Current evidence passed: `make toolchain`, `make build`, + `make check`, `cargo test -p robowbc-ort gear_sonic_planner_command`, + `make test`, `make clippy`, and `make fmt-check`. +- 2026-05-15: Added GEAR-Sonic planner status to live velocity monitor output + and report/replay frames. The status is mode-id/name plus planner target + velocity for velocity-command runs; velocity tracking remains a separate + measured behavior signal. +- 2026-05-15: Local MuJoCo speed samples on the protected GEAR-Sonic scene: + with support band disabled, a 0.6 m/s command over 500 ticks produced + `actual_vx_mean=0.511 m/s`, `actual_vx_wall_mean=0.508 m/s`, and + `real_time_factor=0.99x`; with support band enabled, a 0.6 m/s command over + 200 ticks produced `actual_vx_mean=0.002 m/s`, + `actual_vx_wall_mean=0.002 m/s`, and `real_time_factor=0.98x`. +- 2026-05-15: Earlier protected MuJoCo startup guardrail passed: + `MUJOCO_DOWNLOAD_DIR="$(pwd)/.cache/mujoco" MUJOCO_DYNAMIC_LINK_DIR="$(pwd)/.cache/mujoco/mujoco-3.6.0/lib" LD_LIBRARY_PATH="$(pwd)/.cache/mujoco/mujoco-3.6.0/lib:${LD_LIBRARY_PATH:-}" cargo test -p robowbc-sim --features mujoco-auto-download gear_sonic_demo_model_holds_default_pose_for_startup_window`. diff --git a/docs/plans/refactor-readme.md b/docs/plans/refactor-readme.md new file mode 100644 index 0000000..6c483ef --- /dev/null +++ b/docs/plans/refactor-readme.md @@ -0,0 +1,63 @@ +--- +refactor_scope: readme +status: DONE +accepted_severities: + - P1 +last_verified: 2026-05-14 +--- + +# Refactor Scope: README + +## Status + +DONE + +## Target + +Root `README.md` first-run overview and navigation surface. + +## Accepted Severities + +P1 only. The root README is a primary source-of-truth and onboarding surface, so +this pass treats excessive length and buried first-run paths as a documentation +usability gap. Lower-priority documentation architecture ideas are parked. + +## Accepted P0/P1 Checklist + +- Reduce `README.md` to 150 lines or fewer. +- Preserve the project purpose: Linux-first humanoid WBC policy inference with + Python SDK and Rust runtime surfaces. +- Preserve the protected `make demo-keyboard` clone-and-see-it-work path. +- Preserve quick local validation commands and point detailed validation to docs. +- Preserve current live/blocked policy status. +- Preserve public report links, architecture pointers, documentation index, and + license/third-party notice pointers. + +## Parked P2 / Future Ideas + +- Rework the full documentation information architecture. +- Split each workflow into shorter task-specific pages. +- Add a generated README table from package metadata or policy manifests. + +## Evidence Ladder + +Minimum required confidence: L0 static. + +- `wc -l README.md` must report 150 lines or fewer. +- A manual content scan must confirm the accepted checklist remains visible. + +## Stop Condition + +Stop when `README.md` is 150 lines or fewer, accepted P0/P1 checklist items are +present, and this gate is updated to `DONE`. Park P2 ideas instead of expanding +this cleanup. + +## Execution Log + +- 2026-05-14: Opened scope gate for README line-count cleanup. +- 2026-05-14: Condensed `README.md` from 314 lines to 150 lines while + preserving project purpose, first-run commands, protected demo path, policy + status, public reports, Python/Rust entry points, docs links, and license + pointers. +- 2026-05-14: Verified L0 evidence with `wc -l README.md` and required-anchor + scan using `rg`. diff --git a/docs/plans/refactor-scripts.md b/docs/plans/refactor-scripts.md new file mode 100644 index 0000000..750137b --- /dev/null +++ b/docs/plans/refactor-scripts.md @@ -0,0 +1,61 @@ +--- +refactor_scope: scripts +status: DONE +accepted_severities: + - P1 + - P2 +last_verified: 2026-05-14 +--- + +# Refactor Scope: Scripts + +## Status + +DONE + +## Target + +Make the post-move script layout easier to maintain and discover while keeping +all public root-level script paths working. + +## Accepted Severities + +- P1: Preserve documented script paths and Makefile targets after the workflow + folder split. +- P2: Reduce wrapper duplication and make the canonical script folders easier + for humans and agents to navigate. + +## Accepted P0/P1 Checklist + +- [x] Keep existing root-level script entrypoints as compatibility wrappers. +- [x] Document canonical workflow folders and wrapper policy. +- [x] Add the scripts map to the mdBook navigation. + +## Parked P2 / Future Ideas + +- Add per-folder script READMEs if individual workflow folders grow beyond the + current single-screen map. +- Consider a dedicated script contract test if root compatibility wrappers gain + argument rewriting or environment setup logic later. + +## Evidence Ladder + +- L0: Compile Python scripts with `python3 -m py_compile`. +- L0: Syntax-check shell scripts with `bash -n`. +- L0: Build mdBook with `make docs-book`. +- L1/L2: Run `make python-test` because integration tests exercise script + contracts and report outputs. + +## Stop Condition + +Stop when compatibility wrappers still dispatch to canonical workflow folders, +the script layout is documented in both the repo script index and mdBook, and +the focused script/docs verification commands pass. + +## Execution Log + +- 2026-05-14: Added shared Python compatibility dispatch, updated root Python + wrappers, documented script workflow folders, and added the mdBook script map. + Verified with `python3 -m py_compile scripts/*.py scripts/*/*.py`, + `bash -n scripts/*.sh scripts/*/*.sh`, `make docs-book`, + `make python-test`, and representative root wrapper `--help` smoke checks. diff --git a/docs/roboharness-integration.md b/docs/roboharness-integration.md index 50fbdc6..eabee19 100644 --- a/docs/roboharness-integration.md +++ b/docs/roboharness-integration.md @@ -40,7 +40,7 @@ motion. ## Running a proof pack ```bash -python3 scripts/roboharness_report.py \ +python3 scripts/reports/roboharness_report.py \ --robowbc-binary target/release/robowbc \ --config configs/sonic_g1.toml \ --output-dir artifacts/roboharness-reports/sonic_g1 \ @@ -128,10 +128,10 @@ The authority chain is intentionally one-way: semantic phases once in the authored config 2. the Rust CLI emits that data as authoritative `phase_timeline` metadata in the run artifacts -3. `scripts/roboharness_report.py` copies the contract into +3. `scripts/reports/roboharness_report.py` copies the contract into `proof_pack_manifest.json` -4. `scripts/generate_policy_showcase.py` and - `scripts/validate_site_bundle.py` consume only the manifest contract +4. `scripts/site/generate_policy_showcase.py` and + `scripts/site/validate_site_bundle.py` consume only the manifest contract When `phase_review.enabled = true`, the manifest adds these fields: @@ -231,7 +231,7 @@ outside `0..5`. ## Replay behavior -`scripts/roboharness_report.py` now prefers the canonical replay trace when it +`scripts/reports/roboharness_report.py` now prefers the canonical replay trace when it exists and only falls back to `run_report.json.frames` for backward compatibility with older CLI artifacts. @@ -248,7 +248,7 @@ For MuJoCo replay: The showcase and benchmark pages remain overview surfaces. Proof packs are the drill-down layer. -`scripts/generate_policy_showcase.py` can render optional proof-pack links when +`scripts/site/generate_policy_showcase.py` can render optional proof-pack links when a proof-pack artifact is co-located beside the showcase output. The simplest convention is to copy the proof-pack directory next to the showcase as: diff --git a/docs/scripts.md b/docs/scripts.md new file mode 100644 index 0000000..6f661a1 --- /dev/null +++ b/docs/scripts.md @@ -0,0 +1,52 @@ +# Maintainer Scripts + +Prefer Makefile targets for stable workflows. Use direct script calls when you +are changing a script, debugging its arguments, or reproducing a lower-level CI +step. + +## Layout + +Canonical script implementations live in workflow folders under `scripts/`. +Root-level scripts remain compatibility entrypoints for older commands and +public references. + +| Folder | Purpose | Stable Make targets | +|--------|---------|---------------------| +| `scripts/models/` | Download and normalize public policy assets | `make models-public` | +| `scripts/mujoco/` | Ensure and validate the MuJoCo runtime | `make sim-feature-test`, `make site-render-check`, `make demo-keyboard` | +| `scripts/benchmarks/` | Generate and render NVIDIA comparison artifacts | `make benchmark-nvidia` | +| `scripts/site/` | Build, validate, serve, and smoke-test the static policy site | `make site`, `make site-smoke`, `make site-serve` | +| `scripts/reports/` | Produce RoboWBC proof-pack report contracts | Direct script calls while debugging reports | +| `scripts/sdk/` | Smoke-test the installed Python SDK | `make python-sdk-verify` | + +## Compatibility Policy + +- New implementation logic belongs in the workflow folders. +- Existing root-level `scripts/*.py` and `scripts/*.sh` paths stay as + compatibility entrypoints. +- Documentation and Makefile recipes should prefer the workflow folders unless + they are intentionally showing a legacy-compatible command. +- Downloaded model files, benchmark output, site bundles, MuJoCo runtimes, and + Python caches should stay untracked. + +## Common Direct Calls + +```bash +python3 scripts/site/build_site.py --repo-root . --output-dir /tmp/robowbc-site +python3 scripts/site/validate_site_bundle.py --root /tmp/robowbc-site +python3 scripts/benchmarks/bench_robowbc_compare.py --all +python3 scripts/benchmarks/render_nvidia_benchmark_summary.py \ + --root artifacts/benchmarks/nvidia \ + --output artifacts/benchmarks/nvidia/SUMMARY.md +bash scripts/models/download_gear_sonic_models.sh models/gear-sonic +``` + +## Edit Checks + +Run the focused script checks before committing script changes: + +```bash +python3 -m py_compile scripts/*.py scripts/*/*.py +bash -n scripts/*.sh scripts/*/*.sh +make python-test +``` diff --git a/docs/third-party-notices.md b/docs/third-party-notices.md index abb6e4c..f6eed45 100644 --- a/docs/third-party-notices.md +++ b/docs/third-party-notices.md @@ -54,7 +54,18 @@ licensed code is fine — the obligation only attaches to changed files. source-disclosure obligation is satisfied by linking back to upstream. -### 3. Model licenses +### 3. Bundled UI font licenses + +SIL Open Font License 1.1 and Ubuntu Font Licence 1.0 apply to default +font assets pulled in by egui through the MuJoCo viewer stack. These +licenses allow bundling and redistribution with robowbc binaries as long +as the license text and notices are preserved. + +* **epaint_default_fonts** (`OFL-1.1`, `Ubuntu-font-1.0`, plus + `MIT OR Apache-2.0`) — default egui fonts used by the viewer path + behind `mujoco-rs`. + +### 4. Model licenses These govern *trained model weights*, not source code. They are distinct from any of robowbc's software licenses and have their own @@ -63,7 +74,7 @@ restrictions. * **NVIDIA Open Model License** — covers GEAR-SONIC weights (`model_encoder.onnx`, `model_decoder.onnx`, `planner_sonic.onnx`). robowbc **never bundles these weights**. Users fetch them on first - run via `scripts/download_gear_sonic_models.sh` from HuggingFace and + run via `scripts/models/download_gear_sonic_models.sh` from HuggingFace and accept the license at fetch time. If you build a derived distribution that bundles the weights, you take on the redistribution obligations of the NVIDIA Open Model License — see diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 0000000..6ebb088 --- /dev/null +++ b/examples/README.md @@ -0,0 +1,21 @@ +# Examples + +Examples are runnable entry points. Keep existing paths stable because docs and +users may call them directly. + +## Shell Examples + +- `shell/list_and_switch_policies.sh` lists registry policies and shows config-driven + switching. +- `shell/run_gear_sonic.sh` runs the GEAR-Sonic policy path. +- `shell/run_decoupled_wbc.sh` runs the Decoupled WBC policy path. + +## Python Examples + +- `python/gear_sonic_inference.py` demonstrates Python SDK policy inference. +- `python/mujoco_kinematic_pose_session.py` demonstrates `MujocoSession` with a + kinematic-pose command. +- `python/roboharness_backend.py` demonstrates the visual testing adapter path. + +Additional SDK adapter examples live in `crates/robowbc-py/examples/` because +they are packaged with the Python project. diff --git a/examples/list_and_switch_policies.sh b/examples/list_and_switch_policies.sh old mode 100644 new mode 100755 index ecc4432..9074441 --- a/examples/list_and_switch_policies.sh +++ b/examples/list_and_switch_policies.sh @@ -1,41 +1,4 @@ #!/usr/bin/env bash -# Example 3: List registered policies and switch between them via config -# -# This example demonstrates robowbc's core value proposition: switching WBC -# models by changing a single TOML field, without recompiling. -# -# Usage: bash examples/list_and_switch_policies.sh - set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" -cd "$REPO_ROOT" - -echo "=== RoboWBC policy switching demo ===" -echo "" - -# Validate both configs to show they differ only in policy.name. -echo "--- Config 1: GEAR-SONIC (sonic_g1.toml) ---" -cargo run --bin robowbc -- validate --config configs/sonic_g1.toml -echo "Policy: gear_sonic ✓" -echo "" - -echo "--- Config 2: Decoupled WBC (decoupled_g1.toml) ---" -cargo run --bin robowbc -- validate --config configs/decoupled_g1.toml -echo "Policy: decoupled_wbc ✓" -echo "" - -# Show the diff between the two configs. -echo "--- Diff between the two configs ---" -diff configs/sonic_g1.toml configs/decoupled_g1.toml || true -echo "" - -echo "Both policies share the same runtime interface: Observation in, JointPositionTargets out." -echo "" -echo "Running Decoupled WBC (1 tick, instant exit)..." -cargo run --bin robowbc -- run --config configs/decoupled_g1.toml -echo "" - -echo "To add a new policy, see: docs/adding-a-model.md" -echo "To run GEAR-SONIC: bash examples/run_gear_sonic.sh" +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +exec bash "${SCRIPT_DIR}/shell/list_and_switch_policies.sh" "$@" diff --git a/examples/run_decoupled_wbc.sh b/examples/run_decoupled_wbc.sh old mode 100644 new mode 100755 index 053295a..5e3bbcb --- a/examples/run_decoupled_wbc.sh +++ b/examples/run_decoupled_wbc.sh @@ -1,35 +1,4 @@ #!/usr/bin/env bash -# Example 1: Run Decoupled WBC policy (no downloads required) -# -# Decoupled WBC combines an RL locomotion model (lower body) with an -# analytical IK baseline (upper body). This example uses a small test -# fixture bundled in the repository, so it runs without any external models. -# -# Usage: bash examples/run_decoupled_wbc.sh [--release] - set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" -cd "$REPO_ROOT" - -BUILD_MODE="${1:-}" -CARGO_FLAGS="" -if [[ "$BUILD_MODE" == "--release" ]]; then - CARGO_FLAGS="--release" -fi - -echo "Building robowbc..." -cargo build $CARGO_FLAGS --bin robowbc 2>&1 - -echo "" -echo "Running Decoupled WBC control loop (velocity command [0.2, 0.0, 0.1])..." -echo "Press Ctrl-C to stop early (the config sets max_ticks = 1 for quick demo)." -echo "" - -cargo run $CARGO_FLAGS --bin robowbc -- run --config configs/decoupled_g1.toml - -echo "" -echo "Done. Joint targets were printed above." -echo "" -echo "To run continuously, remove the 'max_ticks' limit from configs/decoupled_g1.toml." +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +exec bash "${SCRIPT_DIR}/shell/run_decoupled_wbc.sh" "$@" diff --git a/examples/run_gear_sonic.sh b/examples/run_gear_sonic.sh old mode 100644 new mode 100755 index 8d10d23..dfa0201 --- a/examples/run_gear_sonic.sh +++ b/examples/run_gear_sonic.sh @@ -1,51 +1,4 @@ #!/usr/bin/env bash -# Example 2: Run GEAR-SONIC policy with real NVIDIA checkpoints -# -# GEAR-SONIC is NVIDIA's universal whole-body controller for Unitree G1. -# It uses three ONNX models (encoder, planner, decoder) in a pipeline. -# -# Prerequisites: -# - ONNX Runtime installed (see README for installation) -# - ~500 MB free disk space for model checkpoints -# -# Usage: bash examples/run_gear_sonic.sh [--release] - set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" -cd "$REPO_ROOT" - -BUILD_MODE="${1:-}" -CARGO_FLAGS="" -if [[ "$BUILD_MODE" == "--release" ]]; then - CARGO_FLAGS="--release" -fi - -# Step 1: Download GEAR-SONIC checkpoints if not already present. -MODELS_DIR="models/gear-sonic" -if [[ ! -f "$MODELS_DIR/model_encoder.onnx" ]]; then - echo "GEAR-SONIC models not found. Downloading from HuggingFace..." - bash scripts/download_gear_sonic_models.sh -else - echo "GEAR-SONIC models already present in $MODELS_DIR/." -fi - -# Step 2: Validate the config before loading models. -echo "" -echo "Validating config..." -cargo run $CARGO_FLAGS --bin robowbc -- validate --config configs/sonic_g1.toml - -# Step 3: Run inference. -echo "" -echo "Running GEAR-SONIC control loop (motion tokens [0.05, -0.1, 0.2, 0.0])..." -echo "Press Ctrl-C to stop early." -echo "" - -cargo run $CARGO_FLAGS --bin robowbc -- run --config configs/sonic_g1.toml - -echo "" -echo "Done. 50 Hz joint target inference complete." -echo "" -echo "To benchmark inference latency, run:" -echo " cargo bench -p robowbc-ort" +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +exec bash "${SCRIPT_DIR}/shell/run_gear_sonic.sh" "$@" diff --git a/examples/shell/list_and_switch_policies.sh b/examples/shell/list_and_switch_policies.sh new file mode 100755 index 0000000..8342a5d --- /dev/null +++ b/examples/shell/list_and_switch_policies.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# Example 3: List registered policies and switch between them via config +# +# This example demonstrates robowbc's core value proposition: switching WBC +# models by changing a single TOML field, without recompiling. +# +# Usage: bash examples/shell/list_and_switch_policies.sh + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +cd "$REPO_ROOT" + +echo "=== RoboWBC policy switching demo ===" +echo "" + +# Validate both configs to show they differ only in policy.name. +echo "--- Config 1: GEAR-SONIC (sonic_g1.toml) ---" +cargo run --bin robowbc -- validate --config configs/sonic_g1.toml +echo "Policy: gear_sonic ✓" +echo "" + +echo "--- Config 2: Decoupled WBC (decoupled_g1.toml) ---" +cargo run --bin robowbc -- validate --config configs/decoupled_g1.toml +echo "Policy: decoupled_wbc ✓" +echo "" + +# Show the diff between the two configs. +echo "--- Diff between the two configs ---" +diff configs/sonic_g1.toml configs/decoupled_g1.toml || true +echo "" + +echo "Both policies share the same runtime interface: Observation in, JointPositionTargets out." +echo "" +echo "Running Decoupled WBC (1 tick, instant exit)..." +cargo run --bin robowbc -- run --config configs/decoupled_g1.toml +echo "" + +echo "To add a new policy, see: docs/adding-a-model.md" +echo "To run GEAR-SONIC: bash examples/shell/run_gear_sonic.sh" diff --git a/examples/shell/run_decoupled_wbc.sh b/examples/shell/run_decoupled_wbc.sh new file mode 100755 index 0000000..b72ab80 --- /dev/null +++ b/examples/shell/run_decoupled_wbc.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# Example 1: Run Decoupled WBC policy (no downloads required) +# +# Decoupled WBC combines an RL locomotion model (lower body) with an +# analytical IK baseline (upper body). This example uses a small test +# fixture bundled in the repository, so it runs without any external models. +# +# Usage: bash examples/shell/run_decoupled_wbc.sh [--release] + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +cd "$REPO_ROOT" + +BUILD_MODE="${1:-}" +CARGO_FLAGS="" +if [[ "$BUILD_MODE" == "--release" ]]; then + CARGO_FLAGS="--release" +fi + +echo "Building robowbc..." +cargo build $CARGO_FLAGS --bin robowbc 2>&1 + +echo "" +echo "Running Decoupled WBC control loop (velocity command [0.2, 0.0, 0.1])..." +echo "Press Ctrl-C to stop early (the config sets max_ticks = 1 for quick demo)." +echo "" + +cargo run $CARGO_FLAGS --bin robowbc -- run --config configs/decoupled_g1.toml + +echo "" +echo "Done. Joint targets were printed above." +echo "" +echo "To run continuously, remove the 'max_ticks' limit from configs/decoupled_g1.toml." diff --git a/examples/shell/run_gear_sonic.sh b/examples/shell/run_gear_sonic.sh new file mode 100755 index 0000000..c429f6d --- /dev/null +++ b/examples/shell/run_gear_sonic.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# Example 2: Run GEAR-SONIC policy with real NVIDIA checkpoints +# +# GEAR-SONIC is NVIDIA's universal whole-body controller for Unitree G1. +# It uses three ONNX models (encoder, planner, decoder) in a pipeline. +# +# Prerequisites: +# - ONNX Runtime installed (see README for installation) +# - ~500 MB free disk space for model checkpoints +# +# Usage: bash examples/shell/run_gear_sonic.sh [--release] + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +cd "$REPO_ROOT" + +BUILD_MODE="${1:-}" +CARGO_FLAGS="" +if [[ "$BUILD_MODE" == "--release" ]]; then + CARGO_FLAGS="--release" +fi + +# Step 1: Download GEAR-SONIC checkpoints if not already present. +MODELS_DIR="models/gear-sonic" +if [[ ! -f "$MODELS_DIR/model_encoder.onnx" ]]; then + echo "GEAR-SONIC models not found. Downloading from HuggingFace..." + bash scripts/models/download_gear_sonic_models.sh +else + echo "GEAR-SONIC models already present in $MODELS_DIR/." +fi + +# Step 2: Validate the config before loading models. +echo "" +echo "Validating config..." +cargo run $CARGO_FLAGS --bin robowbc -- validate --config configs/sonic_g1.toml + +# Step 3: Run inference. +echo "" +echo "Running GEAR-SONIC control loop (motion tokens [0.05, -0.1, 0.2, 0.0])..." +echo "Press Ctrl-C to stop early." +echo "" + +cargo run $CARGO_FLAGS --bin robowbc -- run --config configs/sonic_g1.toml + +echo "" +echo "Done. 50 Hz joint target inference complete." +echo "" +echo "To benchmark inference latency, run:" +echo " cargo bench -p robowbc-ort" diff --git a/pyproject.toml b/pyproject.toml index 7c53ac8..1525b18 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -25,5 +25,14 @@ manifest-path = "crates/robowbc-py/Cargo.toml" module-name = "robowbc" [tool.pytest.ini_options] +addopts = "--strict-markers" testpaths = ["tests"] norecursedirs = ["third_party", "target", ".cache"] +markers = [ + "unit: fast isolated behavior tests", + "contract: public script, schema, generated artifact, and report compatibility tests", + "integration: subprocess, repository, simulator, external CLI, or process-boundary tests", + "regression: known-bug or artifact-shape regression tests", + "local: requires local GPU, simulator, gateway, credentials, display, or other host-specific resource", + "slow: CI-safe but expensive tests excluded from tight loops", +] diff --git a/scripts/README.md b/scripts/README.md new file mode 100644 index 0000000..0d0d991 --- /dev/null +++ b/scripts/README.md @@ -0,0 +1,82 @@ +# Scripts + +Prefer Makefile targets for stable workflows. Call scripts directly when you +are changing or debugging that script. + +Canonical script implementations live in workflow folders. Root-level +`scripts/*.py`, `scripts/*.sh`, and `scripts/bench_nvidia_gear_sonic_official.cpp` +remain compatibility entrypoints for older commands, docs, and local user +habits. + +## Workflow Folders + +| Folder | Purpose | Prefer | +|--------|---------|--------| +| `models/` | Download and normalize public policy assets | `make models-public` or the model-specific scripts | +| `mujoco/` | Ensure and validate the local MuJoCo runtime | `make sim-feature-test`, `make site-render-check`, or `make demo-keyboard` | +| `benchmarks/` | Generate and render NVIDIA comparison artifacts | `make benchmark-nvidia` | +| `site/` | Build, validate, serve, and browser-smoke the static policy site | `make site`, `make site-smoke`, `make site-serve` | +| `reports/` | Convert RoboWBC outputs into proof-pack report contracts | Direct script use while debugging reports | +| `sdk/` | Smoke-test the installed Python SDK | `make python-sdk-verify` | + +## Compatibility Entrypoints + +Root-level Python wrappers use `_compat.py` to dispatch to the canonical +workflow folders. Keep these wrappers small and keep new implementation logic +inside the workflow folders above. + +Shell download wrappers remain at the root for public command compatibility and +delegate to `scripts/models/`. + +## Model Assets + +- `models/download_gear_sonic_models.sh` +- `models/download_gear_sonic_reference_motions.sh` +- `models/download_decoupled_wbc_models.sh` +- `models/download_wbc_agile_models.sh` +- `models/download_bfm_zero_models.sh` +- `models/prepare_bfm_zero_assets.py` + +Downloaded models belong under local model/cache folders and should not be +committed unless the repo already tracks that exact fixture type. + +## MuJoCo And Showcase + +- `mujoco/ensure_mujoco_runtime.py` +- `mujoco/check_mujoco_headless.py` +- `site/build_site.py` +- `site/generate_policy_showcase.py` +- `site/validate_site_bundle.py` +- `site/serve_showcase.py` +- `site/site_browser_smoke.py` + +Use `make site-smoke` for bundle validation and `make showcase-verify` for the +CI-like path that downloads public checkpoints and renders proof packs. + +## Benchmarks + +- `benchmarks/bench_robowbc_compare.py` +- `benchmarks/bench_nvidia_official.py` +- `benchmarks/bench_nvidia_decoupled_official.py` +- `benchmarks/bench_nvidia_gear_sonic_official.cpp` +- `benchmarks/normalize_nvidia_benchmarks.py` +- `benchmarks/render_nvidia_benchmark_summary.py` + +Use `make benchmark-nvidia` for the full comparison package. + +## Reports And SDK + +- `reports/roboharness_report.py` +- `sdk/python_sdk_smoke.py` + +Use `make python-sdk-verify` for the local SDK build/install/smoke path. + +## Script Edit Checks + +Use these focused checks after script changes: + +```bash +python3 -m py_compile scripts/*.py scripts/*/*.py +bash -n scripts/*.sh scripts/*/*.sh +make python-test +``` diff --git a/scripts/_compat.py b/scripts/_compat.py new file mode 100644 index 0000000..9b92095 --- /dev/null +++ b/scripts/_compat.py @@ -0,0 +1,12 @@ +#!/usr/bin/env python3 +"""Compatibility helpers for legacy root-level script entrypoints.""" + +from __future__ import annotations + +import runpy +from pathlib import Path + + +def run_legacy_script(relative_path: str) -> None: + scripts_dir = Path(__file__).resolve().parent + runpy.run_path(str(scripts_dir / relative_path), run_name="__main__") diff --git a/scripts/bench_nvidia_decoupled_official.py b/scripts/bench_nvidia_decoupled_official.py old mode 100644 new mode 100755 index 2f62aa7..fbae318 --- a/scripts/bench_nvidia_decoupled_official.py +++ b/scripts/bench_nvidia_decoupled_official.py @@ -1,276 +1,4 @@ #!/usr/bin/env python3 -"""Headless benchmark harness for the upstream Decoupled WBC policy.""" +from _compat import run_legacy_script -from __future__ import annotations - -import argparse -import collections -import contextlib -import json -import sys -import time -from pathlib import Path -from types import ModuleType -from typing import Any - -import numpy as np -import tomllib - - -def install_torch_shim_if_needed() -> str: - try: - import torch # noqa: F401 - except ModuleNotFoundError: - pass - else: - return "torch" - - class Tensor(np.ndarray): - __array_priority__ = 1000 - - def __new__(cls, value: Any, dtype: Any | None = None) -> "Tensor": - return np.asarray(value, dtype=dtype).view(cls) - - def __array_finalize__(self, obj: Any) -> None: # pragma: no cover - ndarray protocol - del obj - - def unsqueeze(self, dim: int) -> "Tensor": - return np.expand_dims(self, axis=dim).view(Tensor) - - def detach(self) -> "Tensor": - return self - - def cpu(self) -> "Tensor": - return self - - def numpy(self) -> np.ndarray: - return np.asarray(self) - - class NoGrad(contextlib.AbstractContextManager[None]): - def __exit__(self, exc_type, exc, tb) -> None: - del exc_type, exc, tb - return None - - torch_mod = ModuleType("torch") - torch_mod.Tensor = Tensor - torch_mod.float32 = np.float32 - torch_mod.tensor = lambda value, device=None, dtype=None: Tensor(value, dtype=dtype) - torch_mod.zeros = lambda shape, dtype=np.float32: Tensor(np.zeros(shape, dtype=dtype)) - torch_mod.full_like = lambda tensor, fill_value: Tensor( - np.full_like(np.asarray(tensor), fill_value) - ) - torch_mod.remainder = lambda x, y: Tensor(np.remainder(np.asarray(x), y)) - torch_mod.cat = lambda tensors, dim=0: Tensor( - np.concatenate([np.asarray(t) for t in tensors], axis=dim) - ) - torch_mod.stack = lambda tensors, dim=0: Tensor( - np.stack([np.asarray(t) for t in tensors], axis=dim) - ) - torch_mod.sin = lambda tensor: Tensor(np.sin(np.asarray(tensor))) - torch_mod.from_numpy = lambda array: Tensor(array) - torch_mod.no_grad = NoGrad - sys.modules["torch"] = torch_mod - return "torch_shim" - - -class UnitreeG1RobotModelStub: - def __init__(self) -> None: - self.groups = { - "body": list(range(29)), - "lower_body": list(range(15)), - } - - def get_joint_group_indices(self, group: str) -> list[int]: - if group not in self.groups: - raise KeyError(f"unknown joint group: {group}") - return self.groups[group] - - -def load_robot_default_pose(path: Path) -> np.ndarray: - with path.open("rb") as handle: - parsed = tomllib.load(handle) - return np.array(parsed["default_pose"], dtype=np.float32) - - -def resolve_model_paths(model_dir: Path) -> tuple[Path, Path]: - balance_src = (model_dir / "GR00T-WholeBodyControl-Balance.onnx").resolve() - walk_src = (model_dir / "GR00T-WholeBodyControl-Walk.onnx").resolve() - if not balance_src.is_file() or not walk_src.is_file(): - raise FileNotFoundError(f"expected balance and walk ONNX models under {model_dir}") - return balance_src, walk_src - - -@contextlib.contextmanager -def patched_absolute_model_loading(policy_cls: type[Any]): - original = getattr(policy_cls, "load_onnx_policy", None) - if original is None: - yield - return - - def patched(self: Any, model_path: str): - resolved_path = Path(model_path) - if not resolved_path.is_file(): - marker = "/resources/robots/g1/" - if marker in model_path: - suffix = model_path.rsplit(marker, 1)[-1] - suffix_path = Path(suffix) - if suffix_path.is_file(): - model_path = str(suffix_path) - return original(self, model_path) - - setattr(policy_cls, "load_onnx_policy", patched) - try: - yield - finally: - setattr(policy_cls, "load_onnx_policy", original) - - -def build_observation(default_pose: np.ndarray) -> dict[str, np.ndarray]: - return { - "q": default_pose.copy(), - "dq": np.zeros_like(default_pose), - "floating_base_pose": np.array([0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0], dtype=np.float32), - "floating_base_vel": np.zeros(6, dtype=np.float32), - } - - -def reset_policy_state(policy: Any) -> None: - policy.obs_history = collections.deque(maxlen=policy.config["obs_history_len"]) - policy.obs_buffer = np.zeros(policy.config["num_obs"], dtype=np.float32) - policy.counter = 0 - policy.action = np.zeros(policy.config["num_actions"], dtype=np.float32) - policy.target_dof_pos = policy.config["default_angles"].copy() - policy.cmd = policy.config["cmd_init"].copy() - policy.height_cmd = float(policy.config["height_cmd"]) - policy.freq_cmd = float(policy.config["freq_cmd"]) - if "rpy_cmd" in policy.config: - policy.roll_cmd = float(policy.config["rpy_cmd"][0]) - policy.pitch_cmd = float(policy.config["rpy_cmd"][1]) - policy.yaw_cmd = float(policy.config["rpy_cmd"][2]) - else: - policy.roll_cmd = float(policy.config.get("roll_cmd", 0.0)) - policy.pitch_cmd = float(policy.config.get("pitch_cmd", 0.0)) - policy.yaw_cmd = float(policy.config.get("yaw_cmd", 0.0)) - policy.gait_indices = sys.modules["torch"].zeros((1,), dtype=sys.modules["torch"].float32) - policy.obs_tensor = None - policy.use_policy_action = True - policy.set_use_teleop_policy_cmd(True) - - -def single_tick(policy: Any, obs: dict[str, np.ndarray], command: np.ndarray) -> None: - policy.cmd = command.astype(np.float32, copy=True) - policy.set_observation(obs) - policy.get_action(time=time.monotonic()) - - -def run_microbench(policy: Any, obs: dict[str, np.ndarray], command: np.ndarray, samples: int) -> list[int]: - timings: list[int] = [] - for _ in range(samples): - reset_policy_state(policy) - start = time.perf_counter_ns() - single_tick(policy, obs, command) - timings.append(time.perf_counter_ns() - start) - return timings - - -def run_end_to_end_loop( - policy: Any, - obs: dict[str, np.ndarray], - command: np.ndarray, - ticks: int, - control_frequency_hz: int, -) -> tuple[list[int], float]: - timings: list[int] = [] - period_ns = round(1_000_000_000 / control_frequency_hz) - reset_policy_state(policy) - wall_start = time.perf_counter_ns() - for _ in range(ticks): - tick_start = time.perf_counter_ns() - single_tick(policy, obs, command) - tick_end = time.perf_counter_ns() - timings.append(tick_end - tick_start) - remaining_ns = period_ns - (tick_end - tick_start) - if remaining_ns > 0: - time.sleep(remaining_ns / 1_000_000_000) - wall_end = time.perf_counter_ns() - elapsed_s = (wall_end - wall_start) / 1_000_000_000 - achieved_hz = ticks / elapsed_s if elapsed_s > 0 else 0.0 - return timings, achieved_hz - - -def parse_command(case_id: str) -> np.ndarray: - if case_id == "decoupled_wbc/walk_predict" or case_id == "decoupled_wbc/end_to_end_cli_loop": - return np.array([0.25, 0.0, 0.05], dtype=np.float32) - if case_id == "decoupled_wbc/balance_predict": - return np.array([0.0, 0.0, 0.0], dtype=np.float32) - raise ValueError(f"unsupported Decoupled case_id: {case_id}") - - -def load_policy(repo_dir: Path, model_dir: Path) -> tuple[Any, str]: - torch_backend = install_torch_shim_if_needed() - balance_model, walk_model = resolve_model_paths(model_dir) - - sys.path.insert(0, str(repo_dir)) - from decoupled_wbc.control.policy import g1_gear_wbc_policy as policy_module # type: ignore - - robot_model = UnitreeG1RobotModelStub() - config_path = ( - repo_dir - / "decoupled_wbc" - / "sim2mujoco" - / "resources" - / "robots" - / "g1" - / "g1_gear_wbc.yaml" - ) - model_path = f"{balance_model},{walk_model}" - policy_cls = policy_module.G1GearWbcPolicy - with patched_absolute_model_loading(policy_cls): - policy = policy_cls(robot_model=robot_model, config=str(config_path), model_path=model_path) - return policy, torch_backend - - -def main() -> int: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--case-id", required=True) - parser.add_argument("--repo-dir", type=Path, required=True) - parser.add_argument("--model-dir", type=Path, required=True) - parser.add_argument("--robot-config", type=Path, default=Path("configs/robots/unitree_g1.toml")) - parser.add_argument("--samples", type=int, default=100) - parser.add_argument("--ticks", type=int, default=200) - parser.add_argument("--control-frequency-hz", type=int, default=50) - parser.add_argument("--output", type=Path, required=True) - args = parser.parse_args() - - if not args.repo_dir.is_dir(): - raise FileNotFoundError(f"repo-dir does not exist: {args.repo_dir}") - - default_pose = load_robot_default_pose(args.robot_config) - observation = build_observation(default_pose) - command = parse_command(args.case_id) - policy, torch_backend = load_policy(args.repo_dir, args.model_dir) - - if args.case_id == "decoupled_wbc/end_to_end_cli_loop": - samples_ns, hz = run_end_to_end_loop( - policy, observation, command, args.ticks, args.control_frequency_hz - ) - else: - samples_ns = run_microbench(policy, observation, command, args.samples) - hz = None - - payload = { - "case_id": args.case_id, - "samples_ns": samples_ns, - "hz": hz, - "notes": ( - "Measured via upstream decoupled_wbc.control.policy.g1_gear_wbc_policy.G1GearWbcPolicy " - f"with {torch_backend}." - ), - } - args.output.parent.mkdir(parents=True, exist_ok=True) - args.output.write_text(json.dumps(payload, indent=2), encoding="utf-8") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) +run_legacy_script("benchmarks/bench_nvidia_decoupled_official.py") diff --git a/scripts/bench_nvidia_gear_sonic_official.cpp b/scripts/bench_nvidia_gear_sonic_official.cpp index 433ff20..d001bfe 100644 --- a/scripts/bench_nvidia_gear_sonic_official.cpp +++ b/scripts/bench_nvidia_gear_sonic_official.cpp @@ -1,2200 +1 @@ -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include - -#include "math_utils.hpp" -#include "policy_parameters.hpp" -#include "robot_parameters.hpp" - -namespace fs = std::filesystem; - -namespace { - -constexpr std::size_t kPlannerQposDim = 36; -constexpr std::size_t kPlannerJointOffset = 7; -constexpr std::size_t kPlannerContextLen = 4; -constexpr std::size_t kReplanIntervalTicksDefault = 50; -constexpr std::size_t kReplanIntervalTicksRunning = 5; -constexpr std::size_t kPlannerThreadIntervalTicks = 5; -constexpr std::size_t kPlannerLookAheadSteps = 2; -constexpr std::size_t kPlannerBlendFrames = 8; -constexpr std::size_t kAllowedPredNumTokens = 11; -constexpr std::array kAllowedPredNumTokensMask = { - 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, -}; -constexpr float kDefaultHeightMeters = 0.74f; -constexpr float kOfficialDefaultHeightMeters = 0.788740f; -constexpr float kDefaultHeightSentinel = -1.0f; -constexpr std::int64_t kDefaultModeWalk = 2; -constexpr std::int64_t kDefaultModeRun = 3; -constexpr std::int64_t kDefaultModeIdle = 0; -constexpr std::int64_t kDefaultModeSlowWalk = 1; -constexpr std::size_t kReferenceFutureFrames = 10; -constexpr std::size_t kReferenceFrameStep = 5; -constexpr float kControlDtSeconds = 1.0f / 50.0f; -constexpr std::size_t kEncoderDim = 64; -constexpr std::size_t kEncoderObsDictDim = 1762; -constexpr std::size_t kDecoderObsDictDim = 994; -constexpr std::size_t kDecoderHistoryLen = 10; -constexpr std::size_t kLaterMotionProbeTick = 25; -constexpr std::size_t kEncoderModeOffset = 0; -constexpr std::size_t kEncoderMotionJointPositionsOffset = 4; -constexpr std::size_t kEncoderMotionJointVelocitiesOffset = 294; -constexpr std::size_t kEncoderMotionAnchorOrientationOffset = 601; - -volatile float g_sink = 0.0f; - -enum class ProviderKind { - Cpu, - Cuda, - TensorRt, -}; - -struct Options { - std::string case_id; - ProviderKind provider = ProviderKind::Cpu; - fs::path model_dir; - fs::path output; - std::optional dump_dir; - int samples = 100; - int ticks = 200; - int control_frequency_hz = 50; -}; - -struct Observation { - std::vector joint_positions; - std::vector joint_velocities; - std::array gravity_vector{}; - std::array angular_velocity{}; - std::array base_quat_wxyz{1.0, 0.0, 0.0, 0.0}; -}; - -struct Twist { - std::array linear{}; - std::array angular{}; -}; - -struct PlannerCommand { - std::int64_t mode = kDefaultModeIdle; - float target_vel = kDefaultHeightSentinel; - float height = kDefaultHeightSentinel; - std::array movement_direction{0.0f, 0.0f, 0.0f}; - std::array facing_direction{1.0f, 0.0f, 0.0f}; -}; - -float wrap_angle_rad(float angle) { - while (angle > static_cast(M_PI)) { - angle -= 2.0f * static_cast(M_PI); - } - while (angle < -static_cast(M_PI)) { - angle += 2.0f * static_cast(M_PI); - } - return angle; -} - -std::pair bin_angle_to_8_directions(float angle) { - constexpr float kBinSize = static_cast(M_PI / 4.0); - - const float normalized = wrap_angle_rad(angle); - int bin_index = static_cast(std::lround(normalized / kBinSize)); - if (bin_index > 4) { - bin_index -= 8; - } - if (bin_index < -4) { - bin_index += 8; - } - - float slow_walk_speed = 0.2f; - switch (bin_index) { - case 0: - case 1: - case -1: - slow_walk_speed = 0.3f; - break; - case 2: - case -2: - slow_walk_speed = 0.35f; - break; - case 3: - case -3: - slow_walk_speed = 0.25f; - break; - case 4: - case -4: - default: - slow_walk_speed = 0.2f; - break; - } - - return {static_cast(bin_index) * kBinSize, slow_walk_speed}; -} - -PlannerCommand idle_planner_command() { - return PlannerCommand{}; -} - -PlannerCommand derive_planner_command(float& facing_yaw_rad, const Twist& twist) { - facing_yaw_rad = wrap_angle_rad(facing_yaw_rad + twist.angular[2] * kControlDtSeconds); - const std::array facing_direction = { - std::cos(facing_yaw_rad), - std::sin(facing_yaw_rad), - 0.0f, - }; - const float command_norm = std::sqrt( - twist.linear[0] * twist.linear[0] + twist.linear[1] * twist.linear[1] - ); - - if (command_norm <= 0.01f) { - return PlannerCommand{ - kDefaultModeIdle, - kDefaultHeightSentinel, - kDefaultHeightSentinel, - {0.0f, 0.0f, 0.0f}, - facing_direction, - }; - } - - const float local_movement_angle = std::atan2(twist.linear[1], twist.linear[0]); - const auto [movement_angle, slow_walk_speed] = - bin_angle_to_8_directions(facing_yaw_rad + local_movement_angle); - const auto [mode, target_vel] = - command_norm < 0.8f - ? std::pair{kDefaultModeSlowWalk, slow_walk_speed} - : (command_norm < 2.5f - ? std::pair{kDefaultModeWalk, -1.0f} - : std::pair{kDefaultModeRun, -1.0f}); - - return PlannerCommand{ - mode, - target_vel, - kDefaultHeightSentinel, - {std::cos(movement_angle), std::sin(movement_angle), 0.0f}, - facing_direction, - }; -} - -bool planner_command_changed( - const std::optional& previous, - const PlannerCommand& next -) { - const auto vec3_distance = [](const std::array& a, const std::array& b) { - const float dx = a[0] - b[0]; - const float dy = a[1] - b[1]; - const float dz = a[2] - b[2]; - return std::sqrt(dx * dx + dy * dy + dz * dz); - }; - - if (!previous.has_value()) { - return true; - } - - return previous->mode != next.mode - || std::abs(previous->target_vel - next.target_vel) > 0.05f - || std::abs(previous->height - next.height) > 1e-3f - || vec3_distance(previous->movement_direction, next.movement_direction) > 0.1f - || vec3_distance(previous->facing_direction, next.facing_direction) > 0.1f; -} - -std::size_t planner_replan_interval_ticks(const PlannerCommand& command) { - return command.mode == kDefaultModeRun - ? kReplanIntervalTicksRunning - : kReplanIntervalTicksDefault; -} - -std::vector default_pose() { - std::vector pose; - pose.reserve(G1_NUM_MOTOR); - for (double angle : default_angles) { - pose.push_back(static_cast(angle)); - } - return pose; -} - -std::vector mujoco_to_isaaclab_values(const std::vector& values) { - if (values.size() != G1_NUM_MOTOR) { - throw std::runtime_error("values must match G1_NUM_MOTOR"); - } - - std::vector remapped(G1_NUM_MOTOR, 0.0f); - for (std::size_t mujoco_index = 0; mujoco_index < G1_NUM_MOTOR; ++mujoco_index) { - remapped[static_cast(isaaclab_to_mujoco[mujoco_index])] = - values[mujoco_index]; - } - return remapped; -} - -std::vector mujoco_to_isaaclab_positions(const std::vector& joint_positions) { - return mujoco_to_isaaclab_values(joint_positions); -} - -std::vector mujoco_to_isaaclab_joint_offsets(const std::vector& joint_positions) { - auto remapped = mujoco_to_isaaclab_values(joint_positions); - for (std::size_t mujoco_index = 0; mujoco_index < G1_NUM_MOTOR; ++mujoco_index) { - remapped[static_cast(isaaclab_to_mujoco[mujoco_index])] -= - static_cast(default_angles[mujoco_index]); - } - return remapped; -} - -std::vector isaaclab_to_mujoco_values(const std::vector& values) { - if (values.size() != G1_NUM_MOTOR) { - throw std::runtime_error("values must match G1_NUM_MOTOR"); - } - - std::vector remapped(G1_NUM_MOTOR, 0.0f); - for (std::size_t mujoco_index = 0; mujoco_index < G1_NUM_MOTOR; ++mujoco_index) { - remapped[mujoco_index] = values[static_cast(isaaclab_to_mujoco[mujoco_index])]; - } - return remapped; -} - -Observation zero_tracking_observation() { - Observation obs; - obs.joint_positions.assign(G1_NUM_MOTOR, 0.0f); - obs.joint_velocities.assign(G1_NUM_MOTOR, 0.0f); - obs.gravity_vector = {0.0f, 0.0f, -1.0f}; - obs.angular_velocity = {0.0f, 0.0f, 0.0f}; - return obs; -} - -Observation standing_velocity_observation() { - Observation obs; - obs.joint_positions = default_pose(); - obs.joint_velocities.assign(G1_NUM_MOTOR, 0.0f); - obs.gravity_vector = {0.0f, 0.0f, -1.0f}; - obs.angular_velocity = {0.0f, 0.0f, 0.0f}; - return obs; -} - -std::string json_escape(const std::string& input) { - std::string escaped; - escaped.reserve(input.size()); - for (char ch : input) { - switch (ch) { - case '\\': - escaped += "\\\\"; - break; - case '"': - escaped += "\\\""; - break; - case '\n': - escaped += "\\n"; - break; - default: - escaped.push_back(ch); - break; - } - } - return escaped; -} - -std::vector session_names(Ort::Session& session, bool inputs) { - Ort::AllocatorWithDefaultOptions allocator; - const std::size_t count = inputs ? session.GetInputCount() : session.GetOutputCount(); - std::vector names; - names.reserve(count); - for (std::size_t index = 0; index < count; ++index) { - auto name = inputs ? session.GetInputNameAllocated(index, allocator) - : session.GetOutputNameAllocated(index, allocator); - names.emplace_back(name.get()); - } - return names; -} - -void require_name(const std::vector& names, std::string_view needle, std::string_view kind) { - if (std::find(names.begin(), names.end(), needle) == names.end()) { - throw std::runtime_error( - "missing required " + std::string(kind) + " tensor '" + std::string(needle) + "'" - ); - } -} - -std::string provider_label(ProviderKind provider) { - switch (provider) { - case ProviderKind::Cpu: - return "cpu"; - case ProviderKind::Cuda: - return "cuda"; - case ProviderKind::TensorRt: - return "tensor_rt"; - } - throw std::runtime_error("unreachable provider label state"); -} - -std::string provider_identifier(ProviderKind provider) { - switch (provider) { - case ProviderKind::Cpu: - return "CPUExecutionProvider"; - case ProviderKind::Cuda: - return "CUDAExecutionProvider"; - case ProviderKind::TensorRt: - return "TensorrtExecutionProvider"; - } - throw std::runtime_error("unreachable provider identifier state"); -} - -std::vector available_providers() { - return Ort::GetAvailableProviders(); -} - -std::string format_provider_list(const std::vector& providers) { - if (providers.empty()) { - return "(none)"; - } - std::string joined; - for (std::size_t index = 0; index < providers.size(); ++index) { - if (index > 0) { - joined += ", "; - } - joined += providers[index]; - } - return joined; -} - -constexpr const char* kTensorRtExcludedOpTypes = "Cast"; - -void configure_provider(Ort::SessionOptions& options, ProviderKind provider) { - if (provider == ProviderKind::Cpu) { - return; - } - - try { - if (provider == ProviderKind::Cuda) { - Ort::CUDAProviderOptions cuda_options; - cuda_options.Update({{"device_id", "0"}}); - options.AppendExecutionProvider_CUDA_V2(*cuda_options); - return; - } - - Ort::TensorRTProviderOptions tensorrt_options; - // GEAR-Sonic's planner graph includes Float->Int64 Cast paths that - // TensorRT cannot compile inside a fused subgraph. Excluding Cast keeps - // those nodes on CUDA/CPU while still exercising TensorRT for the rest - // of the graph. - tensorrt_options.Update({ - {"device_id", "0"}, - {"trt_op_types_to_exclude", kTensorRtExcludedOpTypes}, - }); - options.AppendExecutionProvider_TensorRT_V2(*tensorrt_options); - Ort::CUDAProviderOptions cuda_options; - cuda_options.Update({{"device_id", "0"}}); - options.AppendExecutionProvider_CUDA_V2(*cuda_options); - } catch (const Ort::Exception& error) { - throw std::runtime_error( - std::string("failed to configure provider `") + provider_label(provider) - + "`: " + error.what() - ); - } -} - -Ort::Session make_session(Ort::Env& env, const fs::path& model_path, ProviderKind provider) { - if (!fs::is_regular_file(model_path)) { - throw std::runtime_error("model file does not exist: " + model_path.string()); - } - - Ort::SessionOptions options; - options.SetIntraOpNumThreads(1); - options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED); - configure_provider(options, provider); - const std::string model_path_str = model_path.string(); - try { - return Ort::Session(env, model_path_str.c_str(), options); - } catch (const Ort::Exception& error) { - const auto providers = available_providers(); - throw std::runtime_error( - std::string("failed to initialize provider `") + provider_label(provider) - + "` for model " + model_path.string() + ": " + error.what() - + " (advertised providers: " + format_provider_list(providers) + ")" - ); - } -} - -struct PlannerState { - std::deque> context; - std::size_t steps_since_plan = kReplanIntervalTicksDefault; - std::vector last_context_frame; - std::vector> motion_qpos_50hz; - std::vector> motion_joint_velocities_isaaclab; - std::size_t current_motion_frame = 0; - float facing_yaw_rad = 0.0f; - std::optional> init_base_quat_wxyz; - std::optional> init_ref_root_quat_wxyz; - std::optional last_command; - - struct PendingPlannerReplan { - std::size_t request_motion_frame = 0; - PlannerCommand command{}; - std::future>> future; - }; - - std::optional pending_replan; - - PlannerState() - : context(), last_context_frame() - { - reset(); - } - - void reset() { - const auto standing = make_standing_qpos(); - context.clear(); - for (std::size_t index = 0; index < kPlannerContextLen; ++index) { - context.push_back(standing); - } - steps_since_plan = kReplanIntervalTicksDefault; - last_context_frame = standing; - motion_qpos_50hz.clear(); - motion_joint_velocities_isaaclab.clear(); - current_motion_frame = 0; - facing_yaw_rad = 0.0f; - init_base_quat_wxyz.reset(); - init_ref_root_quat_wxyz.reset(); - last_command.reset(); - pending_replan.reset(); - } - - static std::vector make_standing_qpos() { - std::vector qpos(kPlannerQposDim, 0.0f); - qpos[2] = kDefaultHeightMeters; - qpos[3] = 1.0f; - const auto pose = default_pose(); - std::copy(pose.begin(), pose.end(), qpos.begin() + static_cast(kPlannerJointOffset)); - return qpos; - } -}; - -struct TrackingState { - std::deque> gravity; - std::deque> angular_velocity; - std::deque> joint_positions; - std::deque> joint_velocities; - std::deque> last_actions; - - TrackingState() - : gravity(), angular_velocity(), joint_positions(), joint_velocities(), last_actions() - { - reset(); - } - - void reset() { - gravity.clear(); - angular_velocity.clear(); - joint_positions.clear(); - joint_velocities.clear(); - last_actions.clear(); - for (std::size_t index = 0; index < kDecoderHistoryLen; ++index) { - gravity.push_back({0.0f, 0.0f, 1.0f}); - angular_velocity.push_back({0.0f, 0.0f, 0.0f}); - joint_positions.emplace_back(G1_NUM_MOTOR, 0.0f); - joint_velocities.emplace_back(G1_NUM_MOTOR, 0.0f); - last_actions.emplace_back(G1_NUM_MOTOR, 0.0f); - } - } - - void push(const Observation& obs, const std::vector& actions) { - if (gravity.size() >= kDecoderHistoryLen) { - gravity.pop_front(); - angular_velocity.pop_front(); - joint_positions.pop_front(); - joint_velocities.pop_front(); - last_actions.pop_front(); - } - gravity.push_back(obs.gravity_vector); - angular_velocity.push_back(obs.angular_velocity); - joint_positions.push_back(mujoco_to_isaaclab_joint_offsets(obs.joint_positions)); - joint_velocities.push_back(mujoco_to_isaaclab_values(obs.joint_velocities)); - last_actions.push_back(actions); - } -}; - -class GearSonicOfficialHarness { -public: - explicit GearSonicOfficialHarness( - const fs::path& model_dir, - ProviderKind provider, - std::optional dump_dir = std::nullopt - ) - : env_(ORT_LOGGING_LEVEL_WARNING, "gear_sonic_official"), - memory_info_(Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault)), - encoder_(make_session(env_, model_dir / "model_encoder.onnx", provider)), - decoder_(make_session(env_, model_dir / "model_decoder.onnx", provider)), - planner_(make_session(env_, model_dir / "planner_sonic.onnx", provider)), - planner_state_(), - tracking_state_(), - velocity_obs_(standing_velocity_observation()), - tracking_obs_(zero_tracking_observation()), - latest_action_(G1_NUM_MOTOR, 0.0f), - dump_dir_(std::move(dump_dir)) - { - validate_contracts(); - } - - void reset() { - planner_state_.reset(); - tracking_state_.reset(); - velocity_obs_ = standing_velocity_observation(); - tracking_obs_ = zero_tracking_observation(); - latest_action_.assign(G1_NUM_MOTOR, 0.0f); - dumped_tracking_tensors_ = false; - } - - std::vector velocity_tick() { - maybe_apply_pending_planner_replan(); - - const Twist twist = velocity_command(); - const float command_speed = std::sqrt( - twist.linear[0] * twist.linear[0] + twist.linear[1] * twist.linear[1] - ); - const auto live_command = derive_planner_command(planner_state_.facing_yaw_rad, twist); - const bool initializing_planner = planner_state_.motion_qpos_50hz.empty(); - const auto planner_command = - initializing_planner && command_speed <= 0.01f - ? idle_planner_command() - : live_command; - const bool needs_replan = - initializing_planner - || (planner_state_.pending_replan.has_value() - ? false - : (planner_command_changed(planner_state_.last_command, live_command) - || planner_state_.steps_since_plan - >= planner_replan_interval_ticks(live_command))); - - if (needs_replan) { - if (!planner_state_.motion_qpos_50hz.empty()) { - planner_state_.context = rebuild_planner_context_from_motion( - planner_state_.motion_qpos_50hz, - planner_state_.current_motion_frame - ); - if (!planner_state_.context.empty()) { - planner_state_.last_context_frame = planner_state_.context.back(); - } - } - if (initializing_planner) { - commit_planner_motion( - 0, - planner_command, - resample_planner_trajectory_to_50hz( - run_planner_command(planner_state_.context, planner_command) - ) - ); - } else { - start_async_planner_replan(planner_command); - } - planner_state_.steps_since_plan = 0; - } - - planner_state_.steps_since_plan += 1; - if (planner_state_.motion_qpos_50hz.empty()) { - throw std::runtime_error("planner motion buffer is empty after velocity bootstrap"); - } - - const auto obs = motion_observation_from_planner_frame( - planner_state_.motion_qpos_50hz, - planner_state_.motion_joint_velocities_isaaclab, - planner_state_.current_motion_frame - ); - const auto init_base_quat = - planner_state_.init_base_quat_wxyz.value_or(obs.base_quat_wxyz); - const auto init_ref_root_quat = - planner_state_.init_ref_root_quat_wxyz.value_or( - planner_frame_root_quaternion(planner_state_.motion_qpos_50hz.front()) - ); - const auto encoder_obs = build_velocity_encoder_obs_dict( - planner_state_.motion_qpos_50hz, - planner_state_.motion_joint_velocities_isaaclab, - planner_state_.current_motion_frame, - obs.base_quat_wxyz, - init_base_quat, - init_ref_root_quat - ); - const auto tokens = run_single_f32( - encoder_, - "obs_dict", - "encoded_tokens", - encoder_obs, - kEncoderObsDictDim - ); - if (tokens.size() != kEncoderDim) { - throw std::runtime_error( - "encoder output dimension mismatch: expected " + std::to_string(kEncoderDim) + - ", got " + std::to_string(tokens.size()) - ); - } - - tracking_state_.push(obs, latest_action_); - const auto decoder_obs = build_decoder_obs_dict(tokens); - const auto raw_actions = run_single_f32( - decoder_, - "obs_dict", - "action", - decoder_obs, - kDecoderObsDictDim - ); - if (raw_actions.size() != G1_NUM_MOTOR) { - throw std::runtime_error( - "decoder output dimension mismatch: expected " + std::to_string(G1_NUM_MOTOR) + - ", got " + std::to_string(raw_actions.size()) - ); - } - - std::vector positions(G1_NUM_MOTOR, 0.0f); - for (int mujoco_index = 0; mujoco_index < G1_NUM_MOTOR; ++mujoco_index) { - const int isaaclab_index = isaaclab_to_mujoco[static_cast(mujoco_index)]; - const float action = raw_actions[static_cast(isaaclab_index)]; - const float scaled = action * static_cast(g1_action_scale[static_cast(mujoco_index)]); - positions[static_cast(mujoco_index)] = - static_cast(default_angles[static_cast(mujoco_index)]) + scaled; - } - - latest_action_ = raw_actions; - advance_planner_motion_frame(); - return positions; - } - - std::vector tracking_tick() { - const auto encoder_obs = build_encoder_obs_dict(); - const auto tokens = run_single_f32(encoder_, "obs_dict", "encoded_tokens", encoder_obs, kEncoderObsDictDim); - if (tokens.size() != kEncoderDim) { - throw std::runtime_error( - "encoder output dimension mismatch: expected " + std::to_string(kEncoderDim) + - ", got " + std::to_string(tokens.size()) - ); - } - - tracking_state_.push(tracking_obs_, latest_action_); - const auto decoder_obs = build_decoder_obs_dict(tokens); - const auto raw_actions = run_single_f32(decoder_, "obs_dict", "action", decoder_obs, kDecoderObsDictDim); - if (raw_actions.size() != G1_NUM_MOTOR) { - throw std::runtime_error( - "decoder output dimension mismatch: expected " + std::to_string(G1_NUM_MOTOR) + - ", got " + std::to_string(raw_actions.size()) - ); - } - maybe_dump_tracking_tensors(encoder_obs, tokens, decoder_obs, raw_actions); - - std::vector positions(G1_NUM_MOTOR, 0.0f); - for (int mujoco_index = 0; mujoco_index < G1_NUM_MOTOR; ++mujoco_index) { - const int isaaclab_index = isaaclab_to_mujoco[static_cast(mujoco_index)]; - const float action = raw_actions[static_cast(isaaclab_index)]; - const float scaled = action * static_cast(g1_action_scale[static_cast(mujoco_index)]); - positions[static_cast(mujoco_index)] = - static_cast(default_angles[static_cast(mujoco_index)]) + scaled; - } - - latest_action_ = raw_actions; - return positions; - } - - std::vector planner_only_tick() { - std::deque> context; - const auto standing = PlannerState::make_standing_qpos(); - for (std::size_t index = 0; index < kPlannerContextLen; ++index) { - context.push_back(standing); - } - - float facing_yaw_rad = 0.0f; - const auto planner_command = derive_planner_command(facing_yaw_rad, velocity_command()); - const auto trajectory = run_planner_command(context, planner_command); - if (trajectory.empty()) { - throw std::runtime_error("planner-only benchmark produced an empty trajectory"); - } - return trajectory.front(); - } - - std::vector velocity_first_live_replan_dump() { - if (!dump_dir_.has_value()) { - throw std::runtime_error("--dump-dir is required for gear_sonic_velocity/first_live_replan_dump"); - } - - reset(); - tracking_state_.reset(); - latest_action_.assign(G1_NUM_MOTOR, 0.0f); - - std::deque> context; - const auto standing = make_official_standing_qpos(); - for (std::size_t index = 0; index < kPlannerContextLen; ++index) { - context.push_back(standing); - } - - const auto idle_planned_30hz = run_planner_command(context, idle_planner_command()); - const auto bootstrap_motion_50hz = resample_planner_trajectory_to_50hz(idle_planned_30hz); - - constexpr std::size_t kFirstLiveReplanTick = kPlannerThreadIntervalTicks; - const auto planner_context = - rebuild_planner_context_from_motion(bootstrap_motion_50hz, kFirstLiveReplanTick); - - float facing_yaw_rad = 0.0f; - Twist twist; - twist.linear = {0.6f, 0.0f, 0.0f}; - twist.angular = {0.0f, 0.0f, 0.0f}; - const auto live_command = derive_planner_command(facing_yaw_rad, twist); - const auto live_planned_30hz = run_planner_command(planner_context, live_command); - const auto live_planned_50hz = resample_planner_trajectory_to_50hz(live_planned_30hz); - const auto committed_motion_50hz = blend_planner_motion( - bootstrap_motion_50hz, - kFirstLiveReplanTick, - kFirstLiveReplanTick, - live_planned_50hz - ); - const auto committed_joint_velocities = - compute_motion_joint_velocities_isaaclab(committed_motion_50hz); - - const std::array init_base_quat_wxyz = {1.0, 0.0, 0.0, 0.0}; - const auto init_ref_root_quat_wxyz = - planner_frame_root_quaternion(committed_motion_50hz.front()); - const auto encoder_obs = build_velocity_encoder_obs_dict( - committed_motion_50hz, - committed_joint_velocities, - 0, - init_base_quat_wxyz, - init_base_quat_wxyz, - init_ref_root_quat_wxyz - ); - const auto tokens = run_single_f32( - encoder_, - "obs_dict", - "encoded_tokens", - encoder_obs, - kEncoderObsDictDim - ); - if (tokens.size() != kEncoderDim) { - throw std::runtime_error( - "encoder output dimension mismatch: expected " + std::to_string(kEncoderDim) + - ", got " + std::to_string(tokens.size()) - ); - } - - tracking_state_.push(tracking_obs_, latest_action_); - const auto decoder_obs = build_decoder_obs_dict(tokens); - const auto raw_actions = run_single_f32( - decoder_, - "obs_dict", - "action", - decoder_obs, - kDecoderObsDictDim - ); - if (raw_actions.size() != G1_NUM_MOTOR) { - throw std::runtime_error( - "decoder output dimension mismatch: expected " + std::to_string(G1_NUM_MOTOR) + - ", got " + std::to_string(raw_actions.size()) - ); - } - - const std::vector planner_command = { - static_cast(live_command.mode), - live_command.target_vel, - live_command.height, - live_command.movement_direction[0], - live_command.movement_direction[1], - live_command.movement_direction[2], - live_command.facing_direction[0], - live_command.facing_direction[1], - live_command.facing_direction[2], - }; - - write_matrix_json(*dump_dir_ / "bootstrap_motion_50hz.json", bootstrap_motion_50hz); - write_matrix_json(*dump_dir_ / "planner_context.json", deque_to_rows(planner_context)); - write_vector_json(*dump_dir_ / "planner_command.json", planner_command); - write_matrix_json(*dump_dir_ / "planner_motion_30hz.json", live_planned_30hz); - write_matrix_json(*dump_dir_ / "planner_motion_50hz.json", live_planned_50hz); - write_matrix_json( - *dump_dir_ / "planner_motion_50hz_committed.json", - committed_motion_50hz - ); - write_matrix_json( - *dump_dir_ / "planner_joint_velocities_50hz.json", - committed_joint_velocities - ); - write_vector_json(*dump_dir_ / "velocity_encoder_obs.json", encoder_obs); - write_vector_json(*dump_dir_ / "velocity_tokens.json", tokens); - write_vector_json(*dump_dir_ / "velocity_decoder_obs.json", decoder_obs); - write_vector_json(*dump_dir_ / "velocity_raw_actions.json", raw_actions); - - std::vector positions(G1_NUM_MOTOR, 0.0f); - for (int mujoco_index = 0; mujoco_index < G1_NUM_MOTOR; ++mujoco_index) { - const int isaaclab_index = isaaclab_to_mujoco[static_cast(mujoco_index)]; - const float action = raw_actions[static_cast(isaaclab_index)]; - const float scaled = - action * static_cast(g1_action_scale[static_cast(mujoco_index)]); - positions[static_cast(mujoco_index)] = - static_cast(default_angles[static_cast(mujoco_index)]) + scaled; - } - - latest_action_ = raw_actions; - return positions; - } - - std::vector velocity_later_motion_dump() { - if (!dump_dir_.has_value()) { - throw std::runtime_error("--dump-dir is required for gear_sonic_velocity/later_motion_dump"); - } - - reset(); - tracking_state_.reset(); - latest_action_.assign(G1_NUM_MOTOR, 0.0f); - - std::deque> context; - const auto standing = make_official_standing_qpos(); - for (std::size_t index = 0; index < kPlannerContextLen; ++index) { - context.push_back(standing); - } - - const auto idle_planned_30hz = run_planner_command(context, idle_planner_command()); - const auto bootstrap_motion_50hz = resample_planner_trajectory_to_50hz(idle_planned_30hz); - - constexpr std::size_t kFirstLiveReplanTick = kPlannerThreadIntervalTicks; - const auto planner_context = - rebuild_planner_context_from_motion(bootstrap_motion_50hz, kFirstLiveReplanTick); - - float facing_yaw_rad = 0.0f; - Twist twist; - twist.linear = {0.6f, 0.0f, 0.0f}; - twist.angular = {0.0f, 0.0f, 0.0f}; - const auto live_command = derive_planner_command(facing_yaw_rad, twist); - const auto live_planned_30hz = run_planner_command(planner_context, live_command); - const auto live_planned_50hz = resample_planner_trajectory_to_50hz(live_planned_30hz); - const auto committed_motion_50hz = blend_planner_motion( - bootstrap_motion_50hz, - kFirstLiveReplanTick, - kFirstLiveReplanTick, - live_planned_50hz - ); - const auto committed_joint_velocities = - compute_motion_joint_velocities_isaaclab(committed_motion_50hz); - if (committed_motion_50hz.size() <= kLaterMotionProbeTick) { - throw std::runtime_error("later motion probe tick exceeds committed motion length"); - } - - const std::array init_base_quat_wxyz = {1.0, 0.0, 0.0, 0.0}; - const auto init_ref_root_quat_wxyz = - planner_frame_root_quaternion(committed_motion_50hz.front()); - - Observation probe_obs; - std::vector encoder_obs; - std::vector tokens; - std::vector decoder_obs; - std::vector raw_actions; - for (std::size_t tick = 0; tick <= kLaterMotionProbeTick; ++tick) { - probe_obs = motion_observation_from_planner_frame( - committed_motion_50hz, - committed_joint_velocities, - tick - ); - encoder_obs = build_velocity_encoder_obs_dict( - committed_motion_50hz, - committed_joint_velocities, - tick, - probe_obs.base_quat_wxyz, - init_base_quat_wxyz, - init_ref_root_quat_wxyz - ); - tokens = run_single_f32( - encoder_, - "obs_dict", - "encoded_tokens", - encoder_obs, - kEncoderObsDictDim - ); - if (tokens.size() != kEncoderDim) { - throw std::runtime_error( - "encoder output dimension mismatch: expected " + std::to_string(kEncoderDim) + - ", got " + std::to_string(tokens.size()) - ); - } - - tracking_state_.push(probe_obs, latest_action_); - decoder_obs = build_decoder_obs_dict(tokens); - raw_actions = run_single_f32( - decoder_, - "obs_dict", - "action", - decoder_obs, - kDecoderObsDictDim - ); - if (raw_actions.size() != G1_NUM_MOTOR) { - throw std::runtime_error( - "decoder output dimension mismatch: expected " + std::to_string(G1_NUM_MOTOR) + - ", got " + std::to_string(raw_actions.size()) - ); - } - latest_action_ = raw_actions; - } - - write_vector_json( - *dump_dir_ / "velocity_probe_tick.json", - std::vector{static_cast(kLaterMotionProbeTick)} - ); - write_vector_json(*dump_dir_ / "current_joint_positions_mujoco.json", probe_obs.joint_positions); - write_vector_json(*dump_dir_ / "current_joint_velocities_mujoco.json", probe_obs.joint_velocities); - write_vector_json( - *dump_dir_ / "current_base_quat_wxyz.json", - std::vector{ - static_cast(probe_obs.base_quat_wxyz[0]), - static_cast(probe_obs.base_quat_wxyz[1]), - static_cast(probe_obs.base_quat_wxyz[2]), - static_cast(probe_obs.base_quat_wxyz[3]), - } - ); - write_vector_json( - *dump_dir_ / "current_gravity.json", - std::vector{ - probe_obs.gravity_vector[0], - probe_obs.gravity_vector[1], - probe_obs.gravity_vector[2], - } - ); - write_vector_json( - *dump_dir_ / "current_angular_velocity.json", - std::vector{ - probe_obs.angular_velocity[0], - probe_obs.angular_velocity[1], - probe_obs.angular_velocity[2], - } - ); - write_matrix_json( - *dump_dir_ / "history_joint_positions_isaaclab_offsets.json", - deque_to_rows(tracking_state_.joint_positions) - ); - write_matrix_json( - *dump_dir_ / "history_joint_velocities_isaaclab.json", - deque_to_rows(tracking_state_.joint_velocities) - ); - write_matrix_json( - *dump_dir_ / "history_last_actions.json", - deque_to_rows(tracking_state_.last_actions) - ); - write_matrix_json( - *dump_dir_ / "history_gravity.json", - deque_to_rows(tracking_state_.gravity) - ); - write_matrix_json( - *dump_dir_ / "history_angular_velocity.json", - deque_to_rows(tracking_state_.angular_velocity) - ); - write_vector_json(*dump_dir_ / "velocity_encoder_obs.json", encoder_obs); - write_vector_json(*dump_dir_ / "velocity_tokens.json", tokens); - write_vector_json(*dump_dir_ / "velocity_decoder_obs.json", decoder_obs); - write_vector_json(*dump_dir_ / "velocity_raw_actions.json", raw_actions); - - std::vector positions(G1_NUM_MOTOR, 0.0f); - for (int mujoco_index = 0; mujoco_index < G1_NUM_MOTOR; ++mujoco_index) { - const int isaaclab_index = isaaclab_to_mujoco[static_cast(mujoco_index)]; - const float action = raw_actions[static_cast(isaaclab_index)]; - const float scaled = - action * static_cast(g1_action_scale[static_cast(mujoco_index)]); - positions[static_cast(mujoco_index)] = - static_cast(default_angles[static_cast(mujoco_index)]) + scaled; - } - - return positions; - } - -private: - Ort::Env env_; - Ort::MemoryInfo memory_info_; - Ort::Session encoder_; - Ort::Session decoder_; - Ort::Session planner_; - PlannerState planner_state_; - TrackingState tracking_state_; - Observation velocity_obs_; - Observation tracking_obs_; - std::vector latest_action_; - std::optional dump_dir_; - bool dumped_tracking_tensors_ = false; - - void validate_contracts() { - const auto planner_inputs = session_names(planner_, true); - const auto planner_outputs = session_names(planner_, false); - for (const auto* name : { - "context_mujoco_qpos", - "target_vel", - "mode", - "movement_direction", - "facing_direction", - "random_seed", - "has_specific_target", - "specific_target_positions", - "specific_target_headings", - "allowed_pred_num_tokens", - "height", - }) { - require_name(planner_inputs, name, "planner input"); - } - require_name(planner_outputs, "mujoco_qpos", "planner output"); - require_name(planner_outputs, "num_pred_frames", "planner output"); - - const auto encoder_inputs = session_names(encoder_, true); - const auto encoder_outputs = session_names(encoder_, false); - require_name(encoder_inputs, "obs_dict", "encoder input"); - require_name(encoder_outputs, "encoded_tokens", "encoder output"); - - const auto decoder_inputs = session_names(decoder_, true); - const auto decoder_outputs = session_names(decoder_, false); - require_name(decoder_inputs, "obs_dict", "decoder input"); - require_name(decoder_outputs, "action", "decoder output"); - } - - std::vector planner_context_frame( - const std::vector& template_frame, - const Observation& obs - ) const { - std::vector frame = - template_frame.size() == kPlannerQposDim ? template_frame : std::vector(kPlannerQposDim, 0.0f); - if (frame[2] == 0.0f) { - frame[2] = kDefaultHeightMeters; - } - frame[3] = static_cast(obs.base_quat_wxyz[0]); - frame[4] = static_cast(obs.base_quat_wxyz[1]); - frame[5] = static_cast(obs.base_quat_wxyz[2]); - frame[6] = static_cast(obs.base_quat_wxyz[3]); - std::copy(obs.joint_positions.begin(), obs.joint_positions.end(), frame.begin() + static_cast(kPlannerJointOffset)); - return frame; - } - - Twist velocity_command() const { - Twist twist; - twist.linear = {0.3f, 0.0f, 0.0f}; - twist.angular = {0.0f, 0.0f, 0.0f}; - return twist; - } - - static std::vector make_official_standing_qpos() { - std::vector qpos(kPlannerQposDim, 0.0f); - qpos[2] = kOfficialDefaultHeightMeters; - qpos[3] = 1.0f; - const auto pose = default_pose(); - std::copy( - pose.begin(), - pose.end(), - qpos.begin() + static_cast(kPlannerJointOffset) - ); - return qpos; - } - - void commit_planner_motion( - std::size_t request_motion_frame, - const PlannerCommand& planner_command, - const std::vector>& planned_50hz - ) { - if (planned_50hz.empty()) { - throw std::runtime_error("planner produced an empty 50Hz motion sequence"); - } - - planner_state_.motion_qpos_50hz = - planner_state_.motion_qpos_50hz.empty() - ? planned_50hz - : blend_planner_motion( - planner_state_.motion_qpos_50hz, - planner_state_.current_motion_frame, - request_motion_frame, - planned_50hz - ); - planner_state_.motion_joint_velocities_isaaclab = - compute_motion_joint_velocities_isaaclab(planner_state_.motion_qpos_50hz); - planner_state_.current_motion_frame = 0; - if (planner_state_.motion_qpos_50hz.empty()) { - planner_state_.init_ref_root_quat_wxyz.reset(); - } else { - planner_state_.init_ref_root_quat_wxyz = - planner_frame_root_quaternion(planner_state_.motion_qpos_50hz.front()); - } - if (!planner_state_.init_base_quat_wxyz.has_value()) { - planner_state_.init_base_quat_wxyz = velocity_obs_.base_quat_wxyz; - } - planner_state_.last_command = planner_command; - } - - void maybe_apply_pending_planner_replan() { - if (!planner_state_.pending_replan.has_value()) { - return; - } - - auto& pending = planner_state_.pending_replan.value(); - if (pending.future.wait_for(std::chrono::seconds(0)) != std::future_status::ready) { - return; - } - - auto ready = std::move(pending); - planner_state_.pending_replan.reset(); - commit_planner_motion( - ready.request_motion_frame, - ready.command, - ready.future.get() - ); - } - - void start_async_planner_replan(const PlannerCommand& planner_command) { - PlannerState::PendingPlannerReplan pending; - pending.request_motion_frame = planner_state_.current_motion_frame; - pending.command = planner_command; - const auto context = planner_state_.context; - pending.future = std::async( - std::launch::async, - [this, context, planner_command]() { - return resample_planner_trajectory_to_50hz( - run_planner_command(context, planner_command) - ); - } - ); - planner_state_.last_command = planner_command; - planner_state_.pending_replan = std::move(pending); - } - - void advance_planner_motion_frame() { - if (planner_state_.motion_qpos_50hz.empty()) { - return; - } - planner_state_.current_motion_frame = std::min( - planner_state_.current_motion_frame + 1, - planner_state_.motion_qpos_50hz.size() - 1 - ); - } - - std::vector> run_planner( - const std::deque>& context, - const Twist& twist - ) { - const float cmd_norm = - std::sqrt(twist.linear[0] * twist.linear[0] + twist.linear[1] * twist.linear[1]); - const std::array movement_direction = - cmd_norm > 1e-6f - ? std::array{ - twist.linear[0] / cmd_norm, - twist.linear[1] / cmd_norm, - 0.0f, - } - : std::array{1.0f, 0.0f, 0.0f}; - const float yaw = twist.angular[2]; - const std::array facing_direction = { - std::cos(yaw), - std::sin(yaw), - 0.0f, - }; - return run_planner_command( - context, - PlannerCommand{ - kDefaultModeWalk, - cmd_norm, - kDefaultHeightMeters, - movement_direction, - facing_direction, - } - ); - } - - std::vector> run_planner_command( - const std::deque>& context, - const PlannerCommand& command - ) { - std::vector context_data; - context_data.reserve(context.size() * kPlannerQposDim); - for (const auto& frame : context) { - context_data.insert(context_data.end(), frame.begin(), frame.end()); - } - - const std::array target_vel = {command.target_vel}; - const std::array mode = {command.mode}; - const std::array height = {command.height}; - const std::array random_seed = {0}; - const std::array has_specific_target = {0}; - const std::vector specific_target_positions(12, 0.0f); - const std::vector specific_target_headings(4, 0.0f); - const auto allowed_pred_num_tokens = kAllowedPredNumTokensMask; - - const std::array context_shape = { - 1, - static_cast(kPlannerContextLen), - static_cast(kPlannerQposDim), - }; - const std::array vec3_shape = {1, 3}; - const std::array scalar_shape = {1}; - const std::array has_target_shape = {1, 1}; - const std::array specific_positions_shape = {1, 4, 3}; - const std::array specific_headings_shape = {1, 4}; - const std::array allowed_tokens_shape = { - 1, - static_cast(kAllowedPredNumTokens), - }; - - std::vector input_tensors; - input_tensors.reserve(11); - input_tensors.push_back(Ort::Value::CreateTensor( - memory_info_, - context_data.data(), - context_data.size(), - context_shape.data(), - context_shape.size() - )); - input_tensors.push_back(Ort::Value::CreateTensor( - memory_info_, - const_cast(target_vel.data()), - target_vel.size(), - scalar_shape.data(), - scalar_shape.size() - )); - input_tensors.push_back(Ort::Value::CreateTensor( - memory_info_, - const_cast(mode.data()), - mode.size(), - scalar_shape.data(), - scalar_shape.size() - )); - input_tensors.push_back(Ort::Value::CreateTensor( - memory_info_, - const_cast(command.movement_direction.data()), - command.movement_direction.size(), - vec3_shape.data(), - vec3_shape.size() - )); - input_tensors.push_back(Ort::Value::CreateTensor( - memory_info_, - const_cast(command.facing_direction.data()), - command.facing_direction.size(), - vec3_shape.data(), - vec3_shape.size() - )); - input_tensors.push_back(Ort::Value::CreateTensor( - memory_info_, - const_cast(random_seed.data()), - random_seed.size(), - scalar_shape.data(), - scalar_shape.size() - )); - input_tensors.push_back(Ort::Value::CreateTensor( - memory_info_, - const_cast(has_specific_target.data()), - has_specific_target.size(), - has_target_shape.data(), - has_target_shape.size() - )); - input_tensors.push_back(Ort::Value::CreateTensor( - memory_info_, - const_cast(specific_target_positions.data()), - specific_target_positions.size(), - specific_positions_shape.data(), - specific_positions_shape.size() - )); - input_tensors.push_back(Ort::Value::CreateTensor( - memory_info_, - const_cast(specific_target_headings.data()), - specific_target_headings.size(), - specific_headings_shape.data(), - specific_headings_shape.size() - )); - input_tensors.push_back(Ort::Value::CreateTensor( - memory_info_, - const_cast(allowed_pred_num_tokens.data()), - allowed_pred_num_tokens.size(), - allowed_tokens_shape.data(), - allowed_tokens_shape.size() - )); - input_tensors.push_back(Ort::Value::CreateTensor( - memory_info_, - const_cast(height.data()), - height.size(), - scalar_shape.data(), - scalar_shape.size() - )); - - static constexpr const char* kPlannerInputNames[] = { - "context_mujoco_qpos", - "target_vel", - "mode", - "movement_direction", - "facing_direction", - "random_seed", - "has_specific_target", - "specific_target_positions", - "specific_target_headings", - "allowed_pred_num_tokens", - "height", - }; - static constexpr const char* kPlannerOutputNames[] = { - "mujoco_qpos", - "num_pred_frames", - }; - - auto outputs = planner_.Run( - Ort::RunOptions{nullptr}, - kPlannerInputNames, - input_tensors.data(), - input_tensors.size(), - kPlannerOutputNames, - std::size(kPlannerOutputNames) - ); - - const Ort::Value& mujoco_qpos = outputs[0]; - const Ort::Value& num_pred_frames = outputs[1]; - const auto qpos_info = mujoco_qpos.GetTensorTypeAndShapeInfo(); - const auto qpos_shape = qpos_info.GetShape(); - const std::size_t available_frames = qpos_info.GetElementCount() / kPlannerQposDim; - const float* qpos_data = mujoco_qpos.GetTensorData(); - - std::int64_t predicted_frames = 1; - const auto frames_info = num_pred_frames.GetTensorTypeAndShapeInfo(); - if (frames_info.GetElementType() == ONNX_TENSOR_ELEMENT_DATA_TYPE_INT32) { - predicted_frames = - static_cast(num_pred_frames.GetTensorData()[0]); - } else if (frames_info.GetElementType() == ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64) { - predicted_frames = num_pred_frames.GetTensorData()[0]; - } else { - throw std::runtime_error("planner num_pred_frames output must be int32 or int64"); - } - predicted_frames = std::max(predicted_frames, 1); - const std::size_t frame_count = - std::min(static_cast(predicted_frames), available_frames); - - if (!(qpos_shape.size() == 3 || qpos_shape.size() == 2)) { - throw std::runtime_error("unexpected planner mujoco_qpos rank"); - } - - std::vector> trajectory; - trajectory.reserve(frame_count); - for (std::size_t frame = 0; frame < frame_count; ++frame) { - const float* begin = qpos_data + frame * kPlannerQposDim; - trajectory.emplace_back( - begin, - begin + static_cast(kPlannerQposDim) - ); - } - return trajectory; - } - - static std::array planner_frame_root_quaternion(const std::vector& frame) { - return quat_unit_d({ - static_cast(frame[3]), - static_cast(frame[4]), - static_cast(frame[5]), - static_cast(frame[6]), - }); - } - - static std::vector interpolate_planner_qpos( - const std::vector& frame_a, - const std::vector& frame_b, - float alpha - ) { - std::vector frame(frame_a.size(), 0.0f); - for (std::size_t index = 0; index < frame.size(); ++index) { - frame[index] = frame_a[index] + alpha * (frame_b[index] - frame_a[index]); - } - - const auto quat = quat_slerp_d( - planner_frame_root_quaternion(frame_a), - planner_frame_root_quaternion(frame_b), - static_cast(alpha) - ); - frame[3] = static_cast(quat[0]); - frame[4] = static_cast(quat[1]); - frame[5] = static_cast(quat[2]); - frame[6] = static_cast(quat[3]); - return frame; - } - - static std::vector sample_motion_qpos_50hz( - const std::vector>& motion_qpos, - float frame_idx - ) { - if (motion_qpos.empty()) { - return {}; - } - if (motion_qpos.size() == 1) { - return motion_qpos.front(); - } - - const float clamped = std::clamp( - frame_idx, - 0.0f, - static_cast(motion_qpos.size() - 1) - ); - const auto frame_a_idx = static_cast(std::floor(clamped)); - const auto frame_b_idx = std::min(frame_a_idx + 1, motion_qpos.size() - 1); - const float alpha = clamped - static_cast(frame_a_idx); - return interpolate_planner_qpos( - motion_qpos[frame_a_idx], - motion_qpos[frame_b_idx], - alpha - ); - } - - static std::vector> resample_planner_trajectory_to_50hz( - const std::vector>& trajectory - ) { - if (trajectory.empty()) { - return {}; - } - - const float motion_seconds = static_cast(trajectory.size()) / 30.0f; - const auto frame_count = std::max( - static_cast(std::floor(motion_seconds * 50.0f)), - static_cast(1) - ); - std::vector> motion_qpos; - motion_qpos.reserve(frame_count); - for (std::size_t frame_50hz = 0; frame_50hz < frame_count; ++frame_50hz) { - const float frame_30hz = static_cast(frame_50hz) * 30.0f / 50.0f; - const auto frame_a_idx = static_cast(std::floor(frame_30hz)); - const auto frame_b_idx = std::min(frame_a_idx + 1, trajectory.size() - 1); - const float alpha = frame_30hz - static_cast(frame_a_idx); - motion_qpos.push_back(interpolate_planner_qpos( - trajectory[frame_a_idx], - trajectory[frame_b_idx], - alpha - )); - } - return motion_qpos; - } - - static std::deque> rebuild_planner_context_from_motion( - const std::vector>& motion_qpos_50hz, - std::size_t current_motion_frame - ) { - std::deque> context; - if (motion_qpos_50hz.empty()) { - return context; - } - - const float gen_time = static_cast(current_motion_frame + 2) * kControlDtSeconds; - for (std::size_t frame_idx = 0; frame_idx < kPlannerContextLen; ++frame_idx) { - const float sample_time = gen_time + static_cast(frame_idx) / 30.0f; - context.push_back(sample_motion_qpos_50hz( - motion_qpos_50hz, - sample_time * 50.0f - )); - } - return context; - } - - static std::vector planner_joint_positions_isaaclab(const std::vector& frame) { - std::vector positions(G1_NUM_MOTOR, 0.0f); - for (std::size_t mujoco_idx = 0; mujoco_idx < G1_NUM_MOTOR; ++mujoco_idx) { - const auto isaaclab_idx = static_cast(isaaclab_to_mujoco[mujoco_idx]); - positions[isaaclab_idx] = frame[kPlannerJointOffset + mujoco_idx]; - } - return positions; - } - - static std::vector> compute_motion_joint_velocities_isaaclab( - const std::vector>& motion_qpos - ) { - if (motion_qpos.empty()) { - return {}; - } - - std::vector> positions; - positions.reserve(motion_qpos.size()); - for (const auto& frame : motion_qpos) { - positions.push_back(planner_joint_positions_isaaclab(frame)); - } - - std::vector> velocities( - positions.size(), - std::vector(G1_NUM_MOTOR, 0.0f) - ); - for (std::size_t frame_idx = 0; frame_idx + 1 < positions.size(); ++frame_idx) { - for (std::size_t joint_idx = 0; joint_idx < G1_NUM_MOTOR; ++joint_idx) { - velocities[frame_idx][joint_idx] = - (positions[frame_idx + 1][joint_idx] - positions[frame_idx][joint_idx]) * 50.0f; - } - } - if (positions.size() > 1) { - velocities.back() = velocities[velocities.size() - 2]; - } - return velocities; - } - - static std::vector> blend_planner_motion( - const std::vector>& existing_motion_qpos, - std::size_t current_motion_frame, - std::size_t request_motion_frame, - const std::vector>& new_motion_qpos - ) { - if (existing_motion_qpos.empty()) { - return new_motion_qpos; - } - - const auto gen_frame = request_motion_frame + kPlannerLookAheadSteps; - const auto lead_frames = - gen_frame > current_motion_frame ? gen_frame - current_motion_frame : 0; - const auto new_anim_length = lead_frames + new_motion_qpos.size(); - const auto blend_start_frame = lead_frames; - - std::vector> blended; - blended.reserve(new_anim_length); - for (std::size_t frame_idx = 0; frame_idx < new_anim_length; ++frame_idx) { - auto old_frame_idx = frame_idx + current_motion_frame; - if (old_frame_idx >= existing_motion_qpos.size()) { - old_frame_idx = existing_motion_qpos.size() - 1; - } - - std::size_t new_frame_idx = 0; - if (frame_idx + current_motion_frame >= gen_frame) { - new_frame_idx = frame_idx + current_motion_frame - gen_frame; - } - if (new_frame_idx >= new_motion_qpos.size()) { - new_frame_idx = new_motion_qpos.size() - 1; - } - - const auto weight_new = std::clamp( - (static_cast(frame_idx) - static_cast(blend_start_frame)) / - static_cast(kPlannerBlendFrames), - 0.0f, - 1.0f - ); - if (weight_new <= std::numeric_limits::epsilon()) { - blended.push_back(existing_motion_qpos[old_frame_idx]); - } else if (std::abs(1.0f - weight_new) <= std::numeric_limits::epsilon()) { - blended.push_back(new_motion_qpos[new_frame_idx]); - } else { - blended.push_back(interpolate_planner_qpos( - existing_motion_qpos[old_frame_idx], - new_motion_qpos[new_frame_idx], - weight_new - )); - } - } - - return blended; - } - - static std::vector build_velocity_encoder_obs_dict( - const std::vector>& motion_qpos_50hz, - const std::vector>& motion_joint_velocities_isaaclab, - std::size_t current_motion_frame, - const std::array& base_quat_wxyz, - const std::array& init_base_quat_wxyz, - const std::array& init_ref_root_quat_wxyz - ) { - if (motion_qpos_50hz.empty()) { - throw std::runtime_error("planner motion buffer is empty"); - } - - const auto apply_delta_heading = quat_mul_d( - calc_heading_quat_d(init_base_quat_wxyz), - calc_heading_quat_inv_d(init_ref_root_quat_wxyz) - ); - const auto base_quat = quat_unit_d(base_quat_wxyz); - - std::vector buf(kEncoderObsDictDim, 0.0f); - buf[kEncoderModeOffset] = 0.0f; - - for (std::size_t frame_idx = 0; frame_idx < kReferenceFutureFrames; ++frame_idx) { - const auto target_frame = std::min( - current_motion_frame + frame_idx * kReferenceFrameStep, - motion_qpos_50hz.size() - 1 - ); - const auto& motion_frame = motion_qpos_50hz[target_frame]; - const auto isaaclab_positions = planner_joint_positions_isaaclab(motion_frame); - - const auto pos_offset = - kEncoderMotionJointPositionsOffset + frame_idx * isaaclab_positions.size(); - std::copy( - isaaclab_positions.begin(), - isaaclab_positions.end(), - buf.begin() + static_cast(pos_offset) - ); - - if (target_frame < motion_joint_velocities_isaaclab.size()) { - const auto& joint_velocities = motion_joint_velocities_isaaclab[target_frame]; - const auto vel_offset = - kEncoderMotionJointVelocitiesOffset + frame_idx * joint_velocities.size(); - std::copy( - joint_velocities.begin(), - joint_velocities.end(), - buf.begin() + static_cast(vel_offset) - ); - } - - const auto ref_root_quat = - quat_mul_d(apply_delta_heading, planner_frame_root_quaternion(motion_frame)); - const auto base_to_ref = quat_mul_d(quat_conjugate_d(base_quat), ref_root_quat); - const auto rotation_matrix = quat_to_rotation_matrix_d(base_to_ref); - const auto orn_offset = kEncoderMotionAnchorOrientationOffset + frame_idx * 6; - buf[orn_offset] = static_cast(rotation_matrix[0][0]); - buf[orn_offset + 1] = static_cast(rotation_matrix[0][1]); - buf[orn_offset + 2] = static_cast(rotation_matrix[1][0]); - buf[orn_offset + 3] = static_cast(rotation_matrix[1][1]); - buf[orn_offset + 4] = static_cast(rotation_matrix[2][0]); - buf[orn_offset + 5] = static_cast(rotation_matrix[2][1]); - } - - return buf; - } - - static Observation motion_observation_from_planner_frame( - const std::vector>& motion_qpos_50hz, - const std::vector>& motion_joint_velocities_isaaclab, - std::size_t frame_idx - ) { - if (motion_qpos_50hz.empty()) { - throw std::runtime_error("planner motion buffer is empty"); - } - - const auto clamped_index = std::min(frame_idx, motion_qpos_50hz.size() - 1); - const auto& motion_frame = motion_qpos_50hz[clamped_index]; - Observation obs; - obs.joint_positions.assign( - motion_frame.begin() + static_cast(kPlannerJointOffset), - motion_frame.end() - ); - obs.joint_velocities = - clamped_index < motion_joint_velocities_isaaclab.size() - ? isaaclab_to_mujoco_values(motion_joint_velocities_isaaclab[clamped_index]) - : std::vector(G1_NUM_MOTOR, 0.0f); - obs.base_quat_wxyz = planner_frame_root_quaternion(motion_frame); - obs.gravity_vector = double_to_float(GetGravityOrientation_d(obs.base_quat_wxyz)); - obs.angular_velocity = - angular_velocity_from_motion_quaternions(motion_qpos_50hz, clamped_index); - return obs; - } - - static std::array angular_velocity_from_motion_quaternions( - const std::vector>& motion_qpos_50hz, - std::size_t frame_idx - ) { - if (frame_idx == 0 || motion_qpos_50hz.empty()) { - return {0.0f, 0.0f, 0.0f}; - } - - const auto prev_quat = planner_frame_root_quaternion(motion_qpos_50hz[frame_idx - 1]); - const auto curr_quat = planner_frame_root_quaternion(motion_qpos_50hz[frame_idx]); - auto delta = quat_mul_d(quat_conjugate_d(prev_quat), curr_quat); - if (delta[0] < 0.0) { - delta = {-delta[0], -delta[1], -delta[2], -delta[3]}; - } - delta = quat_unit_d(delta); - - const auto sin_half = std::sqrt( - delta[1] * delta[1] + delta[2] * delta[2] + delta[3] * delta[3] - ); - if (sin_half <= 1e-6) { - return {0.0f, 0.0f, 0.0f}; - } - - const auto [angle, axis] = quat_to_angle_axis(delta); - if (!std::isfinite(angle)) { - return {0.0f, 0.0f, 0.0f}; - } - - return { - static_cast(axis[0] * angle / kControlDtSeconds), - static_cast(axis[1] * angle / kControlDtSeconds), - static_cast(axis[2] * angle / kControlDtSeconds), - }; - } - - std::vector build_encoder_obs_dict() const { - std::vector buf(kEncoderObsDictDim, 0.0f); - - const auto pose = mujoco_to_isaaclab_positions(default_pose()); - const std::size_t pos_offset = 4; - const std::size_t vel_offset = 294; - const std::size_t orn_offset = 601; - - for (std::size_t frame = 0; frame < kDecoderHistoryLen; ++frame) { - const std::size_t pose_index = pos_offset + frame * G1_NUM_MOTOR; - const std::size_t velocity_index = vel_offset + frame * G1_NUM_MOTOR; - std::copy(pose.begin(), pose.end(), buf.begin() + static_cast(pose_index)); - std::fill_n(buf.begin() + static_cast(velocity_index), G1_NUM_MOTOR, 0.0f); - } - - for (std::size_t frame = 0; frame < kDecoderHistoryLen; ++frame) { - const std::size_t orientation_index = orn_offset + frame * 6; - buf[orientation_index] = 1.0f; - buf[orientation_index + 4] = 1.0f; - } - - return buf; - } - - std::vector build_decoder_obs_dict(const std::vector& tokens) const { - std::vector buf; - buf.reserve(kDecoderObsDictDim); - buf.insert(buf.end(), tokens.begin(), tokens.end()); - - const std::size_t history_skip = tracking_state_.gravity.size() > kDecoderHistoryLen - ? tracking_state_.gravity.size() - kDecoderHistoryLen - : 0; - - append_history_vectors(buf, tracking_state_.angular_velocity, history_skip); - append_history_vectors(buf, tracking_state_.joint_positions, history_skip); - append_history_vectors(buf, tracking_state_.joint_velocities, history_skip); - append_history_vectors(buf, tracking_state_.last_actions, history_skip); - append_history_vectors(buf, tracking_state_.gravity, history_skip); - - return buf; - } - - template - static void append_history_vectors( - std::vector& destination, - const std::deque& history, - std::size_t skip - ) { - for (std::size_t index = skip; index < history.size(); ++index) { - destination.insert(destination.end(), history[index].begin(), history[index].end()); - } - } - - std::vector run_single_f32( - Ort::Session& session, - const char* input_name, - const char* output_name, - const std::vector& input, - std::size_t expected_input_dim - ) { - if (input.size() != expected_input_dim) { - throw std::runtime_error( - "input dimension mismatch for " + std::string(output_name) + - ": expected " + std::to_string(expected_input_dim) + - ", got " + std::to_string(input.size()) - ); - } - - const std::array shape = {1, static_cast(expected_input_dim)}; - auto input_tensor = Ort::Value::CreateTensor( - memory_info_, - const_cast(input.data()), - input.size(), - shape.data(), - shape.size() - ); - - const char* input_names[] = {input_name}; - const char* output_names[] = {output_name}; - auto outputs = session.Run( - Ort::RunOptions{nullptr}, - input_names, - &input_tensor, - 1, - output_names, - 1 - ); - - const Ort::Value& output = outputs.front(); - const auto info = output.GetTensorTypeAndShapeInfo(); - const std::size_t count = info.GetElementCount(); - const float* data = output.GetTensorData(); - return std::vector(data, data + static_cast(count)); - } - - static void write_vector_json(const fs::path& path, const std::vector& values) { - fs::create_directories(path.parent_path()); - std::ofstream out(path); - if (!out) { - throw std::runtime_error("failed to open tensor dump file: " + path.string()); - } - - out << "[\n"; - for (std::size_t index = 0; index < values.size(); ++index) { - out << " " << values[index]; - if (index + 1 < values.size()) { - out << ","; - } - out << "\n"; - } - out << "]\n"; - } - - static void write_matrix_json( - const fs::path& path, - const std::vector>& rows - ) { - fs::create_directories(path.parent_path()); - std::ofstream out(path); - if (!out) { - throw std::runtime_error("failed to open tensor dump file: " + path.string()); - } - - out << "[\n"; - for (std::size_t row_index = 0; row_index < rows.size(); ++row_index) { - out << " [\n"; - for (std::size_t value_index = 0; value_index < rows[row_index].size(); ++value_index) { - out << " " << rows[row_index][value_index]; - if (value_index + 1 < rows[row_index].size()) { - out << ","; - } - out << "\n"; - } - out << " ]"; - if (row_index + 1 < rows.size()) { - out << ","; - } - out << "\n"; - } - out << "]\n"; - } - - static std::vector> deque_to_rows( - const std::deque>& rows - ) { - return {rows.begin(), rows.end()}; - } - - template - static std::vector> deque_to_rows( - const std::deque>& rows - ) { - std::vector> out; - out.reserve(rows.size()); - for (const auto& row : rows) { - out.emplace_back(row.begin(), row.end()); - } - return out; - } - - void maybe_dump_tracking_tensors( - const std::vector& encoder_obs, - const std::vector& tokens, - const std::vector& decoder_obs, - const std::vector& raw_actions - ) { - if (!dump_dir_.has_value() || dumped_tracking_tensors_) { - return; - } - - write_vector_json(*dump_dir_ / "tracking_encoder_obs.json", encoder_obs); - write_vector_json(*dump_dir_ / "tracking_tokens.json", tokens); - write_vector_json(*dump_dir_ / "tracking_decoder_obs.json", decoder_obs); - write_vector_json(*dump_dir_ / "tracking_raw_actions.json", raw_actions); - dumped_tracking_tensors_ = true; - } -}; - -enum class CaseKind { - PlannerOnlyColdStart, - PlannerOnlySteadyState, - EncoderDecoderOnlyTrackingTick, - FullVelocityTickColdStart, - FullVelocityTickSteadyState, - FullVelocityTickReplanBoundary, - FirstLiveReplanDump, - LaterMotionDump, - EndToEndLoop, -}; - -CaseKind parse_case_kind(const std::string& case_id) { - if (case_id == "gear_sonic/planner_only_cold_start") { - return CaseKind::PlannerOnlyColdStart; - } - if (case_id == "gear_sonic/planner_only_steady_state") { - return CaseKind::PlannerOnlySteadyState; - } - if (case_id == "gear_sonic/encoder_decoder_only_tracking_tick") { - return CaseKind::EncoderDecoderOnlyTrackingTick; - } - if (case_id == "gear_sonic/full_velocity_tick_cold_start") { - return CaseKind::FullVelocityTickColdStart; - } - if (case_id == "gear_sonic/full_velocity_tick_steady_state") { - return CaseKind::FullVelocityTickSteadyState; - } - if (case_id == "gear_sonic/full_velocity_tick_replan_boundary") { - return CaseKind::FullVelocityTickReplanBoundary; - } - if (case_id == "gear_sonic_velocity/first_live_replan_dump") { - return CaseKind::FirstLiveReplanDump; - } - if (case_id == "gear_sonic_velocity/later_motion_dump") { - return CaseKind::LaterMotionDump; - } - if (case_id == "gear_sonic/end_to_end_cli_loop") { - return CaseKind::EndToEndLoop; - } - throw std::runtime_error("unsupported GEAR-Sonic case_id: " + case_id); -} - -template -std::vector run_microbench( - GearSonicOfficialHarness& harness, - int samples, - SetupFn&& setup, - MeasureFn&& measure -) { - std::vector timings; - timings.reserve(static_cast(samples)); - for (int sample = 0; sample < samples; ++sample) { - harness.reset(); - setup(); - const auto start = std::chrono::steady_clock::now(); - const auto result = measure(); - const auto end = std::chrono::steady_clock::now(); - g_sink = g_sink + (result.empty() ? 0.0f : result.front()); - timings.push_back( - static_cast( - std::chrono::duration_cast(end - start).count() - ) - ); - } - return timings; -} - -template -std::pair, double> run_end_to_end_loop( - GearSonicOfficialHarness& harness, - int ticks, - int control_frequency_hz, - Fn&& fn -) { - const auto period_ns = static_cast(std::llround(1'000'000'000.0 / static_cast(control_frequency_hz))); - std::vector timings; - timings.reserve(static_cast(ticks)); - - harness.reset(); - const auto wall_start = std::chrono::steady_clock::now(); - for (int tick = 0; tick < ticks; ++tick) { - const auto start = std::chrono::steady_clock::now(); - const auto result = fn(); - const auto end = std::chrono::steady_clock::now(); - g_sink = g_sink + (result.empty() ? 0.0f : result.front()); - - const auto elapsed_ns = - std::chrono::duration_cast(end - start).count(); - timings.push_back(static_cast(elapsed_ns)); - - const auto remaining_ns = period_ns - elapsed_ns; - if (remaining_ns > 0) { - std::this_thread::sleep_for(std::chrono::nanoseconds(remaining_ns)); - } - } - const auto wall_end = std::chrono::steady_clock::now(); - const auto elapsed_seconds = std::chrono::duration(wall_end - wall_start).count(); - const double achieved_hz = elapsed_seconds > 0.0 ? static_cast(ticks) / elapsed_seconds : 0.0; - return {timings, achieved_hz}; -} - -ProviderKind parse_provider(const std::string& value) { - if (value == "cpu") { - return ProviderKind::Cpu; - } - if (value == "cuda") { - return ProviderKind::Cuda; - } - if (value == "tensor_rt") { - return ProviderKind::TensorRt; - } - throw std::runtime_error( - std::string("unsupported provider `") + value - + "`; expected one of: cpu, cuda, tensor_rt" - ); -} - -Options parse_args(int argc, char** argv) { - Options options; - for (int index = 1; index < argc; ++index) { - const std::string arg = argv[index]; - auto require_value = [&](const std::string& flag) -> std::string { - if (index + 1 >= argc) { - throw std::runtime_error(flag + " requires a value"); - } - return argv[++index]; - }; - - if (arg == "--case-id") { - options.case_id = require_value(arg); - } else if (arg == "--provider") { - options.provider = parse_provider(require_value(arg)); - } else if (arg == "--model-dir") { - options.model_dir = require_value(arg); - } else if (arg == "--output") { - options.output = require_value(arg); - } else if (arg == "--dump-dir") { - options.dump_dir = fs::path(require_value(arg)); - } else if (arg == "--samples") { - options.samples = std::stoi(require_value(arg)); - } else if (arg == "--ticks") { - options.ticks = std::stoi(require_value(arg)); - } else if (arg == "--control-frequency-hz") { - options.control_frequency_hz = std::stoi(require_value(arg)); - } else { - throw std::runtime_error("unknown argument: " + arg); - } - } - - if (options.case_id.empty()) { - throw std::runtime_error("--case-id is required"); - } - if (options.model_dir.empty()) { - throw std::runtime_error("--model-dir is required"); - } - if (options.output.empty()) { - throw std::runtime_error("--output is required"); - } - if (options.samples <= 0) { - throw std::runtime_error("--samples must be positive"); - } - if (options.ticks <= 0) { - throw std::runtime_error("--ticks must be positive"); - } - if (options.control_frequency_hz <= 0) { - throw std::runtime_error("--control-frequency-hz must be positive"); - } - return options; -} - -void write_json( - const Options& options, - const std::vector& samples_ns, - const std::optional& hz -) { - fs::create_directories(options.output.parent_path()); - std::ofstream out(options.output); - if (!out) { - throw std::runtime_error("failed to open output file: " + options.output.string()); - } - - out << "{\n"; - out << " \"case_id\": \"" << json_escape(options.case_id) << "\",\n"; - out << " \"samples_ns\": ["; - for (std::size_t index = 0; index < samples_ns.size(); ++index) { - if (index == 0) { - out << "\n "; - } else { - out << ",\n "; - } - out << samples_ns[index]; - } - if (!samples_ns.empty()) { - out << '\n'; - } - out << " ],\n"; - out << " \"hz\": "; - if (hz.has_value()) { - out << *hz; - } else { - out << "null"; - } - out << ",\n"; - out << " \"notes\": \"Measured via official GEAR-Sonic C++ ONNX Runtime harness on the published planner and encoder/decoder contracts.\"\n"; - out << "}\n"; -} - -} // namespace - -int main(int argc, char** argv) { - try { - const Options options = parse_args(argc, argv); - const CaseKind case_kind = parse_case_kind(options.case_id); - GearSonicOfficialHarness harness(options.model_dir, options.provider, options.dump_dir); - - std::vector samples_ns; - std::optional hz; - - switch (case_kind) { - case CaseKind::PlannerOnlyColdStart: - samples_ns = run_microbench( - harness, - options.samples, - []() {}, - [&]() { return harness.planner_only_tick(); } - ); - break; - case CaseKind::PlannerOnlySteadyState: - samples_ns = run_microbench( - harness, - options.samples, - [&]() { harness.planner_only_tick(); }, - [&]() { return harness.planner_only_tick(); } - ); - break; - case CaseKind::EncoderDecoderOnlyTrackingTick: - samples_ns = run_microbench( - harness, - options.samples, - []() {}, - [&]() { return harness.tracking_tick(); } - ); - break; - case CaseKind::FullVelocityTickColdStart: - samples_ns = run_microbench( - harness, - options.samples, - []() {}, - [&]() { return harness.velocity_tick(); } - ); - break; - case CaseKind::FullVelocityTickSteadyState: - samples_ns = run_microbench( - harness, - options.samples, - [&]() { harness.velocity_tick(); }, - [&]() { return harness.velocity_tick(); } - ); - break; - case CaseKind::FullVelocityTickReplanBoundary: - samples_ns = run_microbench( - harness, - options.samples, - [&]() { - for (std::size_t tick = 0; tick < kReplanIntervalTicksDefault; ++tick) { - harness.velocity_tick(); - } - }, - [&]() { return harness.velocity_tick(); } - ); - break; - case CaseKind::FirstLiveReplanDump: { - const auto start = std::chrono::steady_clock::now(); - const auto result = harness.velocity_first_live_replan_dump(); - const auto end = std::chrono::steady_clock::now(); - g_sink = g_sink + (result.empty() ? 0.0f : result.front()); - samples_ns.push_back( - static_cast( - std::chrono::duration_cast(end - start).count() - ) - ); - break; - } - case CaseKind::LaterMotionDump: { - const auto start = std::chrono::steady_clock::now(); - const auto result = harness.velocity_later_motion_dump(); - const auto end = std::chrono::steady_clock::now(); - g_sink = g_sink + (result.empty() ? 0.0f : result.front()); - samples_ns.push_back( - static_cast( - std::chrono::duration_cast(end - start).count() - ) - ); - break; - } - case CaseKind::EndToEndLoop: { - auto [timings, achieved_hz] = run_end_to_end_loop( - harness, - options.ticks, - options.control_frequency_hz, - [&]() { return harness.velocity_tick(); } - ); - samples_ns = std::move(timings); - hz = achieved_hz; - break; - } - } - - write_json(options, samples_ns, hz); - return 0; - } catch (const std::exception& error) { - std::cerr << "error: " << error.what() << '\n'; - return 1; - } -} +#include "benchmarks/bench_nvidia_gear_sonic_official.cpp" diff --git a/scripts/bench_nvidia_official.py b/scripts/bench_nvidia_official.py old mode 100644 new mode 100755 index ef903f4..1736934 --- a/scripts/bench_nvidia_official.py +++ b/scripts/bench_nvidia_official.py @@ -1,633 +1,4 @@ #!/usr/bin/env python3 -"""Run the official NVIDIA comparison cases and emit normalized artifacts.""" +from _compat import run_legacy_script -from __future__ import annotations - -import argparse -import importlib.util -import os -import shlex -import subprocess -import sys -from pathlib import Path -from typing import Any - - -ROOT_DIR = Path(__file__).resolve().parents[1] -REGISTRY_PATH = ROOT_DIR / "benchmarks/nvidia/cases.json" -NORMALIZER_PATH = ROOT_DIR / "scripts/normalize_nvidia_benchmarks.py" -DECOUPLED_HARNESS = ROOT_DIR / "scripts/bench_nvidia_decoupled_official.py" -GEAR_SONIC_HARNESS_SRC = ROOT_DIR / "scripts/bench_nvidia_gear_sonic_official.cpp" -DEFAULT_GEAR_SONIC_HARNESS_BIN = ( - ROOT_DIR / "target/nvidia-bench/bench_nvidia_gear_sonic_official" -) -DEFAULT_OFFICIAL_REPO_DIR = ROOT_DIR / "third_party/GR00T-WholeBodyControl" -IMPLEMENTATION_ID = "ort-cpp-sonic" -DEFAULT_OFFICIAL_OUTPUT_ROOT = ROOT_DIR / "artifacts/benchmarks/nvidia" / IMPLEMENTATION_ID -DEFAULT_DECOUPLED_MODEL_DIR = Path( - os.environ.get("DECOUPLED_WBC_MODEL_DIR", str(ROOT_DIR / "models/decoupled-wbc")) -) -DEFAULT_GEAR_SONIC_MODEL_DIR = Path( - os.environ.get("GEAR_SONIC_MODEL_DIR", str(ROOT_DIR / "models/gear-sonic")) -) -PROVIDER_CHOICES = ("cpu", "cuda", "tensor_rt") - - -def load_normalizer() -> Any: - spec = importlib.util.spec_from_file_location("normalize_nvidia_benchmarks", NORMALIZER_PATH) - if spec is None or spec.loader is None: - raise RuntimeError(f"failed to load normalizer module from {NORMALIZER_PATH}") - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - return module - - -NORMALIZER = load_normalizer() -REGISTRY = NORMALIZER.load_registry(REGISTRY_PATH) - - -def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser(description=__doc__) - mode = parser.add_mutually_exclusive_group(required=True) - mode.add_argument("--list-cases", action="store_true") - mode.add_argument("--case") - mode.add_argument("--all", action="store_true") - parser.add_argument( - "--output-root", - type=Path, - default=DEFAULT_OFFICIAL_OUTPUT_ROOT, - help="Base directory for normalized artifacts; provider subdirectories are created automatically", - ) - parser.add_argument( - "--provider", - default="cpu", - choices=PROVIDER_CHOICES, - help="Execution provider requested for the benchmark row", - ) - parser.add_argument( - "--repo-dir", - type=Path, - default=DEFAULT_OFFICIAL_REPO_DIR, - help="Pinned GR00T-WholeBodyControl source checkout", - ) - parser.add_argument( - "--samples", - type=int, - default=100, - help="Microbenchmark samples for official harnesses", - ) - parser.add_argument( - "--ticks", - type=int, - default=200, - help="End-to-end loop ticks for official harnesses", - ) - parser.add_argument( - "--control-frequency-hz", - type=int, - default=50, - help="End-to-end control frequency label for official harnesses", - ) - return parser.parse_args() - - -def list_cases() -> None: - for case in REGISTRY["cases"]: - print(case["case_id"]) - - -def relative_to_root(path: Path) -> str: - try: - return str(path.relative_to(ROOT_DIR)) - except ValueError: - return str(path) - - -def provider_output_root(output_root: Path, provider: str) -> Path: - return output_root / provider - - -def run( - argv: list[str], - *, - cwd: Path | None = None, - capture_output: bool = True, -) -> subprocess.CompletedProcess[str]: - return subprocess.run( - argv, - cwd=cwd or ROOT_DIR, - check=True, - text=True, - capture_output=capture_output, - ) - - -def git_rev_parse(repo_dir: Path) -> str | None: - try: - result = run( - ["git", "-C", str(repo_dir), "rev-parse", "HEAD"], - capture_output=True, - ) - except (subprocess.CalledProcessError, FileNotFoundError): - return None - return result.stdout.strip() - - -def repo_checkout_blocker(repo_dir: Path) -> str: - if repo_dir == DEFAULT_OFFICIAL_REPO_DIR: - rel = relative_to_root(repo_dir) - return ( - f"Official NVIDIA source checkout is unavailable at {repo_dir}; " - f"run `git submodule update --init --recursive {rel}` first." - ) - return ( - f"Official NVIDIA source checkout is unavailable at {repo_dir}; " - "pass --repo-dir ." - ) - - -def have_decoupled_models(model_dir: Path) -> bool: - return ( - (model_dir / "GR00T-WholeBodyControl-Walk.onnx").is_file() - and (model_dir / "GR00T-WholeBodyControl-Balance.onnx").is_file() - ) - - -def have_gear_sonic_models(model_dir: Path) -> bool: - return ( - (model_dir / "planner_sonic.onnx").is_file() - and (model_dir / "model_encoder.onnx").is_file() - and (model_dir / "model_decoder.onnx").is_file() - ) - - -def configured_onnxruntime_root() -> Path | None: - for env_name in ("ROBOWBC_ORT_DYLIB_PATH", "ORT_DYLIB_PATH"): - configured = os.environ.get(env_name) - if not configured: - continue - dylib_path = Path(configured).expanduser().resolve() - if not dylib_path.is_file(): - continue - root = dylib_path.parent.parent - if (root / "include" / "onnxruntime_cxx_api.h").is_file(): - return root - return None - - -def discover_onnxruntime_root() -> Path | None: - configured_root = configured_onnxruntime_root() - if configured_root is not None: - return configured_root - - build_root = ROOT_DIR / "target/debug/build" - candidates = sorted( - path - for path in build_root.glob("**/onnxruntime-linux-x64-*") - if path.is_dir() - ) - if not candidates: - return None - return candidates[-1] - - -def gear_sonic_harness_bin() -> Path: - configured = os.environ.get("GEAR_SONIC_OFFICIAL_HARNESS_BIN") - if not configured: - return DEFAULT_GEAR_SONIC_HARNESS_BIN - return Path(configured).expanduser().resolve() - - -def describe_process_failure(error: subprocess.CalledProcessError) -> str: - chunks = [chunk.strip() for chunk in (error.stdout, error.stderr) if chunk and chunk.strip()] - if not chunks: - return str(error) - combined = "\n".join(chunks) - lines = combined.splitlines() - if len(lines) > 40: - combined = "\n".join(lines[-40:]) - return combined - - -def blocked_reason_for_provider(case_id: str, provider: str) -> str | None: - if provider != "cpu" and case_id.startswith("decoupled_wbc/"): - return ( - f"{case_id} stays CPU-only in this phase. Provider `{provider}` is not wired on " - "both benchmark implementations for Decoupled WBC, so the row is blocked instead of " - "quietly relabeling a CPU measurement." - ) - return None - - -def emit_blocked( - *, - case_id: str, - provider: str, - upstream_commit: str | None, - robowbc_commit: str, - reason: str, - output_root: Path, - source_command: str, -) -> None: - case = NORMALIZER.registry_case(REGISTRY, case_id) - artifact = NORMALIZER.build_artifact( - case=case, - implementation=IMPLEMENTATION_ID, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - provider=provider, - host_fingerprint=None, - samples_ns=[], - hz=None, - notes=reason, - source_command=source_command, - raw_source=reason, - status="blocked", - ) - output_path = output_root / f"{case_id.replace('/', '__')}.json" - NORMALIZER.dump_json(output_path, artifact) - print(f"[blocked] {case_id} -> {output_path}") - - -def normalize_manual_case( - *, - case_id: str, - provider: str, - upstream_commit: str | None, - robowbc_commit: str, - input_path: Path, - notes: str, - output_root: Path, - source_command: str, -) -> None: - case = NORMALIZER.registry_case(REGISTRY, case_id) - samples_ns, hz, raw_source = NORMALIZER.manual_samples_payload(input_path) - artifact = NORMALIZER.build_artifact( - case=case, - implementation=IMPLEMENTATION_ID, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - provider=provider, - host_fingerprint=None, - samples_ns=samples_ns, - hz=hz, - notes=notes, - source_command=source_command, - raw_source=raw_source, - status="ok", - ) - output_path = output_root / f"{case_id.replace('/', '__')}.json" - NORMALIZER.dump_json(output_path, artifact) - print(f"[ok] {case_id} -> {output_path}") - - -def ensure_gear_sonic_harness(repo_dir: Path, output_root: Path) -> str | None: - harness_bin = gear_sonic_harness_bin() - if "GEAR_SONIC_OFFICIAL_HARNESS_BIN" in os.environ: - if harness_bin.is_file(): - return None - return f"Configured GEAR_SONIC_OFFICIAL_HARNESS_BIN does not exist: {harness_bin}" - - upstream_include = ( - repo_dir / "gear_sonic_deploy/src/g1/g1_deploy_onnx_ref/include" - ) - policy_header = upstream_include / "policy_parameters.hpp" - robot_header = upstream_include / "robot_parameters.hpp" - if not policy_header.is_file() or not robot_header.is_file(): - return ( - f"Official GEAR-Sonic reference headers are missing under {upstream_include}; " - "the pinned source checkout does not expose the required C++ contract." - ) - - ort_root = discover_onnxruntime_root() - if ort_root is None: - return ( - "ONNX Runtime headers/libs were not found under " - f"{ROOT_DIR / 'target/debug/build'} and no compatible ROBOWBC_ORT_DYLIB_PATH / " - "ORT_DYLIB_PATH override was detected; run `cargo build` first or point the " - "benchmark wrapper at a full ONNX Runtime distribution." - ) - - inputs = [GEAR_SONIC_HARNESS_SRC, policy_header, robot_header] - needs_rebuild = configured_onnxruntime_root() is not None or not harness_bin.exists() - if not needs_rebuild: - built_at = harness_bin.stat().st_mtime_ns - needs_rebuild = any(path.stat().st_mtime_ns > built_at for path in inputs) - if not needs_rebuild: - return None - - build_log = output_root / "raw/gear_sonic_build.log" - build_log.parent.mkdir(parents=True, exist_ok=True) - harness_bin.parent.mkdir(parents=True, exist_ok=True) - compile_cmd = [ - "c++", - "-std=c++20", - "-O3", - "-I", - str(ort_root / "include"), - "-I", - str(upstream_include), - str(GEAR_SONIC_HARNESS_SRC), - "-L", - str(ort_root / "lib"), - f"-Wl,-rpath,{ort_root / 'lib'}", - "-lonnxruntime", - "-lpthread", - "-o", - str(harness_bin), - ] - with build_log.open("w", encoding="utf-8") as handle: - result = subprocess.run( - compile_cmd, - cwd=ROOT_DIR, - stdout=handle, - stderr=subprocess.STDOUT, - text=True, - ) - if result.returncode != 0: - return ( - f"Failed to compile {GEAR_SONIC_HARNESS_SRC} against {ort_root}; " - f"see {build_log} for the compiler output." - ) - return None - - -def run_decoupled_case( - *, - case_id: str, - repo_dir: Path, - model_dir: Path, - output_root: Path, - provider: str, - upstream_commit: str, - robowbc_commit: str, - samples: int, - ticks: int, - control_frequency_hz: int, - source_command: str, -) -> None: - raw_output = output_root / "raw" / f"{case_id.replace('/', '__')}.json" - raw_output.parent.mkdir(parents=True, exist_ok=True) - harness_cmd = [ - "python3", - str(DECOUPLED_HARNESS), - "--case-id", - case_id, - "--repo-dir", - str(repo_dir), - "--model-dir", - str(model_dir), - "--robot-config", - str(ROOT_DIR / "configs/robots/unitree_g1.toml"), - "--samples", - str(samples), - "--ticks", - str(ticks), - "--control-frequency-hz", - str(control_frequency_hz), - "--output", - str(raw_output), - ] - run(harness_cmd) - normalize_manual_case( - case_id=case_id, - provider=provider, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - input_path=raw_output, - notes="Measured via upstream Decoupled WBC headless harness on the pinned source checkout.", - output_root=output_root, - source_command=source_command, - ) - - -def run_gear_sonic_case( - *, - case_id: str, - model_dir: Path, - output_root: Path, - provider: str, - upstream_commit: str, - robowbc_commit: str, - samples: int, - ticks: int, - control_frequency_hz: int, - source_command: str, -) -> None: - harness_bin = gear_sonic_harness_bin() - raw_output = output_root / "raw" / f"{case_id.replace('/', '__')}.json" - raw_output.parent.mkdir(parents=True, exist_ok=True) - harness_cmd = [ - str(harness_bin), - "--case-id", - case_id, - "--provider", - provider, - "--model-dir", - str(model_dir), - "--samples", - str(samples), - "--ticks", - str(ticks), - "--control-frequency-hz", - str(control_frequency_hz), - "--output", - str(raw_output), - ] - run(harness_cmd) - normalize_manual_case( - case_id=case_id, - provider=provider, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - input_path=raw_output, - notes="Measured via the ORT-cpp-sonic GEAR-Sonic C++ ONNX Runtime harness on the pinned source checkout.", - output_root=output_root, - source_command=source_command, - ) - - -def source_command_for_case(args: argparse.Namespace, case_id: str) -> str: - argv = [ - "python3", - "scripts/bench_nvidia_official.py", - "--case", - case_id, - "--provider", - args.provider, - ] - if args.repo_dir != DEFAULT_OFFICIAL_REPO_DIR: - argv.extend(["--repo-dir", str(args.repo_dir)]) - if args.output_root != DEFAULT_OFFICIAL_OUTPUT_ROOT: - argv.extend(["--output-root", str(args.output_root)]) - if args.samples != 100: - argv.extend(["--samples", str(args.samples)]) - if args.ticks != 200: - argv.extend(["--ticks", str(args.ticks)]) - if args.control_frequency_hz != 50: - argv.extend(["--control-frequency-hz", str(args.control_frequency_hz)]) - return shlex.join(argv) - - -def run_case(args: argparse.Namespace, case_id: str, robowbc_commit: str) -> None: - upstream_commit = git_rev_parse(args.repo_dir) - source_command = source_command_for_case(args, case_id) - output_root = provider_output_root(args.output_root, args.provider) - if upstream_commit is None: - emit_blocked( - case_id=case_id, - provider=args.provider, - upstream_commit=None, - robowbc_commit=robowbc_commit, - reason=repo_checkout_blocker(args.repo_dir), - output_root=output_root, - source_command=source_command, - ) - return - - blocked_reason = blocked_reason_for_provider(case_id, args.provider) - if blocked_reason is not None: - emit_blocked( - case_id=case_id, - provider=args.provider, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - reason=blocked_reason, - output_root=output_root, - source_command=source_command, - ) - return - - if case_id.startswith("gear_sonic_") or case_id.startswith("gear_sonic/"): - if not have_gear_sonic_models(DEFAULT_GEAR_SONIC_MODEL_DIR): - emit_blocked( - case_id=case_id, - provider=args.provider, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - reason=( - "Official GEAR-Sonic checkpoints are missing under " - f"{DEFAULT_GEAR_SONIC_MODEL_DIR}; run scripts/download_gear_sonic_models.sh first." - ), - output_root=output_root, - source_command=source_command, - ) - return - build_reason = ensure_gear_sonic_harness(args.repo_dir, output_root) - if build_reason is not None: - emit_blocked( - case_id=case_id, - provider=args.provider, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - reason=build_reason, - output_root=output_root, - source_command=source_command, - ) - return - try: - run_gear_sonic_case( - case_id=case_id, - model_dir=DEFAULT_GEAR_SONIC_MODEL_DIR, - output_root=output_root, - provider=args.provider, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - samples=args.samples, - ticks=args.ticks, - control_frequency_hz=args.control_frequency_hz, - source_command=source_command, - ) - except subprocess.CalledProcessError as error: - emit_blocked( - case_id=case_id, - provider=args.provider, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - reason=( - f"Requested provider `{args.provider}` could not run on the ORT-cpp-sonic " - f"GEAR-Sonic harness. Exact runtime output:\n{describe_process_failure(error)}" - ), - output_root=output_root, - source_command=source_command, - ) - return - - if case_id.startswith("decoupled_wbc/"): - model_dir = DEFAULT_DECOUPLED_MODEL_DIR - if not have_decoupled_models(model_dir): - emit_blocked( - case_id=case_id, - provider=args.provider, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - reason=( - "Official Decoupled WBC models are missing under " - f"{model_dir}; run scripts/download_decoupled_wbc_models.sh first." - ), - output_root=output_root, - source_command=source_command, - ) - return - try: - run_decoupled_case( - case_id=case_id, - repo_dir=args.repo_dir, - model_dir=model_dir, - output_root=output_root, - provider=args.provider, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - samples=args.samples, - ticks=args.ticks, - control_frequency_hz=args.control_frequency_hz, - source_command=source_command, - ) - except subprocess.CalledProcessError as error: - emit_blocked( - case_id=case_id, - provider=args.provider, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - reason=( - f"Requested provider `{args.provider}` could not run on the official " - f"Decoupled WBC harness. Exact runtime output:\n{describe_process_failure(error)}" - ), - output_root=output_root, - source_command=source_command, - ) - return - - emit_blocked( - case_id=case_id, - provider=args.provider, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - reason=f"No official-wrapper mapping has been defined for {case_id}.", - output_root=output_root, - source_command=source_command, - ) - - -def main() -> int: - args = parse_args() - if args.list_cases: - list_cases() - return 0 - - robowbc_commit = run( - ["git", "-C", str(ROOT_DIR), "rev-parse", "HEAD"], - capture_output=True, - ).stdout.strip() - - if args.case is not None: - run_case(args, args.case, robowbc_commit) - return 0 - - for case in REGISTRY["cases"]: - run_case(args, case["case_id"], robowbc_commit) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) +run_legacy_script("benchmarks/bench_nvidia_official.py") diff --git a/scripts/bench_robowbc_compare.py b/scripts/bench_robowbc_compare.py old mode 100644 new mode 100755 index 862f023..8ec7596 --- a/scripts/bench_robowbc_compare.py +++ b/scripts/bench_robowbc_compare.py @@ -1,728 +1,4 @@ #!/usr/bin/env python3 -"""Run the RoboWBC comparison cases and emit normalized artifacts.""" +from _compat import run_legacy_script -from __future__ import annotations - -import argparse -import importlib.util -import os -import shlex -import subprocess -import tempfile -from pathlib import Path -from typing import Any - - -ROOT_DIR = Path(__file__).resolve().parents[1] -REGISTRY_PATH = ROOT_DIR / "benchmarks/nvidia/cases.json" -NORMALIZER_PATH = ROOT_DIR / "scripts/normalize_nvidia_benchmarks.py" -IMPLEMENTATION_ID = "ort-rs" -DEFAULT_OUTPUT_ROOT = ROOT_DIR / "artifacts/benchmarks/nvidia" / IMPLEMENTATION_ID -DEFAULT_MUJOCO_DOWNLOAD_DIR = ROOT_DIR / ".cache" / "mujoco" -DEFAULT_GEAR_SONIC_REVISION = "cc80d505b7e055fd6ae26426ae8bfa0a74c26011" -DEFAULT_DECOUPLED_COMMIT = "bc38f6d0ce6cab4589e025037ad0bfbab7ba73d8" -DEFAULT_GEAR_SONIC_MODEL_DIR = Path( - os.environ.get("GEAR_SONIC_MODEL_DIR", str(ROOT_DIR / "models/gear-sonic")) -) -DEFAULT_DECOUPLED_WBC_MODEL_DIR = Path( - os.environ.get("DECOUPLED_WBC_MODEL_DIR", str(ROOT_DIR / "models/decoupled-wbc")) -) -PROVIDER_CHOICES = ("cpu", "cuda", "tensor_rt") -BENCH_PROVIDER_ENV = "ROBOWBC_BENCH_PROVIDER" -GEAR_SONIC_PROVIDER_SECTIONS = ( - "policy.config.encoder", - "policy.config.decoder", - "policy.config.planner", -) -MICROBENCH_CASES: dict[str, dict[str, str]] = { - "gear_sonic/planner_only_cold_start": { - "criterion_filter": "policy/gear_sonic/planner_only_cold_start", - "family": "gear_sonic", - }, - "gear_sonic/planner_only_steady_state": { - "criterion_filter": "policy/gear_sonic/planner_only_steady_state", - "family": "gear_sonic", - }, - "gear_sonic/encoder_decoder_only_tracking_tick": { - "criterion_filter": "policy/gear_sonic/encoder_decoder_only_tracking_tick", - "family": "gear_sonic", - }, - "gear_sonic/full_velocity_tick_cold_start": { - "criterion_filter": "policy/gear_sonic/full_velocity_tick_cold_start", - "family": "gear_sonic", - }, - "gear_sonic/full_velocity_tick_steady_state": { - "criterion_filter": "policy/gear_sonic/full_velocity_tick_steady_state", - "family": "gear_sonic", - }, - "gear_sonic/full_velocity_tick_replan_boundary": { - "criterion_filter": "policy/gear_sonic/full_velocity_tick_replan_boundary", - "family": "gear_sonic", - }, - "decoupled_wbc/walk_predict": { - "criterion_filter": "policy/decoupled_wbc/walk_predict", - "family": "decoupled_wbc", - }, - "decoupled_wbc/balance_predict": { - "criterion_filter": "policy/decoupled_wbc/balance_predict", - "family": "decoupled_wbc", - }, -} -CLI_CASES: dict[str, dict[str, str]] = { - "gear_sonic/end_to_end_cli_loop": { - "config_path": "configs/sonic_g1.toml", - "family": "gear_sonic", - }, - "decoupled_wbc/end_to_end_cli_loop": { - "config_path": "configs/decoupled_g1.toml", - "family": "decoupled_wbc", - }, -} - - -def load_normalizer() -> Any: - spec = importlib.util.spec_from_file_location("normalize_nvidia_benchmarks", NORMALIZER_PATH) - if spec is None or spec.loader is None: - raise RuntimeError(f"failed to load normalizer module from {NORMALIZER_PATH}") - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - return module - - -NORMALIZER = load_normalizer() -REGISTRY = NORMALIZER.load_registry(REGISTRY_PATH) - - -def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser(description=__doc__) - mode = parser.add_mutually_exclusive_group(required=True) - mode.add_argument("--list-cases", action="store_true") - mode.add_argument("--case") - mode.add_argument("--all", action="store_true") - parser.add_argument( - "--output-root", - type=Path, - default=DEFAULT_OUTPUT_ROOT, - help="Base directory for normalized artifacts; provider subdirectories are created automatically", - ) - parser.add_argument( - "--provider", - default="cpu", - choices=PROVIDER_CHOICES, - help="Execution provider requested for the benchmark row", - ) - return parser.parse_args() - - -def list_cases() -> None: - for case in REGISTRY["cases"]: - print(case["case_id"]) - - -def run( - argv: list[str], - *, - env: dict[str, str] | None = None, -) -> subprocess.CompletedProcess[str]: - return subprocess.run( - argv, - cwd=ROOT_DIR, - env=env, - check=True, - text=True, - capture_output=True, - ) - - -def prepend_env_path(env: dict[str, str], key: str, value: Path) -> None: - current = env.get(key) - env[key] = f"{value}{os.pathsep}{current}" if current else str(value) - - -def resolve_mujoco_download_dir(env: dict[str, str]) -> Path: - configured = env.get("MUJOCO_DOWNLOAD_DIR") - download_dir = ( - Path(configured).expanduser().resolve() - if configured - else DEFAULT_MUJOCO_DOWNLOAD_DIR.resolve() - ) - download_dir.mkdir(parents=True, exist_ok=True) - env["MUJOCO_DOWNLOAD_DIR"] = str(download_dir) - return download_dir - - -def resolve_mujoco_runtime_libdir(env: dict[str, str]) -> Path | None: - download_dir = resolve_mujoco_download_dir(env) - candidates = sorted( - download_dir.glob("mujoco-*/lib/libmujoco.so"), - key=lambda path: path.stat().st_mtime, - reverse=True, - ) - return candidates[0].parent if candidates else None - - -def configure_mujoco_runtime_env(env: dict[str, str]) -> dict[str, str]: - resolve_mujoco_download_dir(env) - libdir = resolve_mujoco_runtime_libdir(env) - if libdir is not None: - prepend_env_path(env, "LD_LIBRARY_PATH", libdir) - return env - - -def git_rev_parse(repo_dir: Path) -> str: - result = subprocess.run( - ["git", "-C", str(repo_dir), "rev-parse", "HEAD"], - cwd=ROOT_DIR, - check=True, - text=True, - capture_output=True, - ) - return result.stdout.strip() - - -def read_revision_file(path: Path, fallback: str) -> str: - if not path.is_file(): - return fallback - revision = path.read_text(encoding="utf-8").strip() - return revision or fallback - - -def gear_sonic_revision(model_dir: Path) -> str: - return read_revision_file(model_dir / "REVISION", DEFAULT_GEAR_SONIC_REVISION) - - -def decoupled_revision(model_dir: Path) -> str: - return read_revision_file(model_dir / "REVISION", DEFAULT_DECOUPLED_COMMIT) - - -def have_gear_sonic_models(model_dir: Path) -> bool: - return ( - (model_dir / "model_encoder.onnx").is_file() - and (model_dir / "model_decoder.onnx").is_file() - and (model_dir / "planner_sonic.onnx").is_file() - ) - - -def have_decoupled_models(model_dir: Path) -> bool: - return ( - (model_dir / "GR00T-WholeBodyControl-Walk.onnx").is_file() - and (model_dir / "GR00T-WholeBodyControl-Balance.onnx").is_file() - ) - - -def output_path_for(case_id: str, output_root: Path) -> Path: - return output_root / f"{case_id.replace('/', '__')}.json" - - -def provider_output_root(output_root: Path, provider: str) -> Path: - return output_root / provider - - -def case_ids() -> set[str]: - return {case["case_id"] for case in REGISTRY["cases"]} - - -def source_command_for_case(args: argparse.Namespace, case_id: str) -> str: - argv = [ - "python3", - "scripts/bench_robowbc_compare.py", - "--case", - case_id, - "--provider", - args.provider, - ] - if args.output_root != DEFAULT_OUTPUT_ROOT: - argv.extend(["--output-root", str(args.output_root)]) - return shlex.join(argv) - - -def provider_inline_table(provider: str) -> str: - if provider == "cpu": - return '{ type = "cpu" }' - return f'{{ type = "{provider}", device_id = 0 }}' - - -def rewrite_gear_sonic_provider_blocks(config_text: str, provider: str) -> str: - lines = config_text.splitlines() - rewritten: list[str] = [] - current_section: str | None = None - replaced_sections: set[str] = set() - for line in lines: - stripped = line.strip() - if stripped.startswith("[") and stripped.endswith("]"): - current_section = stripped[1:-1] - if current_section in GEAR_SONIC_PROVIDER_SECTIONS and stripped.startswith( - "execution_provider" - ): - indent = line[: len(line) - len(line.lstrip())] - rewritten.append( - f"{indent}execution_provider = {provider_inline_table(provider)}" - ) - replaced_sections.add(current_section) - continue - rewritten.append(line) - - missing = set(GEAR_SONIC_PROVIDER_SECTIONS) - replaced_sections - if missing: - missing_str = ", ".join(sorted(missing)) - raise ValueError( - "failed to rewrite all GEAR-Sonic execution_provider blocks; " - f"missing sections: {missing_str}" - ) - - rewritten_text = "\n".join(rewritten) - if config_text.endswith("\n"): - rewritten_text += "\n" - return rewritten_text - - -def append_report_section(config_text: str, report_path: Path) -> str: - report_block = f'\n[report]\noutput_path = "{report_path}"\nmax_frames = 200\n' - return config_text.rstrip() + "\n" + report_block - - -def compose_benchmark_cli_config( - config_text: str, - *, - provider: str, - report_path: Path, - rewrite_gear_sonic_providers: bool, -) -> str: - updated = config_text - if rewrite_gear_sonic_providers: - updated = rewrite_gear_sonic_provider_blocks(updated, provider) - return append_report_section(updated, report_path) - - -def describe_process_failure(error: subprocess.CalledProcessError) -> str: - chunks = [chunk.strip() for chunk in (error.stdout, error.stderr) if chunk and chunk.strip()] - if not chunks: - return str(error) - combined = "\n".join(chunks) - lines = combined.splitlines() - if len(lines) > 40: - combined = "\n".join(lines[-40:]) - return combined - - -def benchmark_failure_reason(provider: str, error: subprocess.CalledProcessError) -> str: - details = describe_process_failure(error) - return ( - f"Requested provider `{provider}` could not run on the ORT-rs benchmark path. " - f"Exact runtime output:\n{details}" - ) - - -def blocked_reason_for_provider(case_id: str, provider: str) -> str | None: - if provider != "cpu" and case_id.startswith("decoupled_wbc/"): - return ( - f"{case_id} stays CPU-only in this phase. Provider `{provider}` is not wired on " - "both benchmark implementations for Decoupled WBC, so the row is blocked instead of " - "quietly relabeling a CPU measurement." - ) - return None - - -def emit_blocked( - *, - case_id: str, - provider: str, - upstream_commit: str, - robowbc_commit: str, - output_root: Path, - reason: str, - source_command: str, -) -> None: - case = NORMALIZER.registry_case(REGISTRY, case_id) - artifact = NORMALIZER.build_artifact( - case=case, - implementation=IMPLEMENTATION_ID, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - provider=provider, - host_fingerprint=None, - samples_ns=[], - hz=None, - notes=reason, - source_command=source_command, - raw_source=reason, - status="blocked", - ) - output_path = output_path_for(case_id, output_root) - NORMALIZER.dump_json(output_path, artifact) - print(f"[blocked] {case_id} -> {output_path}") - - -def normalize_criterion_case( - *, - case_id: str, - provider: str, - upstream_commit: str, - robowbc_commit: str, - output_root: Path, - source_command: str, -) -> None: - case = NORMALIZER.registry_case(REGISTRY, case_id) - criterion_id = case["criterion_id"] - samples_ns, raw_source = NORMALIZER.criterion_samples_ns( - ROOT_DIR / "target/criterion", criterion_id - ) - artifact = NORMALIZER.build_artifact( - case=case, - implementation=IMPLEMENTATION_ID, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - provider=provider, - host_fingerprint=None, - samples_ns=samples_ns, - hz=None, - notes="Normalized from ORT-rs Criterion sample.json per-iteration timings.", - source_command=source_command, - raw_source=raw_source, - status="ok", - ) - output_path = output_path_for(case_id, output_root) - NORMALIZER.dump_json(output_path, artifact) - print(f"[ok] {case_id} -> {output_path}") - - -def normalize_run_report( - *, - case_id: str, - provider: str, - upstream_commit: str, - robowbc_commit: str, - output_root: Path, - report_path: Path, - source_command: str, -) -> None: - case = NORMALIZER.registry_case(REGISTRY, case_id) - samples_ns, hz, raw_source = NORMALIZER.run_report_samples_ns(report_path) - artifact = NORMALIZER.build_artifact( - case=case, - implementation=IMPLEMENTATION_ID, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - provider=provider, - host_fingerprint=None, - samples_ns=samples_ns, - hz=hz, - notes="Normalized from the ORT-rs robowbc-cli JSON run report.", - source_command=source_command, - raw_source=raw_source, - status="ok", - ) - output_path = output_path_for(case_id, output_root) - NORMALIZER.dump_json(output_path, artifact) - print(f"[ok] {case_id} -> {output_path}") - - -def run_microbench_case( - *, - case_id: str, - criterion_filter: str, - provider: str, - upstream_commit: str, - robowbc_commit: str, - output_root: Path, - env_name: str, - env_value: Path, - source_command: str, -) -> None: - env = os.environ.copy() - env[env_name] = str(env_value) - env[BENCH_PROVIDER_ENV] = provider - run( - [ - "cargo", - "bench", - "-p", - "robowbc-ort", - "--bench", - "inference", - "--", - "--output-format", - "bencher", - criterion_filter, - ], - env=env, - ) - normalize_criterion_case( - case_id=case_id, - provider=provider, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - output_root=output_root, - source_command=source_command, - ) - - -def run_cli_case( - *, - case_id: str, - config_path: str, - provider: str, - upstream_commit: str, - robowbc_commit: str, - output_root: Path, - source_command: str, -) -> None: - source_config = ROOT_DIR / config_path - with tempfile.TemporaryDirectory() as temp_dir: - temp_root = Path(temp_dir) - temp_config = temp_root / source_config.name - raw_report = temp_root / "report.json" - env = configure_mujoco_runtime_env(os.environ.copy()) - temp_config.write_text( - compose_benchmark_cli_config( - source_config.read_text(encoding="utf-8"), - provider=provider, - report_path=raw_report, - rewrite_gear_sonic_providers=case_id.startswith("gear_sonic/"), - ), - encoding="utf-8", - ) - run( - [ - "cargo", - "run", - "-p", - "robowbc-cli", - "--features", - "sim-auto-download,vis", - "--bin", - "robowbc", - "--", - "run", - "--config", - str(temp_config), - ], - env=env, - ) - normalize_run_report( - case_id=case_id, - provider=provider, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - output_root=output_root, - report_path=raw_report, - source_command=source_command, - ) - - -def run_case(case_id: str, args: argparse.Namespace) -> None: - if case_id not in case_ids(): - raise ValueError(f"unknown case_id: {case_id}") - - source_command = source_command_for_case(args, case_id) - output_root = provider_output_root(args.output_root, args.provider) - gear_model_dir = DEFAULT_GEAR_SONIC_MODEL_DIR - decoupled_model_dir = DEFAULT_DECOUPLED_WBC_MODEL_DIR - robowbc_commit = git_rev_parse(ROOT_DIR) - - blocked_reason = blocked_reason_for_provider(case_id, args.provider) - if blocked_reason is not None: - upstream_commit = ( - gear_sonic_revision(gear_model_dir) - if case_id.startswith("gear_sonic/") - else decoupled_revision(decoupled_model_dir) - ) - emit_blocked( - case_id=case_id, - provider=args.provider, - upstream_commit=upstream_commit, - robowbc_commit=robowbc_commit, - output_root=output_root, - reason=blocked_reason, - source_command=source_command, - ) - return - - if case_id in MICROBENCH_CASES: - spec = MICROBENCH_CASES[case_id] - if spec["family"] == "gear_sonic": - if not have_gear_sonic_models(gear_model_dir): - emit_blocked( - case_id=case_id, - provider=args.provider, - upstream_commit=gear_sonic_revision(gear_model_dir), - robowbc_commit=robowbc_commit, - output_root=output_root, - reason=( - "GEAR-Sonic checkpoints not found under " - f"{gear_model_dir}; run scripts/download_gear_sonic_models.sh first." - ), - source_command=source_command, - ) - return - try: - run_microbench_case( - case_id=case_id, - criterion_filter=spec["criterion_filter"], - provider=args.provider, - upstream_commit=gear_sonic_revision(gear_model_dir), - robowbc_commit=robowbc_commit, - output_root=output_root, - env_name="GEAR_SONIC_MODEL_DIR", - env_value=gear_model_dir, - source_command=source_command, - ) - except subprocess.CalledProcessError as error: - emit_blocked( - case_id=case_id, - provider=args.provider, - upstream_commit=gear_sonic_revision(gear_model_dir), - robowbc_commit=robowbc_commit, - output_root=output_root, - reason=benchmark_failure_reason(args.provider, error), - source_command=source_command, - ) - return - - if not have_decoupled_models(decoupled_model_dir): - emit_blocked( - case_id=case_id, - provider=args.provider, - upstream_commit=decoupled_revision(decoupled_model_dir), - robowbc_commit=robowbc_commit, - output_root=output_root, - reason=( - "Decoupled WBC checkpoints not found under " - f"{decoupled_model_dir}; run scripts/download_decoupled_wbc_models.sh first." - ), - source_command=source_command, - ) - return - try: - run_microbench_case( - case_id=case_id, - criterion_filter=spec["criterion_filter"], - provider=args.provider, - upstream_commit=decoupled_revision(decoupled_model_dir), - robowbc_commit=robowbc_commit, - output_root=output_root, - env_name="DECOUPLED_WBC_MODEL_DIR", - env_value=decoupled_model_dir, - source_command=source_command, - ) - except subprocess.CalledProcessError as error: - emit_blocked( - case_id=case_id, - provider=args.provider, - upstream_commit=decoupled_revision(decoupled_model_dir), - robowbc_commit=robowbc_commit, - output_root=output_root, - reason=benchmark_failure_reason(args.provider, error), - source_command=source_command, - ) - return - - if case_id in CLI_CASES: - spec = CLI_CASES[case_id] - if spec["family"] == "gear_sonic": - if not have_gear_sonic_models(gear_model_dir): - emit_blocked( - case_id=case_id, - provider=args.provider, - upstream_commit=gear_sonic_revision(gear_model_dir), - robowbc_commit=robowbc_commit, - output_root=output_root, - reason=( - "GEAR-Sonic checkpoints not found under " - f"{gear_model_dir}; run scripts/download_gear_sonic_models.sh first." - ), - source_command=source_command, - ) - return - try: - run_cli_case( - case_id=case_id, - config_path=spec["config_path"], - provider=args.provider, - upstream_commit=gear_sonic_revision(gear_model_dir), - robowbc_commit=robowbc_commit, - output_root=output_root, - source_command=source_command, - ) - except (subprocess.CalledProcessError, ValueError) as error: - reason = ( - benchmark_failure_reason(args.provider, error) - if isinstance(error, subprocess.CalledProcessError) - else str(error) - ) - emit_blocked( - case_id=case_id, - provider=args.provider, - upstream_commit=gear_sonic_revision(gear_model_dir), - robowbc_commit=robowbc_commit, - output_root=output_root, - reason=reason, - source_command=source_command, - ) - return - - if not have_decoupled_models(decoupled_model_dir): - emit_blocked( - case_id=case_id, - provider=args.provider, - upstream_commit=decoupled_revision(decoupled_model_dir), - robowbc_commit=robowbc_commit, - output_root=output_root, - reason=( - "Decoupled WBC checkpoints not found under " - f"{decoupled_model_dir}; run scripts/download_decoupled_wbc_models.sh first." - ), - source_command=source_command, - ) - return - try: - run_cli_case( - case_id=case_id, - config_path=spec["config_path"], - provider=args.provider, - upstream_commit=decoupled_revision(decoupled_model_dir), - robowbc_commit=robowbc_commit, - output_root=output_root, - source_command=source_command, - ) - except subprocess.CalledProcessError as error: - emit_blocked( - case_id=case_id, - provider=args.provider, - upstream_commit=decoupled_revision(decoupled_model_dir), - robowbc_commit=robowbc_commit, - output_root=output_root, - reason=benchmark_failure_reason(args.provider, error), - source_command=source_command, - ) - return - - emit_blocked( - case_id=case_id, - provider=args.provider, - upstream_commit="unknown-upstream", - robowbc_commit=robowbc_commit, - output_root=output_root, - reason=f"No RoboWBC benchmark mapping has been defined for {case_id}.", - source_command=source_command, - ) - - -def main() -> int: - args = parse_args() - - if args.list_cases: - list_cases() - return 0 - - if args.case: - run_case(args.case, args) - return 0 - - for case in REGISTRY["cases"]: - run_case(case["case_id"], args) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) +run_legacy_script("benchmarks/bench_robowbc_compare.py") diff --git a/scripts/benchmarks/bench_nvidia_decoupled_official.py b/scripts/benchmarks/bench_nvidia_decoupled_official.py new file mode 100644 index 0000000..2f62aa7 --- /dev/null +++ b/scripts/benchmarks/bench_nvidia_decoupled_official.py @@ -0,0 +1,276 @@ +#!/usr/bin/env python3 +"""Headless benchmark harness for the upstream Decoupled WBC policy.""" + +from __future__ import annotations + +import argparse +import collections +import contextlib +import json +import sys +import time +from pathlib import Path +from types import ModuleType +from typing import Any + +import numpy as np +import tomllib + + +def install_torch_shim_if_needed() -> str: + try: + import torch # noqa: F401 + except ModuleNotFoundError: + pass + else: + return "torch" + + class Tensor(np.ndarray): + __array_priority__ = 1000 + + def __new__(cls, value: Any, dtype: Any | None = None) -> "Tensor": + return np.asarray(value, dtype=dtype).view(cls) + + def __array_finalize__(self, obj: Any) -> None: # pragma: no cover - ndarray protocol + del obj + + def unsqueeze(self, dim: int) -> "Tensor": + return np.expand_dims(self, axis=dim).view(Tensor) + + def detach(self) -> "Tensor": + return self + + def cpu(self) -> "Tensor": + return self + + def numpy(self) -> np.ndarray: + return np.asarray(self) + + class NoGrad(contextlib.AbstractContextManager[None]): + def __exit__(self, exc_type, exc, tb) -> None: + del exc_type, exc, tb + return None + + torch_mod = ModuleType("torch") + torch_mod.Tensor = Tensor + torch_mod.float32 = np.float32 + torch_mod.tensor = lambda value, device=None, dtype=None: Tensor(value, dtype=dtype) + torch_mod.zeros = lambda shape, dtype=np.float32: Tensor(np.zeros(shape, dtype=dtype)) + torch_mod.full_like = lambda tensor, fill_value: Tensor( + np.full_like(np.asarray(tensor), fill_value) + ) + torch_mod.remainder = lambda x, y: Tensor(np.remainder(np.asarray(x), y)) + torch_mod.cat = lambda tensors, dim=0: Tensor( + np.concatenate([np.asarray(t) for t in tensors], axis=dim) + ) + torch_mod.stack = lambda tensors, dim=0: Tensor( + np.stack([np.asarray(t) for t in tensors], axis=dim) + ) + torch_mod.sin = lambda tensor: Tensor(np.sin(np.asarray(tensor))) + torch_mod.from_numpy = lambda array: Tensor(array) + torch_mod.no_grad = NoGrad + sys.modules["torch"] = torch_mod + return "torch_shim" + + +class UnitreeG1RobotModelStub: + def __init__(self) -> None: + self.groups = { + "body": list(range(29)), + "lower_body": list(range(15)), + } + + def get_joint_group_indices(self, group: str) -> list[int]: + if group not in self.groups: + raise KeyError(f"unknown joint group: {group}") + return self.groups[group] + + +def load_robot_default_pose(path: Path) -> np.ndarray: + with path.open("rb") as handle: + parsed = tomllib.load(handle) + return np.array(parsed["default_pose"], dtype=np.float32) + + +def resolve_model_paths(model_dir: Path) -> tuple[Path, Path]: + balance_src = (model_dir / "GR00T-WholeBodyControl-Balance.onnx").resolve() + walk_src = (model_dir / "GR00T-WholeBodyControl-Walk.onnx").resolve() + if not balance_src.is_file() or not walk_src.is_file(): + raise FileNotFoundError(f"expected balance and walk ONNX models under {model_dir}") + return balance_src, walk_src + + +@contextlib.contextmanager +def patched_absolute_model_loading(policy_cls: type[Any]): + original = getattr(policy_cls, "load_onnx_policy", None) + if original is None: + yield + return + + def patched(self: Any, model_path: str): + resolved_path = Path(model_path) + if not resolved_path.is_file(): + marker = "/resources/robots/g1/" + if marker in model_path: + suffix = model_path.rsplit(marker, 1)[-1] + suffix_path = Path(suffix) + if suffix_path.is_file(): + model_path = str(suffix_path) + return original(self, model_path) + + setattr(policy_cls, "load_onnx_policy", patched) + try: + yield + finally: + setattr(policy_cls, "load_onnx_policy", original) + + +def build_observation(default_pose: np.ndarray) -> dict[str, np.ndarray]: + return { + "q": default_pose.copy(), + "dq": np.zeros_like(default_pose), + "floating_base_pose": np.array([0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0], dtype=np.float32), + "floating_base_vel": np.zeros(6, dtype=np.float32), + } + + +def reset_policy_state(policy: Any) -> None: + policy.obs_history = collections.deque(maxlen=policy.config["obs_history_len"]) + policy.obs_buffer = np.zeros(policy.config["num_obs"], dtype=np.float32) + policy.counter = 0 + policy.action = np.zeros(policy.config["num_actions"], dtype=np.float32) + policy.target_dof_pos = policy.config["default_angles"].copy() + policy.cmd = policy.config["cmd_init"].copy() + policy.height_cmd = float(policy.config["height_cmd"]) + policy.freq_cmd = float(policy.config["freq_cmd"]) + if "rpy_cmd" in policy.config: + policy.roll_cmd = float(policy.config["rpy_cmd"][0]) + policy.pitch_cmd = float(policy.config["rpy_cmd"][1]) + policy.yaw_cmd = float(policy.config["rpy_cmd"][2]) + else: + policy.roll_cmd = float(policy.config.get("roll_cmd", 0.0)) + policy.pitch_cmd = float(policy.config.get("pitch_cmd", 0.0)) + policy.yaw_cmd = float(policy.config.get("yaw_cmd", 0.0)) + policy.gait_indices = sys.modules["torch"].zeros((1,), dtype=sys.modules["torch"].float32) + policy.obs_tensor = None + policy.use_policy_action = True + policy.set_use_teleop_policy_cmd(True) + + +def single_tick(policy: Any, obs: dict[str, np.ndarray], command: np.ndarray) -> None: + policy.cmd = command.astype(np.float32, copy=True) + policy.set_observation(obs) + policy.get_action(time=time.monotonic()) + + +def run_microbench(policy: Any, obs: dict[str, np.ndarray], command: np.ndarray, samples: int) -> list[int]: + timings: list[int] = [] + for _ in range(samples): + reset_policy_state(policy) + start = time.perf_counter_ns() + single_tick(policy, obs, command) + timings.append(time.perf_counter_ns() - start) + return timings + + +def run_end_to_end_loop( + policy: Any, + obs: dict[str, np.ndarray], + command: np.ndarray, + ticks: int, + control_frequency_hz: int, +) -> tuple[list[int], float]: + timings: list[int] = [] + period_ns = round(1_000_000_000 / control_frequency_hz) + reset_policy_state(policy) + wall_start = time.perf_counter_ns() + for _ in range(ticks): + tick_start = time.perf_counter_ns() + single_tick(policy, obs, command) + tick_end = time.perf_counter_ns() + timings.append(tick_end - tick_start) + remaining_ns = period_ns - (tick_end - tick_start) + if remaining_ns > 0: + time.sleep(remaining_ns / 1_000_000_000) + wall_end = time.perf_counter_ns() + elapsed_s = (wall_end - wall_start) / 1_000_000_000 + achieved_hz = ticks / elapsed_s if elapsed_s > 0 else 0.0 + return timings, achieved_hz + + +def parse_command(case_id: str) -> np.ndarray: + if case_id == "decoupled_wbc/walk_predict" or case_id == "decoupled_wbc/end_to_end_cli_loop": + return np.array([0.25, 0.0, 0.05], dtype=np.float32) + if case_id == "decoupled_wbc/balance_predict": + return np.array([0.0, 0.0, 0.0], dtype=np.float32) + raise ValueError(f"unsupported Decoupled case_id: {case_id}") + + +def load_policy(repo_dir: Path, model_dir: Path) -> tuple[Any, str]: + torch_backend = install_torch_shim_if_needed() + balance_model, walk_model = resolve_model_paths(model_dir) + + sys.path.insert(0, str(repo_dir)) + from decoupled_wbc.control.policy import g1_gear_wbc_policy as policy_module # type: ignore + + robot_model = UnitreeG1RobotModelStub() + config_path = ( + repo_dir + / "decoupled_wbc" + / "sim2mujoco" + / "resources" + / "robots" + / "g1" + / "g1_gear_wbc.yaml" + ) + model_path = f"{balance_model},{walk_model}" + policy_cls = policy_module.G1GearWbcPolicy + with patched_absolute_model_loading(policy_cls): + policy = policy_cls(robot_model=robot_model, config=str(config_path), model_path=model_path) + return policy, torch_backend + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--case-id", required=True) + parser.add_argument("--repo-dir", type=Path, required=True) + parser.add_argument("--model-dir", type=Path, required=True) + parser.add_argument("--robot-config", type=Path, default=Path("configs/robots/unitree_g1.toml")) + parser.add_argument("--samples", type=int, default=100) + parser.add_argument("--ticks", type=int, default=200) + parser.add_argument("--control-frequency-hz", type=int, default=50) + parser.add_argument("--output", type=Path, required=True) + args = parser.parse_args() + + if not args.repo_dir.is_dir(): + raise FileNotFoundError(f"repo-dir does not exist: {args.repo_dir}") + + default_pose = load_robot_default_pose(args.robot_config) + observation = build_observation(default_pose) + command = parse_command(args.case_id) + policy, torch_backend = load_policy(args.repo_dir, args.model_dir) + + if args.case_id == "decoupled_wbc/end_to_end_cli_loop": + samples_ns, hz = run_end_to_end_loop( + policy, observation, command, args.ticks, args.control_frequency_hz + ) + else: + samples_ns = run_microbench(policy, observation, command, args.samples) + hz = None + + payload = { + "case_id": args.case_id, + "samples_ns": samples_ns, + "hz": hz, + "notes": ( + "Measured via upstream decoupled_wbc.control.policy.g1_gear_wbc_policy.G1GearWbcPolicy " + f"with {torch_backend}." + ), + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(payload, indent=2), encoding="utf-8") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/benchmarks/bench_nvidia_gear_sonic_official.cpp b/scripts/benchmarks/bench_nvidia_gear_sonic_official.cpp new file mode 100644 index 0000000..433ff20 --- /dev/null +++ b/scripts/benchmarks/bench_nvidia_gear_sonic_official.cpp @@ -0,0 +1,2200 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "math_utils.hpp" +#include "policy_parameters.hpp" +#include "robot_parameters.hpp" + +namespace fs = std::filesystem; + +namespace { + +constexpr std::size_t kPlannerQposDim = 36; +constexpr std::size_t kPlannerJointOffset = 7; +constexpr std::size_t kPlannerContextLen = 4; +constexpr std::size_t kReplanIntervalTicksDefault = 50; +constexpr std::size_t kReplanIntervalTicksRunning = 5; +constexpr std::size_t kPlannerThreadIntervalTicks = 5; +constexpr std::size_t kPlannerLookAheadSteps = 2; +constexpr std::size_t kPlannerBlendFrames = 8; +constexpr std::size_t kAllowedPredNumTokens = 11; +constexpr std::array kAllowedPredNumTokensMask = { + 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, +}; +constexpr float kDefaultHeightMeters = 0.74f; +constexpr float kOfficialDefaultHeightMeters = 0.788740f; +constexpr float kDefaultHeightSentinel = -1.0f; +constexpr std::int64_t kDefaultModeWalk = 2; +constexpr std::int64_t kDefaultModeRun = 3; +constexpr std::int64_t kDefaultModeIdle = 0; +constexpr std::int64_t kDefaultModeSlowWalk = 1; +constexpr std::size_t kReferenceFutureFrames = 10; +constexpr std::size_t kReferenceFrameStep = 5; +constexpr float kControlDtSeconds = 1.0f / 50.0f; +constexpr std::size_t kEncoderDim = 64; +constexpr std::size_t kEncoderObsDictDim = 1762; +constexpr std::size_t kDecoderObsDictDim = 994; +constexpr std::size_t kDecoderHistoryLen = 10; +constexpr std::size_t kLaterMotionProbeTick = 25; +constexpr std::size_t kEncoderModeOffset = 0; +constexpr std::size_t kEncoderMotionJointPositionsOffset = 4; +constexpr std::size_t kEncoderMotionJointVelocitiesOffset = 294; +constexpr std::size_t kEncoderMotionAnchorOrientationOffset = 601; + +volatile float g_sink = 0.0f; + +enum class ProviderKind { + Cpu, + Cuda, + TensorRt, +}; + +struct Options { + std::string case_id; + ProviderKind provider = ProviderKind::Cpu; + fs::path model_dir; + fs::path output; + std::optional dump_dir; + int samples = 100; + int ticks = 200; + int control_frequency_hz = 50; +}; + +struct Observation { + std::vector joint_positions; + std::vector joint_velocities; + std::array gravity_vector{}; + std::array angular_velocity{}; + std::array base_quat_wxyz{1.0, 0.0, 0.0, 0.0}; +}; + +struct Twist { + std::array linear{}; + std::array angular{}; +}; + +struct PlannerCommand { + std::int64_t mode = kDefaultModeIdle; + float target_vel = kDefaultHeightSentinel; + float height = kDefaultHeightSentinel; + std::array movement_direction{0.0f, 0.0f, 0.0f}; + std::array facing_direction{1.0f, 0.0f, 0.0f}; +}; + +float wrap_angle_rad(float angle) { + while (angle > static_cast(M_PI)) { + angle -= 2.0f * static_cast(M_PI); + } + while (angle < -static_cast(M_PI)) { + angle += 2.0f * static_cast(M_PI); + } + return angle; +} + +std::pair bin_angle_to_8_directions(float angle) { + constexpr float kBinSize = static_cast(M_PI / 4.0); + + const float normalized = wrap_angle_rad(angle); + int bin_index = static_cast(std::lround(normalized / kBinSize)); + if (bin_index > 4) { + bin_index -= 8; + } + if (bin_index < -4) { + bin_index += 8; + } + + float slow_walk_speed = 0.2f; + switch (bin_index) { + case 0: + case 1: + case -1: + slow_walk_speed = 0.3f; + break; + case 2: + case -2: + slow_walk_speed = 0.35f; + break; + case 3: + case -3: + slow_walk_speed = 0.25f; + break; + case 4: + case -4: + default: + slow_walk_speed = 0.2f; + break; + } + + return {static_cast(bin_index) * kBinSize, slow_walk_speed}; +} + +PlannerCommand idle_planner_command() { + return PlannerCommand{}; +} + +PlannerCommand derive_planner_command(float& facing_yaw_rad, const Twist& twist) { + facing_yaw_rad = wrap_angle_rad(facing_yaw_rad + twist.angular[2] * kControlDtSeconds); + const std::array facing_direction = { + std::cos(facing_yaw_rad), + std::sin(facing_yaw_rad), + 0.0f, + }; + const float command_norm = std::sqrt( + twist.linear[0] * twist.linear[0] + twist.linear[1] * twist.linear[1] + ); + + if (command_norm <= 0.01f) { + return PlannerCommand{ + kDefaultModeIdle, + kDefaultHeightSentinel, + kDefaultHeightSentinel, + {0.0f, 0.0f, 0.0f}, + facing_direction, + }; + } + + const float local_movement_angle = std::atan2(twist.linear[1], twist.linear[0]); + const auto [movement_angle, slow_walk_speed] = + bin_angle_to_8_directions(facing_yaw_rad + local_movement_angle); + const auto [mode, target_vel] = + command_norm < 0.8f + ? std::pair{kDefaultModeSlowWalk, slow_walk_speed} + : (command_norm < 2.5f + ? std::pair{kDefaultModeWalk, -1.0f} + : std::pair{kDefaultModeRun, -1.0f}); + + return PlannerCommand{ + mode, + target_vel, + kDefaultHeightSentinel, + {std::cos(movement_angle), std::sin(movement_angle), 0.0f}, + facing_direction, + }; +} + +bool planner_command_changed( + const std::optional& previous, + const PlannerCommand& next +) { + const auto vec3_distance = [](const std::array& a, const std::array& b) { + const float dx = a[0] - b[0]; + const float dy = a[1] - b[1]; + const float dz = a[2] - b[2]; + return std::sqrt(dx * dx + dy * dy + dz * dz); + }; + + if (!previous.has_value()) { + return true; + } + + return previous->mode != next.mode + || std::abs(previous->target_vel - next.target_vel) > 0.05f + || std::abs(previous->height - next.height) > 1e-3f + || vec3_distance(previous->movement_direction, next.movement_direction) > 0.1f + || vec3_distance(previous->facing_direction, next.facing_direction) > 0.1f; +} + +std::size_t planner_replan_interval_ticks(const PlannerCommand& command) { + return command.mode == kDefaultModeRun + ? kReplanIntervalTicksRunning + : kReplanIntervalTicksDefault; +} + +std::vector default_pose() { + std::vector pose; + pose.reserve(G1_NUM_MOTOR); + for (double angle : default_angles) { + pose.push_back(static_cast(angle)); + } + return pose; +} + +std::vector mujoco_to_isaaclab_values(const std::vector& values) { + if (values.size() != G1_NUM_MOTOR) { + throw std::runtime_error("values must match G1_NUM_MOTOR"); + } + + std::vector remapped(G1_NUM_MOTOR, 0.0f); + for (std::size_t mujoco_index = 0; mujoco_index < G1_NUM_MOTOR; ++mujoco_index) { + remapped[static_cast(isaaclab_to_mujoco[mujoco_index])] = + values[mujoco_index]; + } + return remapped; +} + +std::vector mujoco_to_isaaclab_positions(const std::vector& joint_positions) { + return mujoco_to_isaaclab_values(joint_positions); +} + +std::vector mujoco_to_isaaclab_joint_offsets(const std::vector& joint_positions) { + auto remapped = mujoco_to_isaaclab_values(joint_positions); + for (std::size_t mujoco_index = 0; mujoco_index < G1_NUM_MOTOR; ++mujoco_index) { + remapped[static_cast(isaaclab_to_mujoco[mujoco_index])] -= + static_cast(default_angles[mujoco_index]); + } + return remapped; +} + +std::vector isaaclab_to_mujoco_values(const std::vector& values) { + if (values.size() != G1_NUM_MOTOR) { + throw std::runtime_error("values must match G1_NUM_MOTOR"); + } + + std::vector remapped(G1_NUM_MOTOR, 0.0f); + for (std::size_t mujoco_index = 0; mujoco_index < G1_NUM_MOTOR; ++mujoco_index) { + remapped[mujoco_index] = values[static_cast(isaaclab_to_mujoco[mujoco_index])]; + } + return remapped; +} + +Observation zero_tracking_observation() { + Observation obs; + obs.joint_positions.assign(G1_NUM_MOTOR, 0.0f); + obs.joint_velocities.assign(G1_NUM_MOTOR, 0.0f); + obs.gravity_vector = {0.0f, 0.0f, -1.0f}; + obs.angular_velocity = {0.0f, 0.0f, 0.0f}; + return obs; +} + +Observation standing_velocity_observation() { + Observation obs; + obs.joint_positions = default_pose(); + obs.joint_velocities.assign(G1_NUM_MOTOR, 0.0f); + obs.gravity_vector = {0.0f, 0.0f, -1.0f}; + obs.angular_velocity = {0.0f, 0.0f, 0.0f}; + return obs; +} + +std::string json_escape(const std::string& input) { + std::string escaped; + escaped.reserve(input.size()); + for (char ch : input) { + switch (ch) { + case '\\': + escaped += "\\\\"; + break; + case '"': + escaped += "\\\""; + break; + case '\n': + escaped += "\\n"; + break; + default: + escaped.push_back(ch); + break; + } + } + return escaped; +} + +std::vector session_names(Ort::Session& session, bool inputs) { + Ort::AllocatorWithDefaultOptions allocator; + const std::size_t count = inputs ? session.GetInputCount() : session.GetOutputCount(); + std::vector names; + names.reserve(count); + for (std::size_t index = 0; index < count; ++index) { + auto name = inputs ? session.GetInputNameAllocated(index, allocator) + : session.GetOutputNameAllocated(index, allocator); + names.emplace_back(name.get()); + } + return names; +} + +void require_name(const std::vector& names, std::string_view needle, std::string_view kind) { + if (std::find(names.begin(), names.end(), needle) == names.end()) { + throw std::runtime_error( + "missing required " + std::string(kind) + " tensor '" + std::string(needle) + "'" + ); + } +} + +std::string provider_label(ProviderKind provider) { + switch (provider) { + case ProviderKind::Cpu: + return "cpu"; + case ProviderKind::Cuda: + return "cuda"; + case ProviderKind::TensorRt: + return "tensor_rt"; + } + throw std::runtime_error("unreachable provider label state"); +} + +std::string provider_identifier(ProviderKind provider) { + switch (provider) { + case ProviderKind::Cpu: + return "CPUExecutionProvider"; + case ProviderKind::Cuda: + return "CUDAExecutionProvider"; + case ProviderKind::TensorRt: + return "TensorrtExecutionProvider"; + } + throw std::runtime_error("unreachable provider identifier state"); +} + +std::vector available_providers() { + return Ort::GetAvailableProviders(); +} + +std::string format_provider_list(const std::vector& providers) { + if (providers.empty()) { + return "(none)"; + } + std::string joined; + for (std::size_t index = 0; index < providers.size(); ++index) { + if (index > 0) { + joined += ", "; + } + joined += providers[index]; + } + return joined; +} + +constexpr const char* kTensorRtExcludedOpTypes = "Cast"; + +void configure_provider(Ort::SessionOptions& options, ProviderKind provider) { + if (provider == ProviderKind::Cpu) { + return; + } + + try { + if (provider == ProviderKind::Cuda) { + Ort::CUDAProviderOptions cuda_options; + cuda_options.Update({{"device_id", "0"}}); + options.AppendExecutionProvider_CUDA_V2(*cuda_options); + return; + } + + Ort::TensorRTProviderOptions tensorrt_options; + // GEAR-Sonic's planner graph includes Float->Int64 Cast paths that + // TensorRT cannot compile inside a fused subgraph. Excluding Cast keeps + // those nodes on CUDA/CPU while still exercising TensorRT for the rest + // of the graph. + tensorrt_options.Update({ + {"device_id", "0"}, + {"trt_op_types_to_exclude", kTensorRtExcludedOpTypes}, + }); + options.AppendExecutionProvider_TensorRT_V2(*tensorrt_options); + Ort::CUDAProviderOptions cuda_options; + cuda_options.Update({{"device_id", "0"}}); + options.AppendExecutionProvider_CUDA_V2(*cuda_options); + } catch (const Ort::Exception& error) { + throw std::runtime_error( + std::string("failed to configure provider `") + provider_label(provider) + + "`: " + error.what() + ); + } +} + +Ort::Session make_session(Ort::Env& env, const fs::path& model_path, ProviderKind provider) { + if (!fs::is_regular_file(model_path)) { + throw std::runtime_error("model file does not exist: " + model_path.string()); + } + + Ort::SessionOptions options; + options.SetIntraOpNumThreads(1); + options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED); + configure_provider(options, provider); + const std::string model_path_str = model_path.string(); + try { + return Ort::Session(env, model_path_str.c_str(), options); + } catch (const Ort::Exception& error) { + const auto providers = available_providers(); + throw std::runtime_error( + std::string("failed to initialize provider `") + provider_label(provider) + + "` for model " + model_path.string() + ": " + error.what() + + " (advertised providers: " + format_provider_list(providers) + ")" + ); + } +} + +struct PlannerState { + std::deque> context; + std::size_t steps_since_plan = kReplanIntervalTicksDefault; + std::vector last_context_frame; + std::vector> motion_qpos_50hz; + std::vector> motion_joint_velocities_isaaclab; + std::size_t current_motion_frame = 0; + float facing_yaw_rad = 0.0f; + std::optional> init_base_quat_wxyz; + std::optional> init_ref_root_quat_wxyz; + std::optional last_command; + + struct PendingPlannerReplan { + std::size_t request_motion_frame = 0; + PlannerCommand command{}; + std::future>> future; + }; + + std::optional pending_replan; + + PlannerState() + : context(), last_context_frame() + { + reset(); + } + + void reset() { + const auto standing = make_standing_qpos(); + context.clear(); + for (std::size_t index = 0; index < kPlannerContextLen; ++index) { + context.push_back(standing); + } + steps_since_plan = kReplanIntervalTicksDefault; + last_context_frame = standing; + motion_qpos_50hz.clear(); + motion_joint_velocities_isaaclab.clear(); + current_motion_frame = 0; + facing_yaw_rad = 0.0f; + init_base_quat_wxyz.reset(); + init_ref_root_quat_wxyz.reset(); + last_command.reset(); + pending_replan.reset(); + } + + static std::vector make_standing_qpos() { + std::vector qpos(kPlannerQposDim, 0.0f); + qpos[2] = kDefaultHeightMeters; + qpos[3] = 1.0f; + const auto pose = default_pose(); + std::copy(pose.begin(), pose.end(), qpos.begin() + static_cast(kPlannerJointOffset)); + return qpos; + } +}; + +struct TrackingState { + std::deque> gravity; + std::deque> angular_velocity; + std::deque> joint_positions; + std::deque> joint_velocities; + std::deque> last_actions; + + TrackingState() + : gravity(), angular_velocity(), joint_positions(), joint_velocities(), last_actions() + { + reset(); + } + + void reset() { + gravity.clear(); + angular_velocity.clear(); + joint_positions.clear(); + joint_velocities.clear(); + last_actions.clear(); + for (std::size_t index = 0; index < kDecoderHistoryLen; ++index) { + gravity.push_back({0.0f, 0.0f, 1.0f}); + angular_velocity.push_back({0.0f, 0.0f, 0.0f}); + joint_positions.emplace_back(G1_NUM_MOTOR, 0.0f); + joint_velocities.emplace_back(G1_NUM_MOTOR, 0.0f); + last_actions.emplace_back(G1_NUM_MOTOR, 0.0f); + } + } + + void push(const Observation& obs, const std::vector& actions) { + if (gravity.size() >= kDecoderHistoryLen) { + gravity.pop_front(); + angular_velocity.pop_front(); + joint_positions.pop_front(); + joint_velocities.pop_front(); + last_actions.pop_front(); + } + gravity.push_back(obs.gravity_vector); + angular_velocity.push_back(obs.angular_velocity); + joint_positions.push_back(mujoco_to_isaaclab_joint_offsets(obs.joint_positions)); + joint_velocities.push_back(mujoco_to_isaaclab_values(obs.joint_velocities)); + last_actions.push_back(actions); + } +}; + +class GearSonicOfficialHarness { +public: + explicit GearSonicOfficialHarness( + const fs::path& model_dir, + ProviderKind provider, + std::optional dump_dir = std::nullopt + ) + : env_(ORT_LOGGING_LEVEL_WARNING, "gear_sonic_official"), + memory_info_(Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault)), + encoder_(make_session(env_, model_dir / "model_encoder.onnx", provider)), + decoder_(make_session(env_, model_dir / "model_decoder.onnx", provider)), + planner_(make_session(env_, model_dir / "planner_sonic.onnx", provider)), + planner_state_(), + tracking_state_(), + velocity_obs_(standing_velocity_observation()), + tracking_obs_(zero_tracking_observation()), + latest_action_(G1_NUM_MOTOR, 0.0f), + dump_dir_(std::move(dump_dir)) + { + validate_contracts(); + } + + void reset() { + planner_state_.reset(); + tracking_state_.reset(); + velocity_obs_ = standing_velocity_observation(); + tracking_obs_ = zero_tracking_observation(); + latest_action_.assign(G1_NUM_MOTOR, 0.0f); + dumped_tracking_tensors_ = false; + } + + std::vector velocity_tick() { + maybe_apply_pending_planner_replan(); + + const Twist twist = velocity_command(); + const float command_speed = std::sqrt( + twist.linear[0] * twist.linear[0] + twist.linear[1] * twist.linear[1] + ); + const auto live_command = derive_planner_command(planner_state_.facing_yaw_rad, twist); + const bool initializing_planner = planner_state_.motion_qpos_50hz.empty(); + const auto planner_command = + initializing_planner && command_speed <= 0.01f + ? idle_planner_command() + : live_command; + const bool needs_replan = + initializing_planner + || (planner_state_.pending_replan.has_value() + ? false + : (planner_command_changed(planner_state_.last_command, live_command) + || planner_state_.steps_since_plan + >= planner_replan_interval_ticks(live_command))); + + if (needs_replan) { + if (!planner_state_.motion_qpos_50hz.empty()) { + planner_state_.context = rebuild_planner_context_from_motion( + planner_state_.motion_qpos_50hz, + planner_state_.current_motion_frame + ); + if (!planner_state_.context.empty()) { + planner_state_.last_context_frame = planner_state_.context.back(); + } + } + if (initializing_planner) { + commit_planner_motion( + 0, + planner_command, + resample_planner_trajectory_to_50hz( + run_planner_command(planner_state_.context, planner_command) + ) + ); + } else { + start_async_planner_replan(planner_command); + } + planner_state_.steps_since_plan = 0; + } + + planner_state_.steps_since_plan += 1; + if (planner_state_.motion_qpos_50hz.empty()) { + throw std::runtime_error("planner motion buffer is empty after velocity bootstrap"); + } + + const auto obs = motion_observation_from_planner_frame( + planner_state_.motion_qpos_50hz, + planner_state_.motion_joint_velocities_isaaclab, + planner_state_.current_motion_frame + ); + const auto init_base_quat = + planner_state_.init_base_quat_wxyz.value_or(obs.base_quat_wxyz); + const auto init_ref_root_quat = + planner_state_.init_ref_root_quat_wxyz.value_or( + planner_frame_root_quaternion(planner_state_.motion_qpos_50hz.front()) + ); + const auto encoder_obs = build_velocity_encoder_obs_dict( + planner_state_.motion_qpos_50hz, + planner_state_.motion_joint_velocities_isaaclab, + planner_state_.current_motion_frame, + obs.base_quat_wxyz, + init_base_quat, + init_ref_root_quat + ); + const auto tokens = run_single_f32( + encoder_, + "obs_dict", + "encoded_tokens", + encoder_obs, + kEncoderObsDictDim + ); + if (tokens.size() != kEncoderDim) { + throw std::runtime_error( + "encoder output dimension mismatch: expected " + std::to_string(kEncoderDim) + + ", got " + std::to_string(tokens.size()) + ); + } + + tracking_state_.push(obs, latest_action_); + const auto decoder_obs = build_decoder_obs_dict(tokens); + const auto raw_actions = run_single_f32( + decoder_, + "obs_dict", + "action", + decoder_obs, + kDecoderObsDictDim + ); + if (raw_actions.size() != G1_NUM_MOTOR) { + throw std::runtime_error( + "decoder output dimension mismatch: expected " + std::to_string(G1_NUM_MOTOR) + + ", got " + std::to_string(raw_actions.size()) + ); + } + + std::vector positions(G1_NUM_MOTOR, 0.0f); + for (int mujoco_index = 0; mujoco_index < G1_NUM_MOTOR; ++mujoco_index) { + const int isaaclab_index = isaaclab_to_mujoco[static_cast(mujoco_index)]; + const float action = raw_actions[static_cast(isaaclab_index)]; + const float scaled = action * static_cast(g1_action_scale[static_cast(mujoco_index)]); + positions[static_cast(mujoco_index)] = + static_cast(default_angles[static_cast(mujoco_index)]) + scaled; + } + + latest_action_ = raw_actions; + advance_planner_motion_frame(); + return positions; + } + + std::vector tracking_tick() { + const auto encoder_obs = build_encoder_obs_dict(); + const auto tokens = run_single_f32(encoder_, "obs_dict", "encoded_tokens", encoder_obs, kEncoderObsDictDim); + if (tokens.size() != kEncoderDim) { + throw std::runtime_error( + "encoder output dimension mismatch: expected " + std::to_string(kEncoderDim) + + ", got " + std::to_string(tokens.size()) + ); + } + + tracking_state_.push(tracking_obs_, latest_action_); + const auto decoder_obs = build_decoder_obs_dict(tokens); + const auto raw_actions = run_single_f32(decoder_, "obs_dict", "action", decoder_obs, kDecoderObsDictDim); + if (raw_actions.size() != G1_NUM_MOTOR) { + throw std::runtime_error( + "decoder output dimension mismatch: expected " + std::to_string(G1_NUM_MOTOR) + + ", got " + std::to_string(raw_actions.size()) + ); + } + maybe_dump_tracking_tensors(encoder_obs, tokens, decoder_obs, raw_actions); + + std::vector positions(G1_NUM_MOTOR, 0.0f); + for (int mujoco_index = 0; mujoco_index < G1_NUM_MOTOR; ++mujoco_index) { + const int isaaclab_index = isaaclab_to_mujoco[static_cast(mujoco_index)]; + const float action = raw_actions[static_cast(isaaclab_index)]; + const float scaled = action * static_cast(g1_action_scale[static_cast(mujoco_index)]); + positions[static_cast(mujoco_index)] = + static_cast(default_angles[static_cast(mujoco_index)]) + scaled; + } + + latest_action_ = raw_actions; + return positions; + } + + std::vector planner_only_tick() { + std::deque> context; + const auto standing = PlannerState::make_standing_qpos(); + for (std::size_t index = 0; index < kPlannerContextLen; ++index) { + context.push_back(standing); + } + + float facing_yaw_rad = 0.0f; + const auto planner_command = derive_planner_command(facing_yaw_rad, velocity_command()); + const auto trajectory = run_planner_command(context, planner_command); + if (trajectory.empty()) { + throw std::runtime_error("planner-only benchmark produced an empty trajectory"); + } + return trajectory.front(); + } + + std::vector velocity_first_live_replan_dump() { + if (!dump_dir_.has_value()) { + throw std::runtime_error("--dump-dir is required for gear_sonic_velocity/first_live_replan_dump"); + } + + reset(); + tracking_state_.reset(); + latest_action_.assign(G1_NUM_MOTOR, 0.0f); + + std::deque> context; + const auto standing = make_official_standing_qpos(); + for (std::size_t index = 0; index < kPlannerContextLen; ++index) { + context.push_back(standing); + } + + const auto idle_planned_30hz = run_planner_command(context, idle_planner_command()); + const auto bootstrap_motion_50hz = resample_planner_trajectory_to_50hz(idle_planned_30hz); + + constexpr std::size_t kFirstLiveReplanTick = kPlannerThreadIntervalTicks; + const auto planner_context = + rebuild_planner_context_from_motion(bootstrap_motion_50hz, kFirstLiveReplanTick); + + float facing_yaw_rad = 0.0f; + Twist twist; + twist.linear = {0.6f, 0.0f, 0.0f}; + twist.angular = {0.0f, 0.0f, 0.0f}; + const auto live_command = derive_planner_command(facing_yaw_rad, twist); + const auto live_planned_30hz = run_planner_command(planner_context, live_command); + const auto live_planned_50hz = resample_planner_trajectory_to_50hz(live_planned_30hz); + const auto committed_motion_50hz = blend_planner_motion( + bootstrap_motion_50hz, + kFirstLiveReplanTick, + kFirstLiveReplanTick, + live_planned_50hz + ); + const auto committed_joint_velocities = + compute_motion_joint_velocities_isaaclab(committed_motion_50hz); + + const std::array init_base_quat_wxyz = {1.0, 0.0, 0.0, 0.0}; + const auto init_ref_root_quat_wxyz = + planner_frame_root_quaternion(committed_motion_50hz.front()); + const auto encoder_obs = build_velocity_encoder_obs_dict( + committed_motion_50hz, + committed_joint_velocities, + 0, + init_base_quat_wxyz, + init_base_quat_wxyz, + init_ref_root_quat_wxyz + ); + const auto tokens = run_single_f32( + encoder_, + "obs_dict", + "encoded_tokens", + encoder_obs, + kEncoderObsDictDim + ); + if (tokens.size() != kEncoderDim) { + throw std::runtime_error( + "encoder output dimension mismatch: expected " + std::to_string(kEncoderDim) + + ", got " + std::to_string(tokens.size()) + ); + } + + tracking_state_.push(tracking_obs_, latest_action_); + const auto decoder_obs = build_decoder_obs_dict(tokens); + const auto raw_actions = run_single_f32( + decoder_, + "obs_dict", + "action", + decoder_obs, + kDecoderObsDictDim + ); + if (raw_actions.size() != G1_NUM_MOTOR) { + throw std::runtime_error( + "decoder output dimension mismatch: expected " + std::to_string(G1_NUM_MOTOR) + + ", got " + std::to_string(raw_actions.size()) + ); + } + + const std::vector planner_command = { + static_cast(live_command.mode), + live_command.target_vel, + live_command.height, + live_command.movement_direction[0], + live_command.movement_direction[1], + live_command.movement_direction[2], + live_command.facing_direction[0], + live_command.facing_direction[1], + live_command.facing_direction[2], + }; + + write_matrix_json(*dump_dir_ / "bootstrap_motion_50hz.json", bootstrap_motion_50hz); + write_matrix_json(*dump_dir_ / "planner_context.json", deque_to_rows(planner_context)); + write_vector_json(*dump_dir_ / "planner_command.json", planner_command); + write_matrix_json(*dump_dir_ / "planner_motion_30hz.json", live_planned_30hz); + write_matrix_json(*dump_dir_ / "planner_motion_50hz.json", live_planned_50hz); + write_matrix_json( + *dump_dir_ / "planner_motion_50hz_committed.json", + committed_motion_50hz + ); + write_matrix_json( + *dump_dir_ / "planner_joint_velocities_50hz.json", + committed_joint_velocities + ); + write_vector_json(*dump_dir_ / "velocity_encoder_obs.json", encoder_obs); + write_vector_json(*dump_dir_ / "velocity_tokens.json", tokens); + write_vector_json(*dump_dir_ / "velocity_decoder_obs.json", decoder_obs); + write_vector_json(*dump_dir_ / "velocity_raw_actions.json", raw_actions); + + std::vector positions(G1_NUM_MOTOR, 0.0f); + for (int mujoco_index = 0; mujoco_index < G1_NUM_MOTOR; ++mujoco_index) { + const int isaaclab_index = isaaclab_to_mujoco[static_cast(mujoco_index)]; + const float action = raw_actions[static_cast(isaaclab_index)]; + const float scaled = + action * static_cast(g1_action_scale[static_cast(mujoco_index)]); + positions[static_cast(mujoco_index)] = + static_cast(default_angles[static_cast(mujoco_index)]) + scaled; + } + + latest_action_ = raw_actions; + return positions; + } + + std::vector velocity_later_motion_dump() { + if (!dump_dir_.has_value()) { + throw std::runtime_error("--dump-dir is required for gear_sonic_velocity/later_motion_dump"); + } + + reset(); + tracking_state_.reset(); + latest_action_.assign(G1_NUM_MOTOR, 0.0f); + + std::deque> context; + const auto standing = make_official_standing_qpos(); + for (std::size_t index = 0; index < kPlannerContextLen; ++index) { + context.push_back(standing); + } + + const auto idle_planned_30hz = run_planner_command(context, idle_planner_command()); + const auto bootstrap_motion_50hz = resample_planner_trajectory_to_50hz(idle_planned_30hz); + + constexpr std::size_t kFirstLiveReplanTick = kPlannerThreadIntervalTicks; + const auto planner_context = + rebuild_planner_context_from_motion(bootstrap_motion_50hz, kFirstLiveReplanTick); + + float facing_yaw_rad = 0.0f; + Twist twist; + twist.linear = {0.6f, 0.0f, 0.0f}; + twist.angular = {0.0f, 0.0f, 0.0f}; + const auto live_command = derive_planner_command(facing_yaw_rad, twist); + const auto live_planned_30hz = run_planner_command(planner_context, live_command); + const auto live_planned_50hz = resample_planner_trajectory_to_50hz(live_planned_30hz); + const auto committed_motion_50hz = blend_planner_motion( + bootstrap_motion_50hz, + kFirstLiveReplanTick, + kFirstLiveReplanTick, + live_planned_50hz + ); + const auto committed_joint_velocities = + compute_motion_joint_velocities_isaaclab(committed_motion_50hz); + if (committed_motion_50hz.size() <= kLaterMotionProbeTick) { + throw std::runtime_error("later motion probe tick exceeds committed motion length"); + } + + const std::array init_base_quat_wxyz = {1.0, 0.0, 0.0, 0.0}; + const auto init_ref_root_quat_wxyz = + planner_frame_root_quaternion(committed_motion_50hz.front()); + + Observation probe_obs; + std::vector encoder_obs; + std::vector tokens; + std::vector decoder_obs; + std::vector raw_actions; + for (std::size_t tick = 0; tick <= kLaterMotionProbeTick; ++tick) { + probe_obs = motion_observation_from_planner_frame( + committed_motion_50hz, + committed_joint_velocities, + tick + ); + encoder_obs = build_velocity_encoder_obs_dict( + committed_motion_50hz, + committed_joint_velocities, + tick, + probe_obs.base_quat_wxyz, + init_base_quat_wxyz, + init_ref_root_quat_wxyz + ); + tokens = run_single_f32( + encoder_, + "obs_dict", + "encoded_tokens", + encoder_obs, + kEncoderObsDictDim + ); + if (tokens.size() != kEncoderDim) { + throw std::runtime_error( + "encoder output dimension mismatch: expected " + std::to_string(kEncoderDim) + + ", got " + std::to_string(tokens.size()) + ); + } + + tracking_state_.push(probe_obs, latest_action_); + decoder_obs = build_decoder_obs_dict(tokens); + raw_actions = run_single_f32( + decoder_, + "obs_dict", + "action", + decoder_obs, + kDecoderObsDictDim + ); + if (raw_actions.size() != G1_NUM_MOTOR) { + throw std::runtime_error( + "decoder output dimension mismatch: expected " + std::to_string(G1_NUM_MOTOR) + + ", got " + std::to_string(raw_actions.size()) + ); + } + latest_action_ = raw_actions; + } + + write_vector_json( + *dump_dir_ / "velocity_probe_tick.json", + std::vector{static_cast(kLaterMotionProbeTick)} + ); + write_vector_json(*dump_dir_ / "current_joint_positions_mujoco.json", probe_obs.joint_positions); + write_vector_json(*dump_dir_ / "current_joint_velocities_mujoco.json", probe_obs.joint_velocities); + write_vector_json( + *dump_dir_ / "current_base_quat_wxyz.json", + std::vector{ + static_cast(probe_obs.base_quat_wxyz[0]), + static_cast(probe_obs.base_quat_wxyz[1]), + static_cast(probe_obs.base_quat_wxyz[2]), + static_cast(probe_obs.base_quat_wxyz[3]), + } + ); + write_vector_json( + *dump_dir_ / "current_gravity.json", + std::vector{ + probe_obs.gravity_vector[0], + probe_obs.gravity_vector[1], + probe_obs.gravity_vector[2], + } + ); + write_vector_json( + *dump_dir_ / "current_angular_velocity.json", + std::vector{ + probe_obs.angular_velocity[0], + probe_obs.angular_velocity[1], + probe_obs.angular_velocity[2], + } + ); + write_matrix_json( + *dump_dir_ / "history_joint_positions_isaaclab_offsets.json", + deque_to_rows(tracking_state_.joint_positions) + ); + write_matrix_json( + *dump_dir_ / "history_joint_velocities_isaaclab.json", + deque_to_rows(tracking_state_.joint_velocities) + ); + write_matrix_json( + *dump_dir_ / "history_last_actions.json", + deque_to_rows(tracking_state_.last_actions) + ); + write_matrix_json( + *dump_dir_ / "history_gravity.json", + deque_to_rows(tracking_state_.gravity) + ); + write_matrix_json( + *dump_dir_ / "history_angular_velocity.json", + deque_to_rows(tracking_state_.angular_velocity) + ); + write_vector_json(*dump_dir_ / "velocity_encoder_obs.json", encoder_obs); + write_vector_json(*dump_dir_ / "velocity_tokens.json", tokens); + write_vector_json(*dump_dir_ / "velocity_decoder_obs.json", decoder_obs); + write_vector_json(*dump_dir_ / "velocity_raw_actions.json", raw_actions); + + std::vector positions(G1_NUM_MOTOR, 0.0f); + for (int mujoco_index = 0; mujoco_index < G1_NUM_MOTOR; ++mujoco_index) { + const int isaaclab_index = isaaclab_to_mujoco[static_cast(mujoco_index)]; + const float action = raw_actions[static_cast(isaaclab_index)]; + const float scaled = + action * static_cast(g1_action_scale[static_cast(mujoco_index)]); + positions[static_cast(mujoco_index)] = + static_cast(default_angles[static_cast(mujoco_index)]) + scaled; + } + + return positions; + } + +private: + Ort::Env env_; + Ort::MemoryInfo memory_info_; + Ort::Session encoder_; + Ort::Session decoder_; + Ort::Session planner_; + PlannerState planner_state_; + TrackingState tracking_state_; + Observation velocity_obs_; + Observation tracking_obs_; + std::vector latest_action_; + std::optional dump_dir_; + bool dumped_tracking_tensors_ = false; + + void validate_contracts() { + const auto planner_inputs = session_names(planner_, true); + const auto planner_outputs = session_names(planner_, false); + for (const auto* name : { + "context_mujoco_qpos", + "target_vel", + "mode", + "movement_direction", + "facing_direction", + "random_seed", + "has_specific_target", + "specific_target_positions", + "specific_target_headings", + "allowed_pred_num_tokens", + "height", + }) { + require_name(planner_inputs, name, "planner input"); + } + require_name(planner_outputs, "mujoco_qpos", "planner output"); + require_name(planner_outputs, "num_pred_frames", "planner output"); + + const auto encoder_inputs = session_names(encoder_, true); + const auto encoder_outputs = session_names(encoder_, false); + require_name(encoder_inputs, "obs_dict", "encoder input"); + require_name(encoder_outputs, "encoded_tokens", "encoder output"); + + const auto decoder_inputs = session_names(decoder_, true); + const auto decoder_outputs = session_names(decoder_, false); + require_name(decoder_inputs, "obs_dict", "decoder input"); + require_name(decoder_outputs, "action", "decoder output"); + } + + std::vector planner_context_frame( + const std::vector& template_frame, + const Observation& obs + ) const { + std::vector frame = + template_frame.size() == kPlannerQposDim ? template_frame : std::vector(kPlannerQposDim, 0.0f); + if (frame[2] == 0.0f) { + frame[2] = kDefaultHeightMeters; + } + frame[3] = static_cast(obs.base_quat_wxyz[0]); + frame[4] = static_cast(obs.base_quat_wxyz[1]); + frame[5] = static_cast(obs.base_quat_wxyz[2]); + frame[6] = static_cast(obs.base_quat_wxyz[3]); + std::copy(obs.joint_positions.begin(), obs.joint_positions.end(), frame.begin() + static_cast(kPlannerJointOffset)); + return frame; + } + + Twist velocity_command() const { + Twist twist; + twist.linear = {0.3f, 0.0f, 0.0f}; + twist.angular = {0.0f, 0.0f, 0.0f}; + return twist; + } + + static std::vector make_official_standing_qpos() { + std::vector qpos(kPlannerQposDim, 0.0f); + qpos[2] = kOfficialDefaultHeightMeters; + qpos[3] = 1.0f; + const auto pose = default_pose(); + std::copy( + pose.begin(), + pose.end(), + qpos.begin() + static_cast(kPlannerJointOffset) + ); + return qpos; + } + + void commit_planner_motion( + std::size_t request_motion_frame, + const PlannerCommand& planner_command, + const std::vector>& planned_50hz + ) { + if (planned_50hz.empty()) { + throw std::runtime_error("planner produced an empty 50Hz motion sequence"); + } + + planner_state_.motion_qpos_50hz = + planner_state_.motion_qpos_50hz.empty() + ? planned_50hz + : blend_planner_motion( + planner_state_.motion_qpos_50hz, + planner_state_.current_motion_frame, + request_motion_frame, + planned_50hz + ); + planner_state_.motion_joint_velocities_isaaclab = + compute_motion_joint_velocities_isaaclab(planner_state_.motion_qpos_50hz); + planner_state_.current_motion_frame = 0; + if (planner_state_.motion_qpos_50hz.empty()) { + planner_state_.init_ref_root_quat_wxyz.reset(); + } else { + planner_state_.init_ref_root_quat_wxyz = + planner_frame_root_quaternion(planner_state_.motion_qpos_50hz.front()); + } + if (!planner_state_.init_base_quat_wxyz.has_value()) { + planner_state_.init_base_quat_wxyz = velocity_obs_.base_quat_wxyz; + } + planner_state_.last_command = planner_command; + } + + void maybe_apply_pending_planner_replan() { + if (!planner_state_.pending_replan.has_value()) { + return; + } + + auto& pending = planner_state_.pending_replan.value(); + if (pending.future.wait_for(std::chrono::seconds(0)) != std::future_status::ready) { + return; + } + + auto ready = std::move(pending); + planner_state_.pending_replan.reset(); + commit_planner_motion( + ready.request_motion_frame, + ready.command, + ready.future.get() + ); + } + + void start_async_planner_replan(const PlannerCommand& planner_command) { + PlannerState::PendingPlannerReplan pending; + pending.request_motion_frame = planner_state_.current_motion_frame; + pending.command = planner_command; + const auto context = planner_state_.context; + pending.future = std::async( + std::launch::async, + [this, context, planner_command]() { + return resample_planner_trajectory_to_50hz( + run_planner_command(context, planner_command) + ); + } + ); + planner_state_.last_command = planner_command; + planner_state_.pending_replan = std::move(pending); + } + + void advance_planner_motion_frame() { + if (planner_state_.motion_qpos_50hz.empty()) { + return; + } + planner_state_.current_motion_frame = std::min( + planner_state_.current_motion_frame + 1, + planner_state_.motion_qpos_50hz.size() - 1 + ); + } + + std::vector> run_planner( + const std::deque>& context, + const Twist& twist + ) { + const float cmd_norm = + std::sqrt(twist.linear[0] * twist.linear[0] + twist.linear[1] * twist.linear[1]); + const std::array movement_direction = + cmd_norm > 1e-6f + ? std::array{ + twist.linear[0] / cmd_norm, + twist.linear[1] / cmd_norm, + 0.0f, + } + : std::array{1.0f, 0.0f, 0.0f}; + const float yaw = twist.angular[2]; + const std::array facing_direction = { + std::cos(yaw), + std::sin(yaw), + 0.0f, + }; + return run_planner_command( + context, + PlannerCommand{ + kDefaultModeWalk, + cmd_norm, + kDefaultHeightMeters, + movement_direction, + facing_direction, + } + ); + } + + std::vector> run_planner_command( + const std::deque>& context, + const PlannerCommand& command + ) { + std::vector context_data; + context_data.reserve(context.size() * kPlannerQposDim); + for (const auto& frame : context) { + context_data.insert(context_data.end(), frame.begin(), frame.end()); + } + + const std::array target_vel = {command.target_vel}; + const std::array mode = {command.mode}; + const std::array height = {command.height}; + const std::array random_seed = {0}; + const std::array has_specific_target = {0}; + const std::vector specific_target_positions(12, 0.0f); + const std::vector specific_target_headings(4, 0.0f); + const auto allowed_pred_num_tokens = kAllowedPredNumTokensMask; + + const std::array context_shape = { + 1, + static_cast(kPlannerContextLen), + static_cast(kPlannerQposDim), + }; + const std::array vec3_shape = {1, 3}; + const std::array scalar_shape = {1}; + const std::array has_target_shape = {1, 1}; + const std::array specific_positions_shape = {1, 4, 3}; + const std::array specific_headings_shape = {1, 4}; + const std::array allowed_tokens_shape = { + 1, + static_cast(kAllowedPredNumTokens), + }; + + std::vector input_tensors; + input_tensors.reserve(11); + input_tensors.push_back(Ort::Value::CreateTensor( + memory_info_, + context_data.data(), + context_data.size(), + context_shape.data(), + context_shape.size() + )); + input_tensors.push_back(Ort::Value::CreateTensor( + memory_info_, + const_cast(target_vel.data()), + target_vel.size(), + scalar_shape.data(), + scalar_shape.size() + )); + input_tensors.push_back(Ort::Value::CreateTensor( + memory_info_, + const_cast(mode.data()), + mode.size(), + scalar_shape.data(), + scalar_shape.size() + )); + input_tensors.push_back(Ort::Value::CreateTensor( + memory_info_, + const_cast(command.movement_direction.data()), + command.movement_direction.size(), + vec3_shape.data(), + vec3_shape.size() + )); + input_tensors.push_back(Ort::Value::CreateTensor( + memory_info_, + const_cast(command.facing_direction.data()), + command.facing_direction.size(), + vec3_shape.data(), + vec3_shape.size() + )); + input_tensors.push_back(Ort::Value::CreateTensor( + memory_info_, + const_cast(random_seed.data()), + random_seed.size(), + scalar_shape.data(), + scalar_shape.size() + )); + input_tensors.push_back(Ort::Value::CreateTensor( + memory_info_, + const_cast(has_specific_target.data()), + has_specific_target.size(), + has_target_shape.data(), + has_target_shape.size() + )); + input_tensors.push_back(Ort::Value::CreateTensor( + memory_info_, + const_cast(specific_target_positions.data()), + specific_target_positions.size(), + specific_positions_shape.data(), + specific_positions_shape.size() + )); + input_tensors.push_back(Ort::Value::CreateTensor( + memory_info_, + const_cast(specific_target_headings.data()), + specific_target_headings.size(), + specific_headings_shape.data(), + specific_headings_shape.size() + )); + input_tensors.push_back(Ort::Value::CreateTensor( + memory_info_, + const_cast(allowed_pred_num_tokens.data()), + allowed_pred_num_tokens.size(), + allowed_tokens_shape.data(), + allowed_tokens_shape.size() + )); + input_tensors.push_back(Ort::Value::CreateTensor( + memory_info_, + const_cast(height.data()), + height.size(), + scalar_shape.data(), + scalar_shape.size() + )); + + static constexpr const char* kPlannerInputNames[] = { + "context_mujoco_qpos", + "target_vel", + "mode", + "movement_direction", + "facing_direction", + "random_seed", + "has_specific_target", + "specific_target_positions", + "specific_target_headings", + "allowed_pred_num_tokens", + "height", + }; + static constexpr const char* kPlannerOutputNames[] = { + "mujoco_qpos", + "num_pred_frames", + }; + + auto outputs = planner_.Run( + Ort::RunOptions{nullptr}, + kPlannerInputNames, + input_tensors.data(), + input_tensors.size(), + kPlannerOutputNames, + std::size(kPlannerOutputNames) + ); + + const Ort::Value& mujoco_qpos = outputs[0]; + const Ort::Value& num_pred_frames = outputs[1]; + const auto qpos_info = mujoco_qpos.GetTensorTypeAndShapeInfo(); + const auto qpos_shape = qpos_info.GetShape(); + const std::size_t available_frames = qpos_info.GetElementCount() / kPlannerQposDim; + const float* qpos_data = mujoco_qpos.GetTensorData(); + + std::int64_t predicted_frames = 1; + const auto frames_info = num_pred_frames.GetTensorTypeAndShapeInfo(); + if (frames_info.GetElementType() == ONNX_TENSOR_ELEMENT_DATA_TYPE_INT32) { + predicted_frames = + static_cast(num_pred_frames.GetTensorData()[0]); + } else if (frames_info.GetElementType() == ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64) { + predicted_frames = num_pred_frames.GetTensorData()[0]; + } else { + throw std::runtime_error("planner num_pred_frames output must be int32 or int64"); + } + predicted_frames = std::max(predicted_frames, 1); + const std::size_t frame_count = + std::min(static_cast(predicted_frames), available_frames); + + if (!(qpos_shape.size() == 3 || qpos_shape.size() == 2)) { + throw std::runtime_error("unexpected planner mujoco_qpos rank"); + } + + std::vector> trajectory; + trajectory.reserve(frame_count); + for (std::size_t frame = 0; frame < frame_count; ++frame) { + const float* begin = qpos_data + frame * kPlannerQposDim; + trajectory.emplace_back( + begin, + begin + static_cast(kPlannerQposDim) + ); + } + return trajectory; + } + + static std::array planner_frame_root_quaternion(const std::vector& frame) { + return quat_unit_d({ + static_cast(frame[3]), + static_cast(frame[4]), + static_cast(frame[5]), + static_cast(frame[6]), + }); + } + + static std::vector interpolate_planner_qpos( + const std::vector& frame_a, + const std::vector& frame_b, + float alpha + ) { + std::vector frame(frame_a.size(), 0.0f); + for (std::size_t index = 0; index < frame.size(); ++index) { + frame[index] = frame_a[index] + alpha * (frame_b[index] - frame_a[index]); + } + + const auto quat = quat_slerp_d( + planner_frame_root_quaternion(frame_a), + planner_frame_root_quaternion(frame_b), + static_cast(alpha) + ); + frame[3] = static_cast(quat[0]); + frame[4] = static_cast(quat[1]); + frame[5] = static_cast(quat[2]); + frame[6] = static_cast(quat[3]); + return frame; + } + + static std::vector sample_motion_qpos_50hz( + const std::vector>& motion_qpos, + float frame_idx + ) { + if (motion_qpos.empty()) { + return {}; + } + if (motion_qpos.size() == 1) { + return motion_qpos.front(); + } + + const float clamped = std::clamp( + frame_idx, + 0.0f, + static_cast(motion_qpos.size() - 1) + ); + const auto frame_a_idx = static_cast(std::floor(clamped)); + const auto frame_b_idx = std::min(frame_a_idx + 1, motion_qpos.size() - 1); + const float alpha = clamped - static_cast(frame_a_idx); + return interpolate_planner_qpos( + motion_qpos[frame_a_idx], + motion_qpos[frame_b_idx], + alpha + ); + } + + static std::vector> resample_planner_trajectory_to_50hz( + const std::vector>& trajectory + ) { + if (trajectory.empty()) { + return {}; + } + + const float motion_seconds = static_cast(trajectory.size()) / 30.0f; + const auto frame_count = std::max( + static_cast(std::floor(motion_seconds * 50.0f)), + static_cast(1) + ); + std::vector> motion_qpos; + motion_qpos.reserve(frame_count); + for (std::size_t frame_50hz = 0; frame_50hz < frame_count; ++frame_50hz) { + const float frame_30hz = static_cast(frame_50hz) * 30.0f / 50.0f; + const auto frame_a_idx = static_cast(std::floor(frame_30hz)); + const auto frame_b_idx = std::min(frame_a_idx + 1, trajectory.size() - 1); + const float alpha = frame_30hz - static_cast(frame_a_idx); + motion_qpos.push_back(interpolate_planner_qpos( + trajectory[frame_a_idx], + trajectory[frame_b_idx], + alpha + )); + } + return motion_qpos; + } + + static std::deque> rebuild_planner_context_from_motion( + const std::vector>& motion_qpos_50hz, + std::size_t current_motion_frame + ) { + std::deque> context; + if (motion_qpos_50hz.empty()) { + return context; + } + + const float gen_time = static_cast(current_motion_frame + 2) * kControlDtSeconds; + for (std::size_t frame_idx = 0; frame_idx < kPlannerContextLen; ++frame_idx) { + const float sample_time = gen_time + static_cast(frame_idx) / 30.0f; + context.push_back(sample_motion_qpos_50hz( + motion_qpos_50hz, + sample_time * 50.0f + )); + } + return context; + } + + static std::vector planner_joint_positions_isaaclab(const std::vector& frame) { + std::vector positions(G1_NUM_MOTOR, 0.0f); + for (std::size_t mujoco_idx = 0; mujoco_idx < G1_NUM_MOTOR; ++mujoco_idx) { + const auto isaaclab_idx = static_cast(isaaclab_to_mujoco[mujoco_idx]); + positions[isaaclab_idx] = frame[kPlannerJointOffset + mujoco_idx]; + } + return positions; + } + + static std::vector> compute_motion_joint_velocities_isaaclab( + const std::vector>& motion_qpos + ) { + if (motion_qpos.empty()) { + return {}; + } + + std::vector> positions; + positions.reserve(motion_qpos.size()); + for (const auto& frame : motion_qpos) { + positions.push_back(planner_joint_positions_isaaclab(frame)); + } + + std::vector> velocities( + positions.size(), + std::vector(G1_NUM_MOTOR, 0.0f) + ); + for (std::size_t frame_idx = 0; frame_idx + 1 < positions.size(); ++frame_idx) { + for (std::size_t joint_idx = 0; joint_idx < G1_NUM_MOTOR; ++joint_idx) { + velocities[frame_idx][joint_idx] = + (positions[frame_idx + 1][joint_idx] - positions[frame_idx][joint_idx]) * 50.0f; + } + } + if (positions.size() > 1) { + velocities.back() = velocities[velocities.size() - 2]; + } + return velocities; + } + + static std::vector> blend_planner_motion( + const std::vector>& existing_motion_qpos, + std::size_t current_motion_frame, + std::size_t request_motion_frame, + const std::vector>& new_motion_qpos + ) { + if (existing_motion_qpos.empty()) { + return new_motion_qpos; + } + + const auto gen_frame = request_motion_frame + kPlannerLookAheadSteps; + const auto lead_frames = + gen_frame > current_motion_frame ? gen_frame - current_motion_frame : 0; + const auto new_anim_length = lead_frames + new_motion_qpos.size(); + const auto blend_start_frame = lead_frames; + + std::vector> blended; + blended.reserve(new_anim_length); + for (std::size_t frame_idx = 0; frame_idx < new_anim_length; ++frame_idx) { + auto old_frame_idx = frame_idx + current_motion_frame; + if (old_frame_idx >= existing_motion_qpos.size()) { + old_frame_idx = existing_motion_qpos.size() - 1; + } + + std::size_t new_frame_idx = 0; + if (frame_idx + current_motion_frame >= gen_frame) { + new_frame_idx = frame_idx + current_motion_frame - gen_frame; + } + if (new_frame_idx >= new_motion_qpos.size()) { + new_frame_idx = new_motion_qpos.size() - 1; + } + + const auto weight_new = std::clamp( + (static_cast(frame_idx) - static_cast(blend_start_frame)) / + static_cast(kPlannerBlendFrames), + 0.0f, + 1.0f + ); + if (weight_new <= std::numeric_limits::epsilon()) { + blended.push_back(existing_motion_qpos[old_frame_idx]); + } else if (std::abs(1.0f - weight_new) <= std::numeric_limits::epsilon()) { + blended.push_back(new_motion_qpos[new_frame_idx]); + } else { + blended.push_back(interpolate_planner_qpos( + existing_motion_qpos[old_frame_idx], + new_motion_qpos[new_frame_idx], + weight_new + )); + } + } + + return blended; + } + + static std::vector build_velocity_encoder_obs_dict( + const std::vector>& motion_qpos_50hz, + const std::vector>& motion_joint_velocities_isaaclab, + std::size_t current_motion_frame, + const std::array& base_quat_wxyz, + const std::array& init_base_quat_wxyz, + const std::array& init_ref_root_quat_wxyz + ) { + if (motion_qpos_50hz.empty()) { + throw std::runtime_error("planner motion buffer is empty"); + } + + const auto apply_delta_heading = quat_mul_d( + calc_heading_quat_d(init_base_quat_wxyz), + calc_heading_quat_inv_d(init_ref_root_quat_wxyz) + ); + const auto base_quat = quat_unit_d(base_quat_wxyz); + + std::vector buf(kEncoderObsDictDim, 0.0f); + buf[kEncoderModeOffset] = 0.0f; + + for (std::size_t frame_idx = 0; frame_idx < kReferenceFutureFrames; ++frame_idx) { + const auto target_frame = std::min( + current_motion_frame + frame_idx * kReferenceFrameStep, + motion_qpos_50hz.size() - 1 + ); + const auto& motion_frame = motion_qpos_50hz[target_frame]; + const auto isaaclab_positions = planner_joint_positions_isaaclab(motion_frame); + + const auto pos_offset = + kEncoderMotionJointPositionsOffset + frame_idx * isaaclab_positions.size(); + std::copy( + isaaclab_positions.begin(), + isaaclab_positions.end(), + buf.begin() + static_cast(pos_offset) + ); + + if (target_frame < motion_joint_velocities_isaaclab.size()) { + const auto& joint_velocities = motion_joint_velocities_isaaclab[target_frame]; + const auto vel_offset = + kEncoderMotionJointVelocitiesOffset + frame_idx * joint_velocities.size(); + std::copy( + joint_velocities.begin(), + joint_velocities.end(), + buf.begin() + static_cast(vel_offset) + ); + } + + const auto ref_root_quat = + quat_mul_d(apply_delta_heading, planner_frame_root_quaternion(motion_frame)); + const auto base_to_ref = quat_mul_d(quat_conjugate_d(base_quat), ref_root_quat); + const auto rotation_matrix = quat_to_rotation_matrix_d(base_to_ref); + const auto orn_offset = kEncoderMotionAnchorOrientationOffset + frame_idx * 6; + buf[orn_offset] = static_cast(rotation_matrix[0][0]); + buf[orn_offset + 1] = static_cast(rotation_matrix[0][1]); + buf[orn_offset + 2] = static_cast(rotation_matrix[1][0]); + buf[orn_offset + 3] = static_cast(rotation_matrix[1][1]); + buf[orn_offset + 4] = static_cast(rotation_matrix[2][0]); + buf[orn_offset + 5] = static_cast(rotation_matrix[2][1]); + } + + return buf; + } + + static Observation motion_observation_from_planner_frame( + const std::vector>& motion_qpos_50hz, + const std::vector>& motion_joint_velocities_isaaclab, + std::size_t frame_idx + ) { + if (motion_qpos_50hz.empty()) { + throw std::runtime_error("planner motion buffer is empty"); + } + + const auto clamped_index = std::min(frame_idx, motion_qpos_50hz.size() - 1); + const auto& motion_frame = motion_qpos_50hz[clamped_index]; + Observation obs; + obs.joint_positions.assign( + motion_frame.begin() + static_cast(kPlannerJointOffset), + motion_frame.end() + ); + obs.joint_velocities = + clamped_index < motion_joint_velocities_isaaclab.size() + ? isaaclab_to_mujoco_values(motion_joint_velocities_isaaclab[clamped_index]) + : std::vector(G1_NUM_MOTOR, 0.0f); + obs.base_quat_wxyz = planner_frame_root_quaternion(motion_frame); + obs.gravity_vector = double_to_float(GetGravityOrientation_d(obs.base_quat_wxyz)); + obs.angular_velocity = + angular_velocity_from_motion_quaternions(motion_qpos_50hz, clamped_index); + return obs; + } + + static std::array angular_velocity_from_motion_quaternions( + const std::vector>& motion_qpos_50hz, + std::size_t frame_idx + ) { + if (frame_idx == 0 || motion_qpos_50hz.empty()) { + return {0.0f, 0.0f, 0.0f}; + } + + const auto prev_quat = planner_frame_root_quaternion(motion_qpos_50hz[frame_idx - 1]); + const auto curr_quat = planner_frame_root_quaternion(motion_qpos_50hz[frame_idx]); + auto delta = quat_mul_d(quat_conjugate_d(prev_quat), curr_quat); + if (delta[0] < 0.0) { + delta = {-delta[0], -delta[1], -delta[2], -delta[3]}; + } + delta = quat_unit_d(delta); + + const auto sin_half = std::sqrt( + delta[1] * delta[1] + delta[2] * delta[2] + delta[3] * delta[3] + ); + if (sin_half <= 1e-6) { + return {0.0f, 0.0f, 0.0f}; + } + + const auto [angle, axis] = quat_to_angle_axis(delta); + if (!std::isfinite(angle)) { + return {0.0f, 0.0f, 0.0f}; + } + + return { + static_cast(axis[0] * angle / kControlDtSeconds), + static_cast(axis[1] * angle / kControlDtSeconds), + static_cast(axis[2] * angle / kControlDtSeconds), + }; + } + + std::vector build_encoder_obs_dict() const { + std::vector buf(kEncoderObsDictDim, 0.0f); + + const auto pose = mujoco_to_isaaclab_positions(default_pose()); + const std::size_t pos_offset = 4; + const std::size_t vel_offset = 294; + const std::size_t orn_offset = 601; + + for (std::size_t frame = 0; frame < kDecoderHistoryLen; ++frame) { + const std::size_t pose_index = pos_offset + frame * G1_NUM_MOTOR; + const std::size_t velocity_index = vel_offset + frame * G1_NUM_MOTOR; + std::copy(pose.begin(), pose.end(), buf.begin() + static_cast(pose_index)); + std::fill_n(buf.begin() + static_cast(velocity_index), G1_NUM_MOTOR, 0.0f); + } + + for (std::size_t frame = 0; frame < kDecoderHistoryLen; ++frame) { + const std::size_t orientation_index = orn_offset + frame * 6; + buf[orientation_index] = 1.0f; + buf[orientation_index + 4] = 1.0f; + } + + return buf; + } + + std::vector build_decoder_obs_dict(const std::vector& tokens) const { + std::vector buf; + buf.reserve(kDecoderObsDictDim); + buf.insert(buf.end(), tokens.begin(), tokens.end()); + + const std::size_t history_skip = tracking_state_.gravity.size() > kDecoderHistoryLen + ? tracking_state_.gravity.size() - kDecoderHistoryLen + : 0; + + append_history_vectors(buf, tracking_state_.angular_velocity, history_skip); + append_history_vectors(buf, tracking_state_.joint_positions, history_skip); + append_history_vectors(buf, tracking_state_.joint_velocities, history_skip); + append_history_vectors(buf, tracking_state_.last_actions, history_skip); + append_history_vectors(buf, tracking_state_.gravity, history_skip); + + return buf; + } + + template + static void append_history_vectors( + std::vector& destination, + const std::deque& history, + std::size_t skip + ) { + for (std::size_t index = skip; index < history.size(); ++index) { + destination.insert(destination.end(), history[index].begin(), history[index].end()); + } + } + + std::vector run_single_f32( + Ort::Session& session, + const char* input_name, + const char* output_name, + const std::vector& input, + std::size_t expected_input_dim + ) { + if (input.size() != expected_input_dim) { + throw std::runtime_error( + "input dimension mismatch for " + std::string(output_name) + + ": expected " + std::to_string(expected_input_dim) + + ", got " + std::to_string(input.size()) + ); + } + + const std::array shape = {1, static_cast(expected_input_dim)}; + auto input_tensor = Ort::Value::CreateTensor( + memory_info_, + const_cast(input.data()), + input.size(), + shape.data(), + shape.size() + ); + + const char* input_names[] = {input_name}; + const char* output_names[] = {output_name}; + auto outputs = session.Run( + Ort::RunOptions{nullptr}, + input_names, + &input_tensor, + 1, + output_names, + 1 + ); + + const Ort::Value& output = outputs.front(); + const auto info = output.GetTensorTypeAndShapeInfo(); + const std::size_t count = info.GetElementCount(); + const float* data = output.GetTensorData(); + return std::vector(data, data + static_cast(count)); + } + + static void write_vector_json(const fs::path& path, const std::vector& values) { + fs::create_directories(path.parent_path()); + std::ofstream out(path); + if (!out) { + throw std::runtime_error("failed to open tensor dump file: " + path.string()); + } + + out << "[\n"; + for (std::size_t index = 0; index < values.size(); ++index) { + out << " " << values[index]; + if (index + 1 < values.size()) { + out << ","; + } + out << "\n"; + } + out << "]\n"; + } + + static void write_matrix_json( + const fs::path& path, + const std::vector>& rows + ) { + fs::create_directories(path.parent_path()); + std::ofstream out(path); + if (!out) { + throw std::runtime_error("failed to open tensor dump file: " + path.string()); + } + + out << "[\n"; + for (std::size_t row_index = 0; row_index < rows.size(); ++row_index) { + out << " [\n"; + for (std::size_t value_index = 0; value_index < rows[row_index].size(); ++value_index) { + out << " " << rows[row_index][value_index]; + if (value_index + 1 < rows[row_index].size()) { + out << ","; + } + out << "\n"; + } + out << " ]"; + if (row_index + 1 < rows.size()) { + out << ","; + } + out << "\n"; + } + out << "]\n"; + } + + static std::vector> deque_to_rows( + const std::deque>& rows + ) { + return {rows.begin(), rows.end()}; + } + + template + static std::vector> deque_to_rows( + const std::deque>& rows + ) { + std::vector> out; + out.reserve(rows.size()); + for (const auto& row : rows) { + out.emplace_back(row.begin(), row.end()); + } + return out; + } + + void maybe_dump_tracking_tensors( + const std::vector& encoder_obs, + const std::vector& tokens, + const std::vector& decoder_obs, + const std::vector& raw_actions + ) { + if (!dump_dir_.has_value() || dumped_tracking_tensors_) { + return; + } + + write_vector_json(*dump_dir_ / "tracking_encoder_obs.json", encoder_obs); + write_vector_json(*dump_dir_ / "tracking_tokens.json", tokens); + write_vector_json(*dump_dir_ / "tracking_decoder_obs.json", decoder_obs); + write_vector_json(*dump_dir_ / "tracking_raw_actions.json", raw_actions); + dumped_tracking_tensors_ = true; + } +}; + +enum class CaseKind { + PlannerOnlyColdStart, + PlannerOnlySteadyState, + EncoderDecoderOnlyTrackingTick, + FullVelocityTickColdStart, + FullVelocityTickSteadyState, + FullVelocityTickReplanBoundary, + FirstLiveReplanDump, + LaterMotionDump, + EndToEndLoop, +}; + +CaseKind parse_case_kind(const std::string& case_id) { + if (case_id == "gear_sonic/planner_only_cold_start") { + return CaseKind::PlannerOnlyColdStart; + } + if (case_id == "gear_sonic/planner_only_steady_state") { + return CaseKind::PlannerOnlySteadyState; + } + if (case_id == "gear_sonic/encoder_decoder_only_tracking_tick") { + return CaseKind::EncoderDecoderOnlyTrackingTick; + } + if (case_id == "gear_sonic/full_velocity_tick_cold_start") { + return CaseKind::FullVelocityTickColdStart; + } + if (case_id == "gear_sonic/full_velocity_tick_steady_state") { + return CaseKind::FullVelocityTickSteadyState; + } + if (case_id == "gear_sonic/full_velocity_tick_replan_boundary") { + return CaseKind::FullVelocityTickReplanBoundary; + } + if (case_id == "gear_sonic_velocity/first_live_replan_dump") { + return CaseKind::FirstLiveReplanDump; + } + if (case_id == "gear_sonic_velocity/later_motion_dump") { + return CaseKind::LaterMotionDump; + } + if (case_id == "gear_sonic/end_to_end_cli_loop") { + return CaseKind::EndToEndLoop; + } + throw std::runtime_error("unsupported GEAR-Sonic case_id: " + case_id); +} + +template +std::vector run_microbench( + GearSonicOfficialHarness& harness, + int samples, + SetupFn&& setup, + MeasureFn&& measure +) { + std::vector timings; + timings.reserve(static_cast(samples)); + for (int sample = 0; sample < samples; ++sample) { + harness.reset(); + setup(); + const auto start = std::chrono::steady_clock::now(); + const auto result = measure(); + const auto end = std::chrono::steady_clock::now(); + g_sink = g_sink + (result.empty() ? 0.0f : result.front()); + timings.push_back( + static_cast( + std::chrono::duration_cast(end - start).count() + ) + ); + } + return timings; +} + +template +std::pair, double> run_end_to_end_loop( + GearSonicOfficialHarness& harness, + int ticks, + int control_frequency_hz, + Fn&& fn +) { + const auto period_ns = static_cast(std::llround(1'000'000'000.0 / static_cast(control_frequency_hz))); + std::vector timings; + timings.reserve(static_cast(ticks)); + + harness.reset(); + const auto wall_start = std::chrono::steady_clock::now(); + for (int tick = 0; tick < ticks; ++tick) { + const auto start = std::chrono::steady_clock::now(); + const auto result = fn(); + const auto end = std::chrono::steady_clock::now(); + g_sink = g_sink + (result.empty() ? 0.0f : result.front()); + + const auto elapsed_ns = + std::chrono::duration_cast(end - start).count(); + timings.push_back(static_cast(elapsed_ns)); + + const auto remaining_ns = period_ns - elapsed_ns; + if (remaining_ns > 0) { + std::this_thread::sleep_for(std::chrono::nanoseconds(remaining_ns)); + } + } + const auto wall_end = std::chrono::steady_clock::now(); + const auto elapsed_seconds = std::chrono::duration(wall_end - wall_start).count(); + const double achieved_hz = elapsed_seconds > 0.0 ? static_cast(ticks) / elapsed_seconds : 0.0; + return {timings, achieved_hz}; +} + +ProviderKind parse_provider(const std::string& value) { + if (value == "cpu") { + return ProviderKind::Cpu; + } + if (value == "cuda") { + return ProviderKind::Cuda; + } + if (value == "tensor_rt") { + return ProviderKind::TensorRt; + } + throw std::runtime_error( + std::string("unsupported provider `") + value + + "`; expected one of: cpu, cuda, tensor_rt" + ); +} + +Options parse_args(int argc, char** argv) { + Options options; + for (int index = 1; index < argc; ++index) { + const std::string arg = argv[index]; + auto require_value = [&](const std::string& flag) -> std::string { + if (index + 1 >= argc) { + throw std::runtime_error(flag + " requires a value"); + } + return argv[++index]; + }; + + if (arg == "--case-id") { + options.case_id = require_value(arg); + } else if (arg == "--provider") { + options.provider = parse_provider(require_value(arg)); + } else if (arg == "--model-dir") { + options.model_dir = require_value(arg); + } else if (arg == "--output") { + options.output = require_value(arg); + } else if (arg == "--dump-dir") { + options.dump_dir = fs::path(require_value(arg)); + } else if (arg == "--samples") { + options.samples = std::stoi(require_value(arg)); + } else if (arg == "--ticks") { + options.ticks = std::stoi(require_value(arg)); + } else if (arg == "--control-frequency-hz") { + options.control_frequency_hz = std::stoi(require_value(arg)); + } else { + throw std::runtime_error("unknown argument: " + arg); + } + } + + if (options.case_id.empty()) { + throw std::runtime_error("--case-id is required"); + } + if (options.model_dir.empty()) { + throw std::runtime_error("--model-dir is required"); + } + if (options.output.empty()) { + throw std::runtime_error("--output is required"); + } + if (options.samples <= 0) { + throw std::runtime_error("--samples must be positive"); + } + if (options.ticks <= 0) { + throw std::runtime_error("--ticks must be positive"); + } + if (options.control_frequency_hz <= 0) { + throw std::runtime_error("--control-frequency-hz must be positive"); + } + return options; +} + +void write_json( + const Options& options, + const std::vector& samples_ns, + const std::optional& hz +) { + fs::create_directories(options.output.parent_path()); + std::ofstream out(options.output); + if (!out) { + throw std::runtime_error("failed to open output file: " + options.output.string()); + } + + out << "{\n"; + out << " \"case_id\": \"" << json_escape(options.case_id) << "\",\n"; + out << " \"samples_ns\": ["; + for (std::size_t index = 0; index < samples_ns.size(); ++index) { + if (index == 0) { + out << "\n "; + } else { + out << ",\n "; + } + out << samples_ns[index]; + } + if (!samples_ns.empty()) { + out << '\n'; + } + out << " ],\n"; + out << " \"hz\": "; + if (hz.has_value()) { + out << *hz; + } else { + out << "null"; + } + out << ",\n"; + out << " \"notes\": \"Measured via official GEAR-Sonic C++ ONNX Runtime harness on the published planner and encoder/decoder contracts.\"\n"; + out << "}\n"; +} + +} // namespace + +int main(int argc, char** argv) { + try { + const Options options = parse_args(argc, argv); + const CaseKind case_kind = parse_case_kind(options.case_id); + GearSonicOfficialHarness harness(options.model_dir, options.provider, options.dump_dir); + + std::vector samples_ns; + std::optional hz; + + switch (case_kind) { + case CaseKind::PlannerOnlyColdStart: + samples_ns = run_microbench( + harness, + options.samples, + []() {}, + [&]() { return harness.planner_only_tick(); } + ); + break; + case CaseKind::PlannerOnlySteadyState: + samples_ns = run_microbench( + harness, + options.samples, + [&]() { harness.planner_only_tick(); }, + [&]() { return harness.planner_only_tick(); } + ); + break; + case CaseKind::EncoderDecoderOnlyTrackingTick: + samples_ns = run_microbench( + harness, + options.samples, + []() {}, + [&]() { return harness.tracking_tick(); } + ); + break; + case CaseKind::FullVelocityTickColdStart: + samples_ns = run_microbench( + harness, + options.samples, + []() {}, + [&]() { return harness.velocity_tick(); } + ); + break; + case CaseKind::FullVelocityTickSteadyState: + samples_ns = run_microbench( + harness, + options.samples, + [&]() { harness.velocity_tick(); }, + [&]() { return harness.velocity_tick(); } + ); + break; + case CaseKind::FullVelocityTickReplanBoundary: + samples_ns = run_microbench( + harness, + options.samples, + [&]() { + for (std::size_t tick = 0; tick < kReplanIntervalTicksDefault; ++tick) { + harness.velocity_tick(); + } + }, + [&]() { return harness.velocity_tick(); } + ); + break; + case CaseKind::FirstLiveReplanDump: { + const auto start = std::chrono::steady_clock::now(); + const auto result = harness.velocity_first_live_replan_dump(); + const auto end = std::chrono::steady_clock::now(); + g_sink = g_sink + (result.empty() ? 0.0f : result.front()); + samples_ns.push_back( + static_cast( + std::chrono::duration_cast(end - start).count() + ) + ); + break; + } + case CaseKind::LaterMotionDump: { + const auto start = std::chrono::steady_clock::now(); + const auto result = harness.velocity_later_motion_dump(); + const auto end = std::chrono::steady_clock::now(); + g_sink = g_sink + (result.empty() ? 0.0f : result.front()); + samples_ns.push_back( + static_cast( + std::chrono::duration_cast(end - start).count() + ) + ); + break; + } + case CaseKind::EndToEndLoop: { + auto [timings, achieved_hz] = run_end_to_end_loop( + harness, + options.ticks, + options.control_frequency_hz, + [&]() { return harness.velocity_tick(); } + ); + samples_ns = std::move(timings); + hz = achieved_hz; + break; + } + } + + write_json(options, samples_ns, hz); + return 0; + } catch (const std::exception& error) { + std::cerr << "error: " << error.what() << '\n'; + return 1; + } +} diff --git a/scripts/benchmarks/bench_nvidia_official.py b/scripts/benchmarks/bench_nvidia_official.py new file mode 100644 index 0000000..78bff08 --- /dev/null +++ b/scripts/benchmarks/bench_nvidia_official.py @@ -0,0 +1,633 @@ +#!/usr/bin/env python3 +"""Run the official NVIDIA comparison cases and emit normalized artifacts.""" + +from __future__ import annotations + +import argparse +import importlib.util +import os +import shlex +import subprocess +import sys +from pathlib import Path +from typing import Any + + +ROOT_DIR = Path(__file__).resolve().parents[2] +REGISTRY_PATH = ROOT_DIR / "benchmarks/nvidia/cases.json" +NORMALIZER_PATH = ROOT_DIR / "scripts/benchmarks/normalize_nvidia_benchmarks.py" +DECOUPLED_HARNESS = ROOT_DIR / "scripts/benchmarks/bench_nvidia_decoupled_official.py" +GEAR_SONIC_HARNESS_SRC = ROOT_DIR / "scripts/benchmarks/bench_nvidia_gear_sonic_official.cpp" +DEFAULT_GEAR_SONIC_HARNESS_BIN = ( + ROOT_DIR / "target/nvidia-bench/bench_nvidia_gear_sonic_official" +) +DEFAULT_OFFICIAL_REPO_DIR = ROOT_DIR / "third_party/GR00T-WholeBodyControl" +IMPLEMENTATION_ID = "ort-cpp-sonic" +DEFAULT_OFFICIAL_OUTPUT_ROOT = ROOT_DIR / "artifacts/benchmarks/nvidia" / IMPLEMENTATION_ID +DEFAULT_DECOUPLED_MODEL_DIR = Path( + os.environ.get("DECOUPLED_WBC_MODEL_DIR", str(ROOT_DIR / "models/decoupled-wbc")) +) +DEFAULT_GEAR_SONIC_MODEL_DIR = Path( + os.environ.get("GEAR_SONIC_MODEL_DIR", str(ROOT_DIR / "models/gear-sonic")) +) +PROVIDER_CHOICES = ("cpu", "cuda", "tensor_rt") + + +def load_normalizer() -> Any: + spec = importlib.util.spec_from_file_location("normalize_nvidia_benchmarks", NORMALIZER_PATH) + if spec is None or spec.loader is None: + raise RuntimeError(f"failed to load normalizer module from {NORMALIZER_PATH}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +NORMALIZER = load_normalizer() +REGISTRY = NORMALIZER.load_registry(REGISTRY_PATH) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + mode = parser.add_mutually_exclusive_group(required=True) + mode.add_argument("--list-cases", action="store_true") + mode.add_argument("--case") + mode.add_argument("--all", action="store_true") + parser.add_argument( + "--output-root", + type=Path, + default=DEFAULT_OFFICIAL_OUTPUT_ROOT, + help="Base directory for normalized artifacts; provider subdirectories are created automatically", + ) + parser.add_argument( + "--provider", + default="cpu", + choices=PROVIDER_CHOICES, + help="Execution provider requested for the benchmark row", + ) + parser.add_argument( + "--repo-dir", + type=Path, + default=DEFAULT_OFFICIAL_REPO_DIR, + help="Pinned GR00T-WholeBodyControl source checkout", + ) + parser.add_argument( + "--samples", + type=int, + default=100, + help="Microbenchmark samples for official harnesses", + ) + parser.add_argument( + "--ticks", + type=int, + default=200, + help="End-to-end loop ticks for official harnesses", + ) + parser.add_argument( + "--control-frequency-hz", + type=int, + default=50, + help="End-to-end control frequency label for official harnesses", + ) + return parser.parse_args() + + +def list_cases() -> None: + for case in REGISTRY["cases"]: + print(case["case_id"]) + + +def relative_to_root(path: Path) -> str: + try: + return str(path.relative_to(ROOT_DIR)) + except ValueError: + return str(path) + + +def provider_output_root(output_root: Path, provider: str) -> Path: + return output_root / provider + + +def run( + argv: list[str], + *, + cwd: Path | None = None, + capture_output: bool = True, +) -> subprocess.CompletedProcess[str]: + return subprocess.run( + argv, + cwd=cwd or ROOT_DIR, + check=True, + text=True, + capture_output=capture_output, + ) + + +def git_rev_parse(repo_dir: Path) -> str | None: + try: + result = run( + ["git", "-C", str(repo_dir), "rev-parse", "HEAD"], + capture_output=True, + ) + except (subprocess.CalledProcessError, FileNotFoundError): + return None + return result.stdout.strip() + + +def repo_checkout_blocker(repo_dir: Path) -> str: + if repo_dir == DEFAULT_OFFICIAL_REPO_DIR: + rel = relative_to_root(repo_dir) + return ( + f"Official NVIDIA source checkout is unavailable at {repo_dir}; " + f"run `git submodule update --init --recursive {rel}` first." + ) + return ( + f"Official NVIDIA source checkout is unavailable at {repo_dir}; " + "pass --repo-dir ." + ) + + +def have_decoupled_models(model_dir: Path) -> bool: + return ( + (model_dir / "GR00T-WholeBodyControl-Walk.onnx").is_file() + and (model_dir / "GR00T-WholeBodyControl-Balance.onnx").is_file() + ) + + +def have_gear_sonic_models(model_dir: Path) -> bool: + return ( + (model_dir / "planner_sonic.onnx").is_file() + and (model_dir / "model_encoder.onnx").is_file() + and (model_dir / "model_decoder.onnx").is_file() + ) + + +def configured_onnxruntime_root() -> Path | None: + for env_name in ("ROBOWBC_ORT_DYLIB_PATH", "ORT_DYLIB_PATH"): + configured = os.environ.get(env_name) + if not configured: + continue + dylib_path = Path(configured).expanduser().resolve() + if not dylib_path.is_file(): + continue + root = dylib_path.parent.parent + if (root / "include" / "onnxruntime_cxx_api.h").is_file(): + return root + return None + + +def discover_onnxruntime_root() -> Path | None: + configured_root = configured_onnxruntime_root() + if configured_root is not None: + return configured_root + + build_root = ROOT_DIR / "target/debug/build" + candidates = sorted( + path + for path in build_root.glob("**/onnxruntime-linux-x64-*") + if path.is_dir() + ) + if not candidates: + return None + return candidates[-1] + + +def gear_sonic_harness_bin() -> Path: + configured = os.environ.get("GEAR_SONIC_OFFICIAL_HARNESS_BIN") + if not configured: + return DEFAULT_GEAR_SONIC_HARNESS_BIN + return Path(configured).expanduser().resolve() + + +def describe_process_failure(error: subprocess.CalledProcessError) -> str: + chunks = [chunk.strip() for chunk in (error.stdout, error.stderr) if chunk and chunk.strip()] + if not chunks: + return str(error) + combined = "\n".join(chunks) + lines = combined.splitlines() + if len(lines) > 40: + combined = "\n".join(lines[-40:]) + return combined + + +def blocked_reason_for_provider(case_id: str, provider: str) -> str | None: + if provider != "cpu" and case_id.startswith("decoupled_wbc/"): + return ( + f"{case_id} stays CPU-only in this phase. Provider `{provider}` is not wired on " + "both benchmark implementations for Decoupled WBC, so the row is blocked instead of " + "quietly relabeling a CPU measurement." + ) + return None + + +def emit_blocked( + *, + case_id: str, + provider: str, + upstream_commit: str | None, + robowbc_commit: str, + reason: str, + output_root: Path, + source_command: str, +) -> None: + case = NORMALIZER.registry_case(REGISTRY, case_id) + artifact = NORMALIZER.build_artifact( + case=case, + implementation=IMPLEMENTATION_ID, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + provider=provider, + host_fingerprint=None, + samples_ns=[], + hz=None, + notes=reason, + source_command=source_command, + raw_source=reason, + status="blocked", + ) + output_path = output_root / f"{case_id.replace('/', '__')}.json" + NORMALIZER.dump_json(output_path, artifact) + print(f"[blocked] {case_id} -> {output_path}") + + +def normalize_manual_case( + *, + case_id: str, + provider: str, + upstream_commit: str | None, + robowbc_commit: str, + input_path: Path, + notes: str, + output_root: Path, + source_command: str, +) -> None: + case = NORMALIZER.registry_case(REGISTRY, case_id) + samples_ns, hz, raw_source = NORMALIZER.manual_samples_payload(input_path) + artifact = NORMALIZER.build_artifact( + case=case, + implementation=IMPLEMENTATION_ID, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + provider=provider, + host_fingerprint=None, + samples_ns=samples_ns, + hz=hz, + notes=notes, + source_command=source_command, + raw_source=raw_source, + status="ok", + ) + output_path = output_root / f"{case_id.replace('/', '__')}.json" + NORMALIZER.dump_json(output_path, artifact) + print(f"[ok] {case_id} -> {output_path}") + + +def ensure_gear_sonic_harness(repo_dir: Path, output_root: Path) -> str | None: + harness_bin = gear_sonic_harness_bin() + if "GEAR_SONIC_OFFICIAL_HARNESS_BIN" in os.environ: + if harness_bin.is_file(): + return None + return f"Configured GEAR_SONIC_OFFICIAL_HARNESS_BIN does not exist: {harness_bin}" + + upstream_include = ( + repo_dir / "gear_sonic_deploy/src/g1/g1_deploy_onnx_ref/include" + ) + policy_header = upstream_include / "policy_parameters.hpp" + robot_header = upstream_include / "robot_parameters.hpp" + if not policy_header.is_file() or not robot_header.is_file(): + return ( + f"Official GEAR-Sonic reference headers are missing under {upstream_include}; " + "the pinned source checkout does not expose the required C++ contract." + ) + + ort_root = discover_onnxruntime_root() + if ort_root is None: + return ( + "ONNX Runtime headers/libs were not found under " + f"{ROOT_DIR / 'target/debug/build'} and no compatible ROBOWBC_ORT_DYLIB_PATH / " + "ORT_DYLIB_PATH override was detected; run `cargo build` first or point the " + "benchmark wrapper at a full ONNX Runtime distribution." + ) + + inputs = [GEAR_SONIC_HARNESS_SRC, policy_header, robot_header] + needs_rebuild = configured_onnxruntime_root() is not None or not harness_bin.exists() + if not needs_rebuild: + built_at = harness_bin.stat().st_mtime_ns + needs_rebuild = any(path.stat().st_mtime_ns > built_at for path in inputs) + if not needs_rebuild: + return None + + build_log = output_root / "raw/gear_sonic_build.log" + build_log.parent.mkdir(parents=True, exist_ok=True) + harness_bin.parent.mkdir(parents=True, exist_ok=True) + compile_cmd = [ + "c++", + "-std=c++20", + "-O3", + "-I", + str(ort_root / "include"), + "-I", + str(upstream_include), + str(GEAR_SONIC_HARNESS_SRC), + "-L", + str(ort_root / "lib"), + f"-Wl,-rpath,{ort_root / 'lib'}", + "-lonnxruntime", + "-lpthread", + "-o", + str(harness_bin), + ] + with build_log.open("w", encoding="utf-8") as handle: + result = subprocess.run( + compile_cmd, + cwd=ROOT_DIR, + stdout=handle, + stderr=subprocess.STDOUT, + text=True, + ) + if result.returncode != 0: + return ( + f"Failed to compile {GEAR_SONIC_HARNESS_SRC} against {ort_root}; " + f"see {build_log} for the compiler output." + ) + return None + + +def run_decoupled_case( + *, + case_id: str, + repo_dir: Path, + model_dir: Path, + output_root: Path, + provider: str, + upstream_commit: str, + robowbc_commit: str, + samples: int, + ticks: int, + control_frequency_hz: int, + source_command: str, +) -> None: + raw_output = output_root / "raw" / f"{case_id.replace('/', '__')}.json" + raw_output.parent.mkdir(parents=True, exist_ok=True) + harness_cmd = [ + "python3", + str(DECOUPLED_HARNESS), + "--case-id", + case_id, + "--repo-dir", + str(repo_dir), + "--model-dir", + str(model_dir), + "--robot-config", + str(ROOT_DIR / "configs/robots/unitree_g1.toml"), + "--samples", + str(samples), + "--ticks", + str(ticks), + "--control-frequency-hz", + str(control_frequency_hz), + "--output", + str(raw_output), + ] + run(harness_cmd) + normalize_manual_case( + case_id=case_id, + provider=provider, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + input_path=raw_output, + notes="Measured via upstream Decoupled WBC headless harness on the pinned source checkout.", + output_root=output_root, + source_command=source_command, + ) + + +def run_gear_sonic_case( + *, + case_id: str, + model_dir: Path, + output_root: Path, + provider: str, + upstream_commit: str, + robowbc_commit: str, + samples: int, + ticks: int, + control_frequency_hz: int, + source_command: str, +) -> None: + harness_bin = gear_sonic_harness_bin() + raw_output = output_root / "raw" / f"{case_id.replace('/', '__')}.json" + raw_output.parent.mkdir(parents=True, exist_ok=True) + harness_cmd = [ + str(harness_bin), + "--case-id", + case_id, + "--provider", + provider, + "--model-dir", + str(model_dir), + "--samples", + str(samples), + "--ticks", + str(ticks), + "--control-frequency-hz", + str(control_frequency_hz), + "--output", + str(raw_output), + ] + run(harness_cmd) + normalize_manual_case( + case_id=case_id, + provider=provider, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + input_path=raw_output, + notes="Measured via the ORT-cpp-sonic GEAR-Sonic C++ ONNX Runtime harness on the pinned source checkout.", + output_root=output_root, + source_command=source_command, + ) + + +def source_command_for_case(args: argparse.Namespace, case_id: str) -> str: + argv = [ + "python3", + "scripts/benchmarks/bench_nvidia_official.py", + "--case", + case_id, + "--provider", + args.provider, + ] + if args.repo_dir != DEFAULT_OFFICIAL_REPO_DIR: + argv.extend(["--repo-dir", str(args.repo_dir)]) + if args.output_root != DEFAULT_OFFICIAL_OUTPUT_ROOT: + argv.extend(["--output-root", str(args.output_root)]) + if args.samples != 100: + argv.extend(["--samples", str(args.samples)]) + if args.ticks != 200: + argv.extend(["--ticks", str(args.ticks)]) + if args.control_frequency_hz != 50: + argv.extend(["--control-frequency-hz", str(args.control_frequency_hz)]) + return shlex.join(argv) + + +def run_case(args: argparse.Namespace, case_id: str, robowbc_commit: str) -> None: + upstream_commit = git_rev_parse(args.repo_dir) + source_command = source_command_for_case(args, case_id) + output_root = provider_output_root(args.output_root, args.provider) + if upstream_commit is None: + emit_blocked( + case_id=case_id, + provider=args.provider, + upstream_commit=None, + robowbc_commit=robowbc_commit, + reason=repo_checkout_blocker(args.repo_dir), + output_root=output_root, + source_command=source_command, + ) + return + + blocked_reason = blocked_reason_for_provider(case_id, args.provider) + if blocked_reason is not None: + emit_blocked( + case_id=case_id, + provider=args.provider, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + reason=blocked_reason, + output_root=output_root, + source_command=source_command, + ) + return + + if case_id.startswith("gear_sonic_") or case_id.startswith("gear_sonic/"): + if not have_gear_sonic_models(DEFAULT_GEAR_SONIC_MODEL_DIR): + emit_blocked( + case_id=case_id, + provider=args.provider, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + reason=( + "Official GEAR-Sonic checkpoints are missing under " + f"{DEFAULT_GEAR_SONIC_MODEL_DIR}; run scripts/models/download_gear_sonic_models.sh first." + ), + output_root=output_root, + source_command=source_command, + ) + return + build_reason = ensure_gear_sonic_harness(args.repo_dir, output_root) + if build_reason is not None: + emit_blocked( + case_id=case_id, + provider=args.provider, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + reason=build_reason, + output_root=output_root, + source_command=source_command, + ) + return + try: + run_gear_sonic_case( + case_id=case_id, + model_dir=DEFAULT_GEAR_SONIC_MODEL_DIR, + output_root=output_root, + provider=args.provider, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + samples=args.samples, + ticks=args.ticks, + control_frequency_hz=args.control_frequency_hz, + source_command=source_command, + ) + except subprocess.CalledProcessError as error: + emit_blocked( + case_id=case_id, + provider=args.provider, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + reason=( + f"Requested provider `{args.provider}` could not run on the ORT-cpp-sonic " + f"GEAR-Sonic harness. Exact runtime output:\n{describe_process_failure(error)}" + ), + output_root=output_root, + source_command=source_command, + ) + return + + if case_id.startswith("decoupled_wbc/"): + model_dir = DEFAULT_DECOUPLED_MODEL_DIR + if not have_decoupled_models(model_dir): + emit_blocked( + case_id=case_id, + provider=args.provider, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + reason=( + "Official Decoupled WBC models are missing under " + f"{model_dir}; run scripts/models/download_decoupled_wbc_models.sh first." + ), + output_root=output_root, + source_command=source_command, + ) + return + try: + run_decoupled_case( + case_id=case_id, + repo_dir=args.repo_dir, + model_dir=model_dir, + output_root=output_root, + provider=args.provider, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + samples=args.samples, + ticks=args.ticks, + control_frequency_hz=args.control_frequency_hz, + source_command=source_command, + ) + except subprocess.CalledProcessError as error: + emit_blocked( + case_id=case_id, + provider=args.provider, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + reason=( + f"Requested provider `{args.provider}` could not run on the official " + f"Decoupled WBC harness. Exact runtime output:\n{describe_process_failure(error)}" + ), + output_root=output_root, + source_command=source_command, + ) + return + + emit_blocked( + case_id=case_id, + provider=args.provider, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + reason=f"No official-wrapper mapping has been defined for {case_id}.", + output_root=output_root, + source_command=source_command, + ) + + +def main() -> int: + args = parse_args() + if args.list_cases: + list_cases() + return 0 + + robowbc_commit = run( + ["git", "-C", str(ROOT_DIR), "rev-parse", "HEAD"], + capture_output=True, + ).stdout.strip() + + if args.case is not None: + run_case(args, args.case, robowbc_commit) + return 0 + + for case in REGISTRY["cases"]: + run_case(args, case["case_id"], robowbc_commit) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/benchmarks/bench_robowbc_compare.py b/scripts/benchmarks/bench_robowbc_compare.py new file mode 100644 index 0000000..f5da42c --- /dev/null +++ b/scripts/benchmarks/bench_robowbc_compare.py @@ -0,0 +1,728 @@ +#!/usr/bin/env python3 +"""Run the RoboWBC comparison cases and emit normalized artifacts.""" + +from __future__ import annotations + +import argparse +import importlib.util +import os +import shlex +import subprocess +import tempfile +from pathlib import Path +from typing import Any + + +ROOT_DIR = Path(__file__).resolve().parents[2] +REGISTRY_PATH = ROOT_DIR / "benchmarks/nvidia/cases.json" +NORMALIZER_PATH = ROOT_DIR / "scripts/benchmarks/normalize_nvidia_benchmarks.py" +IMPLEMENTATION_ID = "ort-rs" +DEFAULT_OUTPUT_ROOT = ROOT_DIR / "artifacts/benchmarks/nvidia" / IMPLEMENTATION_ID +DEFAULT_MUJOCO_DOWNLOAD_DIR = ROOT_DIR / ".cache" / "mujoco" +DEFAULT_GEAR_SONIC_REVISION = "cc80d505b7e055fd6ae26426ae8bfa0a74c26011" +DEFAULT_DECOUPLED_COMMIT = "bc38f6d0ce6cab4589e025037ad0bfbab7ba73d8" +DEFAULT_GEAR_SONIC_MODEL_DIR = Path( + os.environ.get("GEAR_SONIC_MODEL_DIR", str(ROOT_DIR / "models/gear-sonic")) +) +DEFAULT_DECOUPLED_WBC_MODEL_DIR = Path( + os.environ.get("DECOUPLED_WBC_MODEL_DIR", str(ROOT_DIR / "models/decoupled-wbc")) +) +PROVIDER_CHOICES = ("cpu", "cuda", "tensor_rt") +BENCH_PROVIDER_ENV = "ROBOWBC_BENCH_PROVIDER" +GEAR_SONIC_PROVIDER_SECTIONS = ( + "policy.config.encoder", + "policy.config.decoder", + "policy.config.planner", +) +MICROBENCH_CASES: dict[str, dict[str, str]] = { + "gear_sonic/planner_only_cold_start": { + "criterion_filter": "policy/gear_sonic/planner_only_cold_start", + "family": "gear_sonic", + }, + "gear_sonic/planner_only_steady_state": { + "criterion_filter": "policy/gear_sonic/planner_only_steady_state", + "family": "gear_sonic", + }, + "gear_sonic/encoder_decoder_only_tracking_tick": { + "criterion_filter": "policy/gear_sonic/encoder_decoder_only_tracking_tick", + "family": "gear_sonic", + }, + "gear_sonic/full_velocity_tick_cold_start": { + "criterion_filter": "policy/gear_sonic/full_velocity_tick_cold_start", + "family": "gear_sonic", + }, + "gear_sonic/full_velocity_tick_steady_state": { + "criterion_filter": "policy/gear_sonic/full_velocity_tick_steady_state", + "family": "gear_sonic", + }, + "gear_sonic/full_velocity_tick_replan_boundary": { + "criterion_filter": "policy/gear_sonic/full_velocity_tick_replan_boundary", + "family": "gear_sonic", + }, + "decoupled_wbc/walk_predict": { + "criterion_filter": "policy/decoupled_wbc/walk_predict", + "family": "decoupled_wbc", + }, + "decoupled_wbc/balance_predict": { + "criterion_filter": "policy/decoupled_wbc/balance_predict", + "family": "decoupled_wbc", + }, +} +CLI_CASES: dict[str, dict[str, str]] = { + "gear_sonic/end_to_end_cli_loop": { + "config_path": "configs/sonic_g1.toml", + "family": "gear_sonic", + }, + "decoupled_wbc/end_to_end_cli_loop": { + "config_path": "configs/decoupled_g1.toml", + "family": "decoupled_wbc", + }, +} + + +def load_normalizer() -> Any: + spec = importlib.util.spec_from_file_location("normalize_nvidia_benchmarks", NORMALIZER_PATH) + if spec is None or spec.loader is None: + raise RuntimeError(f"failed to load normalizer module from {NORMALIZER_PATH}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +NORMALIZER = load_normalizer() +REGISTRY = NORMALIZER.load_registry(REGISTRY_PATH) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + mode = parser.add_mutually_exclusive_group(required=True) + mode.add_argument("--list-cases", action="store_true") + mode.add_argument("--case") + mode.add_argument("--all", action="store_true") + parser.add_argument( + "--output-root", + type=Path, + default=DEFAULT_OUTPUT_ROOT, + help="Base directory for normalized artifacts; provider subdirectories are created automatically", + ) + parser.add_argument( + "--provider", + default="cpu", + choices=PROVIDER_CHOICES, + help="Execution provider requested for the benchmark row", + ) + return parser.parse_args() + + +def list_cases() -> None: + for case in REGISTRY["cases"]: + print(case["case_id"]) + + +def run( + argv: list[str], + *, + env: dict[str, str] | None = None, +) -> subprocess.CompletedProcess[str]: + return subprocess.run( + argv, + cwd=ROOT_DIR, + env=env, + check=True, + text=True, + capture_output=True, + ) + + +def prepend_env_path(env: dict[str, str], key: str, value: Path) -> None: + current = env.get(key) + env[key] = f"{value}{os.pathsep}{current}" if current else str(value) + + +def resolve_mujoco_download_dir(env: dict[str, str]) -> Path: + configured = env.get("MUJOCO_DOWNLOAD_DIR") + download_dir = ( + Path(configured).expanduser().resolve() + if configured + else DEFAULT_MUJOCO_DOWNLOAD_DIR.resolve() + ) + download_dir.mkdir(parents=True, exist_ok=True) + env["MUJOCO_DOWNLOAD_DIR"] = str(download_dir) + return download_dir + + +def resolve_mujoco_runtime_libdir(env: dict[str, str]) -> Path | None: + download_dir = resolve_mujoco_download_dir(env) + candidates = sorted( + download_dir.glob("mujoco-*/lib/libmujoco.so"), + key=lambda path: path.stat().st_mtime, + reverse=True, + ) + return candidates[0].parent if candidates else None + + +def configure_mujoco_runtime_env(env: dict[str, str]) -> dict[str, str]: + resolve_mujoco_download_dir(env) + libdir = resolve_mujoco_runtime_libdir(env) + if libdir is not None: + prepend_env_path(env, "LD_LIBRARY_PATH", libdir) + return env + + +def git_rev_parse(repo_dir: Path) -> str: + result = subprocess.run( + ["git", "-C", str(repo_dir), "rev-parse", "HEAD"], + cwd=ROOT_DIR, + check=True, + text=True, + capture_output=True, + ) + return result.stdout.strip() + + +def read_revision_file(path: Path, fallback: str) -> str: + if not path.is_file(): + return fallback + revision = path.read_text(encoding="utf-8").strip() + return revision or fallback + + +def gear_sonic_revision(model_dir: Path) -> str: + return read_revision_file(model_dir / "REVISION", DEFAULT_GEAR_SONIC_REVISION) + + +def decoupled_revision(model_dir: Path) -> str: + return read_revision_file(model_dir / "REVISION", DEFAULT_DECOUPLED_COMMIT) + + +def have_gear_sonic_models(model_dir: Path) -> bool: + return ( + (model_dir / "model_encoder.onnx").is_file() + and (model_dir / "model_decoder.onnx").is_file() + and (model_dir / "planner_sonic.onnx").is_file() + ) + + +def have_decoupled_models(model_dir: Path) -> bool: + return ( + (model_dir / "GR00T-WholeBodyControl-Walk.onnx").is_file() + and (model_dir / "GR00T-WholeBodyControl-Balance.onnx").is_file() + ) + + +def output_path_for(case_id: str, output_root: Path) -> Path: + return output_root / f"{case_id.replace('/', '__')}.json" + + +def provider_output_root(output_root: Path, provider: str) -> Path: + return output_root / provider + + +def case_ids() -> set[str]: + return {case["case_id"] for case in REGISTRY["cases"]} + + +def source_command_for_case(args: argparse.Namespace, case_id: str) -> str: + argv = [ + "python3", + "scripts/benchmarks/bench_robowbc_compare.py", + "--case", + case_id, + "--provider", + args.provider, + ] + if args.output_root != DEFAULT_OUTPUT_ROOT: + argv.extend(["--output-root", str(args.output_root)]) + return shlex.join(argv) + + +def provider_inline_table(provider: str) -> str: + if provider == "cpu": + return '{ type = "cpu" }' + return f'{{ type = "{provider}", device_id = 0 }}' + + +def rewrite_gear_sonic_provider_blocks(config_text: str, provider: str) -> str: + lines = config_text.splitlines() + rewritten: list[str] = [] + current_section: str | None = None + replaced_sections: set[str] = set() + for line in lines: + stripped = line.strip() + if stripped.startswith("[") and stripped.endswith("]"): + current_section = stripped[1:-1] + if current_section in GEAR_SONIC_PROVIDER_SECTIONS and stripped.startswith( + "execution_provider" + ): + indent = line[: len(line) - len(line.lstrip())] + rewritten.append( + f"{indent}execution_provider = {provider_inline_table(provider)}" + ) + replaced_sections.add(current_section) + continue + rewritten.append(line) + + missing = set(GEAR_SONIC_PROVIDER_SECTIONS) - replaced_sections + if missing: + missing_str = ", ".join(sorted(missing)) + raise ValueError( + "failed to rewrite all GEAR-Sonic execution_provider blocks; " + f"missing sections: {missing_str}" + ) + + rewritten_text = "\n".join(rewritten) + if config_text.endswith("\n"): + rewritten_text += "\n" + return rewritten_text + + +def append_report_section(config_text: str, report_path: Path) -> str: + report_block = f'\n[report]\noutput_path = "{report_path}"\nmax_frames = 200\n' + return config_text.rstrip() + "\n" + report_block + + +def compose_benchmark_cli_config( + config_text: str, + *, + provider: str, + report_path: Path, + rewrite_gear_sonic_providers: bool, +) -> str: + updated = config_text + if rewrite_gear_sonic_providers: + updated = rewrite_gear_sonic_provider_blocks(updated, provider) + return append_report_section(updated, report_path) + + +def describe_process_failure(error: subprocess.CalledProcessError) -> str: + chunks = [chunk.strip() for chunk in (error.stdout, error.stderr) if chunk and chunk.strip()] + if not chunks: + return str(error) + combined = "\n".join(chunks) + lines = combined.splitlines() + if len(lines) > 40: + combined = "\n".join(lines[-40:]) + return combined + + +def benchmark_failure_reason(provider: str, error: subprocess.CalledProcessError) -> str: + details = describe_process_failure(error) + return ( + f"Requested provider `{provider}` could not run on the ORT-rs benchmark path. " + f"Exact runtime output:\n{details}" + ) + + +def blocked_reason_for_provider(case_id: str, provider: str) -> str | None: + if provider != "cpu" and case_id.startswith("decoupled_wbc/"): + return ( + f"{case_id} stays CPU-only in this phase. Provider `{provider}` is not wired on " + "both benchmark implementations for Decoupled WBC, so the row is blocked instead of " + "quietly relabeling a CPU measurement." + ) + return None + + +def emit_blocked( + *, + case_id: str, + provider: str, + upstream_commit: str, + robowbc_commit: str, + output_root: Path, + reason: str, + source_command: str, +) -> None: + case = NORMALIZER.registry_case(REGISTRY, case_id) + artifact = NORMALIZER.build_artifact( + case=case, + implementation=IMPLEMENTATION_ID, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + provider=provider, + host_fingerprint=None, + samples_ns=[], + hz=None, + notes=reason, + source_command=source_command, + raw_source=reason, + status="blocked", + ) + output_path = output_path_for(case_id, output_root) + NORMALIZER.dump_json(output_path, artifact) + print(f"[blocked] {case_id} -> {output_path}") + + +def normalize_criterion_case( + *, + case_id: str, + provider: str, + upstream_commit: str, + robowbc_commit: str, + output_root: Path, + source_command: str, +) -> None: + case = NORMALIZER.registry_case(REGISTRY, case_id) + criterion_id = case["criterion_id"] + samples_ns, raw_source = NORMALIZER.criterion_samples_ns( + ROOT_DIR / "target/criterion", criterion_id + ) + artifact = NORMALIZER.build_artifact( + case=case, + implementation=IMPLEMENTATION_ID, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + provider=provider, + host_fingerprint=None, + samples_ns=samples_ns, + hz=None, + notes="Normalized from ORT-rs Criterion sample.json per-iteration timings.", + source_command=source_command, + raw_source=raw_source, + status="ok", + ) + output_path = output_path_for(case_id, output_root) + NORMALIZER.dump_json(output_path, artifact) + print(f"[ok] {case_id} -> {output_path}") + + +def normalize_run_report( + *, + case_id: str, + provider: str, + upstream_commit: str, + robowbc_commit: str, + output_root: Path, + report_path: Path, + source_command: str, +) -> None: + case = NORMALIZER.registry_case(REGISTRY, case_id) + samples_ns, hz, raw_source = NORMALIZER.run_report_samples_ns(report_path) + artifact = NORMALIZER.build_artifact( + case=case, + implementation=IMPLEMENTATION_ID, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + provider=provider, + host_fingerprint=None, + samples_ns=samples_ns, + hz=hz, + notes="Normalized from the ORT-rs robowbc-cli JSON run report.", + source_command=source_command, + raw_source=raw_source, + status="ok", + ) + output_path = output_path_for(case_id, output_root) + NORMALIZER.dump_json(output_path, artifact) + print(f"[ok] {case_id} -> {output_path}") + + +def run_microbench_case( + *, + case_id: str, + criterion_filter: str, + provider: str, + upstream_commit: str, + robowbc_commit: str, + output_root: Path, + env_name: str, + env_value: Path, + source_command: str, +) -> None: + env = os.environ.copy() + env[env_name] = str(env_value) + env[BENCH_PROVIDER_ENV] = provider + run( + [ + "cargo", + "bench", + "-p", + "robowbc-ort", + "--bench", + "inference", + "--", + "--output-format", + "bencher", + criterion_filter, + ], + env=env, + ) + normalize_criterion_case( + case_id=case_id, + provider=provider, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + output_root=output_root, + source_command=source_command, + ) + + +def run_cli_case( + *, + case_id: str, + config_path: str, + provider: str, + upstream_commit: str, + robowbc_commit: str, + output_root: Path, + source_command: str, +) -> None: + source_config = ROOT_DIR / config_path + with tempfile.TemporaryDirectory() as temp_dir: + temp_root = Path(temp_dir) + temp_config = temp_root / source_config.name + raw_report = temp_root / "report.json" + env = configure_mujoco_runtime_env(os.environ.copy()) + temp_config.write_text( + compose_benchmark_cli_config( + source_config.read_text(encoding="utf-8"), + provider=provider, + report_path=raw_report, + rewrite_gear_sonic_providers=case_id.startswith("gear_sonic/"), + ), + encoding="utf-8", + ) + run( + [ + "cargo", + "run", + "-p", + "robowbc-cli", + "--features", + "sim-auto-download,vis", + "--bin", + "robowbc", + "--", + "run", + "--config", + str(temp_config), + ], + env=env, + ) + normalize_run_report( + case_id=case_id, + provider=provider, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + output_root=output_root, + report_path=raw_report, + source_command=source_command, + ) + + +def run_case(case_id: str, args: argparse.Namespace) -> None: + if case_id not in case_ids(): + raise ValueError(f"unknown case_id: {case_id}") + + source_command = source_command_for_case(args, case_id) + output_root = provider_output_root(args.output_root, args.provider) + gear_model_dir = DEFAULT_GEAR_SONIC_MODEL_DIR + decoupled_model_dir = DEFAULT_DECOUPLED_WBC_MODEL_DIR + robowbc_commit = git_rev_parse(ROOT_DIR) + + blocked_reason = blocked_reason_for_provider(case_id, args.provider) + if blocked_reason is not None: + upstream_commit = ( + gear_sonic_revision(gear_model_dir) + if case_id.startswith("gear_sonic/") + else decoupled_revision(decoupled_model_dir) + ) + emit_blocked( + case_id=case_id, + provider=args.provider, + upstream_commit=upstream_commit, + robowbc_commit=robowbc_commit, + output_root=output_root, + reason=blocked_reason, + source_command=source_command, + ) + return + + if case_id in MICROBENCH_CASES: + spec = MICROBENCH_CASES[case_id] + if spec["family"] == "gear_sonic": + if not have_gear_sonic_models(gear_model_dir): + emit_blocked( + case_id=case_id, + provider=args.provider, + upstream_commit=gear_sonic_revision(gear_model_dir), + robowbc_commit=robowbc_commit, + output_root=output_root, + reason=( + "GEAR-Sonic checkpoints not found under " + f"{gear_model_dir}; run scripts/models/download_gear_sonic_models.sh first." + ), + source_command=source_command, + ) + return + try: + run_microbench_case( + case_id=case_id, + criterion_filter=spec["criterion_filter"], + provider=args.provider, + upstream_commit=gear_sonic_revision(gear_model_dir), + robowbc_commit=robowbc_commit, + output_root=output_root, + env_name="GEAR_SONIC_MODEL_DIR", + env_value=gear_model_dir, + source_command=source_command, + ) + except subprocess.CalledProcessError as error: + emit_blocked( + case_id=case_id, + provider=args.provider, + upstream_commit=gear_sonic_revision(gear_model_dir), + robowbc_commit=robowbc_commit, + output_root=output_root, + reason=benchmark_failure_reason(args.provider, error), + source_command=source_command, + ) + return + + if not have_decoupled_models(decoupled_model_dir): + emit_blocked( + case_id=case_id, + provider=args.provider, + upstream_commit=decoupled_revision(decoupled_model_dir), + robowbc_commit=robowbc_commit, + output_root=output_root, + reason=( + "Decoupled WBC checkpoints not found under " + f"{decoupled_model_dir}; run scripts/models/download_decoupled_wbc_models.sh first." + ), + source_command=source_command, + ) + return + try: + run_microbench_case( + case_id=case_id, + criterion_filter=spec["criterion_filter"], + provider=args.provider, + upstream_commit=decoupled_revision(decoupled_model_dir), + robowbc_commit=robowbc_commit, + output_root=output_root, + env_name="DECOUPLED_WBC_MODEL_DIR", + env_value=decoupled_model_dir, + source_command=source_command, + ) + except subprocess.CalledProcessError as error: + emit_blocked( + case_id=case_id, + provider=args.provider, + upstream_commit=decoupled_revision(decoupled_model_dir), + robowbc_commit=robowbc_commit, + output_root=output_root, + reason=benchmark_failure_reason(args.provider, error), + source_command=source_command, + ) + return + + if case_id in CLI_CASES: + spec = CLI_CASES[case_id] + if spec["family"] == "gear_sonic": + if not have_gear_sonic_models(gear_model_dir): + emit_blocked( + case_id=case_id, + provider=args.provider, + upstream_commit=gear_sonic_revision(gear_model_dir), + robowbc_commit=robowbc_commit, + output_root=output_root, + reason=( + "GEAR-Sonic checkpoints not found under " + f"{gear_model_dir}; run scripts/models/download_gear_sonic_models.sh first." + ), + source_command=source_command, + ) + return + try: + run_cli_case( + case_id=case_id, + config_path=spec["config_path"], + provider=args.provider, + upstream_commit=gear_sonic_revision(gear_model_dir), + robowbc_commit=robowbc_commit, + output_root=output_root, + source_command=source_command, + ) + except (subprocess.CalledProcessError, ValueError) as error: + reason = ( + benchmark_failure_reason(args.provider, error) + if isinstance(error, subprocess.CalledProcessError) + else str(error) + ) + emit_blocked( + case_id=case_id, + provider=args.provider, + upstream_commit=gear_sonic_revision(gear_model_dir), + robowbc_commit=robowbc_commit, + output_root=output_root, + reason=reason, + source_command=source_command, + ) + return + + if not have_decoupled_models(decoupled_model_dir): + emit_blocked( + case_id=case_id, + provider=args.provider, + upstream_commit=decoupled_revision(decoupled_model_dir), + robowbc_commit=robowbc_commit, + output_root=output_root, + reason=( + "Decoupled WBC checkpoints not found under " + f"{decoupled_model_dir}; run scripts/models/download_decoupled_wbc_models.sh first." + ), + source_command=source_command, + ) + return + try: + run_cli_case( + case_id=case_id, + config_path=spec["config_path"], + provider=args.provider, + upstream_commit=decoupled_revision(decoupled_model_dir), + robowbc_commit=robowbc_commit, + output_root=output_root, + source_command=source_command, + ) + except subprocess.CalledProcessError as error: + emit_blocked( + case_id=case_id, + provider=args.provider, + upstream_commit=decoupled_revision(decoupled_model_dir), + robowbc_commit=robowbc_commit, + output_root=output_root, + reason=benchmark_failure_reason(args.provider, error), + source_command=source_command, + ) + return + + emit_blocked( + case_id=case_id, + provider=args.provider, + upstream_commit="unknown-upstream", + robowbc_commit=robowbc_commit, + output_root=output_root, + reason=f"No RoboWBC benchmark mapping has been defined for {case_id}.", + source_command=source_command, + ) + + +def main() -> int: + args = parse_args() + + if args.list_cases: + list_cases() + return 0 + + if args.case: + run_case(args.case, args) + return 0 + + for case in REGISTRY["cases"]: + run_case(case["case_id"], args) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/benchmarks/normalize_nvidia_benchmarks.py b/scripts/benchmarks/normalize_nvidia_benchmarks.py new file mode 100755 index 0000000..71387a3 --- /dev/null +++ b/scripts/benchmarks/normalize_nvidia_benchmarks.py @@ -0,0 +1,401 @@ +#!/usr/bin/env python3 +"""Normalize NVIDIA comparison artifacts into one schema.""" + +from __future__ import annotations + +import argparse +import json +import math +import os +import platform +import socket +from pathlib import Path +from typing import Any + +PROVIDER_ORDER = ("cpu", "cuda", "tensor_rt") +PROVIDER_ALIASES = { + "trt": "tensor_rt", +} +PROVIDER_FAMILY_IDS = { + "cpu": "cpu-baseline", + "cuda": "cuda", + "tensor_rt": "trt", +} +IMPLEMENTATION_ORDER = ("ort-cpp-sonic", "ort-rs") +IMPLEMENTATION_LABELS = { + "ort-cpp-sonic": "ORT-cpp-sonic", + "ort-rs": "ORT-rs", +} +LEGACY_STACK_TO_IMPLEMENTATION = { + "official_nvidia": "ort-cpp-sonic", + "robowbc": "ort-rs", +} +IMPLEMENTATION_TO_LEGACY_STACK = { + implementation: stack for stack, implementation in LEGACY_STACK_TO_IMPLEMENTATION.items() +} +LEGACY_ARTIFACT_DIRS = { + "ort-cpp-sonic": "official", + "ort-rs": "robowbc", +} + + +def load_json(path: Path) -> Any: + with path.open("r", encoding="utf-8") as handle: + return json.load(handle) + + +def dump_json(path: Path, payload: dict[str, Any]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("w", encoding="utf-8") as handle: + json.dump(payload, handle, indent=2, sort_keys=True) + handle.write("\n") + + +def cpu_model() -> str: + cpuinfo = Path("/proc/cpuinfo") + if cpuinfo.is_file(): + for line in cpuinfo.read_text(encoding="utf-8").splitlines(): + if line.startswith("model name"): + _, value = line.split(":", 1) + return value.strip() + return platform.processor() or "unknown-cpu" + + +def default_host_fingerprint() -> str: + return ( + f"{socket.gethostname()} | {platform.system()} {platform.release()} | " + f"{platform.machine()} | {cpu_model()}" + ) + + +def percentile(values: list[float], pct: float) -> int | None: + if not values: + return None + if len(values) == 1: + return round(values[0]) + ordered = sorted(values) + position = (len(ordered) - 1) * pct + lower = math.floor(position) + upper = math.ceil(position) + if lower == upper: + return round(ordered[lower]) + fraction = position - lower + interpolated = ordered[lower] * (1.0 - fraction) + ordered[upper] * fraction + return round(interpolated) + + +def load_registry(path: Path) -> dict[str, Any]: + registry = load_json(path) + if registry.get("schema_version") != 1: + raise ValueError(f"unsupported registry schema version: {registry.get('schema_version')}") + seen: set[str] = set() + for case in registry.get("cases", []): + case_id = case.get("case_id") + if not case_id: + raise ValueError("registry case is missing case_id") + if case_id in seen: + raise ValueError(f"duplicate case_id in registry: {case_id}") + seen.add(case_id) + for field in ("description", "command_fixture", "warmup_policy", "interpretation"): + if field not in case: + raise ValueError(f"registry case {case_id!r} is missing required field {field!r}") + return registry + + +def registry_case(registry: dict[str, Any], case_id: str) -> dict[str, Any]: + for case in registry["cases"]: + if case["case_id"] == case_id: + return case + raise ValueError(f"unknown case_id: {case_id}") + + +def canonical_provider(provider: str) -> str: + try: + normalized = PROVIDER_ALIASES.get(provider, provider) + if normalized not in PROVIDER_ORDER: + raise KeyError(provider) + return normalized + except KeyError as exc: + expected = ", ".join(PROVIDER_ORDER) + raise ValueError(f"unknown provider {provider!r}; expected one of: {expected}") from exc + + +def provider_family(provider: str) -> str: + return PROVIDER_FAMILY_IDS[canonical_provider(provider)] + + +def canonical_implementation(implementation: str) -> str: + normalized = LEGACY_STACK_TO_IMPLEMENTATION.get(implementation, implementation) + if normalized not in IMPLEMENTATION_ORDER: + expected = ", ".join(IMPLEMENTATION_ORDER) + raise ValueError( + f"unknown implementation {implementation!r}; expected one of: {expected}" + ) + return normalized + + +def implementation_label(implementation: str) -> str: + return IMPLEMENTATION_LABELS[canonical_implementation(implementation)] + + +def variant_label(provider: str, implementation: str) -> str: + provider_id = canonical_provider(provider) + if provider_id == "cpu": + return provider_family(provider_id) + return f"{provider_family(provider_id)}-{implementation_label(implementation)}" + + +def variant_slug(provider: str, implementation: str) -> str: + provider_id = canonical_provider(provider) + if provider_id == "cpu": + return provider_family(provider_id) + return f"{provider_family(provider_id)}-{canonical_implementation(implementation)}" + + +def implementation_artifact_dir(implementation: str) -> str: + return canonical_implementation(implementation) + + +def legacy_artifact_dir(implementation: str) -> str: + return LEGACY_ARTIFACT_DIRS[canonical_implementation(implementation)] + + +def criterion_samples_ns(criterion_root: Path, criterion_id: str) -> tuple[list[float], str]: + for benchmark_json in criterion_root.rglob("benchmark.json"): + benchmark = load_json(benchmark_json) + if benchmark.get("full_id") != criterion_id: + continue + sample_path = benchmark_json.parent / "sample.json" + sample = load_json(sample_path) + iters = sample.get("iters", []) + times = sample.get("times", []) + if len(iters) != len(times): + raise ValueError(f"criterion sample mismatch for {criterion_id}") + samples = [float(total_ns) / float(iter_count) for iter_count, total_ns in zip(iters, times)] + return samples, str(sample_path) + raise FileNotFoundError( + f"could not find criterion benchmark {criterion_id!r} under {criterion_root}" + ) + + +def run_report_samples_ns(report_path: Path) -> tuple[list[float], float | None, str]: + report = load_json(report_path) + frames = report.get("frames", []) + samples = [float(frame["inference_latency_ms"]) * 1_000_000.0 for frame in frames] + hz = report.get("metrics", {}).get("achieved_frequency_hz") + return samples, float(hz) if hz is not None else None, str(report_path) + + +def manual_samples_payload(input_path: Path) -> tuple[list[float], float | None, str]: + payload = load_json(input_path) + if isinstance(payload, list): + return [float(value) for value in payload], None, str(input_path) + samples = payload.get("samples_ns") + if samples is None: + raise ValueError(f"{input_path} must be a JSON list or an object with samples_ns") + hz = payload.get("hz") + return [float(value) for value in samples], float(hz) if hz is not None else None, str(input_path) + + +def build_artifact( + *, + case: dict[str, Any], + implementation: str, + upstream_commit: str | None, + robowbc_commit: str | None, + provider: str, + host_fingerprint: str | None, + samples_ns: list[float], + hz: float | None, + notes: str, + source_command: str | None, + raw_source: str, + status: str, +) -> dict[str, Any]: + implementation_id = canonical_implementation(implementation) + provider_id = canonical_provider(provider) + return { + "schema_version": 1, + "status": status, + "case_id": case["case_id"], + "description": case["description"], + "interpretation": case["interpretation"], + "implementation": implementation_id, + "implementation_label": implementation_label(implementation_id), + "stack": IMPLEMENTATION_TO_LEGACY_STACK[implementation_id], + "upstream_commit": upstream_commit, + "robowbc_commit": robowbc_commit, + "provider": provider_id, + "provider_family": provider_family(provider_id), + "variant_label": variant_label(provider_id, implementation_id), + "variant_slug": variant_slug(provider_id, implementation_id), + "host_fingerprint": host_fingerprint or default_host_fingerprint(), + "command_fixture": case["command_fixture"], + "warmup_policy": case["warmup_policy"], + "samples": len(samples_ns), + "p50_ns": percentile(samples_ns, 0.50), + "p95_ns": percentile(samples_ns, 0.95), + "p99_ns": percentile(samples_ns, 0.99), + "hz": round(hz, 6) if hz is not None else None, + "notes": notes, + "source_command": source_command, + "raw_source": raw_source, + } + + +def cmd_validate_registry(args: argparse.Namespace) -> int: + load_registry(args.registry) + return 0 + + +def common_artifact_args(parser: argparse.ArgumentParser) -> None: + parser.add_argument("--registry", type=Path, required=True) + parser.add_argument("--case-id", required=True) + parser.add_argument( + "--implementation", + required=True, + choices=(*IMPLEMENTATION_ORDER, *LEGACY_STACK_TO_IMPLEMENTATION), + ) + parser.add_argument( + "--stack", + dest="implementation", + help=argparse.SUPPRESS, + ) + parser.add_argument("--provider", required=True) + parser.add_argument("--upstream-commit") + parser.add_argument("--robowbc-commit") + parser.add_argument("--host-fingerprint") + parser.add_argument("--notes", default="") + parser.add_argument("--source-command") + parser.add_argument("--output", type=Path, required=True) + + +def cmd_normalize_criterion(args: argparse.Namespace) -> int: + registry = load_registry(args.registry) + case = registry_case(registry, args.case_id) + criterion_id = case.get("criterion_id") + if not criterion_id: + raise ValueError(f"case {args.case_id!r} does not define criterion_id") + samples_ns, raw_source = criterion_samples_ns(args.criterion_root, criterion_id) + artifact = build_artifact( + case=case, + implementation=args.implementation, + upstream_commit=args.upstream_commit, + robowbc_commit=args.robowbc_commit, + provider=args.provider, + host_fingerprint=args.host_fingerprint, + samples_ns=samples_ns, + hz=None, + notes=args.notes, + source_command=args.source_command, + raw_source=raw_source, + status="ok", + ) + dump_json(args.output, artifact) + return 0 + + +def cmd_normalize_run_report(args: argparse.Namespace) -> int: + registry = load_registry(args.registry) + case = registry_case(registry, args.case_id) + samples_ns, hz, raw_source = run_report_samples_ns(args.input) + artifact = build_artifact( + case=case, + implementation=args.implementation, + upstream_commit=args.upstream_commit, + robowbc_commit=args.robowbc_commit, + provider=args.provider, + host_fingerprint=args.host_fingerprint, + samples_ns=samples_ns, + hz=hz, + notes=args.notes, + source_command=args.source_command, + raw_source=raw_source, + status="ok", + ) + dump_json(args.output, artifact) + return 0 + + +def cmd_normalize_samples(args: argparse.Namespace) -> int: + registry = load_registry(args.registry) + case = registry_case(registry, args.case_id) + samples_ns, hz, raw_source = manual_samples_payload(args.input) + artifact = build_artifact( + case=case, + implementation=args.implementation, + upstream_commit=args.upstream_commit, + robowbc_commit=args.robowbc_commit, + provider=args.provider, + host_fingerprint=args.host_fingerprint, + samples_ns=samples_ns, + hz=hz, + notes=args.notes, + source_command=args.source_command, + raw_source=raw_source, + status="ok", + ) + dump_json(args.output, artifact) + return 0 + + +def cmd_emit_blocked(args: argparse.Namespace) -> int: + registry = load_registry(args.registry) + case = registry_case(registry, args.case_id) + artifact = build_artifact( + case=case, + implementation=args.implementation, + upstream_commit=args.upstream_commit, + robowbc_commit=args.robowbc_commit, + provider=args.provider, + host_fingerprint=args.host_fingerprint, + samples_ns=[], + hz=None, + notes=args.reason if not args.notes else f"{args.notes} | {args.reason}", + source_command=args.source_command, + raw_source=args.reason, + status="blocked", + ) + dump_json(args.output, artifact) + return 0 + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + subparsers = parser.add_subparsers(dest="command", required=True) + + validate = subparsers.add_parser("validate-registry") + validate.add_argument("--registry", type=Path, required=True) + validate.set_defaults(func=cmd_validate_registry) + + normalize_criterion = subparsers.add_parser("normalize-criterion") + common_artifact_args(normalize_criterion) + normalize_criterion.add_argument("--criterion-root", type=Path, required=True) + normalize_criterion.set_defaults(func=cmd_normalize_criterion) + + normalize_run = subparsers.add_parser("normalize-run-report") + common_artifact_args(normalize_run) + normalize_run.add_argument("--input", type=Path, required=True) + normalize_run.set_defaults(func=cmd_normalize_run_report) + + normalize_samples = subparsers.add_parser("normalize-samples") + common_artifact_args(normalize_samples) + normalize_samples.add_argument("--input", type=Path, required=True) + normalize_samples.set_defaults(func=cmd_normalize_samples) + + blocked = subparsers.add_parser("emit-blocked") + common_artifact_args(blocked) + blocked.add_argument("--reason", required=True) + blocked.set_defaults(func=cmd_emit_blocked) + + return parser + + +def main() -> int: + parser = build_parser() + args = parser.parse_args() + return args.func(args) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/benchmarks/render_nvidia_benchmark_summary.py b/scripts/benchmarks/render_nvidia_benchmark_summary.py new file mode 100644 index 0000000..13470bb --- /dev/null +++ b/scripts/benchmarks/render_nvidia_benchmark_summary.py @@ -0,0 +1,629 @@ +#!/usr/bin/env python3 +"""Render Markdown and optional HTML summaries from normalized NVIDIA benchmark artifacts.""" + +from __future__ import annotations + +import argparse +import html +import importlib.util +import json +from pathlib import Path +from typing import Any + +ROOT_DIR = Path(__file__).resolve().parents[2] +NORMALIZER_PATH = ROOT_DIR / "scripts/benchmarks/normalize_nvidia_benchmarks.py" + + +def load_normalizer() -> Any: + spec = importlib.util.spec_from_file_location("normalize_nvidia_benchmarks", NORMALIZER_PATH) + if spec is None or spec.loader is None: + raise RuntimeError(f"failed to load normalizer module from {NORMALIZER_PATH}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +NORMALIZER = load_normalizer() +PROVIDER_ORDER = NORMALIZER.PROVIDER_ORDER +IMPLEMENTATION_ORDER = NORMALIZER.IMPLEMENTATION_ORDER +BASELINE_IMPLEMENTATION = "ort-cpp-sonic" +RUST_IMPLEMENTATION = "ort-rs" + + +def load_json(path: Path) -> Any: + with path.open("r", encoding="utf-8") as handle: + return json.load(handle) + + +def load_registry(path: Path) -> dict[str, Any]: + registry = load_json(path) + if registry.get("schema_version") != 1: + raise ValueError(f"unsupported registry schema version: {registry.get('schema_version')}") + return registry + + +def load_artifact(path: Path) -> dict[str, Any] | None: + if not path.is_file(): + return None + return load_json(path) + + +def format_ns(value: int | None) -> str: + if value is None: + return "n/a" + if value >= 1_000_000: + return f"{value / 1_000_000:.3f} ms" + if value >= 1_000: + return f"{value / 1_000:.3f} us" + return f"{value} ns" + + +def format_hz(value: float | None) -> str: + if value is None: + return "n/a" + return f"{value:.3f} Hz" + + +def format_provider_family(provider: str) -> str: + return NORMALIZER.provider_family(provider) + + +def format_implementation(implementation: str) -> str: + return NORMALIZER.implementation_label(implementation) + + +def rendered_variant_labels() -> list[str]: + variants = [NORMALIZER.provider_family("cpu")] + for provider in PROVIDER_ORDER: + if provider == "cpu": + continue + for implementation in IMPLEMENTATION_ORDER: + variants.append(NORMALIZER.variant_label(provider, implementation)) + return variants + + +def ratio_string( + ort_rs: dict[str, Any] | None, ort_cpp_sonic: dict[str, Any] | None +) -> str: + if not ort_rs or not ort_cpp_sonic: + return "n/a" + if ort_rs.get("status") != "ok" or ort_cpp_sonic.get("status") != "ok": + return "n/a" + ort_rs_p50 = ort_rs.get("p50_ns") + ort_cpp_sonic_p50 = ort_cpp_sonic.get("p50_ns") + if ( + not isinstance(ort_rs_p50, int) + or not isinstance(ort_cpp_sonic_p50, int) + or ort_cpp_sonic_p50 == 0 + ): + return "n/a" + return f"{ort_rs_p50 / ort_cpp_sonic_p50:.2f}x" + + +def artifact_cell(artifact: dict[str, Any] | None) -> str: + if artifact is None: + return "missing" + status = artifact.get("status", "unknown") + if status != "ok": + note = str(artifact.get("notes", "blocked")) + return f"{status}; {note}" + return ( + f"p50 {format_ns(artifact.get('p50_ns'))}; " + f"p95 {format_ns(artifact.get('p95_ns'))}; " + f"hz {format_hz(artifact.get('hz'))}" + ) + + +def canonical_artifact_path( + output_root: Path, implementation: str, provider: str, case_id: str +) -> Path: + return ( + output_root + / NORMALIZER.implementation_artifact_dir(implementation) + / provider + / f"{case_id.replace('/', '__')}.json" + ) + + +def legacy_artifact_path( + output_root: Path, implementation: str, provider: str, case_id: str +) -> Path: + return ( + output_root + / NORMALIZER.legacy_artifact_dir(implementation) + / provider + / f"{case_id.replace('/', '__')}.json" + ) + + +def legacy_cpu_artifact_path(output_root: Path, implementation: str, case_id: str) -> Path: + return ( + output_root + / NORMALIZER.legacy_artifact_dir(implementation) + / f"{case_id.replace('/', '__')}.json" + ) + + +def relpath(root: Path, path: Path) -> str: + return str(path.relative_to(root)).replace("\\", "/") + + +def locate_artifact( + output_root: Path, implementation: str, provider: str, case_id: str +) -> tuple[dict[str, Any] | None, str]: + canonical_path = canonical_artifact_path(output_root, implementation, provider, case_id) + for candidate in ( + canonical_path, + legacy_artifact_path(output_root, implementation, provider, case_id), + legacy_cpu_artifact_path(output_root, implementation, case_id) + if provider == "cpu" + else None, + ): + if candidate is None: + continue + artifact = load_artifact(candidate) + if artifact is not None: + return artifact, relpath(output_root, candidate) + return None, relpath(output_root, canonical_path) + + +def consistent_field(artifacts: list[dict[str, Any]], field: str) -> str: + values = { + str(value) + for artifact in artifacts + if artifact and (value := artifact.get(field)) not in (None, "") + } + if not values: + return "n/a" + if len(values) == 1: + return values.pop() + return "multiple" + + +def status_class(artifact: dict[str, Any] | None) -> str: + if artifact is None: + return "missing" + if artifact.get("status") == "ok": + return "ok" + return "blocked" + + +def case_group(case_id: str) -> str: + if case_id.startswith("gear_sonic/planner_only_"): + return "Planner only" + if case_id == "gear_sonic/encoder_decoder_only_tracking_tick": + return "Encoder + decoder only" + if case_id.startswith("gear_sonic/full_velocity_tick_"): + return "Full velocity tick" + if case_id == "gear_sonic/end_to_end_cli_loop": + return "GEAR-Sonic end-to-end" + if case_id.startswith("decoupled_wbc/"): + return "Decoupled WBC" + return "Other" + + +def build_provider_section( + registry: dict[str, Any], output_root: Path, provider: str +) -> dict[str, Any]: + artifacts_by_implementation: dict[str, list[dict[str, Any]]] = { + implementation: [] for implementation in IMPLEMENTATION_ORDER + } + rows: list[dict[str, Any]] = [] + + for case in registry["cases"]: + case_id = case["case_id"] + row_artifacts: dict[str, dict[str, Any] | None] = {} + row_relpaths: dict[str, str] = {} + for implementation in IMPLEMENTATION_ORDER: + artifact, artifact_relpath = locate_artifact(output_root, implementation, provider, case_id) + row_artifacts[implementation] = artifact + row_relpaths[implementation] = artifact_relpath + if artifact is not None: + artifacts_by_implementation[implementation].append(artifact) + + rows.append( + { + "provider": format_provider_family(provider), + "provider_id": provider, + "provider_label": format_provider_family(provider), + "case_id": case_id, + "group": case_group(case_id), + "description": str(case.get("description", "")), + "interpretation": str(case["interpretation"]), + "artifacts": row_artifacts, + "artifact_relpaths": row_relpaths, + "ratio": ratio_string( + row_artifacts[RUST_IMPLEMENTATION], + row_artifacts[BASELINE_IMPLEMENTATION], + ), + } + ) + + artifacts = [ + artifact + for implementation in IMPLEMENTATION_ORDER + for artifact in artifacts_by_implementation[implementation] + ] + return { + "provider": format_provider_family(provider), + "provider_id": provider, + "provider_label": format_provider_family(provider), + "rows": rows, + "case_count": len(rows), + "ok_pair_count": sum( + 1 + for row in rows + if row["artifacts"][BASELINE_IMPLEMENTATION] is not None + and row["artifacts"][RUST_IMPLEMENTATION] is not None + and row["artifacts"][BASELINE_IMPLEMENTATION].get("status") == "ok" + and row["artifacts"][RUST_IMPLEMENTATION].get("status") == "ok" + ), + "blocked_count": sum( + 1 + for row in rows + if status_class(row["artifacts"][BASELINE_IMPLEMENTATION]) != "ok" + or status_class(row["artifacts"][RUST_IMPLEMENTATION]) != "ok" + ), + "provenance": { + "robowbc_commit": consistent_field(artifacts, "robowbc_commit"), + "upstream_commit": consistent_field(artifacts, "upstream_commit"), + "host_fingerprint": consistent_field(artifacts, "host_fingerprint"), + }, + } + + +def build_summary(registry: dict[str, Any], output_root: Path) -> dict[str, Any]: + provider_sections = [ + build_provider_section(registry, output_root, provider) for provider in PROVIDER_ORDER + ] + artifacts = [ + artifact + for section in provider_sections + for row in section["rows"] + for artifact in row["artifacts"].values() + if artifact is not None + ] + return { + "providers": [format_provider_family(provider) for provider in PROVIDER_ORDER], + "provider_ids": list(PROVIDER_ORDER), + "implementations": [format_implementation(implementation) for implementation in IMPLEMENTATION_ORDER], + "variants": rendered_variant_labels(), + "provider_sections": provider_sections, + "case_count": len(registry["cases"]), + "row_count": sum(section["case_count"] for section in provider_sections), + "ok_pair_count": sum(section["ok_pair_count"] for section in provider_sections), + "blocked_count": sum(section["blocked_count"] for section in provider_sections), + "provenance": { + "robowbc_commit": consistent_field(artifacts, "robowbc_commit"), + "upstream_commit": consistent_field(artifacts, "upstream_commit"), + "host_fingerprint": consistent_field(artifacts, "host_fingerprint"), + }, + } + + +def rerun_commands() -> str: + return "\n".join( + [ + 'for provider in cpu cuda tensor_rt; do', + ' python3 scripts/benchmarks/bench_robowbc_compare.py --all --provider "$provider"', + ' python3 scripts/benchmarks/bench_nvidia_official.py --all --provider "$provider"', + "done", + "python3 scripts/benchmarks/render_nvidia_benchmark_summary.py --output artifacts/benchmarks/nvidia/SUMMARY.md", + "# or build the full static site bundle:", + "python3 scripts/site/build_site.py --output-dir /tmp/robowbc-site", + ] + ) + + +def render_markdown(summary: dict[str, Any]) -> str: + provenance = summary["provenance"] + provider_list = ", ".join(f"`{provider}`" for provider in summary["providers"]) + variant_list = ", ".join(f"`{variant}`" for variant in summary["variants"]) + implementation_list = ", ".join(f"`{implementation}`" for implementation in summary["implementations"]) + lines: list[str] = [ + "# NVIDIA Comparison Summary", + "", + "Generated from normalized artifacts under `artifacts/benchmarks/nvidia/`", + "using the tracked case registry in `benchmarks/nvidia/cases.json`.", + "", + "## Provenance", + "", + f"- Variant families rendered: {provider_list}", + f"- Canonical variants: {variant_list}", + f"- Implementations compared: {implementation_list}", + f"- RoboWBC commit: `{provenance['robowbc_commit']}`", + f"- Official upstream commit: `{provenance['upstream_commit']}`", + f"- Host fingerprint: `{provenance['host_fingerprint']}`", + f"- Canonical cases per provider family: `{summary['case_count']}`", + f"- Total rendered rows: `{summary['row_count']}`", + f"- Matched ok pairs: `{summary['ok_pair_count']}`", + f"- Blocked or missing rows: `{summary['blocked_count']}`", + "", + ] + + for section in summary["provider_sections"]: + section_provenance = section["provenance"] + lines.extend( + [ + f"## {section['provider_label']}", + "", + f"- Provider family id: `{section['provider']}`", + f"- Benchmark provider request: `{section['provider_id']}`", + f"- Matched ok pairs: `{section['ok_pair_count']}`", + f"- Blocked or missing rows: `{section['blocked_count']}`", + f"- RoboWBC commit: `{section_provenance['robowbc_commit']}`", + f"- Official upstream commit: `{section_provenance['upstream_commit']}`", + f"- Host fingerprint: `{section_provenance['host_fingerprint']}`", + "", + "| Path Group | Case ID | ORT-cpp-sonic | ORT-rs | ORT-rs / ORT-cpp-sonic (p50) | Why it matters |", + "|------------|---------|----------------|--------|-------------------------------|----------------|", + ] + ) + + for row in section["rows"]: + lines.append( + "| " + + " | ".join( + [ + row["group"], + f"`{row['case_id']}`", + artifact_cell(row["artifacts"][BASELINE_IMPLEMENTATION]), + artifact_cell(row["artifacts"][RUST_IMPLEMENTATION]), + row["ratio"], + row["interpretation"], + ] + ) + + " |" + ) + + lines.extend( + [ + "", + "### Raw Artifacts", + "", + "| Case ID | ORT-cpp-sonic Artifact | ORT-rs Artifact |", + "|---------|-------------------------|-----------------|", + ] + ) + + for row in section["rows"]: + lines.append( + "| " + + " | ".join( + [ + f"`{row['case_id']}`", + f"`{row['artifact_relpaths'][BASELINE_IMPLEMENTATION]}`", + f"`{row['artifact_relpaths'][RUST_IMPLEMENTATION]}`", + ] + ) + + " |" + ) + + lines.append("") + + lines.extend( + [ + "## Rerun", + "", + "```bash", + rerun_commands(), + "```", + "", + "If a future environment is missing models or build prerequisites, the wrappers will emit", + "blocked artifacts instead of silently substituting a different path.", + "", + ] + ) + + return "\n".join(lines) + + +def render_html(summary: dict[str, Any]) -> str: + provenance = summary["provenance"] + provider_label_text = " / ".join(summary["providers"]) + variant_label_text = " / ".join(summary["variants"]) + + def metric_card(label: str, value: str) -> str: + return ( + f"
{html.escape(label)}" + f"{html.escape(value)}
" + ) + + def render_section(section: dict[str, Any]) -> str: + rows_html: list[str] = [] + for row in section["rows"]: + ort_cpp_sonic = row["artifacts"][BASELINE_IMPLEMENTATION] + ort_rs = row["artifacts"][RUST_IMPLEMENTATION] + rows_html.append( + f""" + {html.escape(row['group'])} + + {html.escape(row['case_id'])} +
{html.escape(row['description'])}
+ + {html.escape(artifact_cell(ort_cpp_sonic))} + {html.escape(artifact_cell(ort_rs))} + {html.escape(row['ratio'])} + {html.escape(row['interpretation'])} + + ORT-cpp-sonic JSON
+ ORT-rs JSON + +""" + ) + + section_provenance = section["provenance"] + return f"""
+
+
+ {html.escape(section['provider_label'])} +

{html.escape(section['provider_label'])} Matrix

+

This family keeps the ORT-cpp-sonic baseline beside ORT-rs on the same requested provider. Decoupled WBC remains CPU-only in this phase, so non-CPU rows stay blocked instead of being relabeled.

+
+
+
+ {metric_card("Rows", str(section["case_count"]))} + {metric_card("Matched ok pairs", str(section["ok_pair_count"]))} + {metric_card("Blocked or missing rows", str(section["blocked_count"]))} + {metric_card("Provider request", section["provider_id"])} + {metric_card("RoboWBC commit", section_provenance["robowbc_commit"])} + {metric_card("Official upstream commit", section_provenance["upstream_commit"])} +
+

Host fingerprint: {html.escape(section_provenance['host_fingerprint'])}

+ + + + + + + + + + + + + + {''.join(rows_html)} + +
Path groupCaseORT-cpp-sonicORT-rsORT-rs / ORT-cpp-sonic (p50)Why it mattersArtifacts
+
""" + + provider_nav = "".join( + f'{html.escape(provider)}' + for provider in summary["providers"] + ) + + return f""" + + + + + RoboWBC NVIDIA Comparison + + + +
+
+

RoboWBC NVIDIA Comparison

+

This page is generated automatically from normalized ORT-cpp-sonic and ORT-rs benchmark artifacts. It keeps the benchmark vocabulary aligned with the codebase instead of mixing provider labels with legacy stack names.

+

Canonical variant families: {html.escape(provider_label_text)}. Canonical rendered variants: {html.escape(variant_label_text)}. Decoupled WBC remains CPU-only in this phase and stays blocked on non-CPU rows rather than being approximated.

+
+ Site home + Markdown summary + Case registry + {provider_nav} +
+
+ {metric_card("Canonical cases per family", str(summary["case_count"]))} + {metric_card("Variant families", provider_label_text)} + {metric_card("Canonical variants", variant_label_text)} + {metric_card("Implementation columns", " / ".join(summary["implementations"]))} + {metric_card("Matched ok pairs", str(summary["ok_pair_count"]))} + {metric_card("Blocked or missing rows", str(summary["blocked_count"]))} + {metric_card("RoboWBC commit", provenance["robowbc_commit"])} + {metric_card("Official upstream commit", provenance["upstream_commit"])} +
+

Host fingerprint(s): {html.escape(provenance['host_fingerprint'])}

+
+ + {''.join(render_section(section) for section in summary["provider_sections"])} + +
+

Rerun Commands

+
+ 
{html.escape(rerun_commands())}
+
+
+
+ + +""" + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--registry", + type=Path, + default=Path("benchmarks/nvidia/cases.json"), + ) + parser.add_argument( + "--root", + type=Path, + default=Path("artifacts/benchmarks/nvidia"), + ) + parser.add_argument( + "--output", + type=Path, + default=Path("artifacts/benchmarks/nvidia/SUMMARY.md"), + ) + parser.add_argument( + "--html-output", + type=Path, + help="Optional path for a static HTML report generated from the same artifacts.", + ) + args = parser.parse_args() + + registry = load_registry(args.registry) + summary = build_summary(registry, args.root) + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(render_markdown(summary), encoding="utf-8") + if args.html_output is not None: + args.html_output.parent.mkdir(parents=True, exist_ok=True) + args.html_output.write_text(render_html(summary), encoding="utf-8") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/build_site.py b/scripts/build_site.py old mode 100644 new mode 100755 index e53cd8c..daa2985 --- a/scripts/build_site.py +++ b/scripts/build_site.py @@ -1,302 +1,4 @@ #!/usr/bin/env python3 -"""Build the full RoboWBC static site bundle in one command.""" +from _compat import run_legacy_script -from __future__ import annotations - -import argparse -import hashlib -import os -import platform -import shutil -import subprocess -import sys -import tarfile -import urllib.request -from pathlib import Path - -import normalize_nvidia_benchmarks as benchmark_schema - -MUJOCO_VERSION = "3.6.0" -BENCHMARK_PROVIDERS = benchmark_schema.PROVIDER_ORDER -BENCHMARK_IMPLEMENTATIONS = benchmark_schema.IMPLEMENTATION_ORDER - - -def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--repo-root", default=".", help="Repository root directory") - parser.add_argument( - "--output-dir", - default="/tmp/robowbc-site", - help="Output directory for the generated static site bundle. The directory is recreated on each run.", - ) - parser.add_argument( - "--robowbc-binary", - default="./target/debug/robowbc", - help="Path to the robowbc binary used for policy runs", - ) - parser.add_argument( - "--skip-benchmarks", - action="store_true", - help="Build only the policy site and skip benchmark generation", - ) - return parser.parse_args() - - -def run(argv: list[str], *, cwd: Path, env: dict[str, str] | None = None) -> None: - subprocess.run(argv, cwd=cwd, check=True, text=True, env=env) - - -def recreate_dir(path: Path) -> None: - if path.exists(): - shutil.rmtree(path) - path.mkdir(parents=True, exist_ok=True) - - -def sync_benchmark_metadata(repo_root: Path) -> None: - source_root = repo_root / "benchmarks" / "nvidia" - if not source_root.is_dir(): - raise SystemExit(f"benchmark source root not found: {source_root}") - - artifact_root = repo_root / "artifacts" / "benchmarks" / "nvidia" - artifact_root.mkdir(parents=True, exist_ok=True) - shutil.copy2(source_root / "cases.json", artifact_root / "cases.json") - shutil.copy2(source_root / "README.md", artifact_root / "README.md") - - source_patches = source_root / "patches" - if source_patches.is_dir(): - shutil.copytree(source_patches, artifact_root / "patches", dirs_exist_ok=True) - - -def reset_benchmark_artifacts(repo_root: Path) -> Path: - artifact_root = repo_root / "artifacts" / "benchmarks" / "nvidia" - for implementation in BENCHMARK_IMPLEMENTATIONS: - impl_root = artifact_root / implementation - if impl_root.exists(): - shutil.rmtree(impl_root) - impl_root.mkdir(parents=True, exist_ok=True) - for legacy_dir in benchmark_schema.LEGACY_ARTIFACT_DIRS.values(): - legacy_root = artifact_root / legacy_dir - if legacy_root.exists(): - shutil.rmtree(legacy_root) - return artifact_root - - -def resolve_build_env(repo_root: Path) -> tuple[dict[str, str], Path]: - env = os.environ.copy() - configured = env.get("MUJOCO_DOWNLOAD_DIR") - if configured: - download_dir = Path(configured).expanduser().resolve() - else: - download_dir = (repo_root / ".cache" / "mujoco").resolve() - download_dir.mkdir(parents=True, exist_ok=True) - env["MUJOCO_DOWNLOAD_DIR"] = str(download_dir) - return env, download_dir - - -def prepend_env_path(env: dict[str, str], key: str, value: Path) -> None: - current = env.get(key) - env[key] = f"{value}{os.pathsep}{current}" if current else str(value) - - -def mujoco_archive_name() -> str: - if sys.platform != "linux": - raise SystemExit("scripts/build_site.py currently supports Linux-only MuJoCo site builds") - - machine = platform.machine().lower() - arch_map = { - "x86_64": "x86_64", - "amd64": "x86_64", - "aarch64": "aarch64", - "arm64": "aarch64", - } - try: - arch = arch_map[machine] - except KeyError as exc: - raise SystemExit(f"unsupported architecture for MuJoCo site build: {machine}") from exc - return f"mujoco-{MUJOCO_VERSION}-linux-{arch}.tar.gz" - - -def sha256sum(path: Path) -> str: - digest = hashlib.sha256() - with path.open("rb") as handle: - for chunk in iter(lambda: handle.read(1024 * 1024), b""): - digest.update(chunk) - return digest.hexdigest() - - -def download_file(url: str, destination: Path) -> None: - with urllib.request.urlopen(url) as response, destination.open("wb") as handle: - shutil.copyfileobj(response, handle) - - -def mujoco_runtime_library(download_dir: Path) -> Path: - return download_dir / f"mujoco-{MUJOCO_VERSION}" / "lib" / "libmujoco.so" - - -def ensure_mujoco_runtime(download_dir: Path) -> None: - library_path = mujoco_runtime_library(download_dir) - if library_path.is_file(): - return - - archive = mujoco_archive_name() - base_url = f"https://github.com/google-deepmind/mujoco/releases/download/{MUJOCO_VERSION}" - archive_path = download_dir / archive - checksum_path = download_dir / f"{archive}.sha256" - - print(f"MuJoCo runtime missing; downloading {archive} into {download_dir}") - download_file(f"{base_url}/{archive}", archive_path) - download_file(f"{base_url}/{archive}.sha256", checksum_path) - - expected_sha = checksum_path.read_text(encoding="utf-8").split()[0] - actual_sha = sha256sum(archive_path) - if actual_sha != expected_sha: - raise SystemExit( - "MuJoCo archive checksum mismatch: " - f"expected {expected_sha}, got {actual_sha} for {archive_path}" - ) - - with tarfile.open(archive_path) as tar: - try: - tar.extractall(download_dir, filter="data") - except TypeError: - tar.extractall(download_dir) - - if not library_path.is_file(): - raise SystemExit(f"MuJoCo runtime library not found after extraction: {library_path}") - - -def configure_mujoco_runtime_env(env: dict[str, str], download_dir: Path) -> dict[str, str]: - library_dir = mujoco_runtime_library(download_dir).parent - if library_dir.is_dir(): - if os.name == "nt": - prepend_env_path(env, "PATH", library_dir) - elif sys.platform == "darwin": - prepend_env_path(env, "DYLD_LIBRARY_PATH", library_dir) - else: - prepend_env_path(env, "LD_LIBRARY_PATH", library_dir) - return env - - -def build_binary(repo_root: Path, binary: Path, env: dict[str, str]) -> None: - run( - [ - "cargo", - "build", - "--bin", - "robowbc", - "--features", - "robowbc-cli/sim-auto-download,robowbc-cli/vis", - ], - cwd=repo_root, - env=env, - ) - if not binary.exists(): - raise SystemExit(f"robowbc binary not found after build: {binary}") - - -def build_benchmarks(repo_root: Path, output_dir: Path, env: dict[str, str]) -> None: - sync_benchmark_metadata(repo_root) - artifact_root = reset_benchmark_artifacts(repo_root) - python = sys.executable - for provider in BENCHMARK_PROVIDERS: - run( - [ - python, - "scripts/bench_robowbc_compare.py", - "--all", - "--provider", - provider, - "--output-root", - str(artifact_root / "ort-rs"), - ], - cwd=repo_root, - env=env, - ) - run( - [ - python, - "scripts/bench_nvidia_official.py", - "--all", - "--provider", - provider, - "--output-root", - str(artifact_root / "ort-cpp-sonic"), - ], - cwd=repo_root, - env=env, - ) - - source_root = repo_root / "artifacts" / "benchmarks" / "nvidia" - if not source_root.is_dir(): - raise SystemExit(f"benchmark artifact root not found: {source_root}") - - run( - [ - python, - "scripts/render_nvidia_benchmark_summary.py", - "--root", - str(source_root), - "--output", - str(source_root / "SUMMARY.md"), - "--html-output", - str(source_root / "index.html"), - ], - cwd=repo_root, - env=env, - ) - - dest_root = output_dir / "benchmarks" / "nvidia" - dest_root.parent.mkdir(parents=True, exist_ok=True) - shutil.copytree(source_root, dest_root, dirs_exist_ok=True) - - -def build_policy_site( - repo_root: Path, - output_dir: Path, - binary: Path, - env: dict[str, str], -) -> None: - run( - [ - sys.executable, - "scripts/generate_policy_showcase.py", - "--repo-root", - str(repo_root), - "--robowbc-binary", - str(binary), - "--output-dir", - str(output_dir), - ], - cwd=repo_root, - env=env, - ) - - -def main() -> int: - args = parse_args() - repo_root = Path(args.repo_root).resolve() - output_dir = Path(args.output_dir).resolve() - binary = (repo_root / args.robowbc_binary).resolve() - env, mujoco_download_dir = resolve_build_env(repo_root) - - recreate_dir(output_dir) - ensure_mujoco_runtime(mujoco_download_dir) - env = configure_mujoco_runtime_env(env, mujoco_download_dir) - build_binary(repo_root, binary, env) - - if not args.skip_benchmarks: - build_benchmarks(repo_root, output_dir, env) - - # Benchmark helpers can invoke `cargo run -p robowbc-cli`, so rebuild the - # final CLI binary with the site feature set before generating policy pages. - build_binary(repo_root, binary, env) - build_policy_site(repo_root, output_dir, binary, env) - - print(f"Built RoboWBC site at {output_dir}") - print(f"Using MUJOCO_DOWNLOAD_DIR={env['MUJOCO_DOWNLOAD_DIR']}") - print(f"Open the home page via: python scripts/serve_showcase.py --dir {output_dir} --open") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) +run_legacy_script("site/build_site.py") diff --git a/scripts/check_mujoco_headless.py b/scripts/check_mujoco_headless.py old mode 100644 new mode 100755 index 6307c87..4acfc54 --- a/scripts/check_mujoco_headless.py +++ b/scripts/check_mujoco_headless.py @@ -1,83 +1,4 @@ #!/usr/bin/env python3 -"""Fail fast when MuJoCo headless rendering is not available. +from _compat import run_legacy_script -This is the same offscreen rendering path used by proof-pack screenshot -capture. `make showcase-verify` runs this first so local developer checks and -the GitHub Actions showcase job fail on the same prerequisite instead of -shipping a site bundle with skipped screenshots. -""" - -from __future__ import annotations - -import os -import sys - -from roboharness_report import ensure_headless_mujoco_env, is_headless_render_backend_error - -PROBE_XML = """ - - - - - - - - - -""".strip() - - -def render_probe() -> tuple[str, tuple[int, ...]]: - ensure_headless_mujoco_env() - - import mujoco - - backend = os.environ.get("MUJOCO_GL", "auto") - model = mujoco.MjModel.from_xml_string(PROBE_XML) - data = mujoco.MjData(model) - renderer = mujoco.Renderer(model, height=64, width=64) - try: - mujoco.mj_forward(model, data) - renderer.update_scene(data) - frame = renderer.render() - finally: - close = getattr(renderer, "close", None) - if callable(close): - close() - return backend, tuple(int(dimension) for dimension in frame.shape) - - -def package_hint() -> str: - if sys.platform != "linux": - return "" - return ( - "Install the EGL/Mesa runtime used by CI before retrying, for example:\n" - " sudo apt-get install -y libegl1 libegl-mesa0 libgles2 libgl1-mesa-dri libgbm1" - ) - - -def main() -> int: - try: - backend, frame_shape = render_probe() - except Exception as exc: - backend = os.environ.get("MUJOCO_GL", "auto") - if is_headless_render_backend_error(exc): - hint = package_hint() - raise SystemExit( - "MuJoCo headless render smoke check failed for the configured " - f"backend ({backend}): {type(exc).__name__}: {exc}\n" - "This blocks proof-pack screenshot capture, so " - "`make showcase-verify` refuses to continue.\n" - f"{hint}".rstrip() - ) from exc - raise - - print( - "MuJoCo headless render smoke check passed: " - f"backend={backend} frame_shape={frame_shape}" - ) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) +run_legacy_script("mujoco/check_mujoco_headless.py") diff --git a/scripts/download_bfm_zero_models.sh b/scripts/download_bfm_zero_models.sh index 2d2a90a..1d5342b 100755 --- a/scripts/download_bfm_zero_models.sh +++ b/scripts/download_bfm_zero_models.sh @@ -1,43 +1,4 @@ #!/usr/bin/env bash set -euo pipefail - -DEST_DIR="${1:-models/bfm_zero}" -BASE_URL="https://huggingface.co/LeCAR-Lab/BFM-Zero/resolve/main" -TMP_DIR="$(mktemp -d)" -UPSTREAM_DIR="${TMP_DIR}/bfm-zero-upstream" -ONNX_TARGET="${DEST_DIR}/bfm_zero_g1.onnx" -CONTEXT_TARGET="${DEST_DIR}/zs_walking.npy" - -cleanup() { - rm -rf "${TMP_DIR}" -} -trap cleanup EXIT - -if [[ -s "${ONNX_TARGET}" && -s "${CONTEXT_TARGET}" ]]; then - echo "[cache] BFM-Zero assets already present in ${DEST_DIR}" - echo "[cache] $(basename "${ONNX_TARGET}") ($(wc -c < "${ONNX_TARGET}") bytes)" - echo "[cache] $(basename "${CONTEXT_TARGET}") ($(wc -c < "${CONTEXT_TARGET}") bytes)" - exit 0 -fi - -mkdir -p "${UPSTREAM_DIR}/model/exported" "${UPSTREAM_DIR}/model/tracking_inference" "${DEST_DIR}" - -download() { - local url="$1" - local target="$2" - echo "[download] ${target}" - curl -L --fail --retry 3 --retry-delay 2 --show-error --output "${target}" "${url}" -} - -download \ - "${BASE_URL}/model/exported/FBcprAuxModel.onnx" \ - "${UPSTREAM_DIR}/model/exported/FBcprAuxModel.onnx" -download \ - "${BASE_URL}/model/tracking_inference/zs_walking.pkl" \ - "${UPSTREAM_DIR}/model/tracking_inference/zs_walking.pkl" - -python scripts/prepare_bfm_zero_assets.py \ - --source "${UPSTREAM_DIR}" \ - --output "${DEST_DIR}" - -echo "BFM-Zero assets ready in ${DEST_DIR}" +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +exec bash "${SCRIPT_DIR}/models/download_bfm_zero_models.sh" "$@" diff --git a/scripts/download_decoupled_wbc_models.sh b/scripts/download_decoupled_wbc_models.sh index f954492..268c8aa 100755 --- a/scripts/download_decoupled_wbc_models.sh +++ b/scripts/download_decoupled_wbc_models.sh @@ -1,24 +1,4 @@ #!/usr/bin/env bash set -euo pipefail - -DEST_DIR="${1:-models/decoupled-wbc}" -UPSTREAM_COMMIT="${GR00T_WBC_UPSTREAM_COMMIT:-bc38f6d0ce6cab4589e025037ad0bfbab7ba73d8}" -BASE_MEDIA_URL="https://media.githubusercontent.com/media/NVlabs/GR00T-WholeBodyControl/${UPSTREAM_COMMIT}/decoupled_wbc/sim2mujoco/resources/robots/g1" -BASE_RAW_URL="https://raw.githubusercontent.com/NVlabs/GR00T-WholeBodyControl/${UPSTREAM_COMMIT}/decoupled_wbc/sim2mujoco/resources/robots/g1" - -mkdir -p "${DEST_DIR}" -printf '%s\n' "${UPSTREAM_COMMIT}" > "${DEST_DIR}/REVISION" -echo "[info] pinned GR00T-WholeBodyControl commit: ${UPSTREAM_COMMIT}" - -download() { - local url="$1" - local target="$2" - echo "[download] ${target}" - curl -L --fail --retry 3 --retry-delay 2 --show-error --output "${target}" "${url}" -} - -download "${BASE_MEDIA_URL}/policy/GR00T-WholeBodyControl-Balance.onnx" "${DEST_DIR}/GR00T-WholeBodyControl-Balance.onnx" -download "${BASE_MEDIA_URL}/policy/GR00T-WholeBodyControl-Walk.onnx" "${DEST_DIR}/GR00T-WholeBodyControl-Walk.onnx" -download "${BASE_RAW_URL}/g1_gear_wbc.yaml" "${DEST_DIR}/g1_gear_wbc.yaml" - -echo "Decoupled WBC models ready in ${DEST_DIR}" +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +exec bash "${SCRIPT_DIR}/models/download_decoupled_wbc_models.sh" "$@" diff --git a/scripts/download_gear_sonic_models.sh b/scripts/download_gear_sonic_models.sh index 487eb89..ced3722 100755 --- a/scripts/download_gear_sonic_models.sh +++ b/scripts/download_gear_sonic_models.sh @@ -1,35 +1,4 @@ #!/usr/bin/env bash set -euo pipefail - -DEST_DIR="${1:-models/gear-sonic}" -HF_REVISION="${GEAR_SONIC_HF_REVISION:-cc80d505b7e055fd6ae26426ae8bfa0a74c26011}" -BASE_URL="https://huggingface.co/nvidia/GEAR-SONIC/resolve/${HF_REVISION}" - -mkdir -p "${DEST_DIR}" -printf '%s\n' "${HF_REVISION}" > "${DEST_DIR}/REVISION" -echo "[info] pinned Hugging Face revision: ${HF_REVISION}" - -models=( - "model_encoder.onnx" - "model_decoder.onnx" - "planner_sonic.onnx" -) - -for model in "${models[@]}"; do - target="${DEST_DIR}/${model}" - if [[ -s "${target}" ]]; then - echo "[cache] ${model} already present at ${target} ($(wc -c < "${target}") bytes)" - continue - fi - echo "[download] ${model} -> ${target}" - curl --fail --location --retry 3 --retry-delay 2 \ - "${BASE_URL}/${model}" \ - --output "${target}" - if [[ ! -s "${target}" ]]; then - echo "[error] downloaded file is empty: ${target}" >&2 - exit 1 - fi - echo "[ok] ${model} ($(wc -c < "${target}") bytes)" -done - -echo "Downloaded GEAR-SONIC ONNX models to ${DEST_DIR}." +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +exec bash "${SCRIPT_DIR}/models/download_gear_sonic_models.sh" "$@" diff --git a/scripts/download_gear_sonic_reference_motions.sh b/scripts/download_gear_sonic_reference_motions.sh old mode 100644 new mode 100755 index ba43fec..c3bb538 --- a/scripts/download_gear_sonic_reference_motions.sh +++ b/scripts/download_gear_sonic_reference_motions.sh @@ -1,66 +1,4 @@ #!/usr/bin/env bash set -euo pipefail - -DEST_ROOT="${1:-models/gear-sonic/reference/example}" -shift || true - -UPSTREAM_COMMIT="${GEAR_SONIC_REFERENCE_COMMIT:-bc38f6d0ce6cab4589e025037ad0bfbab7ba73d8}" -BASE_MEDIA_URL="https://media.githubusercontent.com/media/NVlabs/GR00T-WholeBodyControl/${UPSTREAM_COMMIT}/gear_sonic_deploy/reference/example" -BASE_RAW_URL="https://raw.githubusercontent.com/NVlabs/GR00T-WholeBodyControl/${UPSTREAM_COMMIT}/gear_sonic_deploy/reference/example" - -if [[ $# -eq 0 ]]; then - clips=( - "macarena_001__A545" - ) -else - clips=("$@") -fi - -files=( - "joint_pos.csv" - "joint_vel.csv" - "body_pos.csv" - "body_quat.csv" - "body_lin_vel.csv" - "body_ang_vel.csv" - "metadata.txt" - "info.txt" -) - -mkdir -p "${DEST_ROOT}" -printf '%s\n' "${UPSTREAM_COMMIT}" > "${DEST_ROOT}/UPSTREAM_COMMIT" -echo "[info] pinned GR00T-WholeBodyControl commit: ${UPSTREAM_COMMIT}" - -for clip in "${clips[@]}"; do - clip_dir="${DEST_ROOT}/${clip}" - mkdir -p "${clip_dir}" - echo "[clip] ${clip}" - - for file in "${files[@]}"; do - target="${clip_dir}/${file}" - if [[ -s "${target}" ]] && ! head -n 1 "${target}" | grep -q '^version https://git-lfs.github.com/spec/v1$'; then - echo " [cache] ${file} already present" - continue - fi - - case "${file}" in - metadata.txt|info.txt) - source_url="${BASE_RAW_URL}/${clip}/${file}" - ;; - *) - source_url="${BASE_MEDIA_URL}/${clip}/${file}" - ;; - esac - - echo " [download] ${file}" - curl --fail --location --retry 3 --retry-delay 2 \ - "${source_url}" \ - --output "${target}" - if [[ ! -s "${target}" ]]; then - echo " [error] downloaded file is empty: ${target}" >&2 - exit 1 - fi - done -done - -echo "Downloaded GEAR-Sonic reference motions to ${DEST_ROOT}." +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +exec bash "${SCRIPT_DIR}/models/download_gear_sonic_reference_motions.sh" "$@" diff --git a/scripts/download_wbc_agile_models.sh b/scripts/download_wbc_agile_models.sh index 59d8cdd..028d449 100755 --- a/scripts/download_wbc_agile_models.sh +++ b/scripts/download_wbc_agile_models.sh @@ -1,20 +1,4 @@ #!/usr/bin/env bash set -euo pipefail - -DEST_DIR="${1:-models/wbc-agile}" -BASE_MEDIA_URL="https://media.githubusercontent.com/media/nvidia-isaac/WBC-AGILE/main/agile/data/policy/velocity_g1" -BASE_RAW_URL="https://raw.githubusercontent.com/nvidia-isaac/WBC-AGILE/main/agile/data/policy/velocity_g1" - -mkdir -p "${DEST_DIR}" - -download() { - local url="$1" - local target="$2" - echo "[download] ${target}" - curl -L --fail --retry 3 --retry-delay 2 --show-error --output "${target}" "${url}" -} - -download "${BASE_MEDIA_URL}/unitree_g1_velocity_e2e.onnx" "${DEST_DIR}/unitree_g1_velocity_e2e.onnx" -download "${BASE_RAW_URL}/unitree_g1_velocity_e2e.yaml" "${DEST_DIR}/unitree_g1_velocity_e2e.yaml" - -echo "WBC-AGILE models ready in ${DEST_DIR}" +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +exec bash "${SCRIPT_DIR}/models/download_wbc_agile_models.sh" "$@" diff --git a/scripts/ensure_mujoco_runtime.py b/scripts/ensure_mujoco_runtime.py old mode 100644 new mode 100755 index 6ffd83c..d1025bb --- a/scripts/ensure_mujoco_runtime.py +++ b/scripts/ensure_mujoco_runtime.py @@ -1,157 +1,4 @@ #!/usr/bin/env python3 -"""Ensure the MuJoCo runtime is available in a local download directory.""" +from _compat import run_legacy_script -from __future__ import annotations - -import argparse -import hashlib -import os -from pathlib import Path -import platform -import shutil -import tarfile -import urllib.request - -MUJOCO_VERSION = "3.6.0" - - -def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser( - description=( - "Download and extract the MuJoCo runtime into the given directory " - "when it is missing." - ) - ) - parser.add_argument( - "--download-dir", - type=Path, - default=os.environ.get("MUJOCO_DOWNLOAD_DIR"), - help="absolute directory used to store the downloaded MuJoCo runtime", - ) - return parser.parse_args() - - -def archive_name() -> str: - if sys_platform() != "linux": - raise SystemExit("scripts/ensure_mujoco_runtime.py currently supports Linux only") - - machine = platform.machine().lower() - arch_map = { - "x86_64": "x86_64", - "amd64": "x86_64", - "aarch64": "aarch64", - "arm64": "aarch64", - } - try: - arch = arch_map[machine] - except KeyError as exc: - raise SystemExit(f"unsupported architecture for MuJoCo runtime download: {machine}") from exc - return f"mujoco-{MUJOCO_VERSION}-linux-{arch}.tar.gz" - - -def sys_platform() -> str: - return platform.system().lower() - - -def sha256sum(path: Path) -> str: - digest = hashlib.sha256() - with path.open("rb") as handle: - for chunk in iter(lambda: handle.read(1024 * 1024), b""): - digest.update(chunk) - return digest.hexdigest() - - -def download_file(url: str, destination: Path) -> None: - with urllib.request.urlopen(url) as response, destination.open("wb") as handle: - shutil.copyfileobj(response, handle) - - -def runtime_root(download_dir: Path) -> Path: - return download_dir / f"mujoco-{MUJOCO_VERSION}" - - -def runtime_library(download_dir: Path) -> Path: - return runtime_root(download_dir) / "lib" / "libmujoco.so" - - -def versioned_runtime_library(download_dir: Path) -> Path: - return runtime_root(download_dir) / "lib" / f"libmujoco.so.{MUJOCO_VERSION}" - - -def ensure_runtime_symlink(download_dir: Path) -> None: - symlink_path = runtime_library(download_dir) - versioned_path = versioned_runtime_library(download_dir) - if symlink_path.exists(): - return - if not versioned_path.is_file(): - raise SystemExit(f"MuJoCo versioned runtime library missing: {versioned_path}") - symlink_path.symlink_to(versioned_path.name) - - -def safe_extract_linux(archive_path: Path, destination: Path) -> None: - destination = destination.resolve() - with tarfile.open(archive_path) as archive: - members = archive.getmembers() - for member in members: - member_path = (destination / member.name).resolve() - if os.path.commonpath([destination, member_path]) != str(destination): - raise SystemExit(f"unsafe tar entry outside extraction root: {member.name}") - archive.extractall(destination, filter="fully_trusted") - - -def ensure_runtime(download_dir: Path) -> Path: - library_path = runtime_library(download_dir) - if library_path.is_file(): - return library_path - - versioned_path = versioned_runtime_library(download_dir) - if versioned_path.is_file(): - ensure_runtime_symlink(download_dir) - return library_path - - archive = archive_name() - base_url = f"https://github.com/google-deepmind/mujoco/releases/download/{MUJOCO_VERSION}" - archive_path = download_dir / archive - checksum_path = download_dir / f"{archive}.sha256" - - print(f"MuJoCo runtime missing; downloading {archive} into {download_dir}") - download_file(f"{base_url}/{archive}", archive_path) - download_file(f"{base_url}/{archive}.sha256", checksum_path) - - expected_sha = checksum_path.read_text(encoding="utf-8").split()[0] - actual_sha = sha256sum(archive_path) - if actual_sha != expected_sha: - raise SystemExit( - "MuJoCo archive checksum mismatch: " - f"expected {expected_sha}, got {actual_sha} for {archive_path}" - ) - - safe_extract_linux(archive_path, download_dir) - archive_path.unlink() - checksum_path.unlink() - - ensure_runtime_symlink(download_dir) - if not library_path.is_file(): - raise SystemExit(f"MuJoCo runtime library not found after extraction: {library_path}") - return library_path - - -def main() -> int: - args = parse_args() - if args.download_dir is None: - raise SystemExit( - "--download-dir is required when MUJOCO_DOWNLOAD_DIR is not set" - ) - - download_dir = args.download_dir.expanduser().resolve() - if not download_dir.is_absolute(): - raise SystemExit(f"download directory must be absolute: {download_dir}") - download_dir.mkdir(parents=True, exist_ok=True) - - library_path = ensure_runtime(download_dir) - print(library_path) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) +run_legacy_script("mujoco/ensure_mujoco_runtime.py") diff --git a/scripts/generate_policy_showcase.py b/scripts/generate_policy_showcase.py index 5c3fe6d..f138664 100755 --- a/scripts/generate_policy_showcase.py +++ b/scripts/generate_policy_showcase.py @@ -1,2899 +1,4 @@ #!/usr/bin/env python3 -"""Generate a mixed-source RoboWBC policy showcase as a static HTML artifact.""" +from _compat import run_legacy_script -from __future__ import annotations - -import argparse -import datetime as dt -import html -import importlib.util -import json -import math -import os -from pathlib import Path -import re -import shutil -import subprocess -import tarfile -import tempfile -import sys -import tomllib -from typing import Iterable - -POLICIES = [ - { - "id": "gear_sonic", - "policy_family": "gear_sonic", - "title": "GEAR-SONIC", - "config": "configs/showcase/gear_sonic_real.toml", - "source": "NVIDIA GR00T", - "summary": "Real CPU planner_sonic.onnx run inside the MuJoCo-backed G1 showcase, driven by an explicit staged velocity-tracking script instead of a single constant command.", - "coverage": "Planner-only velocity tracking on the published G1 planner contract", - "execution_kind": "real", - "checkpoint_source": "Published GEAR-SONIC ONNX checkpoints", - "command_source": "runtime.velocity_schedule", - "demo_family": "Velocity tracking", - "demo_sequence": "Stand, accelerate from 0.0 to 0.6 m/s over 2 s, command a 90 degree right turn over 1 s, accelerate into a 1.0 m/s run over 3 s, then settle back to stand.", - "showcase_gain_profile": "default_pd", - "model_artifact": "models/gear-sonic/planner_sonic.onnx", - "required_paths": [ - "models/gear-sonic/model_encoder.onnx", - "models/gear-sonic/model_decoder.onnx", - "models/gear-sonic/planner_sonic.onnx", - ], - "blocked_reason": "Requires downloaded GEAR-SONIC checkpoints. Run scripts/download_gear_sonic_models.sh or let CI warm the cache first.", - }, - { - "id": "gear_sonic_tracking", - "policy_family": "gear_sonic", - "title": "GEAR-SONIC Reference Motion", - "config": "configs/showcase/gear_sonic_tracking_real.toml", - "source": "NVIDIA GR00T", - "summary": "Real published GEAR-Sonic encoder+decoder tracking on the official `macarena_001__A545` clip, running inside the MuJoCo-backed G1 showcase with the upstream heading-corrected reference orientation contract.", - "coverage": "Published G1 reference-motion tracking with explicit upper-body motion from the official example clip", - "execution_kind": "real", - "checkpoint_source": "Published GEAR-Sonic ONNX checkpoints plus pinned official reference-motion CSVs", - "command_source": "runtime.reference_motion_tracking", - "demo_family": "Reference / pose tracking", - "demo_sequence": "Autoplays the official `macarena_001__A545` reference clip to showcase clip-backed upper-body tracking instead of a standing placeholder.", - "showcase_gain_profile": "simulation_pd", - "model_artifact": "models/gear-sonic/reference/example/macarena_001__A545", - "required_paths": [ - "models/gear-sonic/model_encoder.onnx", - "models/gear-sonic/model_decoder.onnx", - "models/gear-sonic/planner_sonic.onnx", - "models/gear-sonic/reference/example/macarena_001__A545/joint_pos.csv", - "models/gear-sonic/reference/example/macarena_001__A545/joint_vel.csv", - "models/gear-sonic/reference/example/macarena_001__A545/body_quat.csv", - ], - "blocked_reason": "Requires the published GEAR-Sonic checkpoints and the official reference clip payloads. Run scripts/download_gear_sonic_models.sh and scripts/download_gear_sonic_reference_motions.sh first.", - }, - { - "id": "decoupled_wbc", - "title": "Decoupled WBC", - "config": "configs/showcase/decoupled_wbc_real.toml", - "source": "NVIDIA GR00T", - "summary": "Real public GR00T WholeBodyControl run inside the MuJoCo-backed G1 showcase, driven by the same staged locomotion script used for the velocity-only cards.", - "coverage": "Lower-body RL locomotion with default upper-body posture", - "execution_kind": "real", - "checkpoint_source": "Published GR00T WholeBodyControl ONNX checkpoints", - "command_source": "runtime.velocity_schedule", - "demo_family": "Velocity tracking", - "demo_sequence": "Stand, accelerate from 0.0 to 0.6 m/s over 2 s, command a 90 degree right turn over 1 s, accelerate into a 1.0 m/s run over 3 s, then settle back to stand.", - "showcase_gain_profile": "simulation_pd", - "model_artifact": "models/decoupled-wbc/GR00T-WholeBodyControl-Walk.onnx", - "required_paths": [ - "models/decoupled-wbc/GR00T-WholeBodyControl-Balance.onnx", - "models/decoupled-wbc/GR00T-WholeBodyControl-Walk.onnx", - ], - "blocked_reason": "Requires downloaded GR00T WholeBodyControl checkpoints. Run scripts/download_decoupled_wbc_models.sh or let CI warm the cache first.", - }, - { - "id": "bfm_zero", - "title": "BFM-Zero", - "config": "configs/bfm_zero_g1.toml", - "source": "CMU", - "summary": "Real public G1 tracking contract running inside the MuJoCo-backed showcase with a 721D prompt-conditioned observation, IMU gyro/history features, and the shipped walking latent context.", - "coverage": "Reference/context walking tracking", - "execution_kind": "real", - "checkpoint_source": "Prepared BFM-Zero ONNX checkpoint plus tracking context assets", - "command_source": "runtime.motion_tokens", - "demo_family": "Reference / pose tracking", - "demo_sequence": "Replays the shipped `zs_walking.npy` latent walking context. No verified public waving or upper-body mocap clip is bundled in this repo today.", - "showcase_gain_profile": "simulation_pd", - "model_artifact": "models/bfm_zero/bfm_zero_g1.onnx", - "required_paths": [ - "models/bfm_zero/bfm_zero_g1.onnx", - "models/bfm_zero/zs_walking.npy", - ], - "blocked_reason": "Requires public BFM-Zero assets. Run scripts/download_bfm_zero_models.sh or warm the CI cache to fetch the ONNX checkpoint and zs_walking.npy context automatically.", - }, - { - "id": "hover", - "title": "HOVER", - "config": "configs/hover_h1.toml", - "source": "NVIDIA", - "summary": "Real H1 multi-modal masked policy wrapper for locomotion and body-pose commands, enabled when a user-exported checkpoint is available.", - "coverage": "Multi-modal masked H1 controller", - "execution_kind": "real", - "checkpoint_source": "User-exported HOVER ONNX checkpoint", - "command_source": "runtime.velocity", - "demo_family": "Velocity tracking", - "demo_sequence": "Blocked until a compatible public checkpoint exists; intended to use an explicit locomotion command rather than a fabricated upper-body demo.", - "model_artifact": "models/hover/hover_h1.onnx", - "required_paths": [ - "models/hover/hover_h1.onnx", - ], - "blocked_reason": "HOVER ships public code and deployment tooling, but the public repo/releases do not include pretrained checkpoints. Provide your own exported ONNX model to enable this card.", - }, - { - "id": "wbc_agile", - "title": "WBC-AGILE", - "config": "configs/showcase/wbc_agile_real.toml", - "source": "NVIDIA Isaac", - "summary": "Real public G1 checkpoint using the published recurrent history tensors and lower-body target mapping, driven by the staged velocity-tracking script rather than a single constant command.", - "coverage": "Published G1 locomotion checkpoint on the public 29-DOF embodiment", - "execution_kind": "real", - "checkpoint_source": "Published NVIDIA Isaac G1 ONNX checkpoint", - "command_source": "runtime.velocity_schedule", - "demo_family": "Velocity tracking", - "demo_sequence": "Stand, accelerate from 0.0 to 0.6 m/s over 2 s, command a 90 degree right turn over 1 s, accelerate into a 1.0 m/s run over 3 s, then settle back to stand.", - "showcase_gain_profile": "simulation_pd", - "model_artifact": "models/wbc-agile/unitree_g1_velocity_e2e.onnx", - "required_paths": [ - "models/wbc-agile/unitree_g1_velocity_e2e.onnx", - ], - "blocked_reason": "Requires downloaded WBC-AGILE G1 checkpoint. Run scripts/download_wbc_agile_models.sh or let CI warm the cache first.", - }, - { - "id": "wholebody_vla", - "title": "WholeBodyVLA", - "config": "configs/wholebody_vla_x2.toml", - "source": "OpenDriveLab", - "summary": "Experimental AGIBOT X2 kinematic-pose contract wrapper for WholeBodyVLA. The public upstream project does not yet expose a runnable inference release, so this card documents the expected handoff shape and stays blocked until a compatible local model exists.", - "coverage": "Experimental KinematicPose contract placeholder", - "execution_kind": "experimental", - "checkpoint_source": "Local/private WholeBodyVLA ONNX checkpoint", - "command_source": "runtime.kinematic_pose", - "demo_family": "Reference / pose tracking", - "demo_sequence": "Pose-target handoff only. This remains blocked until a runnable upstream model exists; the showcase does not invent a fake upper-body clip.", - "model_artifact": "models/wholebody_vla/wholebody_vla_x2.onnx", - "required_paths": [ - "models/wholebody_vla/wholebody_vla_x2.onnx", - ], - "blocked_reason": "The public WholeBodyVLA repo does not currently provide runnable code or ONNX checkpoints. This wrapper remains blocked until a compatible local model is available.", - }, -] - -NOT_YET_SHOWCASED = [ - { - "name": "wbc_agile_t1", - "reason": "The Booster T1 path exists, but the public upstream release still does not match the ONNX contract used by the Rust CLI today.", - }, - { - "name": "py_model", - "reason": "The showcase job is focused on compiled ORT-backed policies inside the Rust CLI.", - }, -] - -COLORS = ["#0f766e", "#dc2626", "#2563eb", "#d97706", "#7c3aed", "#0891b2"] -RERUN_WEB_VIEWER_DIR = "assets/rerun-web-viewer" -DISPLAY_ORDER = { - "gear_sonic": 0, - "gear_sonic_tracking": 1, - "decoupled_wbc": 2, - "wbc_agile": 3, - "bfm_zero": 4, - "hover": 5, - "wholebody_vla": 6, -} - -DEMO_FAMILY_DESCRIPTIONS = { - "Velocity tracking": "Policies driven by an explicit locomotion command profile. These cards now use the same staged sequence instead of a single constant velocity.", - "Reference / pose tracking": "Policies driven by pose targets, motion references, or latent tracking context. If no verified official clip is wired, the card stays blocked instead of inventing a demo.", -} - - -def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser() - parser.add_argument("--repo-root", default=".") - parser.add_argument("--robowbc-binary", required=True) - parser.add_argument("--output-dir", required=True) - return parser.parse_args() - - -def site_relative_path(root: Path, path: Path) -> str: - return path.relative_to(root).as_posix() - - -def policy_output_dir(output_dir: Path, policy_id: str) -> Path: - return output_dir / "policies" / policy_id - - -def detail_page_path(output_dir: Path, card_id: str) -> Path: - return policy_output_dir(output_dir, card_id) / "index.html" - - -def resolve_ort_dylib(repo_root: Path) -> str | None: - explicit = os.environ.get("ROBOWBC_ORT_DYLIB_PATH") - if explicit: - return explicit - - candidates = sorted( - repo_root.glob( - "target/debug/build/robowbc-ort-*/out/onnxruntime-linux-x64-1.24.2/lib/libonnxruntime.so.1.24.2" - ), - key=lambda path: path.stat().st_mtime, - reverse=True, - ) - for path in candidates: - providers = path.parent / "libonnxruntime_providers_shared.so" - if providers.exists(): - return str(path) - return str(candidates[0]) if candidates else None - - -def resolve_mujoco_runtime_libdir(env: dict[str, str]) -> Path | None: - explicit = env.get("MUJOCO_DYNAMIC_LINK_DIR") - if explicit: - return Path(explicit) - - download_dir = env.get("MUJOCO_DOWNLOAD_DIR") - if not download_dir: - return None - - if os.name == "nt": - library_name = "mujoco.dll" - elif sys.platform == "darwin": - library_name = "libmujoco.dylib" - else: - library_name = "libmujoco.so" - - candidates = sorted( - Path(download_dir).glob(f"mujoco-*/lib/{library_name}"), - key=lambda path: path.stat().st_mtime, - reverse=True, - ) - return candidates[0].parent if candidates else None - - -def prepend_env_path(env: dict[str, str], key: str, value: Path) -> None: - current = env.get(key) - env[key] = f"{value}{os.pathsep}{current}" if current else str(value) - - -def configure_binary_runtime_env(env: dict[str, str]) -> dict[str, str]: - libdir = resolve_mujoco_runtime_libdir(env) - if libdir is None: - return env - - if os.name == "nt": - prepend_env_path(env, "PATH", libdir) - elif sys.platform == "darwin": - prepend_env_path(env, "DYLD_LIBRARY_PATH", libdir) - else: - prepend_env_path(env, "LD_LIBRARY_PATH", libdir) - return env - - -def resolve_rerun_web_viewer_version(repo_root: Path) -> str: - lock_text = (repo_root / "Cargo.lock").read_text(encoding="utf-8") - match = re.search(r'name = "rerun"\nversion = "([^"]+)"', lock_text) - if match is None: - raise SystemExit("failed to resolve rerun version from Cargo.lock") - return match.group(1) - - -def vendor_rerun_web_viewer(repo_root: Path, output_dir: Path) -> dict[str, str]: - version = resolve_rerun_web_viewer_version(repo_root) - viewer_dir = output_dir / RERUN_WEB_VIEWER_DIR - viewer_dir.mkdir(parents=True, exist_ok=True) - version_file = viewer_dir / "VERSION" - - target_files = { - "index_js": viewer_dir / "index.js", - "viewer_js": viewer_dir / "re_viewer.js", - "viewer_wasm": viewer_dir / "re_viewer_bg.wasm", - } - if ( - all(path.exists() for path in target_files.values()) - and version_file.exists() - and version_file.read_text(encoding="utf-8").strip() == version - ): - return { - "version": version, - "module_path": f"./{RERUN_WEB_VIEWER_DIR}/index.js", - } - - if shutil.which("npm") is None: - raise SystemExit( - "npm is required to vendor the embedded Rerun web viewer assets for the policy showcase" - ) - - with tempfile.TemporaryDirectory(prefix="robowbc-rerun-web-viewer-") as tempdir: - temp_path = Path(tempdir) - proc = subprocess.run( - ["npm", "pack", f"@rerun-io/web-viewer@{version}"], - cwd=temp_path, - capture_output=True, - text=True, - check=False, - ) - if proc.returncode != 0: - raise SystemExit( - "failed to fetch @rerun-io/web-viewer via npm pack:\n" - f"{proc.stdout}\n--- STDERR ---\n{proc.stderr}" - ) - - tgz_files = list(temp_path.glob("rerun-io-web-viewer-*.tgz")) - if len(tgz_files) != 1: - raise SystemExit("unexpected npm pack output for @rerun-io/web-viewer") - - with tarfile.open(tgz_files[0]) as tar: - try: - tar.extractall(temp_path, filter="data") - except TypeError: - tar.extractall(temp_path) - - package_dir = temp_path / "package" - index_text = (package_dir / "index.js").read_text(encoding="utf-8") - index_text = index_text.replace('import("./re_viewer")', 'import("./re_viewer.js")') - target_files["index_js"].write_text(index_text, encoding="utf-8") - shutil.copy2(package_dir / "re_viewer.js", target_files["viewer_js"]) - shutil.copy2(package_dir / "re_viewer_bg.wasm", target_files["viewer_wasm"]) - version_file.write_text(version, encoding="utf-8") - - return { - "version": version, - "module_path": f"./{RERUN_WEB_VIEWER_DIR}/index.js", - } - - -def derive_replay_trace_path(report_path: Path) -> Path: - stem = report_path.stem or "run" - suffix = report_path.suffix or ".json" - return report_path.with_name(f"{stem}_replay_trace{suffix}") - - -def load_roboharness_report_module(): - script_path = Path(__file__).with_name("roboharness_report.py") - spec = importlib.util.spec_from_file_location("roboharness_report", script_path) - if spec is None or spec.loader is None: - raise RuntimeError(f"failed to load roboharness report helpers from {script_path}") - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - return module - - -def resolve_showcase_context(repo_root: Path, policy: dict[str, object]) -> dict[str, object]: - config_path = repo_root / str(policy["config"]) - app_config = tomllib.loads(config_path.read_text(encoding="utf-8")) - comm_cfg = app_config.get("comm") or app_config.get("communication") or {} - frequency_hz = int(comm_cfg.get("frequency_hz", 50) or 50) - runtime_cfg = app_config.get("runtime") or {} - configured_max_ticks = runtime_cfg.get("max_ticks") - report_max_frames = int(policy.get("report_max_frames") or configured_max_ticks or 120) - existing_sim = app_config.get("sim") - - robot_cfg_path = app_config.get("robot", {}).get("config_path") - robot_model_path = None - if robot_cfg_path: - robot_cfg = tomllib.loads((repo_root / str(robot_cfg_path)).read_text(encoding="utf-8")) - robot_model_path = robot_cfg.get("model_path") - - timestep = float(policy.get("showcase_timestep", 0.002)) - derived_substeps = round(1.0 / (max(frequency_hz, 1) * timestep)) - default_substeps = int(policy.get("showcase_substeps", max(derived_substeps, 1))) - default_gain_profile = str(policy.get("showcase_gain_profile", "simulation_pd")) - - if isinstance(existing_sim, dict): - model_path = str(existing_sim.get("model_path") or robot_model_path or "") - timestep = float(existing_sim.get("timestep", timestep)) - substeps = int(existing_sim.get("substeps", default_substeps)) - gain_profile = str(existing_sim.get("gain_profile") or default_gain_profile) - return { - "transport": "mujoco" if model_path else "synthetic", - "model_path": model_path or None, - "timestep": timestep, - "substeps": substeps, - "gain_profile": gain_profile if model_path else None, - "robot_config_path": str(robot_cfg_path) if robot_cfg_path else None, - "config_has_sim_section": True, - "report_max_frames": report_max_frames, - } - - if robot_model_path is None: - return { - "transport": "synthetic", - "model_path": None, - "timestep": None, - "substeps": None, - "gain_profile": None, - "robot_config_path": str(robot_cfg_path) if robot_cfg_path else None, - "config_has_sim_section": False, - "report_max_frames": report_max_frames, - } - - return { - "transport": "mujoco", - "model_path": str(robot_model_path), - "timestep": timestep, - "substeps": default_substeps, - "gain_profile": default_gain_profile, - "robot_config_path": str(robot_cfg_path) if robot_cfg_path else None, - "config_has_sim_section": False, - "report_max_frames": report_max_frames, - } - - -def ensure_showcase_sim_section(base_toml: str, showcase_context: dict[str, object]) -> str: - if showcase_context["transport"] != "mujoco": - return base_toml.rstrip() - - required_lines = [ - (r"^model_path\s*=", f'model_path = "{showcase_context["model_path"]}"'), - (r"^timestep\s*=", f'timestep = {showcase_context["timestep"]:g}'), - (r"^substeps\s*=", f'substeps = {showcase_context["substeps"]}'), - ( - r"^gain_profile\s*=", - f'gain_profile = "{showcase_context["gain_profile"]}"', - ), - ] - sim_section_pattern = re.compile(r"^(\[sim\].*?)(?=^\[|\Z)", re.MULTILINE | re.DOTALL) - match = sim_section_pattern.search(base_toml) - if match is None: - sim_lines = ["[sim]"] - sim_lines.extend(line for _, line in required_lines) - return "\n\n".join([base_toml.rstrip(), "\n".join(sim_lines)]) - - sim_section = match.group(1).rstrip() - for pattern, line in required_lines: - if re.search(pattern, sim_section, re.MULTILINE) is None: - sim_section += "\n" + line - sim_section += "\n" - return base_toml[: match.start()] + sim_section + base_toml[match.end() :] - - -def compose_showcase_config( - base_toml: str, - policy_id: str, - json_path: Path, - rrd_path: Path, - showcase_context: dict[str, object], -) -> str: - sections = [ensure_showcase_sim_section(base_toml, showcase_context).rstrip()] - sections.append( - "\n".join( - [ - "[vis]", - f'app_id = "robowbc-showcase-{policy_id}"', - "spawn_viewer = false", - f'save_path = "{rrd_path.as_posix()}"', - "", - "[report]", - f'output_path = "{json_path.as_posix()}"', - f'max_frames = {int(showcase_context["report_max_frames"])}', - ] - ) - ) - return "\n\n".join(sections) + "\n" - - -def missing_required_paths(repo_root: Path, policy: dict[str, object]) -> list[str]: - required = policy.get("required_paths", []) - assert isinstance(required, list) - missing: list[str] = [] - for rel_path in required: - candidate = repo_root / str(rel_path) - if not candidate.exists(): - missing.append(str(rel_path)) - return missing - - -def detect_transport(log_text: str) -> str: - if "mujoco simulation transport active" in log_text: - return "mujoco" - if "unitree g1 hardware transport active" in log_text: - return "hardware" - return "synthetic" - - -def detect_mujoco_model_variant(log_text: str) -> str | None: - match = re.search(r"model_variant=([^,\s)]+)", log_text) - return match.group(1) if match else None - - -def build_proof_pack_manifest_payload( - policy_dir: Path, - report: dict[str, object], - checkpoints: list[dict[str, object]], -) -> dict[str, object]: - helpers = load_roboharness_report_module() - return helpers.build_proof_pack_manifest_payload( - policy_dir, - report, - checkpoints, - html_entrypoint="index.html", - ) - - -def generate_policy_proof_pack( - repo_root: Path, - policy_dir: Path, - report: dict[str, object], - site_root: Path, -) -> tuple[dict[str, str], dict[str, object]] | None: - report_meta = report.get("_meta") - if not isinstance(report_meta, dict): - return None - - showcase_context = report_meta.get("showcase_context") - if not isinstance(showcase_context, dict) or showcase_context.get("transport") != "mujoco": - return None - - helpers = load_roboharness_report_module() - checkpoints = helpers.capture_frames_from_report(repo_root, report, policy_dir) - manifest_payload = build_proof_pack_manifest_payload(policy_dir, report, checkpoints) - manifest_path = policy_dir / "proof_pack_manifest.json" - manifest_path.write_text(json.dumps(manifest_payload, indent=2), encoding="utf-8") - return ( - { - "proof_pack_manifest_file": site_relative_path(site_root, manifest_path), - }, - manifest_payload, - ) - - -def policy_meta( - policy: dict[str, object], - site_root: Path, - showcase_context: dict[str, object], - actual_transport: str | None = None, - actual_model_variant: str | None = None, - json_path: Path | None = None, - rrd_path: Path | None = None, - log_path: Path | None = None, - proof_pack_artifacts: dict[str, str] | None = None, -) -> dict[str, object]: - meta = { - "card_id": policy["id"], - "policy_family": policy.get("policy_family", policy["id"]), - "title": policy["title"], - "source": policy["source"], - "summary": policy["summary"], - "coverage": policy["coverage"], - "execution_kind": policy["execution_kind"], - "checkpoint_source": policy["checkpoint_source"], - "command_source": policy["command_source"], - "demo_family": policy["demo_family"], - "demo_sequence": policy["demo_sequence"], - "model_artifact": policy.get("model_artifact", ""), - "config_path": policy["config"], - "required_paths": list(policy.get("required_paths", [])), - "blocked_reason": policy.get("blocked_reason"), - "showcase_transport": actual_transport or str(showcase_context["transport"]), - "showcase_model_path": showcase_context.get("model_path"), - "showcase_gain_profile": showcase_context.get("gain_profile"), - "showcase_model_variant": actual_model_variant, - "robot_config_path": showcase_context.get("robot_config_path"), - } - if json_path is not None: - meta["json_file"] = site_relative_path(site_root, json_path) - if rrd_path is not None: - meta["rrd_file"] = site_relative_path(site_root, rrd_path) - if log_path is not None: - meta["log_file"] = site_relative_path(site_root, log_path) - if proof_pack_artifacts is not None: - meta.update(proof_pack_artifacts) - return meta - - - -def blocked_entry(repo_root: Path, policy: dict[str, object]) -> dict[str, object]: - missing = missing_required_paths(repo_root, policy) - showcase_context = resolve_showcase_context(repo_root, policy) - return { - "card_id": policy["id"], - "policy_name": policy.get("policy_family", policy["id"]), - "status": "blocked", - "metrics": None, - "frames": [], - "joint_names": [], - "command_kind": str(policy["command_source"]).removeprefix("runtime."), - "command_data": [], - "_meta": { - **policy_meta(policy, repo_root, showcase_context), - "missing_paths": missing, - }, - } - - - -def run_policy( - repo_root: Path, - binary: Path, - output_dir: Path, - policy: dict[str, object], - env: dict[str, str], -) -> dict[str, object]: - missing = missing_required_paths(repo_root, policy) - if missing: - return blocked_entry(repo_root, policy) - - policy_id = str(policy["id"]) - policy_dir = policy_output_dir(output_dir, policy_id) - policy_dir.mkdir(parents=True, exist_ok=True) - showcase_context = resolve_showcase_context(repo_root, policy) - base_config = (repo_root / str(policy["config"])).read_text(encoding="utf-8") - temp_config = policy_dir / "run.toml" - json_path = policy_dir / "run.json" - replay_path = derive_replay_trace_path(json_path) - rrd_path = policy_dir / "run.rrd" - log_path = policy_dir / "run.log" - temp_config.write_text( - compose_showcase_config( - base_config, - policy_id, - json_path, - rrd_path, - showcase_context, - ), - encoding="utf-8", - ) - - proc = subprocess.run( - [str(binary), "run", "--config", str(temp_config)], - cwd=repo_root, - env=env, - capture_output=True, - text=True, - check=False, - ) - log_text = proc.stdout + "\n--- STDERR ---\n" + proc.stderr - log_path.write_text(log_text, encoding="utf-8") - if proc.returncode != 0: - raise SystemExit( - f"policy showcase run failed for {policy_id} with exit code {proc.returncode}; see {log_path}" - ) - if not json_path.exists(): - raise SystemExit( - f"policy showcase run for {policy_id} did not write the expected report JSON at {json_path}; see {log_path}" - ) - if not rrd_path.exists(): - raise SystemExit( - f"policy showcase run for {policy_id} did not write the expected Rerun recording at {rrd_path}; " - "build robowbc with --features robowbc-cli/sim,robowbc-cli/vis before running the showcase generator" - ) - - actual_transport = detect_transport(log_text) - actual_model_variant = detect_mujoco_model_variant(log_text) - if showcase_context["transport"] == "mujoco" and actual_transport != "mujoco": - raise SystemExit( - f"policy showcase run for {policy_id} did not activate MuJoCo transport; " - f"see {log_path} and build robowbc with sim + vis support before regenerating the showcase" - ) - - report = json.loads(json_path.read_text(encoding="utf-8")) - replay_trace = None - if replay_path.exists(): - replay_trace = json.loads(replay_path.read_text(encoding="utf-8")) - - report["_meta"] = { - "log_path": str(log_path), - "rrd_path": str(rrd_path), - "json_path": str(json_path), - "replay_trace_path": str(replay_path), - "replay_trace_present": replay_trace is not None, - "temp_config": str(temp_config), - "showcase_context": showcase_context, - } - if replay_trace is not None: - report["_replay_trace"] = replay_trace - - proof_pack_artifacts: dict[str, str] | None = None - proof_pack_manifest: dict[str, object] | None = None - proof_pack_result = generate_policy_proof_pack(repo_root, policy_dir, report, output_dir) - if proof_pack_result is not None: - proof_pack_artifacts, proof_pack_manifest = proof_pack_result - - report["card_id"] = policy_id - report.setdefault("policy_name", str(policy.get("policy_family", policy_id))) - report["status"] = "ok" - report["_meta"] = policy_meta( - policy, - output_dir, - showcase_context, - actual_transport=actual_transport, - actual_model_variant=actual_model_variant, - json_path=json_path, - rrd_path=rrd_path, - log_path=log_path, - proof_pack_artifacts=proof_pack_artifacts, - ) - if proof_pack_manifest is not None: - report["_proof_pack_manifest"] = proof_pack_manifest - return report - - -def series_from_frames(frames: list[dict[str, object]], field: str, joint_idx: int) -> list[float]: - values: list[float] = [] - for frame in frames: - data = frame.get(field, []) - if isinstance(data, list) and joint_idx < len(data): - values.append(float(data[joint_idx])) - return values - - -def yaw_from_rotation_xyzw(rotation_xyzw: list[float]) -> float: - x, y, z, w = [float(value) for value in rotation_xyzw] - siny_cosp = 2.0 * (w * z + x * y) - cosy_cosp = 1.0 - 2.0 * (y * y + z * z) - return math.atan2(siny_cosp, cosy_cosp) - - -def wrap_angle_rad(angle_rad: float) -> float: - wrapped = angle_rad - while wrapped > math.pi: - wrapped -= 2.0 * math.pi - while wrapped < -math.pi: - wrapped += 2.0 * math.pi - return wrapped - - -def derive_velocity_tracking_series( - frames: list[dict[str, object]], control_frequency_hz: int -) -> dict[str, list[float]] | None: - if control_frequency_hz <= 0 or len(frames) < 2: - return None - - dt_secs = 1.0 / control_frequency_hz - vx_cmd: list[float] = [] - vx_actual: list[float] = [] - yaw_cmd: list[float] = [] - yaw_actual: list[float] = [] - - for previous, current in zip(frames, frames[1:]): - command_data = previous.get("command_data") - previous_pose = previous.get("base_pose") - current_pose = current.get("base_pose") - if not ( - isinstance(command_data, list) - and len(command_data) >= 3 - and isinstance(previous_pose, dict) - and isinstance(current_pose, dict) - ): - continue - - previous_position = previous_pose.get("position_world") - previous_rotation = previous_pose.get("rotation_xyzw") - current_position = current_pose.get("position_world") - current_rotation = current_pose.get("rotation_xyzw") - if not ( - isinstance(previous_position, list) - and len(previous_position) >= 2 - and isinstance(previous_rotation, list) - and len(previous_rotation) == 4 - and isinstance(current_position, list) - and len(current_position) >= 2 - and isinstance(current_rotation, list) - and len(current_rotation) == 4 - ): - continue - - yaw_prev = yaw_from_rotation_xyzw(previous_rotation) - yaw_curr = yaw_from_rotation_xyzw(current_rotation) - dx_world = float(current_position[0]) - float(previous_position[0]) - dy_world = float(current_position[1]) - float(previous_position[1]) - cos_yaw = math.cos(yaw_prev) - sin_yaw = math.sin(yaw_prev) - dx_body = cos_yaw * dx_world + sin_yaw * dy_world - vx_cmd.append(float(command_data[0])) - vx_actual.append(dx_body / dt_secs) - yaw_cmd.append(float(command_data[2])) - yaw_actual.append(wrap_angle_rad(yaw_curr - yaw_prev) / dt_secs) - - if not vx_cmd: - return None - - return { - "vx_cmd": vx_cmd, - "vx_actual": vx_actual, - "yaw_cmd": yaw_cmd, - "yaw_actual": yaw_actual, - } - - -def derive_target_tracking_metrics( - frames: list[dict[str, object]], -) -> dict[str, float | int | None] | None: - joint_abs_errors: list[float] = [] - joint_sq_error_sum = 0.0 - matched_frame_count = 0 - base_heights_m: list[float] = [] - - for frame in frames: - actual_positions = frame.get("actual_positions") - target_positions = frame.get("target_positions") - if ( - isinstance(actual_positions, list) - and isinstance(target_positions, list) - and actual_positions - and len(actual_positions) == len(target_positions) - ): - matched_frame_count += 1 - for actual_position, target_position in zip(actual_positions, target_positions): - abs_error = abs(float(actual_position) - float(target_position)) - joint_abs_errors.append(abs_error) - joint_sq_error_sum += abs_error * abs_error - - base_pose = frame.get("base_pose") - if isinstance(base_pose, dict): - position_world = base_pose.get("position_world") - if isinstance(position_world, list) and len(position_world) >= 3: - base_heights_m.append(float(position_world[2])) - - if not joint_abs_errors: - return None - - joint_abs_errors_sorted = sorted(joint_abs_errors) - p95_index = max(0, math.ceil(0.95 * len(joint_abs_errors_sorted)) - 1) - joint_sample_count = len(joint_abs_errors) - mean_joint_abs_error_rad = sum(joint_abs_errors) / joint_sample_count - joint_rmse_rad = math.sqrt(joint_sq_error_sum / joint_sample_count) - base_height_min_m = min(base_heights_m) if base_heights_m else None - base_height_max_m = max(base_heights_m) if base_heights_m else None - - return { - "matched_frame_count": matched_frame_count, - "joint_sample_count": joint_sample_count, - "mean_joint_abs_error_rad": mean_joint_abs_error_rad, - "p95_joint_abs_error_rad": joint_abs_errors_sorted[p95_index], - "peak_joint_abs_error_rad": max(joint_abs_errors), - "joint_rmse_rad": joint_rmse_rad, - "base_height_min_m": base_height_min_m, - "base_height_max_m": base_height_max_m, - "frames_below_base_height_0_4m": sum(height < 0.4 for height in base_heights_m), - "frames_below_base_height_0_2m": sum(height < 0.2 for height in base_heights_m), - } - - -def classify_quality_verdict( - status: str, - command_kind: str, - metrics: dict[str, object] | None, -) -> dict[str, str] | None: - if status != "ok": - return None - if not isinstance(metrics, dict): - return {"label": "??", "css_class": "unknown", "summary": "runtime metrics unavailable"} - - dropped_frames = int(metrics.get("dropped_frames", 0) or 0) - achieved_frequency_hz = float(metrics.get("achieved_frequency_hz", 0.0) or 0.0) - target_tracking = metrics.get("target_tracking") - target_tracking_dict = target_tracking if isinstance(target_tracking, dict) else None - - if command_kind in {"velocity", "velocity_schedule"}: - velocity_tracking = metrics.get("velocity_tracking") - if not isinstance(velocity_tracking, dict): - return { - "label": "??", - "css_class": "unknown", - "summary": "velocity-tracking metrics unavailable", - } - - vx_rmse_mps = float(velocity_tracking.get("vx_rmse_mps", math.inf)) - yaw_rate_rmse_rad_s = float( - velocity_tracking.get("yaw_rate_rmse_rad_s", math.inf) - ) - forward_distance_m = float(velocity_tracking.get("forward_distance_m", 0.0)) - heading_change_deg = float(velocity_tracking.get("heading_change_deg", math.nan)) - collapse_frames = ( - int(target_tracking_dict.get("frames_below_base_height_0_4m", 0)) - if target_tracking_dict is not None - else 0 - ) - - if ( - dropped_frames <= 1 - and achieved_frequency_hz >= 47.0 - and vx_rmse_mps < 0.4 - and yaw_rate_rmse_rad_s < 1.5 - and forward_distance_m > 2.5 - and collapse_frames == 0 - ): - return { - "label": "GOOD", - "css_class": "good", - "summary": "meets the showcase velocity gates", - } - - bad_reasons: list[str] = [] - if dropped_frames > 5: - bad_reasons.append(f"dropped frames {dropped_frames} > 5") - if achieved_frequency_hz < 45.0: - bad_reasons.append(f"achieved rate {achieved_frequency_hz:.1f} Hz < 45") - if vx_rmse_mps >= 0.6: - bad_reasons.append(f"vx RMSE {vx_rmse_mps:.3f} >= 0.6") - if yaw_rate_rmse_rad_s >= 1.5: - bad_reasons.append(f"yaw RMSE {yaw_rate_rmse_rad_s:.3f} >= 1.5") - if forward_distance_m < 0.5: - bad_reasons.append(f"forward distance {forward_distance_m:.3f} m < 0.5") - if collapse_frames > 20: - bad_reasons.append(f"collapse frames {collapse_frames} > 20") - if bad_reasons: - return { - "label": "BAD", - "css_class": "bad", - "summary": "; ".join(bad_reasons[:3]), - } - - mixed_reasons: list[str] = [] - if vx_rmse_mps >= 0.4: - mixed_reasons.append(f"vx RMSE {vx_rmse_mps:.3f} above target") - if forward_distance_m <= 2.5: - mixed_reasons.append(f"forward distance {forward_distance_m:.3f} m below target") - if collapse_frames > 0: - mixed_reasons.append(f"collapse frames {collapse_frames} > 0") - - return { - "label": "??", - "css_class": "unknown", - "summary": "; ".join(mixed_reasons[:3]) or "mixed velocity metrics", - } - - if target_tracking_dict is None: - return { - "label": "??", - "css_class": "unknown", - "summary": "joint-tracking metrics unavailable", - } - - mean_joint_abs_error_rad = float(target_tracking_dict["mean_joint_abs_error_rad"]) - p95_joint_abs_error_rad = float(target_tracking_dict["p95_joint_abs_error_rad"]) - base_height_min_m = target_tracking_dict.get("base_height_min_m") - frames_below_base_height_0_4m = int( - target_tracking_dict["frames_below_base_height_0_4m"] - ) - frames_below_base_height_0_2m = int( - target_tracking_dict["frames_below_base_height_0_2m"] - ) - - bad_reasons = [] - if mean_joint_abs_error_rad > 0.35: - bad_reasons.append(f"mean joint error {mean_joint_abs_error_rad:.3f} rad > 0.35") - if p95_joint_abs_error_rad > 1.0: - bad_reasons.append(f"joint error p95 {p95_joint_abs_error_rad:.3f} rad > 1.0") - if frames_below_base_height_0_4m > 20: - bad_reasons.append( - f"collapse frames {frames_below_base_height_0_4m} > 20" - ) - if frames_below_base_height_0_2m > 5: - bad_reasons.append( - f"deep-collapse frames {frames_below_base_height_0_2m} > 5" - ) - if ( - base_height_min_m is not None - and float(base_height_min_m) < 0.4 - ): - bad_reasons.append(f"min base height {float(base_height_min_m):.3f} m < 0.4") - if dropped_frames > 5: - bad_reasons.append(f"dropped frames {dropped_frames} > 5") - if achieved_frequency_hz < 45.0: - bad_reasons.append(f"achieved rate {achieved_frequency_hz:.1f} Hz < 45") - - if bad_reasons: - return { - "label": "BAD", - "css_class": "bad", - "summary": "; ".join(bad_reasons[:3]), - } - - if ( - mean_joint_abs_error_rad <= 0.15 - and p95_joint_abs_error_rad <= 0.45 - and frames_below_base_height_0_4m == 0 - and frames_below_base_height_0_2m == 0 - and (base_height_min_m is None or float(base_height_min_m) >= 0.6) - and dropped_frames == 0 - and achieved_frequency_hz >= 47.0 - ): - return { - "label": "GOOD", - "css_class": "good", - "summary": "stable run with tight joint-target tracking", - } - - return { - "label": "??", - "css_class": "unknown", - "summary": "stable run, but only generic tracking heuristics are available", - } - - -def spark_svg(series_list: list[dict[str, object]], width: int = 360, height: int = 140) -> str: - if not series_list: - return '' - - all_values = [value for series in series_list for value in series["values"]] - if not all_values: - return '' - - min_v = min(all_values) - max_v = max(all_values) - if math.isclose(min_v, max_v): - min_v -= 1.0 - max_v += 1.0 - - pad = 12 - inner_w = width - 2 * pad - inner_h = height - 2 * pad - - def point_x(idx: int, total: int) -> float: - if total <= 1: - return pad + inner_w / 2 - return pad + inner_w * idx / (total - 1) - - def point_y(val: float) -> float: - return pad + inner_h * (1.0 - (val - min_v) / (max_v - min_v)) - - paths = [] - for series in series_list: - values = series["values"] - pts = " ".join( - f"{point_x(idx, len(values)):.2f},{point_y(val):.2f}" - for idx, val in enumerate(values) - ) - dash = ' stroke-dasharray="5 4"' if series.get("dashed") else "" - paths.append( - f'' - ) - - baseline = point_y(0.0) - return ( - f'' - f'' - f'' - + "".join(paths) - + "" - ) - - -def format_vector(values: Iterable[float], limit: int = 6) -> str: - items = list(values) - head = ", ".join(f"{value:.3f}" for value in items[:limit]) - if len(items) > limit: - head += ", ..." - return f"[{head}]" - - -def pill(label: str, css_class: str) -> str: - return f'{html.escape(label)}' - - -def showcase_transport_badge_label(transport: str) -> str: - if transport == "mujoco": - return "MUJOCO SIM" - if transport == "hardware": - return "HARDWARE" - return transport.upper() - - -def showcase_transport_text(transport: str) -> str: - if transport == "mujoco": - return "MuJoCo sim" - if transport == "hardware": - return "Hardware" - return "Synthetic fallback" - - -def showcase_model_variant_text(variant: str | None) -> str: - if variant == "meshless-public-mjcf": - return "Meshless public MJCF" - if variant == "upstream-mjcf": - return "Upstream MJCF" - return variant or "-" - - -def display_sort_key(index: int, entry: dict[str, object]) -> tuple[int, int, int]: - meta = entry["_meta"] - status = entry.get("status", "ok") - execution_kind = str(meta["execution_kind"]) - card_id = entry_card_id(entry) - return ( - 0 if status == "ok" else 1, - 0 if execution_kind == "real" else 1, - DISPLAY_ORDER.get(card_id, index), - ) - - -def entry_card_id(entry: dict[str, object]) -> str: - explicit = entry.get("card_id") - if explicit: - return str(explicit) - - meta = entry.get("_meta") - if isinstance(meta, dict): - for key in ("card_id", "json_file", "rrd_file"): - value = meta.get(key) - if value: - return Path(str(value)).stem - - return str(entry.get("policy_name") or "") - - -def entry_policy_family(entry: dict[str, object]) -> str: - explicit = entry.get("policy_name") - if explicit: - return str(explicit) - - meta = entry.get("_meta") - if isinstance(meta, dict) and meta.get("policy_family"): - return str(meta["policy_family"]) - - return entry_card_id(entry) - - -def entry_identity_label(entry: dict[str, object]) -> str: - card_id = entry_card_id(entry) - policy_family = entry_policy_family(entry) - if policy_family == card_id: - return card_id - return f"{card_id} · family {policy_family}" - - - -def render_demo_section(title: str, cards: list[str]) -> str: - if not cards: - return "" - - description = DEMO_FAMILY_DESCRIPTIONS.get(title, "") - return f'''
-
-

{html.escape(title)}

-

{html.escape(description)}

-
-
- {''.join(cards)} -
-
''' - - -def relative_href(from_path: Path, target_path: Path) -> str: - return os.path.relpath(target_path, start=from_path.parent).replace(os.sep, "/") - -def rebase_meta_artifact_paths( - meta: dict[str, object], - from_page: Path, - output_dir: Path, -) -> dict[str, object]: - rebased = dict(meta) - for key in ( - "json_file", - "rrd_file", - "log_file", - "proof_pack_manifest_file", - ): - value = meta.get(key) - if isinstance(value, str) and value: - rebased[key] = relative_href(from_page, output_dir / value) - return rebased - - -def render_overview_links(meta: dict[str, object], detail_href: str) -> str: - links = [f'Detail page'] - if meta.get("proof_pack_manifest_file"): - links.append(f'Visual checkpoints') - for key, label in ( - ("rrd_file", "Rerun"), - ("json_file", "JSON"), - ("log_file", "log"), - ("proof_pack_manifest_file", "Proof-pack manifest"), - ): - value = meta.get(key) - if isinstance(value, str) and value: - links.append(f'{html.escape(label)}') - return "
".join(links) - - -def discover_benchmark_pages(output_dir: Path, index_path: Path) -> list[dict[str, str]]: - benchmark_specs = [ - ( - output_dir / "benchmarks" / "nvidia" / "index.html", - "NVIDIA Comparison", - "RoboWBC-vs-official benchmark matrix built from the normalized NVIDIA artifacts.", - ) - ] - pages: list[dict[str, str]] = [] - for page_path, title, summary in benchmark_specs: - if page_path.is_file(): - pages.append( - { - "title": title, - "summary": summary, - "href": relative_href(index_path, page_path), - } - ) - return pages - - -def render_benchmark_section(pages: list[dict[str, str]]) -> str: - if not pages: - return "" - - cards = [] - for page in pages: - cards.append( - f'''''' - ) - - return f'''
-
-

Benchmarks

-

Normalized benchmark packages that sit beside the policy pages in the same site bundle.

-
-
- {''.join(cards)} -
-
''' - - -def render_generic_proof_pack_section(proof_pack_manifest: dict[str, object]) -> str: - checkpoints = proof_pack_manifest.get("checkpoints") - if not isinstance(checkpoints, list) or not checkpoints: - capture_warning = proof_pack_manifest.get("capture_warning") - if isinstance(capture_warning, str) and capture_warning: - capture_backend = proof_pack_manifest.get("capture_backend") - backend_html = ( - f'

Configured offscreen backend: {html.escape(str(capture_backend))}.

' - if capture_backend - else "" - ) - return f'''
-

Visual Checkpoints

-
-

Screenshots unavailable for this build.

-

{html.escape(capture_warning)}

- {backend_html} -

The raw run report, replay trace, and Rerun recording are still published above so the result remains reviewable.

-
-
''' - return "" - - checkpoint_cards: list[str] = [] - for checkpoint in checkpoints: - if not isinstance(checkpoint, dict): - continue - relative_dir = checkpoint.get("relative_dir") - cameras = checkpoint.get("cameras") - if not isinstance(relative_dir, str) or not isinstance(cameras, list): - continue - - camera_cards = [] - for camera in cameras: - if not isinstance(camera, str): - continue - image_href = f"{relative_dir}/{camera}_rgb.png" - camera_cards.append( - f'''
- {html.escape(str(checkpoint.get( -
{html.escape(camera)}
-
''' - ) - - if not camera_cards: - continue - - tick = checkpoint.get("tick", "-") - sim_time_secs = checkpoint.get("sim_time_secs") - sim_time_text = ( - f"{float(sim_time_secs):.2f} s" - if isinstance(sim_time_secs, (int, float)) - else "n/a" - ) - selection_reason = str(checkpoint.get("selection_reason", "")) - checkpoint_cards.append( - f'''
-
-
-

{html.escape(str(checkpoint.get("name", "checkpoint")))}

-

{html.escape(selection_reason)}

-
-
- {html.escape(f"tick {tick}")} - {html.escape(sim_time_text)} -
-
-
- {''.join(camera_cards)} -
-
''' - ) - - if not checkpoint_cards: - return "" - - return f'''
-

Visual checkpoints

-

Each image overlays the target pose in blue against the actual replayed pose in orange. The checkpoints are selected from the replay trace so you can cross-check startup, motion onset, peak latency, furthest progress, and final state without opening a second report.

-
- {''.join(checkpoint_cards)} -
-
''' - - -def render_proof_pack_section(proof_pack_manifest: dict[str, object] | None) -> str: - if not isinstance(proof_pack_manifest, dict): - return "" - - phase_review = proof_pack_manifest.get("phase_review") - if not (isinstance(phase_review, dict) and phase_review.get("enabled") is True): - return render_generic_proof_pack_section(proof_pack_manifest) - - phase_timeline = proof_pack_manifest.get("phase_timeline") - phase_checkpoints = proof_pack_manifest.get("phase_checkpoints") - diagnostic_checkpoints = proof_pack_manifest.get("diagnostic_checkpoints") - lag_options = proof_pack_manifest.get("lag_options") - default_lag_ticks = proof_pack_manifest.get("default_lag_ticks") - default_lag_ms = proof_pack_manifest.get("default_lag_ms") - target_lag_options = proof_pack_manifest.get("target_lag_options") - default_target_lag_ticks = proof_pack_manifest.get("default_target_lag_ticks") - default_target_lag_ms = proof_pack_manifest.get("default_target_lag_ms") - if not isinstance(phase_timeline, list) or not isinstance(phase_checkpoints, list): - raise SystemExit("phase-aware proof-pack manifest is missing phase timeline data") - if not isinstance(diagnostic_checkpoints, list): - diagnostic_checkpoints = [] - if not isinstance(lag_options, list) or not all(isinstance(item, int) for item in lag_options): - raise SystemExit("phase-aware proof-pack manifest is missing integer lag_options") - if not isinstance(default_lag_ticks, int): - raise SystemExit("phase-aware proof-pack manifest is missing default_lag_ticks") - if not isinstance(target_lag_options, list) or not all( - isinstance(item, int) for item in target_lag_options - ): - target_lag_options = [0] - if not isinstance(default_target_lag_ticks, int): - default_target_lag_ticks = 0 - - checkpoint_map: dict[str, dict[str, dict[str, object]]] = {} - for checkpoint in phase_checkpoints: - if not isinstance(checkpoint, dict): - continue - phase_name = checkpoint.get("phase_name") - phase_kind = checkpoint.get("phase_kind") - if not isinstance(phase_name, str) or not isinstance(phase_kind, str): - continue - checkpoint_map.setdefault(phase_name, {})[phase_kind] = checkpoint - - def debug_variant_summary(variant: dict[str, object]) -> dict[str, object]: - return { - "lag_ticks": int(variant.get("lag_ticks", 0) or 0), - "lag_ms": float(variant.get("lag_ms", 0.0) or 0.0), - "tick": int(variant.get("tick", 0) or 0), - "frame_index": int(variant.get("frame_index", 0) or 0), - "sim_time_secs": float(variant.get("sim_time_secs", 0.0) or 0.0), - "relative_dir": str(variant.get("relative_dir", "")), - "frame_source": str(variant.get("frame_source", "")), - "selection_reason": str(variant.get("selection_reason", "")), - "cameras": [str(camera) for camera in variant.get("cameras", []) if isinstance(camera, str)], - } - - def render_lag_buttons(lags: list[int], selected_lag: int) -> str: - return "".join( - f'' - for lag in lags - ) - - timeline_cards: list[str] = [] - phase_cards: list[str] = [] - for entry in phase_timeline: - if not isinstance(entry, dict): - continue - phase_name = entry.get("phase_name") - if not isinstance(phase_name, str): - continue - midpoint_checkpoint = checkpoint_map.get(phase_name, {}).get("midpoint") - phase_end_checkpoint = checkpoint_map.get(phase_name, {}).get("phase_end") - if not isinstance(midpoint_checkpoint, dict) or not isinstance(phase_end_checkpoint, dict): - raise SystemExit(f"phase-aware manifest is missing midpoint/end checkpoints for {phase_name}") - - start_tick = int(entry.get("start_tick", 0) or 0) - midpoint_tick = int(entry.get("midpoint_tick", 0) or 0) - end_tick = int(entry.get("end_tick", 0) or 0) - duration_ticks = int(entry.get("duration_ticks", 0) or 0) - duration_secs = float(entry.get("duration_secs", 0.0) or 0.0) - timeline_cards.append( - f'''
-

{html.escape(phase_name)}

-

ticks {start_tick}–{end_tick} · midpoint {midpoint_tick} · {duration_ticks} ticks / {duration_secs:.2f} s

-
''' - ) - - midpoint_relative_dir = midpoint_checkpoint.get("relative_dir") - midpoint_cameras = midpoint_checkpoint.get("cameras") - if not isinstance(midpoint_relative_dir, str) or not isinstance(midpoint_cameras, list): - raise SystemExit(f"midpoint checkpoint for {phase_name} is malformed") - midpoint_views = "".join( - f'''
- {html.escape(phase_name)} midpoint {html.escape(camera)} overlay -
{html.escape(camera)}
-
''' - for camera in midpoint_cameras - if isinstance(camera, str) - ) - - lag_variants = phase_end_checkpoint.get("lag_variants") - if not isinstance(lag_variants, list) or not lag_variants: - raise SystemExit(f"phase-end checkpoint for {phase_name} is missing lag_variants") - lag_variants_by_tick = { - int(variant["lag_ticks"]): variant - for variant in lag_variants - if isinstance(variant, dict) - and isinstance(variant.get("lag_ticks"), int) - and isinstance(variant.get("relative_dir"), str) - } - available_lags = sorted(lag_variants_by_tick) - if not available_lags: - raise SystemExit(f"phase-end checkpoint for {phase_name} has no usable lag variants") - display_lag = default_lag_ticks if default_lag_ticks in lag_variants_by_tick else available_lags[-1] - default_variant = lag_variants_by_tick[display_lag] - default_variant_dir = str(default_variant["relative_dir"]) - default_variant_ms = float(default_variant.get("lag_ms", 0.0) or 0.0) - target_phase_lag_options = phase_end_checkpoint.get("target_lag_options") - if not isinstance(target_phase_lag_options, list) or not all( - isinstance(item, int) for item in target_phase_lag_options - ): - target_phase_lag_options = [0] - target_lag_variants = phase_end_checkpoint.get("target_lag_variants") - target_variant_fallback_dir = str(default_variant["relative_dir"]) - target_variants_by_tick: dict[int, dict[str, object]] = {} - if isinstance(target_lag_variants, list) and target_lag_variants: - target_variants_by_tick = { - int(variant["lag_ticks"]): variant - for variant in target_lag_variants - if isinstance(variant, dict) - and isinstance(variant.get("lag_ticks"), int) - and isinstance(variant.get("relative_dir"), str) - } - if not target_variants_by_tick: - target_variants_by_tick = { - 0: { - "lag_ticks": 0, - "lag_ms": 0.0, - "tick": phase_end_checkpoint.get("phase_end_tick", end_tick), - "frame_index": phase_end_checkpoint.get("frame_index", end_tick), - "sim_time_secs": phase_end_checkpoint.get("sim_time_secs", 0.0), - "selection_reason": f"{phase_name} target pose at canonical phase end", - "frame_source": phase_end_checkpoint.get("frame_source", "canonical_replay_trace"), - "relative_dir": target_variant_fallback_dir, - "cameras": default_variant.get("cameras", []), - "_raw_suffix": "_target_rgb.png", - } - } - available_target_lags = sorted(target_variants_by_tick) - display_target_lag = ( - default_target_lag_ticks - if default_target_lag_ticks in target_variants_by_tick - else available_target_lags[-1] - ) - default_target_variant = target_variants_by_tick[display_target_lag] - default_target_variant_ms = float(default_target_variant.get("lag_ms", 0.0) or 0.0) - default_actual_tick = int(default_variant.get("tick", end_tick) or end_tick) - default_actual_frame_index = int(default_variant.get("frame_index", end_tick) or end_tick) - default_target_tick = int(default_target_variant.get("tick", end_tick) or end_tick) - default_target_frame_index = int( - default_target_variant.get("frame_index", end_tick) or end_tick - ) - end_cameras = default_variant.get("cameras") - if not isinstance(end_cameras, list): - raise SystemExit(f"phase-end checkpoint for {phase_name} is missing camera data") - phase_end_views = [] - for camera in end_cameras: - if not isinstance(camera, str): - continue - overlay_lag_map = { - str(lag): f"{str(variant['relative_dir'])}/{camera}_rgb.png" - for lag, variant in lag_variants_by_tick.items() - if isinstance(variant.get("cameras"), list) and camera in variant["cameras"] - } - actual_lag_map = { - str(lag): f"{str(variant['relative_dir'])}/{camera}_actual_rgb.png" - for lag, variant in lag_variants_by_tick.items() - if isinstance(variant.get("cameras"), list) and camera in variant["cameras"] - } - actual_lag_ms_map = { - str(lag): float(variant.get("lag_ms", 0.0) or 0.0) - for lag, variant in lag_variants_by_tick.items() - if isinstance(variant.get("cameras"), list) and camera in variant["cameras"] - } - target_lag_map = {} - target_lag_ms_map = {} - for lag, variant in target_variants_by_tick.items(): - cameras = variant.get("cameras") - if not isinstance(cameras, list) or camera not in cameras: - continue - suffix = str(variant.get("_raw_suffix") or "_rgb.png") - target_lag_map[str(lag)] = f"{str(variant['relative_dir'])}/{camera}{suffix}" - target_lag_ms_map[str(lag)] = float(variant.get("lag_ms", 0.0) or 0.0) - phase_end_views.append( - f'''
- {html.escape(phase_name)} phase-end {html.escape(camera)} overlay -
{html.escape(camera)} · T+{display_target_lag} ({default_target_variant_ms:.0f} ms) · A+{display_lag} ({default_variant_ms:.0f} ms)
-
''' - ) - - phase_end_anchor = debug_variant_summary(phase_end_checkpoint) - phase_end_anchor["phase_end_tick"] = int( - phase_end_checkpoint.get("phase_end_tick", end_tick) or end_tick - ) - phase_debug_payload = { - "phase_name": phase_name, - "timeline": { - "start_tick": start_tick, - "midpoint_tick": midpoint_tick, - "end_tick": end_tick, - "duration_ticks": duration_ticks, - "duration_secs": duration_secs, - }, - "midpoint": debug_variant_summary(midpoint_checkpoint), - "phase_end_anchor": phase_end_anchor, - "default_review": { - "target_lag_ticks": display_target_lag, - "target_lag_ms": default_target_variant_ms, - "target_tick": default_target_tick, - "target_frame_index": default_target_frame_index, - "target_relative_dir": str(default_target_variant.get("relative_dir", "")), - "actual_lag_ticks": display_lag, - "actual_lag_ms": default_variant_ms, - "actual_tick": default_actual_tick, - "actual_frame_index": default_actual_frame_index, - "actual_relative_dir": default_variant_dir, - }, - "actual_variants": [ - debug_variant_summary(lag_variants_by_tick[lag]) for lag in available_lags - ], - "target_variants": [ - debug_variant_summary(target_variants_by_tick[lag]) - for lag in available_target_lags - ], - } - phase_debug_json = html.escape( - json.dumps(phase_debug_payload, indent=2, sort_keys=True) - ) - - phase_cards.append( - f'''
-
-
-

{html.escape(phase_name)}

-

Midpoint capture at tick {midpoint_tick}; phase-end review anchored at tick {end_tick}.

-
-
- {html.escape(f"start {start_tick}")} - {html.escape(f"end {end_tick}")} -
-
-
-
-

Midpoint

-
- {midpoint_views} -
-
-
-

Phase End

-
- {''.join(phase_end_views)} -
-
-
-
- Debug metadata -

Static tick and asset-path contract for manual debugging without driving the browser controls.

- 
{phase_debug_json}
-
-
''' - ) - - diagnostic_cards: list[str] = [] - for checkpoint in diagnostic_checkpoints: - if not isinstance(checkpoint, dict): - continue - relative_dir = checkpoint.get("relative_dir") - cameras = checkpoint.get("cameras") - if not isinstance(relative_dir, str) or not isinstance(cameras, list): - continue - diagnostic_cards.append( - f'''
-
-
-

{html.escape(str(checkpoint.get("name", "diagnostic")))}

-

{html.escape(str(checkpoint.get("selection_reason", "")))}

-
-
- {html.escape(f"tick {checkpoint.get('tick', '-')}")} -
-
-
- {''.join( - f'
{html.escape(str(checkpoint.get(
{html.escape(camera)}
' - for camera in cameras - if isinstance(camera, str) - )} -
-
''' - ) - - lag_button_html = render_lag_buttons(lag_options, default_lag_ticks) - target_lag_button_html = render_lag_buttons( - target_lag_options, default_target_lag_ticks - ) - default_lag_ms_text = ( - f"{float(default_lag_ms):.0f} ms" - if isinstance(default_lag_ms, (int, float)) - else "n/a" - ) - default_target_lag_ms_text = ( - f"{float(default_target_lag_ms):.0f} ms" - if isinstance(default_target_lag_ms, (int, float)) - else "n/a" - ) - - diagnostics_html = ( - f'''
-

Diagnostics

-

Generic evidence checkpoints stay available as secondary diagnostics instead of the primary story.

-
- {''.join(diagnostic_cards)} -
-
''' - if diagnostic_cards - else "" - ) - - return f'''
-
-
-

Phase review

-

The proof pack follows the authored phase timeline directly, so the staged locomotion story reads as stand → accelerate → turn → run → settle instead of generic checkpoint archaeology.

-
-
- {html.escape(f"default target +{default_target_lag_ticks}")} - {html.escape(default_target_lag_ms_text)} - {html.escape(f"default actual +{default_lag_ticks}")} - {html.escape(default_lag_ms_text)} -
-
-
- {''.join(timeline_cards)} -
-
-
- Target timestamp selector -
- {target_lag_button_html} -
-
-
- Actual / robot timestamp selector -
- {lag_button_html} -
-
-
-
- {''.join(phase_cards)} -
-
-{diagnostics_html}''' - - -def render_policy_link_card( - entry: dict[str, object], - detail_href: str, - quality_html: str, -) -> str: - meta = dict(entry["_meta"]) - status = str(entry.get("status", "ok")) - metrics = entry.get("metrics") or {} - badge_bits = [ - quality_html if status == "ok" else pill("BLOCKED", "blocked"), - pill(str(meta["execution_kind"]).upper(), str(meta["execution_kind"])), - pill(showcase_transport_badge_label(str(meta.get("showcase_transport", "synthetic"))), "transport"), - pill(str(entry.get("command_kind", "")).upper(), "command"), - ] - metric_line = ( - f'{metrics.get("ticks", "-")} ticks · ' - f'{float(metrics.get("average_inference_ms", 0.0)):.3f} ms avg · ' - f'{float(metrics.get("achieved_frequency_hz", 0.0)):.2f} Hz' - if status == "ok" and isinstance(metrics, dict) - else f'{html.escape(str(meta.get("blocked_reason", "Blocked")))}' - "" - ) - links = render_overview_links(meta, detail_href) - return f'''
-
-
-

{html.escape(str(meta["title"]))}

-

{html.escape(str(meta["source"]))} · {html.escape(str(meta["coverage"]))}

-

{html.escape(entry_identity_label(entry))}

-
-
{" ".join(badge_bits)}
-
-

{html.escape(str(meta["summary"]))}

-
- {metric_line} - {html.escape(str(meta["demo_family"]))} -
-

{links}

-
''' - - -def showcase_styles() -> str: - return """\ - :root { - color-scheme: light; - --bg: #f5f7fb; - --panel: #ffffff; - --text: #142033; - --muted: #5f6f85; - --border: #d9e0ea; - --shadow: 0 18px 50px rgba(20, 32, 51, 0.08); - --real-bg: #e7f7ef; - --real-fg: #11643a; - --experimental-bg: #fff4e5; - --experimental-fg: #9a3412; - --fixture-bg: #e7f0ff; - --fixture-fg: #1146a6; - --blocked-bg: #fff1f2; - --blocked-fg: #b42318; - --command-bg: #fff6db; - --command-fg: #8a5b00; - --meta-bg: #edf2f7; - --meta-fg: #334155; - --transport-bg: #e8f1ff; - --transport-fg: #1d4ed8; - --good-bg: #e7f7ef; - --good-fg: #11643a; - --bad-bg: #fff1f2; - --bad-fg: #b42318; - --unknown-bg: #fff6db; - --unknown-fg: #8a5b00; - } - * { box-sizing: border-box; } - body { margin: 0; font-family: "IBM Plex Sans", "Segoe UI", sans-serif; background: radial-gradient(circle at top, #eef7ff, var(--bg) 45%); color: var(--text); } - main { width: min(1180px, calc(100% - 32px)); margin: 0 auto; padding: 40px 0 64px; } - h1, h2, h3, p { margin-top: 0; } - a { color: #0f5bd3; } - .hero { background: linear-gradient(135deg, #ffffff, #ecf4ff); border: 1px solid var(--border); border-radius: 28px; box-shadow: var(--shadow); padding: 32px; margin-bottom: 28px; } - .hero p { max-width: 80ch; line-height: 1.6; } - .meta-row { display: flex; gap: 16px; flex-wrap: wrap; color: var(--muted); font-size: 0.95rem; } - .breadcrumbs { margin-bottom: 12px; font-weight: 700; } - .overview, .footer-panel { background: var(--panel); border: 1px solid var(--border); border-radius: 24px; box-shadow: var(--shadow); padding: 24px; margin-bottom: 28px; } - .demo-section { margin-bottom: 28px; } - .section-header { margin-bottom: 16px; } - .section-header p { max-width: 80ch; } - table { width: 100%; border-collapse: collapse; } - th, td { text-align: left; padding: 12px 10px; border-bottom: 1px solid var(--border); vertical-align: top; } - th { font-size: 0.82rem; text-transform: uppercase; letter-spacing: 0.06em; color: var(--muted); } - .cards { display: grid; gap: 20px; } - .card { background: var(--panel); border: 1px solid var(--border); border-radius: 24px; box-shadow: var(--shadow); padding: 24px; } - .policy-link-card { display: block; text-decoration: none; background: var(--panel); border: 1px solid var(--border); border-radius: 24px; box-shadow: var(--shadow); padding: 24px; color: inherit; } - .policy-link-card:hover { border-color: #b9cae3; box-shadow: 0 20px 55px rgba(20, 32, 51, 0.12); } - .policy-link-header { display: flex; align-items: flex-start; justify-content: space-between; gap: 16px; } - .policy-link-meta { display: flex; gap: 16px; flex-wrap: wrap; color: var(--muted); margin-bottom: 14px; } - .blocked-card { border-color: #f5c2c7; } - .card-header { display: flex; align-items: flex-start; justify-content: space-between; gap: 16px; } - .badge-row { display: flex; flex-wrap: wrap; gap: 8px; justify-content: flex-end; } - .pill { border-radius: 999px; padding: 8px 12px; font-size: 0.82rem; font-weight: 700; text-transform: uppercase; letter-spacing: 0.04em; display: inline-flex; align-items: center; } - .pill.real { background: var(--real-bg); color: var(--real-fg); } - .pill.experimental { background: var(--experimental-bg); color: var(--experimental-fg); } - .pill.fixture { background: var(--fixture-bg); color: var(--fixture-fg); } - .pill.blocked { background: var(--blocked-bg); color: var(--blocked-fg); } - .pill.ok { background: var(--real-bg); color: var(--real-fg); } - .pill.good { background: var(--good-bg); color: var(--good-fg); } - .pill.bad { background: var(--bad-bg); color: var(--bad-fg); } - .pill.unknown { background: var(--unknown-bg); color: var(--unknown-fg); } - .pill.command { background: var(--command-bg); color: var(--command-fg); } - .pill.meta { background: var(--meta-bg); color: var(--meta-fg); text-transform: none; } - .pill.transport { background: var(--transport-bg); color: var(--transport-fg); } - .muted { color: var(--muted); } - .stats { display: grid; grid-template-columns: repeat(auto-fit, minmax(160px, 1fr)); gap: 12px; margin: 16px 0 20px; } - .stats div, .details-grid div { background: #f7f9fc; border: 1px solid var(--border); border-radius: 16px; padding: 12px 14px; } - .stats span, .details-grid span, .blocked-paths span { display: block; color: var(--muted); font-size: 0.82rem; margin-bottom: 6px; text-transform: uppercase; letter-spacing: 0.05em; } - .stats strong { font-size: 1.05rem; } - .charts-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(240px, 1fr)); gap: 16px; margin-bottom: 18px; } - figure { margin: 0; } - figcaption { margin-bottom: 10px; font-weight: 700; } - .chart { width: 100%; height: auto; display: block; } - .chart rect { fill: #fbfdff; stroke: var(--border); } - .chart .baseline { stroke: #d4dae3; stroke-width: 1; } - .details-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(220px, 1fr)); gap: 12px; } - .rerun-block { margin: 0 0 18px; } - .rerun-block-header { display: flex; justify-content: space-between; gap: 12px; align-items: baseline; margin-bottom: 10px; flex-wrap: wrap; } - .rerun-stage { min-height: 420px; border-radius: 18px; border: 1px solid var(--border); background: linear-gradient(180deg, #0f172a, #111827); overflow: hidden; position: relative; } - .rerun-stage canvas { display: block; width: 100%; height: 420px; } - .rerun-stage-placeholder, .rerun-stage-error { position: absolute; inset: 0; display: flex; flex-direction: column; align-items: center; justify-content: center; gap: 8px; padding: 20px; text-align: center; color: #e5edf8; font-size: 0.95rem; } - .rerun-stage-placeholder strong, .rerun-stage-error strong { color: #ffffff; } - .rerun-stage-error { background: linear-gradient(180deg, rgba(127, 29, 29, 0.95), rgba(69, 10, 10, 0.96)); } - .blocked-reason, .blocked-paths { margin-top: 18px; background: #fff7f7; border: 1px solid #f5c2c7; border-radius: 16px; padding: 14px; } - code { font-family: "IBM Plex Mono", "SFMono-Regular", monospace; font-size: 0.9rem; word-break: break-word; } - .links { margin-top: 16px; line-height: 1.35; } - .proof-checkpoint-grid { display: grid; gap: 18px; } - .proof-checkpoint-card { border: 1px solid var(--border); border-radius: 20px; padding: 18px; background: #fbfdff; } - .proof-checkpoint-head { display: flex; justify-content: space-between; gap: 12px; align-items: flex-start; flex-wrap: wrap; margin-bottom: 14px; } - .proof-checkpoint-head h3 { margin-bottom: 6px; } - .proof-checkpoint-meta { display: flex; flex-wrap: wrap; gap: 10px; color: var(--muted); font-size: 0.92rem; } - .proof-view-grid { display: grid; gap: 14px; grid-template-columns: repeat(auto-fit, minmax(220px, 1fr)); } - .proof-view { margin: 0; } - .proof-view img { width: 100%; height: auto; display: block; border-radius: 14px; border: 1px solid var(--border); background: #f8fafc; } - .proof-view figcaption { margin-top: 8px; color: var(--muted); font-weight: 600; text-transform: uppercase; letter-spacing: 0.04em; font-size: 0.78rem; } - .phase-review-header { display: flex; justify-content: space-between; align-items: flex-start; gap: 16px; flex-wrap: wrap; margin-bottom: 18px; } - .phase-timeline-grid { display: grid; gap: 12px; grid-template-columns: repeat(auto-fit, minmax(170px, 1fr)); margin-bottom: 18px; } - .phase-timeline-card { border: 1px solid var(--border); border-radius: 18px; padding: 14px 16px; background: linear-gradient(180deg, #fbfdff, #f2f7ff); } - .phase-timeline-card h3 { margin-bottom: 6px; } - .phase-lag-controls { display: grid; gap: 12px; margin-bottom: 18px; } - .phase-lag-selector { display: flex; justify-content: space-between; align-items: center; gap: 14px; flex-wrap: wrap; padding: 14px 16px; border: 1px solid var(--border); border-radius: 18px; background: #f7f9fc; margin-bottom: 0; } - .phase-lag-buttons { display: flex; gap: 10px; flex-wrap: wrap; } - .phase-lag-button { border: 1px solid var(--border); background: white; color: #0f172a; border-radius: 999px; padding: 8px 12px; font: inherit; font-weight: 700; cursor: pointer; } - .phase-lag-button[data-active="true"] { background: #0f766e; border-color: #0f766e; color: white; } - .phase-checkpoint-stack { display: grid; gap: 18px; } - .phase-checkpoint-stack h4 { margin-bottom: 10px; font-size: 0.96rem; letter-spacing: 0.02em; } - .phase-debug-panel { margin-top: 16px; border: 1px solid var(--border); border-radius: 16px; background: #ffffff; padding: 12px 14px; } - .phase-debug-panel summary { cursor: pointer; font-weight: 700; } - .phase-debug-panel p { margin: 10px 0 0; } - .phase-debug-panel pre { margin: 10px 0 0; padding: 12px; border-radius: 12px; background: #0f172a; color: #e2e8f0; overflow-x: auto; font-size: 0.82rem; line-height: 1.45; } - .diagnostics-section { margin-top: 18px; } - .diagnostic-grid .diagnostic-card { background: #ffffff; border: 1px solid var(--border); border-radius: 18px; padding: 16px; } - ul { margin-bottom: 0; } - @media (max-width: 720px) { - main { width: min(100% - 20px, 1180px); padding-top: 20px; } - .hero, .overview, .card, .footer-panel, .policy-link-card { padding: 20px; border-radius: 20px; } - .card-header, .policy-link-header { flex-direction: column; } - .badge-row { justify-content: flex-start; } - } - """ - - -def viewer_loader_script(viewer_module_path: str) -> str: - return f"""""" - - -def render_policy_detail_page( - page_title: str, - page_summary: str, - body_html: str, - back_href: str, - generated_at: str, - commit_html: str, - run_html: str, - viewer_module_path: str, -) -> str: - return f''' - - - - - {html.escape(page_title)} · RoboWBC Site - - - -
-
-

← Back to site home

-

{html.escape(page_title)}

-

{html.escape(page_summary)}

-
- Generated: {html.escape(generated_at)} - Commit: {commit_html or 'local'} - {run_html} -
-
- {body_html} -
- {viewer_loader_script(viewer_module_path)} - -''' - - -def render_html(entries: list[dict[str, object]], output_dir: Path, repo_root: Path) -> None: - generated_at = dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%d %H:%M:%SZ") - sha = os.environ.get("GITHUB_SHA", "") - repo = os.environ.get("GITHUB_REPOSITORY", "") - server = os.environ.get("GITHUB_SERVER_URL", "https://github.com") - run_id = os.environ.get("GITHUB_RUN_ID", "") - commit_link = f"{server}/{repo}/commit/{sha}" if sha and repo else "" - run_link = f"{server}/{repo}/actions/runs/{run_id}" if run_id and repo else "" - vendor_rerun_web_viewer(repo_root, output_dir) - index_path = output_dir / "index.html" - (output_dir / "policies").mkdir(parents=True, exist_ok=True) - commit_html = ( - f'{html.escape(sha[:12])}' - if commit_link - else html.escape(sha[:12]) - ) - run_html = f'Actions run' if run_link else "" - benchmark_pages = discover_benchmark_pages(output_dir, index_path) - - overview_rows: list[str] = [] - velocity_cards: list[str] = [] - tracking_cards: list[str] = [] - normalized_entries: list[dict[str, object]] = [] - - sorted_entries = [ - entry - for index, entry in sorted( - enumerate(entries), - key=lambda item: display_sort_key(item[0], item[1]), - ) - ] - - for entry in sorted_entries: - card_id = entry_card_id(entry) - policy_family = entry_policy_family(entry) - detail_path = detail_page_path(output_dir, card_id) - detail_path.parent.mkdir(parents=True, exist_ok=True) - detail_href = f"{relative_href(index_path, detail_path.parent)}/" - back_href = relative_href(detail_path, index_path) - viewer_module_path = relative_href( - detail_path, - output_dir / RERUN_WEB_VIEWER_DIR / "index.js", - ) - meta = dict(entry["_meta"]) - meta.setdefault("card_id", card_id) - meta.setdefault("policy_family", policy_family) - meta["detail_page"] = detail_href - normalized_entry = dict(entry) - normalized_entry["card_id"] = card_id - normalized_entry["policy_name"] = policy_family - normalized_entry["_meta"] = meta - normalized_entry["detail_page"] = detail_href - normalized_entries.append(normalized_entry) - - status = normalized_entry.get("status", "ok") - execution_kind = str(meta["execution_kind"]) - identity_label = entry_identity_label(entry) - command_kind = str(normalized_entry.get("command_kind", "")) - transport = str(meta.get("showcase_transport", "synthetic")) - model_variant = showcase_model_variant_text(meta.get("showcase_model_variant")) - transport_html = pill(showcase_transport_badge_label(transport), "transport") - status_html = pill( - "OK" if status == "ok" else "BLOCKED", - "ok" if status == "ok" else "blocked", - ) - provenance_html = " ".join([pill(execution_kind.upper(), execution_kind), transport_html]) - - frames = normalized_entry.get("frames", []) - metrics = normalized_entry.get("metrics") or {} - quality_verdict = None - if status == "ok" and isinstance(metrics, dict): - metrics.setdefault("target_tracking", derive_target_tracking_metrics(frames)) - quality_verdict = classify_quality_verdict(status, command_kind, metrics) - normalized_entry["quality_verdict"] = quality_verdict - quality_html = ( - pill(str(quality_verdict["label"]), str(quality_verdict["css_class"])) - if isinstance(quality_verdict, dict) - else 'n/a' - ) - ticks = metrics.get("ticks", "-") - avg_inference = ( - f"{metrics['average_inference_ms']:.3f} ms" if metrics else "-" - ) - achieved_hz = ( - f"{metrics['achieved_frequency_hz']:.2f} Hz" if metrics else "-" - ) - dropped_frames = metrics.get("dropped_frames", "-") - overview_links = render_overview_links(meta, detail_href) - - overview_rows.append( - f"{html.escape(str(meta['title']))}" - f"
{html.escape(identity_label)}
" - f"

{overview_links}

" - f"{status_html}" - f"{quality_html}" - f"{provenance_html}" - f"{html.escape(str(meta['demo_family']))}" - f"{html.escape(str(meta['coverage']))}" - f"{ticks}" - f"{avg_inference}" - f"{achieved_hz}" - f"{dropped_frames}" - ) - - badge_bits = [] - if isinstance(quality_verdict, dict): - badge_bits.append( - pill(str(quality_verdict["label"]), str(quality_verdict["css_class"])) - ) - elif status != "ok": - badge_bits.append(pill("BLOCKED", "blocked")) - badge_bits.extend( - [ - pill(execution_kind.upper(), execution_kind), - transport_html, - pill(command_kind.upper(), "command"), - pill(str(meta["command_source"]), "meta"), - ] - ) - badge_row = " ".join(badge_bits) - detail_meta = rebase_meta_artifact_paths(meta, detail_path, output_dir) - detail_summary = ( - f"{str(meta['summary'])} This page records the exact blocker and any missing assets." - if status != "ok" - else f"{str(meta['summary'])} This page keeps the full charts, playback, and downloadable artifacts." - ) - overview_card = render_policy_link_card(normalized_entry, detail_href, quality_html) - - if status != "ok": - missing_paths = meta.get("missing_paths", []) - missing_html = "
".join(f"{html.escape(path)}" for path in missing_paths) - detail_body_html = f'''
-
-
-

{html.escape(str(detail_meta['title']))}

-

{html.escape(str(detail_meta['source']))} · {html.escape(str(detail_meta['coverage']))}

-

{html.escape(identity_label)}

-
-
{badge_row}
-
-

{html.escape(str(detail_meta['summary']))}

-
-
- Case key - {html.escape(card_id)} -
-
- Policy family - {html.escape(policy_family)} -
-
- Command kind - {html.escape(command_kind)} -
-
- Expected behavior - {html.escape(str(detail_meta['coverage']))} -
-
- Status - Blocked -
-
- Showcase transport - {html.escape(showcase_transport_text(transport))} -
-
- Embodiment - {html.escape(str(detail_meta['showcase_model_path'] or '-'))} -
-
- MuJoCo model variant - {html.escape(model_variant)} -
-
- Checkpoint source - {html.escape(str(detail_meta['checkpoint_source']))} -
-
- Demo family - {html.escape(str(detail_meta['demo_family']))} -
-
- Demo sequence - {html.escape(str(detail_meta['demo_sequence']))} -
-
- Model artifact - {html.escape(str(detail_meta['model_artifact']))} -
-
- Config - {html.escape(str(detail_meta['config_path']))} -
-
-
- Why blocked: {html.escape(str(detail_meta['blocked_reason']))} -
-
- Missing required paths -
{missing_html or 'None'}
-
-
''' - detail_path.write_text( - render_policy_detail_page( - page_title=str(detail_meta["title"]), - page_summary=detail_summary, - body_html=detail_body_html, - back_href=back_href, - generated_at=generated_at, - commit_html=commit_html, - run_html=run_html, - viewer_module_path=viewer_module_path, - ), - encoding="utf-8", - ) - if str(meta["demo_family"]) == "Velocity tracking": - velocity_cards.append(overview_card) - else: - tracking_cards.append(overview_card) - continue - - metrics = normalized_entry["metrics"] - joint_names = normalized_entry.get("joint_names", []) - velocity_tracking_metrics = None - target_tracking_metrics = None - if isinstance(metrics, dict): - velocity_tracking_metrics = metrics.get("velocity_tracking") - target_tracking_metrics = metrics.get("target_tracking") - - target_series = [] - for idx, joint_name in enumerate(joint_names[:4]): - values = series_from_frames(frames, "target_positions", idx) - target_series.append( - { - "label": joint_name, - "values": values, - "color": COLORS[idx % len(COLORS)], - } - ) - - actual_vs_target = [] - if joint_names: - actual_vs_target = [ - { - "label": f"{joint_names[0]} actual", - "values": series_from_frames(frames, "actual_positions", 0), - "color": COLORS[0], - "dashed": True, - }, - { - "label": f"{joint_names[0]} target", - "values": series_from_frames(frames, "target_positions", 0), - "color": COLORS[1], - }, - ] - - latency_series = [ - { - "label": "latency_ms", - "values": [frame["inference_latency_ms"] for frame in frames], - "color": COLORS[4], - } - ] - - velocity_tracking_series = None - if command_kind in {"velocity", "velocity_schedule"}: - velocity_tracking_series = derive_velocity_tracking_series( - frames, - int(normalized_entry.get("control_frequency_hz", 0) or 0), - ) - command_series = [ - { - "label": "vx_cmd", - "values": series_from_frames(frames, "command_data", 0), - "color": COLORS[2], - }, - { - "label": "yaw_cmd", - "values": series_from_frames(frames, "command_data", 2), - "color": COLORS[3], - }, - ] - command_chart_title = "Velocity command profile" - else: - command_series = [ - { - "label": f"{joint_names[0]} velocity" if joint_names else "joint0_velocity", - "values": series_from_frames(frames, "actual_velocities", 0), - "color": COLORS[2], - } - ] - command_chart_title = "Observed joint velocity" - - if velocity_tracking_series is not None: - second_chart_title = "Body vx command vs actual" - second_chart_series = [ - { - "label": "vx_cmd", - "values": velocity_tracking_series["vx_cmd"], - "color": COLORS[2], - }, - { - "label": "vx_actual", - "values": velocity_tracking_series["vx_actual"], - "color": COLORS[0], - "dashed": True, - }, - ] - third_chart_title = "Yaw rate command vs actual" - third_chart_series = [ - { - "label": "yaw_cmd", - "values": velocity_tracking_series["yaw_cmd"], - "color": COLORS[3], - }, - { - "label": "yaw_actual", - "values": velocity_tracking_series["yaw_actual"], - "color": COLORS[1], - "dashed": True, - }, - ] - else: - second_chart_title = "Joint 0 actual vs target" - second_chart_series = actual_vs_target - third_chart_title = command_chart_title - third_chart_series = command_series - - if isinstance(velocity_tracking_metrics, dict): - velocity_tracking_details = f''' -
- VX RMSE - {float(velocity_tracking_metrics["vx_rmse_mps"]):.3f} m/s -
-
- Yaw RMSE - {float(velocity_tracking_metrics["yaw_rate_rmse_rad_s"]):.3f} rad/s -
-
- Heading change - {float(velocity_tracking_metrics["heading_change_deg"]):.1f} deg -
-
- Forward distance - {float(velocity_tracking_metrics["forward_distance_m"]):.3f} m -
''' - else: - velocity_tracking_details = "" - - if isinstance(target_tracking_metrics, dict): - min_base_height = target_tracking_metrics.get("base_height_min_m") - min_base_height_text = ( - f"{float(min_base_height):.3f} m" - if min_base_height is not None - else "n/a" - ) - target_tracking_details = f''' -
- Mean joint error - {float(target_tracking_metrics["mean_joint_abs_error_rad"]):.3f} rad -
-
- Joint error p95 - {float(target_tracking_metrics["p95_joint_abs_error_rad"]):.3f} rad -
-
- Peak joint error - {float(target_tracking_metrics["peak_joint_abs_error_rad"]):.3f} rad -
-
- Min base height - {html.escape(min_base_height_text)} -
-
- Frames height < 0.4 m - {int(target_tracking_metrics["frames_below_base_height_0_4m"])} -
''' - else: - target_tracking_details = "" - - if isinstance(quality_verdict, dict): - verdict_details = f''' -
- Quality verdict - {html.escape(str(quality_verdict["label"]))} -
-
- Verdict basis - {html.escape(str(quality_verdict["summary"]))} -
''' - else: - verdict_details = "" - - proof_pack_links: list[str] = [] - proof_pack_manifest_file = detail_meta.get("proof_pack_manifest_file") - if proof_pack_manifest_file: - proof_pack_links.append( - f'Proof-pack manifest' - ) - proof_pack_links_html = ( - " · " + " · ".join(proof_pack_links) if proof_pack_links else "" - ) - proof_pack_section = render_proof_pack_section( - normalized_entry.get("_proof_pack_manifest") - if isinstance(normalized_entry.get("_proof_pack_manifest"), dict) - else None - ) - - detail_body_html = f'''
-
-
-

{html.escape(str(detail_meta['title']))}

-

{html.escape(str(detail_meta['source']))} · {html.escape(str(detail_meta['coverage']))}

-

{html.escape(identity_label)}

-
-
{badge_row}
-
-

{html.escape(str(detail_meta['summary']))}

-
-
Robot{html.escape(str(normalized_entry['robot_name']))}
-
Ticks{metrics['ticks']}
-
Avg inference{metrics['average_inference_ms']:.3f} ms
-
Achieved rate{metrics['achieved_frequency_hz']:.2f} Hz
-
-
-
-
Target positions
- {spark_svg(target_series)} -
-
-
{html.escape(second_chart_title)}
- {spark_svg(second_chart_series)} -
-
-
{html.escape(third_chart_title)}
- {spark_svg(third_chart_series)} -
-
-
Inference latency
- {spark_svg(latency_series)} -
-
-
-
- Embedded Rerun viewer - Fetches {html.escape(str(detail_meta['rrd_file']))} lazily when the viewer enters the viewport. -
-
-
- Preparing interactive view - Loads the viewer runtime and recording on demand when visible. -
-
-
-
-
- Case key - {html.escape(card_id)} -
-
- Policy family - {html.escape(policy_family)} -
-
- Command kind - {html.escape(command_kind)} -
-
- Expected behavior - {html.escape(str(detail_meta['coverage']))} -
-
- Showcase transport - {html.escape(showcase_transport_text(transport))} -
-
- Embodiment - {html.escape(str(detail_meta['showcase_model_path'] or '-'))} -
-
- MuJoCo model variant - {html.escape(model_variant)} -
-
- Command data - {html.escape(format_vector(normalized_entry.get('command_data', [])))} -
-
- Checkpoint source - {html.escape(str(detail_meta['checkpoint_source']))} -
-
- Demo family - {html.escape(str(detail_meta['demo_family']))} -
-
- Demo sequence - {html.escape(str(detail_meta['demo_sequence']))} -
-
- Model artifact - {html.escape(str(detail_meta['model_artifact']))} -
-
- Command source - {html.escape(str(detail_meta['command_source']))} -
-
- First target frame - {html.escape(format_vector(frames[0]['target_positions'] if frames else []))} -
-
- Last target frame - {html.escape(format_vector(frames[-1]['target_positions'] if frames else []))} -
- {verdict_details} - {target_tracking_details} - {velocity_tracking_details} -
-

Rerun recording · JSON summary · run log{proof_pack_links_html} · {html.escape(str(detail_meta['config_path']))}

-
-{proof_pack_section}''' - detail_path.write_text( - render_policy_detail_page( - page_title=str(detail_meta["title"]), - page_summary=detail_summary, - body_html=detail_body_html, - back_href=back_href, - generated_at=generated_at, - commit_html=commit_html, - run_html=run_html, - viewer_module_path=viewer_module_path, - ), - encoding="utf-8", - ) - if str(meta["demo_family"]) == "Velocity tracking": - velocity_cards.append(overview_card) - else: - tracking_cards.append(overview_card) - - excluded = "".join( - f"
  • {html.escape(item['name'])}: {html.escape(item['reason'])}
    • " - for item in NOT_YET_SHOWCASED - ) - - html_doc = f''' - - - - - RoboWBC Site - - - -
    -
    -

    RoboWBC Site

    -

    This site is generated automatically in CI from the runnable policy integrations and benchmark packages that exist today. The home page is comparison-first: use it to cross-check policy status, quality, provenance, and demo coverage quickly, then open a policy folder for charts, Rerun playback, logs, and visual checkpoints.

    -

    Each policy now owns its own folder under policies/<policy>/, so the HTML page and raw artifacts live together instead of being scattered at the site root. The same bundle also carries benchmark families under benchmarks/.

    -

    Velocity runs use staged locomotion command profiles instead of a single constant command. Reference or pose-tracking cases stay explicitly blocked unless a verified official asset and runtime path actually exist, so the site does not silently drift into mock output.

    -

    The public G1 cards currently load a meshless MuJoCo MJCF variant because this repository does not redistribute Unitree's upstream STL mesh bundle. The dynamics stay MuJoCo-backed, while the Rerun robot scene is reconstructed from the same open MJCF kinematic tree.

    -

    Serve the generated folder over HTTP for reliable playback. Each policy page lazy-loads the saved .rrd recording and the visual checkpoints inline, so you do not need a second proof-pack HTML flow for normal review.

    -
    - Generated: {html.escape(generated_at)} - Commit: {commit_html or 'local'} - {run_html} -
    -
    - -
    -

    Policy runs

    -

    Successful entries use real checkpoints or public asset bundles cached by CI and must activate the requested MuJoCo transport. The links in each row jump straight to the per-policy folder and raw artifacts. Blocked entries surface the exact missing files or unavailable upstream artifacts instead of falling back to mock output.

    - - - - - - {''.join(overview_rows)} - -
    PolicyStatusQualityRun pathDemo familyCoverageTicksAvg inferenceAchieved rateDropped frames
    -
    - - {render_demo_section("Velocity tracking", velocity_cards)} - {render_demo_section("Reference / pose tracking", tracking_cards)} - {render_benchmark_section(benchmark_pages)} - -
    -

    Not Yet In This Site

    -
      {excluded}
    -
    -
    - -''' - - index_path.write_text(html_doc, encoding="utf-8") - (output_dir / "manifest.json").write_text( - json.dumps(normalized_entries, indent=2), - encoding="utf-8", - ) - - -def main() -> int: - args = parse_args() - repo_root = Path(args.repo_root).resolve() - output_dir = Path(args.output_dir).resolve() - output_dir.mkdir(parents=True, exist_ok=True) - binary = Path(args.robowbc_binary).resolve() - if not binary.exists(): - raise SystemExit(f"robowbc binary not found: {binary}") - - env = os.environ.copy() - dylib = resolve_ort_dylib(repo_root) - if dylib: - env.setdefault("ROBOWBC_ORT_DYLIB_PATH", dylib) - env = configure_binary_runtime_env(env) - - entries = [run_policy(repo_root, binary, output_dir, policy, env) for policy in POLICIES] - render_html(entries, output_dir, repo_root) - print(f"wrote site home to {output_dir / 'index.html'}") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) +run_legacy_script("site/generate_policy_showcase.py") diff --git a/scripts/models/download_bfm_zero_models.sh b/scripts/models/download_bfm_zero_models.sh new file mode 100755 index 0000000..dc75d7f --- /dev/null +++ b/scripts/models/download_bfm_zero_models.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +set -euo pipefail + +DEST_DIR="${1:-models/bfm_zero}" +BASE_URL="https://huggingface.co/LeCAR-Lab/BFM-Zero/resolve/main" +TMP_DIR="$(mktemp -d)" +UPSTREAM_DIR="${TMP_DIR}/bfm-zero-upstream" +ONNX_TARGET="${DEST_DIR}/bfm_zero_g1.onnx" +CONTEXT_TARGET="${DEST_DIR}/zs_walking.npy" + +cleanup() { + rm -rf "${TMP_DIR}" +} +trap cleanup EXIT + +if [[ -s "${ONNX_TARGET}" && -s "${CONTEXT_TARGET}" ]]; then + echo "[cache] BFM-Zero assets already present in ${DEST_DIR}" + echo "[cache] $(basename "${ONNX_TARGET}") ($(wc -c < "${ONNX_TARGET}") bytes)" + echo "[cache] $(basename "${CONTEXT_TARGET}") ($(wc -c < "${CONTEXT_TARGET}") bytes)" + exit 0 +fi + +mkdir -p "${UPSTREAM_DIR}/model/exported" "${UPSTREAM_DIR}/model/tracking_inference" "${DEST_DIR}" + +download() { + local url="$1" + local target="$2" + echo "[download] ${target}" + curl -L --fail --retry 3 --retry-delay 2 --show-error --output "${target}" "${url}" +} + +download \ + "${BASE_URL}/model/exported/FBcprAuxModel.onnx" \ + "${UPSTREAM_DIR}/model/exported/FBcprAuxModel.onnx" +download \ + "${BASE_URL}/model/tracking_inference/zs_walking.pkl" \ + "${UPSTREAM_DIR}/model/tracking_inference/zs_walking.pkl" + +python scripts/models/prepare_bfm_zero_assets.py \ + --source "${UPSTREAM_DIR}" \ + --output "${DEST_DIR}" + +echo "BFM-Zero assets ready in ${DEST_DIR}" diff --git a/scripts/models/download_decoupled_wbc_models.sh b/scripts/models/download_decoupled_wbc_models.sh new file mode 100755 index 0000000..f954492 --- /dev/null +++ b/scripts/models/download_decoupled_wbc_models.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +set -euo pipefail + +DEST_DIR="${1:-models/decoupled-wbc}" +UPSTREAM_COMMIT="${GR00T_WBC_UPSTREAM_COMMIT:-bc38f6d0ce6cab4589e025037ad0bfbab7ba73d8}" +BASE_MEDIA_URL="https://media.githubusercontent.com/media/NVlabs/GR00T-WholeBodyControl/${UPSTREAM_COMMIT}/decoupled_wbc/sim2mujoco/resources/robots/g1" +BASE_RAW_URL="https://raw.githubusercontent.com/NVlabs/GR00T-WholeBodyControl/${UPSTREAM_COMMIT}/decoupled_wbc/sim2mujoco/resources/robots/g1" + +mkdir -p "${DEST_DIR}" +printf '%s\n' "${UPSTREAM_COMMIT}" > "${DEST_DIR}/REVISION" +echo "[info] pinned GR00T-WholeBodyControl commit: ${UPSTREAM_COMMIT}" + +download() { + local url="$1" + local target="$2" + echo "[download] ${target}" + curl -L --fail --retry 3 --retry-delay 2 --show-error --output "${target}" "${url}" +} + +download "${BASE_MEDIA_URL}/policy/GR00T-WholeBodyControl-Balance.onnx" "${DEST_DIR}/GR00T-WholeBodyControl-Balance.onnx" +download "${BASE_MEDIA_URL}/policy/GR00T-WholeBodyControl-Walk.onnx" "${DEST_DIR}/GR00T-WholeBodyControl-Walk.onnx" +download "${BASE_RAW_URL}/g1_gear_wbc.yaml" "${DEST_DIR}/g1_gear_wbc.yaml" + +echo "Decoupled WBC models ready in ${DEST_DIR}" diff --git a/scripts/models/download_gear_sonic_models.sh b/scripts/models/download_gear_sonic_models.sh new file mode 100755 index 0000000..487eb89 --- /dev/null +++ b/scripts/models/download_gear_sonic_models.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +set -euo pipefail + +DEST_DIR="${1:-models/gear-sonic}" +HF_REVISION="${GEAR_SONIC_HF_REVISION:-cc80d505b7e055fd6ae26426ae8bfa0a74c26011}" +BASE_URL="https://huggingface.co/nvidia/GEAR-SONIC/resolve/${HF_REVISION}" + +mkdir -p "${DEST_DIR}" +printf '%s\n' "${HF_REVISION}" > "${DEST_DIR}/REVISION" +echo "[info] pinned Hugging Face revision: ${HF_REVISION}" + +models=( + "model_encoder.onnx" + "model_decoder.onnx" + "planner_sonic.onnx" +) + +for model in "${models[@]}"; do + target="${DEST_DIR}/${model}" + if [[ -s "${target}" ]]; then + echo "[cache] ${model} already present at ${target} ($(wc -c < "${target}") bytes)" + continue + fi + echo "[download] ${model} -> ${target}" + curl --fail --location --retry 3 --retry-delay 2 \ + "${BASE_URL}/${model}" \ + --output "${target}" + if [[ ! -s "${target}" ]]; then + echo "[error] downloaded file is empty: ${target}" >&2 + exit 1 + fi + echo "[ok] ${model} ($(wc -c < "${target}") bytes)" +done + +echo "Downloaded GEAR-SONIC ONNX models to ${DEST_DIR}." diff --git a/scripts/models/download_gear_sonic_reference_motions.sh b/scripts/models/download_gear_sonic_reference_motions.sh new file mode 100755 index 0000000..ba43fec --- /dev/null +++ b/scripts/models/download_gear_sonic_reference_motions.sh @@ -0,0 +1,66 @@ +#!/usr/bin/env bash +set -euo pipefail + +DEST_ROOT="${1:-models/gear-sonic/reference/example}" +shift || true + +UPSTREAM_COMMIT="${GEAR_SONIC_REFERENCE_COMMIT:-bc38f6d0ce6cab4589e025037ad0bfbab7ba73d8}" +BASE_MEDIA_URL="https://media.githubusercontent.com/media/NVlabs/GR00T-WholeBodyControl/${UPSTREAM_COMMIT}/gear_sonic_deploy/reference/example" +BASE_RAW_URL="https://raw.githubusercontent.com/NVlabs/GR00T-WholeBodyControl/${UPSTREAM_COMMIT}/gear_sonic_deploy/reference/example" + +if [[ $# -eq 0 ]]; then + clips=( + "macarena_001__A545" + ) +else + clips=("$@") +fi + +files=( + "joint_pos.csv" + "joint_vel.csv" + "body_pos.csv" + "body_quat.csv" + "body_lin_vel.csv" + "body_ang_vel.csv" + "metadata.txt" + "info.txt" +) + +mkdir -p "${DEST_ROOT}" +printf '%s\n' "${UPSTREAM_COMMIT}" > "${DEST_ROOT}/UPSTREAM_COMMIT" +echo "[info] pinned GR00T-WholeBodyControl commit: ${UPSTREAM_COMMIT}" + +for clip in "${clips[@]}"; do + clip_dir="${DEST_ROOT}/${clip}" + mkdir -p "${clip_dir}" + echo "[clip] ${clip}" + + for file in "${files[@]}"; do + target="${clip_dir}/${file}" + if [[ -s "${target}" ]] && ! head -n 1 "${target}" | grep -q '^version https://git-lfs.github.com/spec/v1$'; then + echo " [cache] ${file} already present" + continue + fi + + case "${file}" in + metadata.txt|info.txt) + source_url="${BASE_RAW_URL}/${clip}/${file}" + ;; + *) + source_url="${BASE_MEDIA_URL}/${clip}/${file}" + ;; + esac + + echo " [download] ${file}" + curl --fail --location --retry 3 --retry-delay 2 \ + "${source_url}" \ + --output "${target}" + if [[ ! -s "${target}" ]]; then + echo " [error] downloaded file is empty: ${target}" >&2 + exit 1 + fi + done +done + +echo "Downloaded GEAR-Sonic reference motions to ${DEST_ROOT}." diff --git a/scripts/models/download_wbc_agile_models.sh b/scripts/models/download_wbc_agile_models.sh new file mode 100755 index 0000000..59d8cdd --- /dev/null +++ b/scripts/models/download_wbc_agile_models.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +set -euo pipefail + +DEST_DIR="${1:-models/wbc-agile}" +BASE_MEDIA_URL="https://media.githubusercontent.com/media/nvidia-isaac/WBC-AGILE/main/agile/data/policy/velocity_g1" +BASE_RAW_URL="https://raw.githubusercontent.com/nvidia-isaac/WBC-AGILE/main/agile/data/policy/velocity_g1" + +mkdir -p "${DEST_DIR}" + +download() { + local url="$1" + local target="$2" + echo "[download] ${target}" + curl -L --fail --retry 3 --retry-delay 2 --show-error --output "${target}" "${url}" +} + +download "${BASE_MEDIA_URL}/unitree_g1_velocity_e2e.onnx" "${DEST_DIR}/unitree_g1_velocity_e2e.onnx" +download "${BASE_RAW_URL}/unitree_g1_velocity_e2e.yaml" "${DEST_DIR}/unitree_g1_velocity_e2e.yaml" + +echo "WBC-AGILE models ready in ${DEST_DIR}" diff --git a/scripts/models/prepare_bfm_zero_assets.py b/scripts/models/prepare_bfm_zero_assets.py new file mode 100755 index 0000000..a2bc7f0 --- /dev/null +++ b/scripts/models/prepare_bfm_zero_assets.py @@ -0,0 +1,104 @@ +#!/usr/bin/env python3 +"""Normalize public BFM-Zero assets into RoboWBC's local model layout.""" + +from __future__ import annotations + +import argparse +from pathlib import Path +import shutil +import sys + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument( + "--source", + required=True, + help="Path to the upstream BFM-Zero model directory or repo root containing model/", + ) + parser.add_argument( + "--output", + default="models/bfm_zero", + help="Destination directory for normalized RoboWBC assets", + ) + parser.add_argument( + "--tracking-context", + default=None, + help="Optional override for the tracking .pkl file to convert", + ) + return parser.parse_args() + + +def resolve_model_root(source: Path) -> Path: + if (source / "exported").is_dir() and (source / "tracking_inference").is_dir(): + return source + model_root = source / "model" + if (model_root / "exported").is_dir() and (model_root / "tracking_inference").is_dir(): + return model_root + raise SystemExit( + "source must be the upstream BFM-Zero model directory or a repo root containing model/" + ) + + +def load_python_deps() -> tuple[object, object]: + try: + import joblib # type: ignore + import numpy as np # type: ignore + except ImportError as exc: + raise SystemExit( + "prepare_bfm_zero_assets.py requires Python packages `joblib` and `numpy`" + ) from exc + return joblib, np + + +def select_onnx(exported_dir: Path) -> Path: + candidates = sorted(exported_dir.glob("*.onnx")) + if not candidates: + raise SystemExit(f"no ONNX checkpoints found under {exported_dir}") + if len(candidates) > 1: + print( + f"warning: multiple ONNX checkpoints found; using {candidates[0].name}", + file=sys.stderr, + ) + return candidates[0] + + +def main() -> None: + args = parse_args() + source = Path(args.source).expanduser().resolve() + output_dir = Path(args.output).expanduser().resolve() + model_root = resolve_model_root(source) + joblib, np = load_python_deps() + + exported_dir = model_root / "exported" + tracking_pkl = ( + Path(args.tracking_context).expanduser().resolve() + if args.tracking_context is not None + else model_root / "tracking_inference" / "zs_walking.pkl" + ) + + if not tracking_pkl.exists(): + raise SystemExit(f"tracking context not found: {tracking_pkl}") + + onnx_src = select_onnx(exported_dir) + output_dir.mkdir(parents=True, exist_ok=True) + + onnx_dst = output_dir / "bfm_zero_g1.onnx" + shutil.copy2(onnx_src, onnx_dst) + + context = np.asarray(joblib.load(tracking_pkl), dtype=np.float32) + if context.ndim != 2 or context.shape[1] != 256: + raise SystemExit( + f"tracking context must have shape [T, 256], got {tuple(context.shape)}" + ) + + context_dst = output_dir / f"{tracking_pkl.stem}.npy" + np.save(context_dst, context) + + print(f"copied ONNX checkpoint: {onnx_dst}") + print(f"converted tracking context: {context_dst}") + print("BFM-Zero assets are ready for configs/bfm_zero_g1.toml") + + +if __name__ == "__main__": + main() diff --git a/scripts/mujoco/check_mujoco_headless.py b/scripts/mujoco/check_mujoco_headless.py new file mode 100755 index 0000000..c0d7ac3 --- /dev/null +++ b/scripts/mujoco/check_mujoco_headless.py @@ -0,0 +1,85 @@ +#!/usr/bin/env python3 +"""Fail fast when MuJoCo headless rendering is not available. + +This is the same offscreen rendering path used by proof-pack screenshot +capture. `make showcase-verify` runs this first so local developer checks and +the GitHub Actions showcase job fail on the same prerequisite instead of +shipping a site bundle with skipped screenshots. +""" + +from __future__ import annotations + +import os +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "reports")) +from roboharness_report import ensure_headless_mujoco_env, is_headless_render_backend_error + +PROBE_XML = """ + + + + + + + + + +""".strip() + + +def render_probe() -> tuple[str, tuple[int, ...]]: + ensure_headless_mujoco_env() + + import mujoco + + backend = os.environ.get("MUJOCO_GL", "auto") + model = mujoco.MjModel.from_xml_string(PROBE_XML) + data = mujoco.MjData(model) + renderer = mujoco.Renderer(model, height=64, width=64) + try: + mujoco.mj_forward(model, data) + renderer.update_scene(data) + frame = renderer.render() + finally: + close = getattr(renderer, "close", None) + if callable(close): + close() + return backend, tuple(int(dimension) for dimension in frame.shape) + + +def package_hint() -> str: + if sys.platform != "linux": + return "" + return ( + "Install the EGL/Mesa runtime used by CI before retrying, for example:\n" + " sudo apt-get install -y libegl1 libegl-mesa0 libgles2 libgl1-mesa-dri libgbm1" + ) + + +def main() -> int: + try: + backend, frame_shape = render_probe() + except Exception as exc: + backend = os.environ.get("MUJOCO_GL", "auto") + if is_headless_render_backend_error(exc): + hint = package_hint() + raise SystemExit( + "MuJoCo headless render smoke check failed for the configured " + f"backend ({backend}): {type(exc).__name__}: {exc}\n" + "This blocks proof-pack screenshot capture, so " + "`make showcase-verify` refuses to continue.\n" + f"{hint}".rstrip() + ) from exc + raise + + print( + "MuJoCo headless render smoke check passed: " + f"backend={backend} frame_shape={frame_shape}" + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/mujoco/ensure_mujoco_runtime.py b/scripts/mujoco/ensure_mujoco_runtime.py new file mode 100755 index 0000000..8eff76b --- /dev/null +++ b/scripts/mujoco/ensure_mujoco_runtime.py @@ -0,0 +1,157 @@ +#!/usr/bin/env python3 +"""Ensure the MuJoCo runtime is available in a local download directory.""" + +from __future__ import annotations + +import argparse +import hashlib +import os +from pathlib import Path +import platform +import shutil +import tarfile +import urllib.request + +MUJOCO_VERSION = "3.6.0" + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description=( + "Download and extract the MuJoCo runtime into the given directory " + "when it is missing." + ) + ) + parser.add_argument( + "--download-dir", + type=Path, + default=os.environ.get("MUJOCO_DOWNLOAD_DIR"), + help="absolute directory used to store the downloaded MuJoCo runtime", + ) + return parser.parse_args() + + +def archive_name() -> str: + if sys_platform() != "linux": + raise SystemExit("scripts/mujoco/ensure_mujoco_runtime.py currently supports Linux only") + + machine = platform.machine().lower() + arch_map = { + "x86_64": "x86_64", + "amd64": "x86_64", + "aarch64": "aarch64", + "arm64": "aarch64", + } + try: + arch = arch_map[machine] + except KeyError as exc: + raise SystemExit(f"unsupported architecture for MuJoCo runtime download: {machine}") from exc + return f"mujoco-{MUJOCO_VERSION}-linux-{arch}.tar.gz" + + +def sys_platform() -> str: + return platform.system().lower() + + +def sha256sum(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def download_file(url: str, destination: Path) -> None: + with urllib.request.urlopen(url) as response, destination.open("wb") as handle: + shutil.copyfileobj(response, handle) + + +def runtime_root(download_dir: Path) -> Path: + return download_dir / f"mujoco-{MUJOCO_VERSION}" + + +def runtime_library(download_dir: Path) -> Path: + return runtime_root(download_dir) / "lib" / "libmujoco.so" + + +def versioned_runtime_library(download_dir: Path) -> Path: + return runtime_root(download_dir) / "lib" / f"libmujoco.so.{MUJOCO_VERSION}" + + +def ensure_runtime_symlink(download_dir: Path) -> None: + symlink_path = runtime_library(download_dir) + versioned_path = versioned_runtime_library(download_dir) + if symlink_path.exists(): + return + if not versioned_path.is_file(): + raise SystemExit(f"MuJoCo versioned runtime library missing: {versioned_path}") + symlink_path.symlink_to(versioned_path.name) + + +def safe_extract_linux(archive_path: Path, destination: Path) -> None: + destination = destination.resolve() + with tarfile.open(archive_path) as archive: + members = archive.getmembers() + for member in members: + member_path = (destination / member.name).resolve() + if os.path.commonpath([destination, member_path]) != str(destination): + raise SystemExit(f"unsafe tar entry outside extraction root: {member.name}") + archive.extractall(destination, filter="fully_trusted") + + +def ensure_runtime(download_dir: Path) -> Path: + library_path = runtime_library(download_dir) + if library_path.is_file(): + return library_path + + versioned_path = versioned_runtime_library(download_dir) + if versioned_path.is_file(): + ensure_runtime_symlink(download_dir) + return library_path + + archive = archive_name() + base_url = f"https://github.com/google-deepmind/mujoco/releases/download/{MUJOCO_VERSION}" + archive_path = download_dir / archive + checksum_path = download_dir / f"{archive}.sha256" + + print(f"MuJoCo runtime missing; downloading {archive} into {download_dir}") + download_file(f"{base_url}/{archive}", archive_path) + download_file(f"{base_url}/{archive}.sha256", checksum_path) + + expected_sha = checksum_path.read_text(encoding="utf-8").split()[0] + actual_sha = sha256sum(archive_path) + if actual_sha != expected_sha: + raise SystemExit( + "MuJoCo archive checksum mismatch: " + f"expected {expected_sha}, got {actual_sha} for {archive_path}" + ) + + safe_extract_linux(archive_path, download_dir) + archive_path.unlink() + checksum_path.unlink() + + ensure_runtime_symlink(download_dir) + if not library_path.is_file(): + raise SystemExit(f"MuJoCo runtime library not found after extraction: {library_path}") + return library_path + + +def main() -> int: + args = parse_args() + if args.download_dir is None: + raise SystemExit( + "--download-dir is required when MUJOCO_DOWNLOAD_DIR is not set" + ) + + download_dir = args.download_dir.expanduser().resolve() + if not download_dir.is_absolute(): + raise SystemExit(f"download directory must be absolute: {download_dir}") + download_dir.mkdir(parents=True, exist_ok=True) + + library_path = ensure_runtime(download_dir) + print(library_path) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/normalize_nvidia_benchmarks.py b/scripts/normalize_nvidia_benchmarks.py index 71387a3..9fc082a 100755 --- a/scripts/normalize_nvidia_benchmarks.py +++ b/scripts/normalize_nvidia_benchmarks.py @@ -1,401 +1,4 @@ #!/usr/bin/env python3 -"""Normalize NVIDIA comparison artifacts into one schema.""" +from _compat import run_legacy_script -from __future__ import annotations - -import argparse -import json -import math -import os -import platform -import socket -from pathlib import Path -from typing import Any - -PROVIDER_ORDER = ("cpu", "cuda", "tensor_rt") -PROVIDER_ALIASES = { - "trt": "tensor_rt", -} -PROVIDER_FAMILY_IDS = { - "cpu": "cpu-baseline", - "cuda": "cuda", - "tensor_rt": "trt", -} -IMPLEMENTATION_ORDER = ("ort-cpp-sonic", "ort-rs") -IMPLEMENTATION_LABELS = { - "ort-cpp-sonic": "ORT-cpp-sonic", - "ort-rs": "ORT-rs", -} -LEGACY_STACK_TO_IMPLEMENTATION = { - "official_nvidia": "ort-cpp-sonic", - "robowbc": "ort-rs", -} -IMPLEMENTATION_TO_LEGACY_STACK = { - implementation: stack for stack, implementation in LEGACY_STACK_TO_IMPLEMENTATION.items() -} -LEGACY_ARTIFACT_DIRS = { - "ort-cpp-sonic": "official", - "ort-rs": "robowbc", -} - - -def load_json(path: Path) -> Any: - with path.open("r", encoding="utf-8") as handle: - return json.load(handle) - - -def dump_json(path: Path, payload: dict[str, Any]) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - with path.open("w", encoding="utf-8") as handle: - json.dump(payload, handle, indent=2, sort_keys=True) - handle.write("\n") - - -def cpu_model() -> str: - cpuinfo = Path("/proc/cpuinfo") - if cpuinfo.is_file(): - for line in cpuinfo.read_text(encoding="utf-8").splitlines(): - if line.startswith("model name"): - _, value = line.split(":", 1) - return value.strip() - return platform.processor() or "unknown-cpu" - - -def default_host_fingerprint() -> str: - return ( - f"{socket.gethostname()} | {platform.system()} {platform.release()} | " - f"{platform.machine()} | {cpu_model()}" - ) - - -def percentile(values: list[float], pct: float) -> int | None: - if not values: - return None - if len(values) == 1: - return round(values[0]) - ordered = sorted(values) - position = (len(ordered) - 1) * pct - lower = math.floor(position) - upper = math.ceil(position) - if lower == upper: - return round(ordered[lower]) - fraction = position - lower - interpolated = ordered[lower] * (1.0 - fraction) + ordered[upper] * fraction - return round(interpolated) - - -def load_registry(path: Path) -> dict[str, Any]: - registry = load_json(path) - if registry.get("schema_version") != 1: - raise ValueError(f"unsupported registry schema version: {registry.get('schema_version')}") - seen: set[str] = set() - for case in registry.get("cases", []): - case_id = case.get("case_id") - if not case_id: - raise ValueError("registry case is missing case_id") - if case_id in seen: - raise ValueError(f"duplicate case_id in registry: {case_id}") - seen.add(case_id) - for field in ("description", "command_fixture", "warmup_policy", "interpretation"): - if field not in case: - raise ValueError(f"registry case {case_id!r} is missing required field {field!r}") - return registry - - -def registry_case(registry: dict[str, Any], case_id: str) -> dict[str, Any]: - for case in registry["cases"]: - if case["case_id"] == case_id: - return case - raise ValueError(f"unknown case_id: {case_id}") - - -def canonical_provider(provider: str) -> str: - try: - normalized = PROVIDER_ALIASES.get(provider, provider) - if normalized not in PROVIDER_ORDER: - raise KeyError(provider) - return normalized - except KeyError as exc: - expected = ", ".join(PROVIDER_ORDER) - raise ValueError(f"unknown provider {provider!r}; expected one of: {expected}") from exc - - -def provider_family(provider: str) -> str: - return PROVIDER_FAMILY_IDS[canonical_provider(provider)] - - -def canonical_implementation(implementation: str) -> str: - normalized = LEGACY_STACK_TO_IMPLEMENTATION.get(implementation, implementation) - if normalized not in IMPLEMENTATION_ORDER: - expected = ", ".join(IMPLEMENTATION_ORDER) - raise ValueError( - f"unknown implementation {implementation!r}; expected one of: {expected}" - ) - return normalized - - -def implementation_label(implementation: str) -> str: - return IMPLEMENTATION_LABELS[canonical_implementation(implementation)] - - -def variant_label(provider: str, implementation: str) -> str: - provider_id = canonical_provider(provider) - if provider_id == "cpu": - return provider_family(provider_id) - return f"{provider_family(provider_id)}-{implementation_label(implementation)}" - - -def variant_slug(provider: str, implementation: str) -> str: - provider_id = canonical_provider(provider) - if provider_id == "cpu": - return provider_family(provider_id) - return f"{provider_family(provider_id)}-{canonical_implementation(implementation)}" - - -def implementation_artifact_dir(implementation: str) -> str: - return canonical_implementation(implementation) - - -def legacy_artifact_dir(implementation: str) -> str: - return LEGACY_ARTIFACT_DIRS[canonical_implementation(implementation)] - - -def criterion_samples_ns(criterion_root: Path, criterion_id: str) -> tuple[list[float], str]: - for benchmark_json in criterion_root.rglob("benchmark.json"): - benchmark = load_json(benchmark_json) - if benchmark.get("full_id") != criterion_id: - continue - sample_path = benchmark_json.parent / "sample.json" - sample = load_json(sample_path) - iters = sample.get("iters", []) - times = sample.get("times", []) - if len(iters) != len(times): - raise ValueError(f"criterion sample mismatch for {criterion_id}") - samples = [float(total_ns) / float(iter_count) for iter_count, total_ns in zip(iters, times)] - return samples, str(sample_path) - raise FileNotFoundError( - f"could not find criterion benchmark {criterion_id!r} under {criterion_root}" - ) - - -def run_report_samples_ns(report_path: Path) -> tuple[list[float], float | None, str]: - report = load_json(report_path) - frames = report.get("frames", []) - samples = [float(frame["inference_latency_ms"]) * 1_000_000.0 for frame in frames] - hz = report.get("metrics", {}).get("achieved_frequency_hz") - return samples, float(hz) if hz is not None else None, str(report_path) - - -def manual_samples_payload(input_path: Path) -> tuple[list[float], float | None, str]: - payload = load_json(input_path) - if isinstance(payload, list): - return [float(value) for value in payload], None, str(input_path) - samples = payload.get("samples_ns") - if samples is None: - raise ValueError(f"{input_path} must be a JSON list or an object with samples_ns") - hz = payload.get("hz") - return [float(value) for value in samples], float(hz) if hz is not None else None, str(input_path) - - -def build_artifact( - *, - case: dict[str, Any], - implementation: str, - upstream_commit: str | None, - robowbc_commit: str | None, - provider: str, - host_fingerprint: str | None, - samples_ns: list[float], - hz: float | None, - notes: str, - source_command: str | None, - raw_source: str, - status: str, -) -> dict[str, Any]: - implementation_id = canonical_implementation(implementation) - provider_id = canonical_provider(provider) - return { - "schema_version": 1, - "status": status, - "case_id": case["case_id"], - "description": case["description"], - "interpretation": case["interpretation"], - "implementation": implementation_id, - "implementation_label": implementation_label(implementation_id), - "stack": IMPLEMENTATION_TO_LEGACY_STACK[implementation_id], - "upstream_commit": upstream_commit, - "robowbc_commit": robowbc_commit, - "provider": provider_id, - "provider_family": provider_family(provider_id), - "variant_label": variant_label(provider_id, implementation_id), - "variant_slug": variant_slug(provider_id, implementation_id), - "host_fingerprint": host_fingerprint or default_host_fingerprint(), - "command_fixture": case["command_fixture"], - "warmup_policy": case["warmup_policy"], - "samples": len(samples_ns), - "p50_ns": percentile(samples_ns, 0.50), - "p95_ns": percentile(samples_ns, 0.95), - "p99_ns": percentile(samples_ns, 0.99), - "hz": round(hz, 6) if hz is not None else None, - "notes": notes, - "source_command": source_command, - "raw_source": raw_source, - } - - -def cmd_validate_registry(args: argparse.Namespace) -> int: - load_registry(args.registry) - return 0 - - -def common_artifact_args(parser: argparse.ArgumentParser) -> None: - parser.add_argument("--registry", type=Path, required=True) - parser.add_argument("--case-id", required=True) - parser.add_argument( - "--implementation", - required=True, - choices=(*IMPLEMENTATION_ORDER, *LEGACY_STACK_TO_IMPLEMENTATION), - ) - parser.add_argument( - "--stack", - dest="implementation", - help=argparse.SUPPRESS, - ) - parser.add_argument("--provider", required=True) - parser.add_argument("--upstream-commit") - parser.add_argument("--robowbc-commit") - parser.add_argument("--host-fingerprint") - parser.add_argument("--notes", default="") - parser.add_argument("--source-command") - parser.add_argument("--output", type=Path, required=True) - - -def cmd_normalize_criterion(args: argparse.Namespace) -> int: - registry = load_registry(args.registry) - case = registry_case(registry, args.case_id) - criterion_id = case.get("criterion_id") - if not criterion_id: - raise ValueError(f"case {args.case_id!r} does not define criterion_id") - samples_ns, raw_source = criterion_samples_ns(args.criterion_root, criterion_id) - artifact = build_artifact( - case=case, - implementation=args.implementation, - upstream_commit=args.upstream_commit, - robowbc_commit=args.robowbc_commit, - provider=args.provider, - host_fingerprint=args.host_fingerprint, - samples_ns=samples_ns, - hz=None, - notes=args.notes, - source_command=args.source_command, - raw_source=raw_source, - status="ok", - ) - dump_json(args.output, artifact) - return 0 - - -def cmd_normalize_run_report(args: argparse.Namespace) -> int: - registry = load_registry(args.registry) - case = registry_case(registry, args.case_id) - samples_ns, hz, raw_source = run_report_samples_ns(args.input) - artifact = build_artifact( - case=case, - implementation=args.implementation, - upstream_commit=args.upstream_commit, - robowbc_commit=args.robowbc_commit, - provider=args.provider, - host_fingerprint=args.host_fingerprint, - samples_ns=samples_ns, - hz=hz, - notes=args.notes, - source_command=args.source_command, - raw_source=raw_source, - status="ok", - ) - dump_json(args.output, artifact) - return 0 - - -def cmd_normalize_samples(args: argparse.Namespace) -> int: - registry = load_registry(args.registry) - case = registry_case(registry, args.case_id) - samples_ns, hz, raw_source = manual_samples_payload(args.input) - artifact = build_artifact( - case=case, - implementation=args.implementation, - upstream_commit=args.upstream_commit, - robowbc_commit=args.robowbc_commit, - provider=args.provider, - host_fingerprint=args.host_fingerprint, - samples_ns=samples_ns, - hz=hz, - notes=args.notes, - source_command=args.source_command, - raw_source=raw_source, - status="ok", - ) - dump_json(args.output, artifact) - return 0 - - -def cmd_emit_blocked(args: argparse.Namespace) -> int: - registry = load_registry(args.registry) - case = registry_case(registry, args.case_id) - artifact = build_artifact( - case=case, - implementation=args.implementation, - upstream_commit=args.upstream_commit, - robowbc_commit=args.robowbc_commit, - provider=args.provider, - host_fingerprint=args.host_fingerprint, - samples_ns=[], - hz=None, - notes=args.reason if not args.notes else f"{args.notes} | {args.reason}", - source_command=args.source_command, - raw_source=args.reason, - status="blocked", - ) - dump_json(args.output, artifact) - return 0 - - -def build_parser() -> argparse.ArgumentParser: - parser = argparse.ArgumentParser(description=__doc__) - subparsers = parser.add_subparsers(dest="command", required=True) - - validate = subparsers.add_parser("validate-registry") - validate.add_argument("--registry", type=Path, required=True) - validate.set_defaults(func=cmd_validate_registry) - - normalize_criterion = subparsers.add_parser("normalize-criterion") - common_artifact_args(normalize_criterion) - normalize_criterion.add_argument("--criterion-root", type=Path, required=True) - normalize_criterion.set_defaults(func=cmd_normalize_criterion) - - normalize_run = subparsers.add_parser("normalize-run-report") - common_artifact_args(normalize_run) - normalize_run.add_argument("--input", type=Path, required=True) - normalize_run.set_defaults(func=cmd_normalize_run_report) - - normalize_samples = subparsers.add_parser("normalize-samples") - common_artifact_args(normalize_samples) - normalize_samples.add_argument("--input", type=Path, required=True) - normalize_samples.set_defaults(func=cmd_normalize_samples) - - blocked = subparsers.add_parser("emit-blocked") - common_artifact_args(blocked) - blocked.add_argument("--reason", required=True) - blocked.set_defaults(func=cmd_emit_blocked) - - return parser - - -def main() -> int: - parser = build_parser() - args = parser.parse_args() - return args.func(args) - - -if __name__ == "__main__": - raise SystemExit(main()) +run_legacy_script("benchmarks/normalize_nvidia_benchmarks.py") diff --git a/scripts/prepare_bfm_zero_assets.py b/scripts/prepare_bfm_zero_assets.py index a2bc7f0..c5fc883 100755 --- a/scripts/prepare_bfm_zero_assets.py +++ b/scripts/prepare_bfm_zero_assets.py @@ -1,104 +1,4 @@ #!/usr/bin/env python3 -"""Normalize public BFM-Zero assets into RoboWBC's local model layout.""" +from _compat import run_legacy_script -from __future__ import annotations - -import argparse -from pathlib import Path -import shutil -import sys - - -def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser() - parser.add_argument( - "--source", - required=True, - help="Path to the upstream BFM-Zero model directory or repo root containing model/", - ) - parser.add_argument( - "--output", - default="models/bfm_zero", - help="Destination directory for normalized RoboWBC assets", - ) - parser.add_argument( - "--tracking-context", - default=None, - help="Optional override for the tracking .pkl file to convert", - ) - return parser.parse_args() - - -def resolve_model_root(source: Path) -> Path: - if (source / "exported").is_dir() and (source / "tracking_inference").is_dir(): - return source - model_root = source / "model" - if (model_root / "exported").is_dir() and (model_root / "tracking_inference").is_dir(): - return model_root - raise SystemExit( - "source must be the upstream BFM-Zero model directory or a repo root containing model/" - ) - - -def load_python_deps() -> tuple[object, object]: - try: - import joblib # type: ignore - import numpy as np # type: ignore - except ImportError as exc: - raise SystemExit( - "prepare_bfm_zero_assets.py requires Python packages `joblib` and `numpy`" - ) from exc - return joblib, np - - -def select_onnx(exported_dir: Path) -> Path: - candidates = sorted(exported_dir.glob("*.onnx")) - if not candidates: - raise SystemExit(f"no ONNX checkpoints found under {exported_dir}") - if len(candidates) > 1: - print( - f"warning: multiple ONNX checkpoints found; using {candidates[0].name}", - file=sys.stderr, - ) - return candidates[0] - - -def main() -> None: - args = parse_args() - source = Path(args.source).expanduser().resolve() - output_dir = Path(args.output).expanduser().resolve() - model_root = resolve_model_root(source) - joblib, np = load_python_deps() - - exported_dir = model_root / "exported" - tracking_pkl = ( - Path(args.tracking_context).expanduser().resolve() - if args.tracking_context is not None - else model_root / "tracking_inference" / "zs_walking.pkl" - ) - - if not tracking_pkl.exists(): - raise SystemExit(f"tracking context not found: {tracking_pkl}") - - onnx_src = select_onnx(exported_dir) - output_dir.mkdir(parents=True, exist_ok=True) - - onnx_dst = output_dir / "bfm_zero_g1.onnx" - shutil.copy2(onnx_src, onnx_dst) - - context = np.asarray(joblib.load(tracking_pkl), dtype=np.float32) - if context.ndim != 2 or context.shape[1] != 256: - raise SystemExit( - f"tracking context must have shape [T, 256], got {tuple(context.shape)}" - ) - - context_dst = output_dir / f"{tracking_pkl.stem}.npy" - np.save(context_dst, context) - - print(f"copied ONNX checkpoint: {onnx_dst}") - print(f"converted tracking context: {context_dst}") - print("BFM-Zero assets are ready for configs/bfm_zero_g1.toml") - - -if __name__ == "__main__": - main() +run_legacy_script("models/prepare_bfm_zero_assets.py") diff --git a/scripts/python_sdk_smoke.py b/scripts/python_sdk_smoke.py old mode 100644 new mode 100755 index 6848cfa..775857c --- a/scripts/python_sdk_smoke.py +++ b/scripts/python_sdk_smoke.py @@ -1,96 +1,4 @@ #!/usr/bin/env python3 -"""Smoke-test the installed RoboWBC Python SDK.""" +from _compat import run_legacy_script -from __future__ import annotations - -from pathlib import Path - -from robowbc import ( - KinematicPoseCommand, - LinkPose, - Observation, - Registry, - VelocityCommand, -) - - -def main() -> int: - names = Registry.list_policies() - assert names, f"expected at least one policy, got: {names}" - print("Registered policies:", names) - - repo_root = Path(__file__).resolve().parents[1] - config_path = repo_root / "configs" / "decoupled_smoke.toml" - - policy = Registry.build("decoupled_wbc", str(config_path)) - capabilities = policy.capabilities() - assert hasattr(capabilities, "supported_commands"), capabilities - assert "velocity" in capabilities.supported_commands, capabilities - print("Policy capabilities:", capabilities.supported_commands) - - structured_obs = Observation( - joint_positions=[0.0] * 4, - joint_velocities=[0.0] * 4, - gravity_vector=[0.0, 0.0, -1.0], - command=VelocityCommand( - linear=[0.2, 0.0, 0.0], - angular=[0.0, 0.0, 0.1], - ), - ) - targets = policy.predict(structured_obs) - assert len(targets.positions) == 4, targets.positions - print("Structured velocity targets:", targets.positions) - - legacy_obs = Observation( - joint_positions=[0.0] * 4, - joint_velocities=[0.0] * 4, - gravity_vector=[0.0, 0.0, -1.0], - command_type="motion_tokens", - command_data=[0.1, 0.2], - ) - assert legacy_obs.joint_positions == [0.0] * 4 - assert legacy_obs.command_type == "motion_tokens" - assert len(legacy_obs.command_data) == 2 - assert all( - abs(actual - expected) < 1e-6 - for actual, expected in zip(legacy_obs.command_data, [0.1, 0.2], strict=True) - ), legacy_obs.command_data - print("Legacy observation:", legacy_obs) - - manipulation_obs = Observation( - joint_positions=[0.0] * 4, - joint_velocities=[0.0] * 4, - gravity_vector=[0.0, 0.0, -1.0], - command=KinematicPoseCommand( - [ - LinkPose( - name="left_wrist", - translation=[0.35, 0.20, 0.95], - rotation_xyzw=[0.0, 0.0, 0.0, 1.0], - ) - ] - ), - ) - assert manipulation_obs.command_type == "kinematic_pose" - assert manipulation_obs.command.links[0].name == "left_wrist" - try: - _ = manipulation_obs.command_data - except ValueError: - pass - else: - raise AssertionError("kinematic_pose should not expose flat command_data") - print("Structured manipulation observation:", manipulation_obs) - - try: - Registry.build_from_str('[policy]\nname = "no_such_policy"') - except RuntimeError as exc: - assert "unknown policy" in str(exc).lower(), f"unexpected error: {exc}" - else: - raise AssertionError("Registry.build_from_str should have raised") - - print("Python SDK smoke tests passed.") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) +run_legacy_script("sdk/python_sdk_smoke.py") diff --git a/scripts/render_nvidia_benchmark_summary.py b/scripts/render_nvidia_benchmark_summary.py old mode 100644 new mode 100755 index e948904..1a7f9fb --- a/scripts/render_nvidia_benchmark_summary.py +++ b/scripts/render_nvidia_benchmark_summary.py @@ -1,629 +1,4 @@ #!/usr/bin/env python3 -"""Render Markdown and optional HTML summaries from normalized NVIDIA benchmark artifacts.""" +from _compat import run_legacy_script -from __future__ import annotations - -import argparse -import html -import importlib.util -import json -from pathlib import Path -from typing import Any - -ROOT_DIR = Path(__file__).resolve().parents[1] -NORMALIZER_PATH = ROOT_DIR / "scripts/normalize_nvidia_benchmarks.py" - - -def load_normalizer() -> Any: - spec = importlib.util.spec_from_file_location("normalize_nvidia_benchmarks", NORMALIZER_PATH) - if spec is None or spec.loader is None: - raise RuntimeError(f"failed to load normalizer module from {NORMALIZER_PATH}") - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - return module - - -NORMALIZER = load_normalizer() -PROVIDER_ORDER = NORMALIZER.PROVIDER_ORDER -IMPLEMENTATION_ORDER = NORMALIZER.IMPLEMENTATION_ORDER -BASELINE_IMPLEMENTATION = "ort-cpp-sonic" -RUST_IMPLEMENTATION = "ort-rs" - - -def load_json(path: Path) -> Any: - with path.open("r", encoding="utf-8") as handle: - return json.load(handle) - - -def load_registry(path: Path) -> dict[str, Any]: - registry = load_json(path) - if registry.get("schema_version") != 1: - raise ValueError(f"unsupported registry schema version: {registry.get('schema_version')}") - return registry - - -def load_artifact(path: Path) -> dict[str, Any] | None: - if not path.is_file(): - return None - return load_json(path) - - -def format_ns(value: int | None) -> str: - if value is None: - return "n/a" - if value >= 1_000_000: - return f"{value / 1_000_000:.3f} ms" - if value >= 1_000: - return f"{value / 1_000:.3f} us" - return f"{value} ns" - - -def format_hz(value: float | None) -> str: - if value is None: - return "n/a" - return f"{value:.3f} Hz" - - -def format_provider_family(provider: str) -> str: - return NORMALIZER.provider_family(provider) - - -def format_implementation(implementation: str) -> str: - return NORMALIZER.implementation_label(implementation) - - -def rendered_variant_labels() -> list[str]: - variants = [NORMALIZER.provider_family("cpu")] - for provider in PROVIDER_ORDER: - if provider == "cpu": - continue - for implementation in IMPLEMENTATION_ORDER: - variants.append(NORMALIZER.variant_label(provider, implementation)) - return variants - - -def ratio_string( - ort_rs: dict[str, Any] | None, ort_cpp_sonic: dict[str, Any] | None -) -> str: - if not ort_rs or not ort_cpp_sonic: - return "n/a" - if ort_rs.get("status") != "ok" or ort_cpp_sonic.get("status") != "ok": - return "n/a" - ort_rs_p50 = ort_rs.get("p50_ns") - ort_cpp_sonic_p50 = ort_cpp_sonic.get("p50_ns") - if ( - not isinstance(ort_rs_p50, int) - or not isinstance(ort_cpp_sonic_p50, int) - or ort_cpp_sonic_p50 == 0 - ): - return "n/a" - return f"{ort_rs_p50 / ort_cpp_sonic_p50:.2f}x" - - -def artifact_cell(artifact: dict[str, Any] | None) -> str: - if artifact is None: - return "missing" - status = artifact.get("status", "unknown") - if status != "ok": - note = str(artifact.get("notes", "blocked")) - return f"{status}; {note}" - return ( - f"p50 {format_ns(artifact.get('p50_ns'))}; " - f"p95 {format_ns(artifact.get('p95_ns'))}; " - f"hz {format_hz(artifact.get('hz'))}" - ) - - -def canonical_artifact_path( - output_root: Path, implementation: str, provider: str, case_id: str -) -> Path: - return ( - output_root - / NORMALIZER.implementation_artifact_dir(implementation) - / provider - / f"{case_id.replace('/', '__')}.json" - ) - - -def legacy_artifact_path( - output_root: Path, implementation: str, provider: str, case_id: str -) -> Path: - return ( - output_root - / NORMALIZER.legacy_artifact_dir(implementation) - / provider - / f"{case_id.replace('/', '__')}.json" - ) - - -def legacy_cpu_artifact_path(output_root: Path, implementation: str, case_id: str) -> Path: - return ( - output_root - / NORMALIZER.legacy_artifact_dir(implementation) - / f"{case_id.replace('/', '__')}.json" - ) - - -def relpath(root: Path, path: Path) -> str: - return str(path.relative_to(root)).replace("\\", "/") - - -def locate_artifact( - output_root: Path, implementation: str, provider: str, case_id: str -) -> tuple[dict[str, Any] | None, str]: - canonical_path = canonical_artifact_path(output_root, implementation, provider, case_id) - for candidate in ( - canonical_path, - legacy_artifact_path(output_root, implementation, provider, case_id), - legacy_cpu_artifact_path(output_root, implementation, case_id) - if provider == "cpu" - else None, - ): - if candidate is None: - continue - artifact = load_artifact(candidate) - if artifact is not None: - return artifact, relpath(output_root, candidate) - return None, relpath(output_root, canonical_path) - - -def consistent_field(artifacts: list[dict[str, Any]], field: str) -> str: - values = { - str(value) - for artifact in artifacts - if artifact and (value := artifact.get(field)) not in (None, "") - } - if not values: - return "n/a" - if len(values) == 1: - return values.pop() - return "multiple" - - -def status_class(artifact: dict[str, Any] | None) -> str: - if artifact is None: - return "missing" - if artifact.get("status") == "ok": - return "ok" - return "blocked" - - -def case_group(case_id: str) -> str: - if case_id.startswith("gear_sonic/planner_only_"): - return "Planner only" - if case_id == "gear_sonic/encoder_decoder_only_tracking_tick": - return "Encoder + decoder only" - if case_id.startswith("gear_sonic/full_velocity_tick_"): - return "Full velocity tick" - if case_id == "gear_sonic/end_to_end_cli_loop": - return "GEAR-Sonic end-to-end" - if case_id.startswith("decoupled_wbc/"): - return "Decoupled WBC" - return "Other" - - -def build_provider_section( - registry: dict[str, Any], output_root: Path, provider: str -) -> dict[str, Any]: - artifacts_by_implementation: dict[str, list[dict[str, Any]]] = { - implementation: [] for implementation in IMPLEMENTATION_ORDER - } - rows: list[dict[str, Any]] = [] - - for case in registry["cases"]: - case_id = case["case_id"] - row_artifacts: dict[str, dict[str, Any] | None] = {} - row_relpaths: dict[str, str] = {} - for implementation in IMPLEMENTATION_ORDER: - artifact, artifact_relpath = locate_artifact(output_root, implementation, provider, case_id) - row_artifacts[implementation] = artifact - row_relpaths[implementation] = artifact_relpath - if artifact is not None: - artifacts_by_implementation[implementation].append(artifact) - - rows.append( - { - "provider": format_provider_family(provider), - "provider_id": provider, - "provider_label": format_provider_family(provider), - "case_id": case_id, - "group": case_group(case_id), - "description": str(case.get("description", "")), - "interpretation": str(case["interpretation"]), - "artifacts": row_artifacts, - "artifact_relpaths": row_relpaths, - "ratio": ratio_string( - row_artifacts[RUST_IMPLEMENTATION], - row_artifacts[BASELINE_IMPLEMENTATION], - ), - } - ) - - artifacts = [ - artifact - for implementation in IMPLEMENTATION_ORDER - for artifact in artifacts_by_implementation[implementation] - ] - return { - "provider": format_provider_family(provider), - "provider_id": provider, - "provider_label": format_provider_family(provider), - "rows": rows, - "case_count": len(rows), - "ok_pair_count": sum( - 1 - for row in rows - if row["artifacts"][BASELINE_IMPLEMENTATION] is not None - and row["artifacts"][RUST_IMPLEMENTATION] is not None - and row["artifacts"][BASELINE_IMPLEMENTATION].get("status") == "ok" - and row["artifacts"][RUST_IMPLEMENTATION].get("status") == "ok" - ), - "blocked_count": sum( - 1 - for row in rows - if status_class(row["artifacts"][BASELINE_IMPLEMENTATION]) != "ok" - or status_class(row["artifacts"][RUST_IMPLEMENTATION]) != "ok" - ), - "provenance": { - "robowbc_commit": consistent_field(artifacts, "robowbc_commit"), - "upstream_commit": consistent_field(artifacts, "upstream_commit"), - "host_fingerprint": consistent_field(artifacts, "host_fingerprint"), - }, - } - - -def build_summary(registry: dict[str, Any], output_root: Path) -> dict[str, Any]: - provider_sections = [ - build_provider_section(registry, output_root, provider) for provider in PROVIDER_ORDER - ] - artifacts = [ - artifact - for section in provider_sections - for row in section["rows"] - for artifact in row["artifacts"].values() - if artifact is not None - ] - return { - "providers": [format_provider_family(provider) for provider in PROVIDER_ORDER], - "provider_ids": list(PROVIDER_ORDER), - "implementations": [format_implementation(implementation) for implementation in IMPLEMENTATION_ORDER], - "variants": rendered_variant_labels(), - "provider_sections": provider_sections, - "case_count": len(registry["cases"]), - "row_count": sum(section["case_count"] for section in provider_sections), - "ok_pair_count": sum(section["ok_pair_count"] for section in provider_sections), - "blocked_count": sum(section["blocked_count"] for section in provider_sections), - "provenance": { - "robowbc_commit": consistent_field(artifacts, "robowbc_commit"), - "upstream_commit": consistent_field(artifacts, "upstream_commit"), - "host_fingerprint": consistent_field(artifacts, "host_fingerprint"), - }, - } - - -def rerun_commands() -> str: - return "\n".join( - [ - 'for provider in cpu cuda tensor_rt; do', - ' python3 scripts/bench_robowbc_compare.py --all --provider "$provider"', - ' python3 scripts/bench_nvidia_official.py --all --provider "$provider"', - "done", - "python3 scripts/render_nvidia_benchmark_summary.py --output artifacts/benchmarks/nvidia/SUMMARY.md", - "# or build the full static site bundle:", - "python3 scripts/build_site.py --output-dir /tmp/robowbc-site", - ] - ) - - -def render_markdown(summary: dict[str, Any]) -> str: - provenance = summary["provenance"] - provider_list = ", ".join(f"`{provider}`" for provider in summary["providers"]) - variant_list = ", ".join(f"`{variant}`" for variant in summary["variants"]) - implementation_list = ", ".join(f"`{implementation}`" for implementation in summary["implementations"]) - lines: list[str] = [ - "# NVIDIA Comparison Summary", - "", - "Generated from normalized artifacts under `artifacts/benchmarks/nvidia/`", - "using the tracked case registry in `benchmarks/nvidia/cases.json`.", - "", - "## Provenance", - "", - f"- Variant families rendered: {provider_list}", - f"- Canonical variants: {variant_list}", - f"- Implementations compared: {implementation_list}", - f"- RoboWBC commit: `{provenance['robowbc_commit']}`", - f"- Official upstream commit: `{provenance['upstream_commit']}`", - f"- Host fingerprint: `{provenance['host_fingerprint']}`", - f"- Canonical cases per provider family: `{summary['case_count']}`", - f"- Total rendered rows: `{summary['row_count']}`", - f"- Matched ok pairs: `{summary['ok_pair_count']}`", - f"- Blocked or missing rows: `{summary['blocked_count']}`", - "", - ] - - for section in summary["provider_sections"]: - section_provenance = section["provenance"] - lines.extend( - [ - f"## {section['provider_label']}", - "", - f"- Provider family id: `{section['provider']}`", - f"- Benchmark provider request: `{section['provider_id']}`", - f"- Matched ok pairs: `{section['ok_pair_count']}`", - f"- Blocked or missing rows: `{section['blocked_count']}`", - f"- RoboWBC commit: `{section_provenance['robowbc_commit']}`", - f"- Official upstream commit: `{section_provenance['upstream_commit']}`", - f"- Host fingerprint: `{section_provenance['host_fingerprint']}`", - "", - "| Path Group | Case ID | ORT-cpp-sonic | ORT-rs | ORT-rs / ORT-cpp-sonic (p50) | Why it matters |", - "|------------|---------|----------------|--------|-------------------------------|----------------|", - ] - ) - - for row in section["rows"]: - lines.append( - "| " - + " | ".join( - [ - row["group"], - f"`{row['case_id']}`", - artifact_cell(row["artifacts"][BASELINE_IMPLEMENTATION]), - artifact_cell(row["artifacts"][RUST_IMPLEMENTATION]), - row["ratio"], - row["interpretation"], - ] - ) - + " |" - ) - - lines.extend( - [ - "", - "### Raw Artifacts", - "", - "| Case ID | ORT-cpp-sonic Artifact | ORT-rs Artifact |", - "|---------|-------------------------|-----------------|", - ] - ) - - for row in section["rows"]: - lines.append( - "| " - + " | ".join( - [ - f"`{row['case_id']}`", - f"`{row['artifact_relpaths'][BASELINE_IMPLEMENTATION]}`", - f"`{row['artifact_relpaths'][RUST_IMPLEMENTATION]}`", - ] - ) - + " |" - ) - - lines.append("") - - lines.extend( - [ - "## Rerun", - "", - "```bash", - rerun_commands(), - "```", - "", - "If a future environment is missing models or build prerequisites, the wrappers will emit", - "blocked artifacts instead of silently substituting a different path.", - "", - ] - ) - - return "\n".join(lines) - - -def render_html(summary: dict[str, Any]) -> str: - provenance = summary["provenance"] - provider_label_text = " / ".join(summary["providers"]) - variant_label_text = " / ".join(summary["variants"]) - - def metric_card(label: str, value: str) -> str: - return ( - f"
    {html.escape(label)}" - f"{html.escape(value)}
    " - ) - - def render_section(section: dict[str, Any]) -> str: - rows_html: list[str] = [] - for row in section["rows"]: - ort_cpp_sonic = row["artifacts"][BASELINE_IMPLEMENTATION] - ort_rs = row["artifacts"][RUST_IMPLEMENTATION] - rows_html.append( - f""" - {html.escape(row['group'])} - - {html.escape(row['case_id'])} -
    {html.escape(row['description'])}
    - - {html.escape(artifact_cell(ort_cpp_sonic))} - {html.escape(artifact_cell(ort_rs))} - {html.escape(row['ratio'])} - {html.escape(row['interpretation'])} - - ORT-cpp-sonic JSON
    - ORT-rs JSON - -""" - ) - - section_provenance = section["provenance"] - return f"""
    -
    -
    - {html.escape(section['provider_label'])} -

    {html.escape(section['provider_label'])} Matrix

    -

    This family keeps the ORT-cpp-sonic baseline beside ORT-rs on the same requested provider. Decoupled WBC remains CPU-only in this phase, so non-CPU rows stay blocked instead of being relabeled.

    -
    -
    -
    - {metric_card("Rows", str(section["case_count"]))} - {metric_card("Matched ok pairs", str(section["ok_pair_count"]))} - {metric_card("Blocked or missing rows", str(section["blocked_count"]))} - {metric_card("Provider request", section["provider_id"])} - {metric_card("RoboWBC commit", section_provenance["robowbc_commit"])} - {metric_card("Official upstream commit", section_provenance["upstream_commit"])} -
    -

    Host fingerprint: {html.escape(section_provenance['host_fingerprint'])}

    - - - - - - - - - - - - - - {''.join(rows_html)} - -
    Path groupCaseORT-cpp-sonicORT-rsORT-rs / ORT-cpp-sonic (p50)Why it mattersArtifacts
    -
    """ - - provider_nav = "".join( - f'{html.escape(provider)}' - for provider in summary["providers"] - ) - - return f""" - - - - - RoboWBC NVIDIA Comparison - - - -
    -
    -

    RoboWBC NVIDIA Comparison

    -

    This page is generated automatically from normalized ORT-cpp-sonic and ORT-rs benchmark artifacts. It keeps the benchmark vocabulary aligned with the codebase instead of mixing provider labels with legacy stack names.

    -

    Canonical variant families: {html.escape(provider_label_text)}. Canonical rendered variants: {html.escape(variant_label_text)}. Decoupled WBC remains CPU-only in this phase and stays blocked on non-CPU rows rather than being approximated.

    -
    - Site home - Markdown summary - Case registry - {provider_nav} -
    -
    - {metric_card("Canonical cases per family", str(summary["case_count"]))} - {metric_card("Variant families", provider_label_text)} - {metric_card("Canonical variants", variant_label_text)} - {metric_card("Implementation columns", " / ".join(summary["implementations"]))} - {metric_card("Matched ok pairs", str(summary["ok_pair_count"]))} - {metric_card("Blocked or missing rows", str(summary["blocked_count"]))} - {metric_card("RoboWBC commit", provenance["robowbc_commit"])} - {metric_card("Official upstream commit", provenance["upstream_commit"])} -
    -

    Host fingerprint(s): {html.escape(provenance['host_fingerprint'])}

    -
    - - {''.join(render_section(section) for section in summary["provider_sections"])} - -
    -

    Rerun Commands

    -
    - 
    {html.escape(rerun_commands())}
    -
    -
    -
    - - -""" - - -def main() -> int: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument( - "--registry", - type=Path, - default=Path("benchmarks/nvidia/cases.json"), - ) - parser.add_argument( - "--root", - type=Path, - default=Path("artifacts/benchmarks/nvidia"), - ) - parser.add_argument( - "--output", - type=Path, - default=Path("artifacts/benchmarks/nvidia/SUMMARY.md"), - ) - parser.add_argument( - "--html-output", - type=Path, - help="Optional path for a static HTML report generated from the same artifacts.", - ) - args = parser.parse_args() - - registry = load_registry(args.registry) - summary = build_summary(registry, args.root) - args.output.parent.mkdir(parents=True, exist_ok=True) - args.output.write_text(render_markdown(summary), encoding="utf-8") - if args.html_output is not None: - args.html_output.parent.mkdir(parents=True, exist_ok=True) - args.html_output.write_text(render_html(summary), encoding="utf-8") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) +run_legacy_script("benchmarks/render_nvidia_benchmark_summary.py") diff --git a/scripts/reports/roboharness_report.py b/scripts/reports/roboharness_report.py new file mode 100755 index 0000000..ed2629e --- /dev/null +++ b/scripts/reports/roboharness_report.py @@ -0,0 +1,1949 @@ +#!/usr/bin/env python3 +"""Generate a roboharness-style visual report for a RoboWBC MuJoCo run. + +This script orchestrates: + 1. Pre-flight dependency checks (binary, models, roboharness, mujoco) + 2. RoboWBC CLI run with [report] and [vis] config injection + 3. Post-run frame capture by replaying the saved joint trajectory in MuJoCo + 4. HTML report generation using roboharness's report builder + +Example: + python scripts/reports/roboharness_report.py \\ + --robowbc-binary target/release/robowbc \\ + --config configs/sonic_g1.toml \\ + --output-dir artifacts/roboharness-reports/sonic_g1 +""" + +from __future__ import annotations + +import argparse +import datetime as dt +import json +import math +import os +import re +import subprocess +import sys +import tempfile +import tomllib +from pathlib import Path +from typing import Any + +# Reuse runtime-env helpers from the existing showcase generator. +# We import them lazily so the script can still run --help even if +# generate_policy_showcase.py has heavy dependencies. +_SCRIPTS_DIR = Path(__file__).resolve().parents[1] +PHASE_REVIEW_VERSION = 1 +DEFAULT_PHASE_REVIEW_LAG_TICKS = 3 +DEFAULT_PHASE_REVIEW_TARGET_LAG_TICKS = 0 +MAX_PHASE_REVIEW_LAG_TICKS = 5 +TRACKING_COMMAND_KINDS = { + "kinematic_pose", + "motion_tokens", + "reference_motion_tracking", + "standing_placeholder_tracking", +} + + +def _import_showcase_helpers(): + import importlib.util + + spec = importlib.util.spec_from_file_location( + "generate_policy_showcase", _SCRIPTS_DIR / "site" / "generate_policy_showcase.py" + ) + module = importlib.util.module_from_spec(spec) # type: ignore[arg-type] + spec.loader.exec_module(module) # type: ignore[union-attr] + return module + + +# --------------------------------------------------------------------------- +# CLI +# --------------------------------------------------------------------------- + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Run RoboWBC policy in MuJoCo and produce a roboharness-style visual report." + ) + parser.add_argument("--repo-root", default=".", help="Repository root directory") + parser.add_argument( + "--robowbc-binary", + default="target/release/robowbc", + help="Path to the robowbc binary", + ) + parser.add_argument( + "--config", + default="configs/sonic_g1.toml", + help="Base TOML config for the policy run", + ) + parser.add_argument( + "--output-dir", + required=True, + help="Directory where the HTML report and artifacts will be written", + ) + parser.add_argument( + "--max-ticks", + type=int, + default=None, + help="Override max_ticks in the runtime config", + ) + return parser.parse_args() + + +# --------------------------------------------------------------------------- +# Pre-flight checks +# --------------------------------------------------------------------------- + + +def ensure_headless_mujoco_env() -> None: + """Default MuJoCo to an explicit offscreen backend for headless Linux runs. + + CI and most developer shells that render proof-pack screenshots do not have + a windowing session. MuJoCo's Python renderer needs an explicit offscreen + backend there or `mujoco.Renderer(...)` fails while creating the GL + context. + """ + backend = os.environ.get("MUJOCO_GL") + if backend: + backend = backend.strip().lower() + if backend in {"egl", "osmesa"} and not os.environ.get("PYOPENGL_PLATFORM"): + os.environ["PYOPENGL_PLATFORM"] = backend + return + + if sys.platform != "linux": + return + if os.environ.get("DISPLAY") or os.environ.get("WAYLAND_DISPLAY"): + return + + os.environ["MUJOCO_GL"] = "egl" + os.environ.setdefault("PYOPENGL_PLATFORM", "egl") + + +def _iter_exception_chain(exc: BaseException) -> list[BaseException]: + chain: list[BaseException] = [] + current: BaseException | None = exc + while current is not None and current not in chain: + chain.append(current) + if current.__cause__ is not None: + current = current.__cause__ + elif current.__context__ is not None and not current.__suppress_context__: + current = current.__context__ + else: + current = None + return chain + + +def is_headless_render_backend_error(exc: BaseException) -> bool: + """Return True for environment-specific GL bootstrap failures. + + These failures happen on some headless runners when MuJoCo's Python + renderer cannot initialize the configured offscreen backend. They are + different from deterministic logic bugs in the replay or image-writing + code, which should still fail loudly. + """ + + markers = ( + "eglquerystring", + "pyopengl_platform", + "opengl platform", + "failed to initialize glfw", + "glfwerror", + "glcontext", + "osmesa", + "no display name and no $display", + "cannot create opengl context", + "failed to create opengl context", + ) + for candidate in _iter_exception_chain(exc): + text = f"{type(candidate).__name__}: {candidate}".lower() + if any(marker in text for marker in markers): + return True + return False + + +def frame_capture_warning(exc: BaseException) -> str: + backend = os.environ.get("MUJOCO_GL", "auto") + return ( + "Skipped proof-pack screenshots because the MuJoCo offscreen renderer " + f"could not initialize the configured backend ({backend}). The raw run " + "report, replay trace, and Rerun recording are still available." + ) + + +def preflight_checks(repo_root: Path, binary: Path, config_path: Path) -> dict[str, str]: + """Verify that all required dependencies are present before running. + + Returns the runtime environment dict (with any necessary vars set). + """ + errors: list[str] = [] + ensure_headless_mujoco_env() + + # 1. Binary exists + if not binary.exists(): + errors.append(f"robowbc binary not found: {binary}") + + # 2. Config exists + if not config_path.exists(): + errors.append(f"config file not found: {config_path}") + + # 3. roboharness importable + try: + import roboharness # noqa: F401 + except ImportError: + errors.append( + "roboharness Python package is not installed. " + "Install it (e.g. pip install /path/to/roboharness) before running this script." + ) + + # 4. mujoco importable + try: + import mujoco # noqa: F401 + except ImportError: + errors.append( + "mujoco Python package is not installed. " + "Install it (e.g. pip install mujoco) before running this script." + ) + + # 5. Required model paths from config + if config_path.exists(): + config_text = config_path.read_text(encoding="utf-8") + config = tomllib.loads(config_text) + for key in ("encoder", "decoder", "planner"): + section = config.get("policy", {}).get("config", {}) + if isinstance(section, dict) and key in section: + model_path = section[key].get("model_path") + if model_path: + full_path = repo_root / model_path + if not full_path.exists(): + errors.append(f"required model missing: {model_path}") + + if errors: + raise SystemExit("Pre-flight checks failed:\n - " + "\n - ".join(errors)) + + # Configure runtime environment (ORT dylib, MuJoCo libdir, etc.) + showcase = _import_showcase_helpers() + env = os.environ.copy() + dylib = showcase.resolve_ort_dylib(repo_root) + if dylib: + env.setdefault("ROBOWBC_ORT_DYLIB_PATH", dylib) + env = showcase.configure_binary_runtime_env(env) + return env + + +# --------------------------------------------------------------------------- +# Config composition +# --------------------------------------------------------------------------- + + +def resolve_showcase_context(repo_root: Path, config_path: Path) -> dict[str, Any]: + """Extract simulation context from the base config.""" + app_config = tomllib.loads(config_path.read_text(encoding="utf-8")) + comm_cfg = app_config.get("comm") or app_config.get("communication") or {} + frequency_hz = int(comm_cfg.get("frequency_hz", 50) or 50) + existing_sim = app_config.get("sim") + + robot_cfg_path = app_config.get("robot", {}).get("config_path") + robot_model_path = None + if robot_cfg_path: + robot_cfg = tomllib.loads((repo_root / str(robot_cfg_path)).read_text(encoding="utf-8")) + robot_model_path = robot_cfg.get("model_path") + + timestep = 0.002 + default_substeps = round(1.0 / (max(frequency_hz, 1) * timestep)) + default_gain_profile = "simulation_pd" + + if isinstance(existing_sim, dict): + model_path = str(existing_sim.get("model_path") or robot_model_path or "") + timestep = float(existing_sim.get("timestep", timestep)) + substeps = int(existing_sim.get("substeps", default_substeps)) + gain_profile = str(existing_sim.get("gain_profile") or default_gain_profile) + return { + "transport": "mujoco" if model_path else "synthetic", + "model_path": model_path or None, + "timestep": timestep, + "substeps": substeps, + "gain_profile": gain_profile if model_path else None, + "robot_config_path": str(robot_cfg_path) if robot_cfg_path else None, + "config_has_sim_section": True, + } + + if robot_model_path is None: + return { + "transport": "synthetic", + "model_path": None, + "timestep": None, + "substeps": None, + "gain_profile": None, + "robot_config_path": str(robot_cfg_path) if robot_cfg_path else None, + "config_has_sim_section": False, + } + + return { + "transport": "mujoco", + "model_path": str(robot_model_path), + "timestep": timestep, + "substeps": default_substeps, + "gain_profile": default_gain_profile, + "robot_config_path": str(robot_cfg_path) if robot_cfg_path else None, + "config_has_sim_section": False, + } + + +def ensure_showcase_sim_section(base_toml: str, showcase_context: dict[str, Any]) -> str: + if showcase_context["transport"] != "mujoco": + return base_toml.rstrip() + + required_lines = [ + (r"^model_path\s*=", f'model_path = "{showcase_context["model_path"]}"'), + (r"^timestep\s*=", f'timestep = {showcase_context["timestep"]:g}'), + (r"^substeps\s*=", f'substeps = {showcase_context["substeps"]}'), + ( + r"^gain_profile\s*=", + f'gain_profile = "{showcase_context["gain_profile"]}"', + ), + ] + sim_section_pattern = re.compile(r"^(\[sim\].*?)(?=^\[|\Z)", re.MULTILINE | re.DOTALL) + match = sim_section_pattern.search(base_toml) + if match is None: + sim_lines = ["[sim]"] + sim_lines.extend(line for _, line in required_lines) + return "\n\n".join([base_toml.rstrip(), "\n".join(sim_lines)]) + + sim_section = match.group(1).rstrip() + for pattern, line in required_lines: + if re.search(pattern, sim_section, re.MULTILINE) is None: + sim_section += "\n" + line + sim_section += "\n" + return base_toml[: match.start()] + sim_section + base_toml[match.end() :] + + +def compose_run_config( + base_toml: str, + json_path: Path, + replay_path: Path, + rrd_path: Path, + showcase_context: dict[str, Any], + max_ticks: int | None = None, +) -> str: + sections = [ensure_showcase_sim_section(base_toml, showcase_context).rstrip()] + + # Inject report and vis sections + sections.append( + "\n".join( + [ + "[vis]", + 'app_id = "robowbc-roboharness"', + "spawn_viewer = false", + f'save_path = "{rrd_path.as_posix()}"', + "", + "[report]", + f'output_path = "{json_path.as_posix()}"', + "max_frames = 200", + f'replay_output_path = "{replay_path.as_posix()}"', + ] + ) + ) + + composed = "\n\n".join(sections) + "\n" + + if max_ticks is not None: + # Patch existing [runtime] max_ticks if present, else append a runtime block + runtime_pattern = re.compile(r"^(\[runtime\].*?)(?=^\[|\Z)", re.MULTILINE | re.DOTALL) + max_ticks_pattern = re.compile(r"^(max_ticks\s*=\s*)\d+", re.MULTILINE) + + match = runtime_pattern.search(composed) + if match: + runtime_section = match.group(1) + if max_ticks_pattern.search(runtime_section): + patched = max_ticks_pattern.sub(r"\g<1>" + str(max_ticks), runtime_section) + else: + patched = runtime_section.rstrip() + f"\nmax_ticks = {max_ticks}\n" + composed = composed[: match.start()] + patched + composed[match.end() :] + else: + composed = composed.rstrip() + f"\n\n[runtime]\nmax_ticks = {max_ticks}\n\n" + + return composed + + +# --------------------------------------------------------------------------- +# Run robowbc CLI +# --------------------------------------------------------------------------- + + +def run_robowbc( + repo_root: Path, + binary: Path, + output_dir: Path, + config_path: Path, + env: dict[str, str], + max_ticks: int | None = None, +) -> dict[str, Any]: + showcase_context = resolve_showcase_context(repo_root, config_path) + if showcase_context["transport"] != "mujoco": + raise SystemExit( + "roboharness_report requires a MuJoCo simulation config. " + f"Config {config_path} does not specify a model_path." + ) + + base_config = config_path.read_text(encoding="utf-8") + temp_config = output_dir / "roboharness_run.toml" + json_path = output_dir / "run_report.json" + replay_path = derive_replay_trace_path(json_path) + rrd_path = output_dir / "run_recording.rrd" + log_path = output_dir / "run.log" + + temp_config.write_text( + compose_run_config( + base_config, + json_path, + replay_path, + rrd_path, + showcase_context, + max_ticks, + ), + encoding="utf-8", + ) + + proc = subprocess.run( + [str(binary), "run", "--config", str(temp_config)], + cwd=repo_root, + env=env, + capture_output=True, + text=True, + check=False, + ) + log_text = proc.stdout + "\n--- STDERR ---\n" + proc.stderr + log_path.write_text(log_text, encoding="utf-8") + + if proc.returncode != 0: + raise SystemExit( + f"robowbc run failed with exit code {proc.returncode}; see {log_path}" + ) + if not json_path.exists(): + raise SystemExit( + f"robowbc run did not write the expected report JSON at {json_path}; see {log_path}" + ) + + report = json.loads(json_path.read_text(encoding="utf-8")) + replay_trace = None + if replay_path.exists(): + replay_trace = json.loads(replay_path.read_text(encoding="utf-8")) + + report["_meta"] = { + "config_path": str(config_path), + "log_path": str(log_path), + "rrd_path": str(rrd_path), + "json_path": str(json_path), + "replay_trace_path": str(replay_path), + "replay_trace_present": replay_trace is not None, + "temp_config": str(temp_config), + "showcase_context": showcase_context, + } + if replay_trace is not None: + report["_replay_trace"] = replay_trace + return report + + +# --------------------------------------------------------------------------- +# MuJoCo frame capture (replay trajectory) +# --------------------------------------------------------------------------- + + +def derive_replay_trace_path(report_path: Path) -> Path: + stem = report_path.stem or "run_report" + suffix = report_path.suffix or ".json" + return report_path.with_name(f"{stem}_replay_trace{suffix}") + + +def load_meshless_mujoco_model(model_path: Path) -> tuple[Any, Any]: + """Load a MuJoCo model, falling back to visible proxy geoms if meshes are missing.""" + import mujoco + import xml.etree.ElementTree as ET + + try: + model = mujoco.MjModel.from_xml_path(str(model_path)) + data = mujoco.MjData(model) + return model, data + except Exception: + pass + + # Build meshless fallback by stripping mesh assets and replacing mesh geoms + # with simple proxy spheres so proof-pack capture still shows a readable body. + xml_text = model_path.read_text(encoding="utf-8") + root = ET.fromstring(xml_text) + + # Remove all elements from + for asset in root.findall("asset"): + for mesh in asset.findall("mesh"): + asset.remove(mesh) + + for geom in root.iter("geom"): + mesh_name = geom.attrib.pop("mesh", None) + if mesh_name is None: + continue + geom.attrib["type"] = "sphere" + geom.attrib["size"] = _meshless_proxy_size(mesh_name) + geom.attrib["rgba"] = _meshless_proxy_rgba(geom.attrib.get("rgba")) + + # Serialize back to string + meshless_xml = ET.tostring(root, encoding="unicode") + model = mujoco.MjModel.from_xml_string(meshless_xml) + data = mujoco.MjData(model) + return model, data + + +def _meshless_proxy_size(mesh_name: str) -> str: + lowered = mesh_name.lower() + if any(token in lowered for token in ("pelvis", "torso", "waist", "head")): + return "0.055" + if any(token in lowered for token in ("thigh", "hip", "knee")): + return "0.05" + if any(token in lowered for token in ("ankle", "foot")): + return "0.045" + if any(token in lowered for token in ("shoulder", "upper_arm", "lower_arm", "elbow")): + return "0.042" + if any(token in lowered for token in ("wrist", "hand", "finger")): + return "0.028" + return "0.04" + + +def _meshless_proxy_rgba(original_rgba: str | None) -> str: + if not original_rgba: + return "0.82 0.82 0.86 1" + + parts = original_rgba.split() + if len(parts) != 4: + return "0.82 0.82 0.86 1" + + try: + rgb = [float(parts[0]), float(parts[1]), float(parts[2])] + alpha = float(parts[3]) + except ValueError: + return "0.82 0.82 0.86 1" + + lifted = [min(1.0, 0.45 + channel * 0.55) for channel in rgb] + return f"{lifted[0]:.3f} {lifted[1]:.3f} {lifted[2]:.3f} {max(alpha, 1.0):.3f}" + + +def replay_payload(report: dict[str, Any]) -> tuple[dict[str, Any], list[dict[str, Any]], str]: + replay_trace = report.get("_replay_trace") + if isinstance(replay_trace, dict): + frames = replay_trace.get("frames") + if isinstance(frames, list) and frames: + return replay_trace, frames, "canonical_replay_trace" + + frames = report.get("frames", []) + if isinstance(frames, list): + return report, frames, "run_report_frames" + + return report, [], "run_report_frames" + + +def frame_sim_time_secs(frame: dict[str, Any], control_frequency_hz: int) -> float: + sim_time_secs = frame.get("sim_time_secs") + if isinstance(sim_time_secs, (int, float)): + return float(sim_time_secs) + + tick = int(frame.get("tick", 0) or 0) + if control_frequency_hz <= 0: + return float(tick) + return tick / float(control_frequency_hz) + + +def planar_position(frame: dict[str, Any]) -> tuple[float, float] | None: + base_pose = frame.get("base_pose") + if isinstance(base_pose, dict): + position_world = base_pose.get("position_world") + if isinstance(position_world, list) and len(position_world) >= 2: + return float(position_world[0]), float(position_world[1]) + + qpos = frame.get("mujoco_qpos") + if isinstance(qpos, list) and len(qpos) >= 2: + return float(qpos[0]), float(qpos[1]) + + return None + + +def planar_displacement(reference: dict[str, Any], frame: dict[str, Any]) -> float | None: + reference_position = planar_position(reference) + current_position = planar_position(frame) + if reference_position is None or current_position is None: + return None + + return math.hypot( + current_position[0] - reference_position[0], + current_position[1] - reference_position[1], + ) + + +def mean_joint_delta(reference: dict[str, Any], frame: dict[str, Any]) -> float | None: + reference_positions = reference.get("actual_positions") + current_positions = frame.get("actual_positions") + if not ( + isinstance(reference_positions, list) + and isinstance(current_positions, list) + and reference_positions + and len(reference_positions) == len(current_positions) + ): + return None + + total = 0.0 + for reference_position, current_position in zip(reference_positions, current_positions): + total += abs(float(current_position) - float(reference_position)) + return total / len(reference_positions) + + +def lag_ticks_to_ms(lag_ticks: int, control_frequency_hz: int) -> float: + if control_frequency_hz <= 0: + return float(lag_ticks) + return (float(lag_ticks) * 1_000.0) / float(control_frequency_hz) + + +def normalize_phase_name(name: object, *, source: str, index: int) -> str: + if not isinstance(name, str): + raise SystemExit(f"{source}: phase #{index + 1} is missing a string phase name") + + trimmed = name.strip() + if not trimmed: + raise SystemExit(f"{source}: phase #{index + 1} has an empty phase name") + if "/" in trimmed or "\\" in trimmed or ".." in trimmed: + raise SystemExit( + f"{source}: phase {trimmed!r} must not contain path separators or '..'" + ) + if any(ord(ch) < 32 for ch in trimmed): + raise SystemExit(f"{source}: phase {trimmed!r} contains control characters") + return trimmed + + +def slugify_phase_name(name: str) -> str: + slug = re.sub(r"[^a-z0-9]+", "-", name.lower()).strip("-") + if not slug: + raise SystemExit(f"phase {name!r} did not produce a safe directory slug") + return slug + + +def normalize_phase_timeline_entries( + entries: object, + *, + frame_count: int, + control_frequency_hz: int, + source: str, +) -> list[dict[str, object]]: + if not isinstance(entries, list) or not entries: + raise SystemExit(f"{source}: phase timeline must be a non-empty list") + + normalized: list[dict[str, object]] = [] + seen_names: set[str] = set() + previous_end = -1 + for index, raw_entry in enumerate(entries): + if not isinstance(raw_entry, dict): + raise SystemExit(f"{source}: phase #{index + 1} must be a TOML/JSON object") + + phase_name = normalize_phase_name( + raw_entry.get("phase_name", raw_entry.get("name")), + source=source, + index=index, + ) + if phase_name in seen_names: + raise SystemExit(f"{source}: duplicate phase name {phase_name!r}") + seen_names.add(phase_name) + + start_tick = raw_entry.get("start_tick") + end_tick = raw_entry.get("end_tick") + if not isinstance(start_tick, int) or start_tick < 0: + raise SystemExit(f"{source}: phase {phase_name!r} must define start_tick >= 0") + if not isinstance(end_tick, int) or end_tick < start_tick: + raise SystemExit( + f"{source}: phase {phase_name!r} must define end_tick >= start_tick" + ) + if start_tick <= previous_end: + raise SystemExit( + f"{source}: phase {phase_name!r} overlaps or is out of order relative to the previous phase" + ) + if end_tick >= frame_count: + raise SystemExit( + f"{source}: phase {phase_name!r} ends at tick {end_tick}, but only {frame_count} frames were recorded" + ) + + midpoint_tick = raw_entry.get("midpoint_tick") + expected_midpoint = start_tick + ((end_tick - start_tick) // 2) + if midpoint_tick is None: + midpoint_tick = expected_midpoint + elif not isinstance(midpoint_tick, int) or midpoint_tick != expected_midpoint: + raise SystemExit( + f"{source}: phase {phase_name!r} midpoint_tick must equal the canonical midpoint {expected_midpoint}" + ) + + duration_ticks = end_tick - start_tick + 1 + raw_duration_ticks = raw_entry.get("duration_ticks") + if raw_duration_ticks is not None and raw_duration_ticks != duration_ticks: + raise SystemExit( + f"{source}: phase {phase_name!r} duration_ticks must equal {duration_ticks}" + ) + + duration_secs = raw_entry.get("duration_secs") + if duration_secs is None: + duration_secs = duration_ticks / float(max(control_frequency_hz, 1)) + elif not isinstance(duration_secs, (int, float)) or float(duration_secs) <= 0.0: + raise SystemExit( + f"{source}: phase {phase_name!r} must define a positive duration_secs when provided" + ) + + normalized.append( + { + "phase_name": phase_name, + "phase_slug": slugify_phase_name(phase_name), + "start_tick": start_tick, + "midpoint_tick": midpoint_tick, + "end_tick": end_tick, + "duration_ticks": duration_ticks, + "duration_secs": float(duration_secs), + } + ) + previous_end = end_tick + + return normalized + + +def resolve_report_config_path(repo_root: Path, report: dict[str, Any]) -> Path | None: + meta = report.get("_meta") + if not isinstance(meta, dict): + return None + + config_path = meta.get("config_path") + if not isinstance(config_path, str) or not config_path: + return None + + candidate = Path(config_path) + if not candidate.is_absolute(): + candidate = (repo_root / candidate).resolve() + else: + candidate = candidate.resolve() + + try: + candidate.relative_to(repo_root.resolve()) + except ValueError as exc: + raise SystemExit( + f"tracking phase sidecars must stay inside the repo root; got config path {candidate}" + ) from exc + return candidate + + +def load_tracking_phase_review_contract( + sidecar_path: Path, + *, + frame_count: int, + control_frequency_hz: int, +) -> dict[str, object]: + raw = tomllib.loads(sidecar_path.read_text(encoding="utf-8")) + default_lag_ticks = raw.get("default_lag_ticks", DEFAULT_PHASE_REVIEW_LAG_TICKS) + if ( + not isinstance(default_lag_ticks, int) + or default_lag_ticks < 0 + or default_lag_ticks > MAX_PHASE_REVIEW_LAG_TICKS + ): + raise SystemExit( + f"{sidecar_path}: default_lag_ticks must be an integer between 0 and {MAX_PHASE_REVIEW_LAG_TICKS}" + ) + + normalized_timeline = normalize_phase_timeline_entries( + raw.get("phases"), + frame_count=frame_count, + control_frequency_hz=control_frequency_hz, + source=str(sidecar_path), + ) + return { + "source": "tracking_sidecar", + "default_lag_ticks": default_lag_ticks, + "default_lag_ms": lag_ticks_to_ms(default_lag_ticks, control_frequency_hz), + "lag_options": list(range(MAX_PHASE_REVIEW_LAG_TICKS + 1)), + "default_target_lag_ticks": DEFAULT_PHASE_REVIEW_TARGET_LAG_TICKS, + "default_target_lag_ms": lag_ticks_to_ms( + DEFAULT_PHASE_REVIEW_TARGET_LAG_TICKS, control_frequency_hz + ), + "target_lag_options": list(range(MAX_PHASE_REVIEW_LAG_TICKS + 1)), + "phase_timeline": normalized_timeline, + } + + +def resolve_phase_review_contract(repo_root: Path, report: dict[str, Any]) -> dict[str, object] | None: + replay_root, frames, _ = replay_payload(report) + if not frames: + return None + + control_frequency_hz = int(replay_root.get("control_frequency_hz", 50) or 50) + command_kind = str(report.get("command_kind") or replay_root.get("command_kind") or "") + + if command_kind == "velocity_schedule": + raw_timeline = report.get("phase_timeline", replay_root.get("phase_timeline")) + if raw_timeline is None: + return None + normalized_timeline = normalize_phase_timeline_entries( + raw_timeline, + frame_count=len(frames), + control_frequency_hz=control_frequency_hz, + source="run artifact phase_timeline", + ) + return { + "source": "velocity_schedule", + "default_lag_ticks": DEFAULT_PHASE_REVIEW_LAG_TICKS, + "default_lag_ms": lag_ticks_to_ms( + DEFAULT_PHASE_REVIEW_LAG_TICKS, control_frequency_hz + ), + "lag_options": list(range(MAX_PHASE_REVIEW_LAG_TICKS + 1)), + "default_target_lag_ticks": DEFAULT_PHASE_REVIEW_TARGET_LAG_TICKS, + "default_target_lag_ms": lag_ticks_to_ms( + DEFAULT_PHASE_REVIEW_TARGET_LAG_TICKS, control_frequency_hz + ), + "target_lag_options": list(range(MAX_PHASE_REVIEW_LAG_TICKS + 1)), + "phase_timeline": normalized_timeline, + } + + if command_kind not in TRACKING_COMMAND_KINDS: + return None + + config_path = resolve_report_config_path(repo_root, report) + if config_path is None: + return None + + sidecar_path = config_path.with_suffix(".phases.toml") + if not sidecar_path.exists(): + return None + if sidecar_path.parent != config_path.parent: + raise SystemExit( + f"tracking phase sidecar must be a sibling of {config_path}; got {sidecar_path}" + ) + + return load_tracking_phase_review_contract( + sidecar_path.resolve(), + frame_count=len(frames), + control_frequency_hz=control_frequency_hz, + ) + + +def build_phase_review_capture_plan( + frames: list[dict[str, Any]], + phase_review: dict[str, object], + *, + frame_source: str, + control_frequency_hz: int, +) -> list[dict[str, object]]: + plan: list[dict[str, object]] = [] + timeline = phase_review["phase_timeline"] + assert isinstance(timeline, list) + lag_options = phase_review["lag_options"] + assert isinstance(lag_options, list) + default_lag_ticks = int(phase_review["default_lag_ticks"]) + target_lag_options = phase_review.get("target_lag_options", lag_options) + assert isinstance(target_lag_options, list) + default_target_lag_ticks = int( + phase_review.get("default_target_lag_ticks", DEFAULT_PHASE_REVIEW_TARGET_LAG_TICKS) + ) + + for phase_entry in timeline: + assert isinstance(phase_entry, dict) + phase_name = str(phase_entry["phase_name"]) + phase_slug = str(phase_entry["phase_slug"]) + midpoint_tick = int(phase_entry["midpoint_tick"]) + end_tick = int(phase_entry["end_tick"]) + + plan.append( + { + "kind": "phase_midpoint", + "name": f"{phase_name}_midpoint", + "phase_name": phase_name, + "phase_slug": phase_slug, + "tick": midpoint_tick, + "frame_index": midpoint_tick, + "selection_reason": f"{phase_name} midpoint from the explicit phase timeline", + "frame_source": frame_source, + } + ) + + available_lag_options = [ + int(lag) + for lag in lag_options + if isinstance(lag, int) and 0 <= lag <= MAX_PHASE_REVIEW_LAG_TICKS + and end_tick + lag < len(frames) + ] + if not available_lag_options: + raise SystemExit( + f"phase {phase_name!r} has no recorded frames at or after phase end tick {end_tick}" + ) + default_display_lag = ( + default_lag_ticks + if default_lag_ticks in available_lag_options + else available_lag_options[-1] + ) + available_target_lag_options = [ + int(lag) + for lag in target_lag_options + if isinstance(lag, int) and 0 <= lag <= MAX_PHASE_REVIEW_LAG_TICKS + and end_tick + lag < len(frames) + ] + if not available_target_lag_options: + raise SystemExit( + f"phase {phase_name!r} has no recorded target frames at or after phase end tick {end_tick}" + ) + default_target_display_lag = ( + default_target_lag_ticks + if default_target_lag_ticks in available_target_lag_options + else available_target_lag_options[-1] + ) + variants = [ + { + "lag_ticks": lag_ticks, + "lag_ms": lag_ticks_to_ms(lag_ticks, control_frequency_hz), + "frame_index": end_tick + lag_ticks, + "tick": int(frames[end_tick + lag_ticks]["tick"]), + "selection_reason": ( + f"{phase_name} phase end actual response at +{lag_ticks} ticks" + ), + "frame_source": frame_source, + } + for lag_ticks in available_lag_options + ] + plan.append( + { + "kind": "phase_end", + "name": f"{phase_name}_end", + "phase_name": phase_name, + "phase_slug": phase_slug, + "phase_end_tick": end_tick, + "frame_index": end_tick, + "selection_reason": ( + f"{phase_name} phase end with positive-lag actual-response review" + ), + "frame_source": frame_source, + "lag_options": available_lag_options, + "default_display_lag": default_display_lag, + "target_lag_options": available_target_lag_options, + "default_target_display_lag": default_target_display_lag, + "variants": variants, + } + ) + + return plan + + +def select_checkpoint_specs( + frames: list[dict[str, Any]], +) -> list[dict[str, Any]]: + if not frames: + return [] + + target_checkpoint_count = 5 + first_frame = frames[0] + last_index = len(frames) - 1 + candidates: list[dict[str, Any]] = [ + { + "name": "start", + "index": 0, + "reason": "initial state before meaningful motion", + } + ] + + first_motion: dict[str, Any] | None = None + for index, frame in enumerate(frames[1:], start=1): + displacement = planar_displacement(first_frame, frame) + if displacement is not None and displacement >= 0.05: + first_motion = { + "name": "first_motion", + "index": index, + "reason": f"first floating-base displacement >= 0.05 m ({displacement:.3f} m)", + } + break + + joint_delta = mean_joint_delta(first_frame, frame) + if joint_delta is not None and joint_delta >= 0.08: + first_motion = { + "name": "first_motion", + "index": index, + "reason": f"first mean joint delta >= 0.08 rad ({joint_delta:.3f} rad)", + } + break + + if first_motion is not None: + candidates.append(first_motion) + + peak_latency_index = max( + range(len(frames)), + key=lambda index: float(frames[index].get("inference_latency_ms", 0.0) or 0.0), + ) + peak_latency_ms = float( + frames[peak_latency_index].get("inference_latency_ms", 0.0) or 0.0 + ) + candidates.append( + { + "name": "peak_latency", + "index": peak_latency_index, + "reason": f"highest inference latency ({peak_latency_ms:.3f} ms)", + } + ) + + furthest_progress: dict[str, Any] | None = None + best_planar_displacement = -1.0 + for index, frame in enumerate(frames[1:], start=1): + displacement = planar_displacement(first_frame, frame) + if displacement is not None and displacement > best_planar_displacement: + best_planar_displacement = displacement + furthest_progress = { + "name": "furthest_progress", + "index": index, + "reason": f"largest planar displacement from start ({displacement:.3f} m)", + } + + if furthest_progress is None: + best_joint_delta = -1.0 + for index, frame in enumerate(frames[1:], start=1): + joint_delta = mean_joint_delta(first_frame, frame) + if joint_delta is not None and joint_delta > best_joint_delta: + best_joint_delta = joint_delta + furthest_progress = { + "name": "furthest_progress", + "index": index, + "reason": f"largest mean joint delta from start ({joint_delta:.3f} rad)", + } + + if furthest_progress is not None: + candidates.append(furthest_progress) + + candidates.append( + { + "name": "final", + "index": last_index, + "reason": "final recorded simulator state", + } + ) + + selected: list[dict[str, Any]] = [] + used_indices: set[int] = set() + for candidate in candidates: + index = int(candidate["index"]) + if index in used_indices: + continue + selected.append(candidate) + used_indices.add(index) + + fallback_specs = [ + ("fallback_mid_25", 0.25), + ("fallback_mid_50", 0.50), + ("fallback_mid_75", 0.75), + ] + for name, fraction in fallback_specs: + if len(selected) >= target_checkpoint_count or last_index <= 0: + break + index = int(round(fraction * last_index)) + if index in used_indices: + continue + selected.append( + { + "name": name, + "index": index, + "reason": ( + f"deterministic fallback at {int(fraction * 100)}% of the replay " + "because the evidence checkpoints collapsed to the same frame" + ), + } + ) + used_indices.add(index) + + return sorted(selected, key=lambda checkpoint: (int(checkpoint["index"]), checkpoint["name"])) + + +def restore_frame_state( + model: Any, + data: Any, + frame: dict[str, Any], + joint_names: list[str], + default_pose: list[float], + joint_qpos_map: dict[str, int], + joint_qvel_map: dict[str, int], + floating_base_state: dict[str, int] | None, +) -> None: + import mujoco + + mujoco.mj_resetData(model, data) + + qpos = frame.get("mujoco_qpos") + qvel = frame.get("mujoco_qvel") + if ( + isinstance(qpos, list) + and isinstance(qvel, list) + and len(qpos) == model.nq + and len(qvel) == model.nv + ): + data.qpos[:] = qpos + data.qvel[:] = qvel + else: + for jnt_id in range(model.njnt): + qpos_adr = int(model.jnt_qposadr[jnt_id]) + name = mujoco.mj_id2name(model, mujoco.mjtObj.mjOBJ_JOINT, jnt_id) + if name in joint_names: + default_index = joint_names.index(name) + data.qpos[qpos_adr] = default_pose[default_index] + + if floating_base_state is not None: + base_pose = frame.get("base_pose") + if isinstance(base_pose, dict): + position_world = base_pose.get("position_world") + rotation_xyzw = base_pose.get("rotation_xyzw") + if ( + isinstance(position_world, list) + and len(position_world) >= 3 + and isinstance(rotation_xyzw, list) + and len(rotation_xyzw) == 4 + ): + base_qpos = [ + float(position_world[0]), + float(position_world[1]), + float(position_world[2]), + float(rotation_xyzw[3]), + float(rotation_xyzw[0]), + float(rotation_xyzw[1]), + float(rotation_xyzw[2]), + ] + qpos_adr = floating_base_state["qpos_adr"] + data.qpos[qpos_adr : qpos_adr + 7] = base_qpos + qvel_adr = floating_base_state["qvel_adr"] + data.qvel[qvel_adr : qvel_adr + 6] = [0.0] * 6 + + positions = frame.get("actual_positions", []) + if isinstance(positions, list): + for joint_name, joint_position in zip(joint_names, positions): + if joint_name in joint_qpos_map: + data.qpos[joint_qpos_map[joint_name]] = float(joint_position) + + velocities = frame.get("actual_velocities", []) + if isinstance(velocities, list): + for joint_name, joint_velocity in zip(joint_names, velocities): + if joint_name in joint_qvel_map: + data.qvel[joint_qvel_map[joint_name]] = float(joint_velocity) + + sim_time_secs = frame.get("sim_time_secs") + if isinstance(sim_time_secs, (int, float)): + data.time = float(sim_time_secs) + + mujoco.mj_forward(model, data) + + +def phase_review_camera_configs(phase_review: dict[str, object] | None) -> list[tuple[str, Any]]: + if isinstance(phase_review, dict) and phase_review.get("source") == "velocity_schedule": + return [ + ("track", _phase_review_track_camera()), + ("side", _side_camera()), + ("top", _phase_review_top_camera()), + ] + return [ + ("track", _track_camera()), + ("side", _side_camera()), + ("top", _top_camera()), + ] + + +def capture_overlay_images( + *, + model: Any, + data: Any, + renderer: Any, + actual_frame: dict[str, Any], + target_frame: dict[str, Any] | None, + joint_names: list[str], + default_pose: list[float], + joint_qpos_map: dict[str, int], + joint_qvel_map: dict[str, int], + floating_base_state: dict[str, int] | None, + camera_configs: list[tuple[str, Any]], + output_dir: Path, + require_target: bool, +) -> list[str]: + if require_target and target_frame is None: + raise SystemExit( + f"phase-aware checkpoint at tick {actual_frame.get('tick')} is missing target_positions" + ) + + cameras: list[str] = [] + for cam_name, cam_obj in camera_configs: + restore_frame_state( + model, + data, + actual_frame, + joint_names, + default_pose, + joint_qpos_map, + joint_qvel_map, + floating_base_state, + ) + renderer.update_scene(data, camera=cam_obj) + actual_rgb = renderer.render() + + image_path = output_dir / f"{cam_name}_rgb.png" + _save_png(output_dir / f"{cam_name}_actual_rgb.png", actual_rgb) + if target_frame is None: + _save_png(image_path, actual_rgb) + else: + restore_frame_state( + model, + data, + target_frame, + joint_names, + default_pose, + joint_qpos_map, + joint_qvel_map, + floating_base_state, + ) + renderer.update_scene(data, camera=cam_obj) + target_rgb = renderer.render() + _save_png(output_dir / f"{cam_name}_target_rgb.png", target_rgb) + _save_comparison_png(image_path, actual_rgb, target_rgb) + cameras.append(cam_name) + + return cameras + + +def write_checkpoint_metadata( + *, + checkpoint_dir: Path, + tick: int, + sim_time_secs: float, + cameras: list[str], + selection_reason: str, + frame_source: str, + comparison_mode: str, + phase_name: str | None = None, + phase_end_tick: int | None = None, + lag_ticks: int | None = None, +) -> None: + metadata = { + "step": tick, + "sim_time": sim_time_secs, + "cameras": cameras, + "camera_capability": "rgb", + "comparison_mode": comparison_mode, + "selection_reason": selection_reason, + "frame_source": frame_source, + } + if phase_name is not None: + metadata["phase_name"] = phase_name + if phase_end_tick is not None: + metadata["phase_end_tick"] = phase_end_tick + if lag_ticks is not None: + metadata["lag_ticks"] = lag_ticks + + (checkpoint_dir / "metadata.json").write_text( + json.dumps(metadata), + encoding="utf-8", + ) + + +def capture_frames_from_report( + repo_root: Path, + report: dict[str, Any], + output_dir: Path, +) -> list[dict[str, Any]]: + """Replay the recorded trajectory in MuJoCo and capture screenshots. + + Returns a list of checkpoint metadata dicts for the HTML report. + """ + ensure_headless_mujoco_env() + renderer = None + try: + import mujoco + + showcase_context = report["_meta"]["showcase_context"] + model_path = repo_root / showcase_context["model_path"] + robot_cfg_path = repo_root / showcase_context["robot_config_path"] + robot_cfg = tomllib.loads(robot_cfg_path.read_text(encoding="utf-8")) + joint_names = robot_cfg.get("joint_names", []) + default_pose = robot_cfg.get("default_pose", [0.0] * len(joint_names)) + + model, data = load_meshless_mujoco_model(model_path) + renderer = mujoco.Renderer(model, height=480, width=640) + replay_root, frames, frame_source = replay_payload(report) + control_frequency_hz = int(replay_root.get("control_frequency_hz", 50) or 50) + phase_review = resolve_phase_review_contract(repo_root, report) + if phase_review is not None: + report["_phase_review"] = { + "source": phase_review["source"], + "default_lag_ticks": phase_review["default_lag_ticks"], + "default_lag_ms": phase_review["default_lag_ms"], + "lag_options": list(phase_review["lag_options"]), + "default_target_lag_ticks": phase_review["default_target_lag_ticks"], + "default_target_lag_ms": phase_review["default_target_lag_ms"], + "target_lag_options": list(phase_review["target_lag_options"]), + "phase_timeline": list(phase_review["phase_timeline"]), + } + + # Build joint name -> qpos address mapping + joint_qpos_map: dict[str, int] = {} + joint_qvel_map: dict[str, int] = {} + floating_base_state: dict[str, int] | None = None + for jnt_id in range(model.njnt): + name = mujoco.mj_id2name(model, mujoco.mjtObj.mjOBJ_JOINT, jnt_id) + qpos_adr = model.jnt_qposadr[jnt_id] + qvel_adr = model.jnt_dofadr[jnt_id] + joint_qpos_map[name] = int(qpos_adr) + joint_qvel_map[name] = int(qvel_adr) + if ( + floating_base_state is None + and model.jnt_type[jnt_id] == mujoco.mjtJoint.mjJNT_FREE + ): + floating_base_state = { + "qpos_adr": int(qpos_adr), + "qvel_adr": int(qvel_adr), + } + + if not frames: + return [] + + capture_plan: list[dict[str, Any]] = [] + primary_ticks: set[int] = set() + if phase_review is not None: + capture_plan.extend( + build_phase_review_capture_plan( + frames, + phase_review, + frame_source=frame_source, + control_frequency_hz=control_frequency_hz, + ) + ) + for checkpoint in capture_plan: + if checkpoint["kind"] == "phase_midpoint": + primary_ticks.add(int(checkpoint["tick"])) + elif checkpoint["kind"] == "phase_end": + primary_ticks.add(int(checkpoint["phase_end_tick"])) + + for checkpoint in select_checkpoint_specs(frames): + frame_index = int(checkpoint["index"]) + if frame_index in primary_ticks: + continue + frame = frames[frame_index] + capture_plan.append( + { + "kind": "diagnostic", + "name": str(checkpoint["name"]), + "tick": int(frame["tick"]), + "frame_index": frame_index, + "selection_reason": str(checkpoint["reason"]), + "frame_source": frame_source, + } + ) + + # roboharness expects: output_dir / task_name / trial_name / checkpoint_name + capture_dir = output_dir / "roboharness_run" / "trial_001" + capture_dir.mkdir(parents=True, exist_ok=True) + + checkpoints: list[dict[str, Any]] = [] + camera_configs = phase_review_camera_configs(phase_review) + + for checkpoint in capture_plan: + kind = str(checkpoint["kind"]) + if kind in {"diagnostic", "phase_midpoint"}: + frame_index = int(checkpoint["frame_index"]) + frame = frames[frame_index] + target_frame = _target_pose_frame(frame) + phase_name = checkpoint.get("phase_name") + if kind == "phase_midpoint": + cp_name = f"{checkpoint['phase_slug']}_midpoint_tick_{int(frame['tick']):04d}" + else: + cp_name = f"{checkpoint['name']}_tick_{int(frame['tick']):04d}" + cp_dir = capture_dir / cp_name + cp_dir.mkdir(parents=True, exist_ok=True) + sim_time_secs = frame_sim_time_secs(frame, control_frequency_hz) + cameras = capture_overlay_images( + model=model, + data=data, + renderer=renderer, + actual_frame=frame, + target_frame=target_frame, + joint_names=joint_names, + default_pose=default_pose, + joint_qpos_map=joint_qpos_map, + joint_qvel_map=joint_qvel_map, + floating_base_state=floating_base_state, + camera_configs=camera_configs, + output_dir=cp_dir, + require_target=kind == "phase_midpoint", + ) + write_checkpoint_metadata( + checkpoint_dir=cp_dir, + tick=int(frame["tick"]), + sim_time_secs=sim_time_secs, + cameras=cameras, + selection_reason=str(checkpoint["selection_reason"]), + frame_source=frame_source, + comparison_mode="target_vs_actual_overlay" + if target_frame is not None + else "actual_only", + phase_name=str(phase_name) if isinstance(phase_name, str) else None, + ) + checkpoints.append( + { + "kind": kind, + "phase_name": phase_name if isinstance(phase_name, str) else None, + "phase_kind": "midpoint" if kind == "phase_midpoint" else None, + "name": str(checkpoint["name"]), + "dir": cp_dir, + "relative_dir": cp_dir.relative_to(output_dir).as_posix(), + "meta": { + "tick": int(frame["tick"]), + "frame_index": frame_index, + "sim_time_secs": sim_time_secs, + "inference_latency_ms": float( + frame.get("inference_latency_ms", 0.0) or 0.0 + ), + "selection_reason": str(checkpoint["selection_reason"]), + "frame_source": frame_source, + "cameras": cameras, + }, + } + ) + continue + + if kind != "phase_end": + raise SystemExit(f"unknown checkpoint kind: {kind}") + + phase_name = str(checkpoint["phase_name"]) + phase_slug = str(checkpoint["phase_slug"]) + phase_end_tick = int(checkpoint["phase_end_tick"]) + canonical_target_source_frame = frames[int(checkpoint["frame_index"])] + target_frame = _target_pose_frame(canonical_target_source_frame) + cp_root = capture_dir / f"{phase_slug}_end_tick_{phase_end_tick:04d}" + cp_root.mkdir(parents=True, exist_ok=True) + lag_variants: list[dict[str, Any]] = [] + for variant in checkpoint["variants"]: + assert isinstance(variant, dict) + lag_ticks = int(variant["lag_ticks"]) + actual_frame = frames[int(variant["frame_index"])] + variant_dir = cp_root / f"lag_{lag_ticks}" + variant_dir.mkdir(parents=True, exist_ok=True) + cameras = capture_overlay_images( + model=model, + data=data, + renderer=renderer, + actual_frame=actual_frame, + target_frame=target_frame, + joint_names=joint_names, + default_pose=default_pose, + joint_qpos_map=joint_qpos_map, + joint_qvel_map=joint_qvel_map, + floating_base_state=floating_base_state, + camera_configs=camera_configs, + output_dir=variant_dir, + require_target=True, + ) + sim_time_secs = frame_sim_time_secs(actual_frame, control_frequency_hz) + write_checkpoint_metadata( + checkpoint_dir=variant_dir, + tick=int(actual_frame["tick"]), + sim_time_secs=sim_time_secs, + cameras=cameras, + selection_reason=str(variant["selection_reason"]), + frame_source=frame_source, + comparison_mode="target_vs_actual_overlay", + phase_name=phase_name, + phase_end_tick=phase_end_tick, + lag_ticks=lag_ticks, + ) + lag_variants.append( + { + "lag_ticks": lag_ticks, + "lag_ms": float(variant["lag_ms"]), + "tick": int(actual_frame["tick"]), + "frame_index": int(variant["frame_index"]), + "sim_time_secs": sim_time_secs, + "selection_reason": str(variant["selection_reason"]), + "frame_source": frame_source, + "relative_dir": variant_dir.relative_to(output_dir).as_posix(), + "cameras": cameras, + } + ) + + default_display_lag = int(checkpoint["default_display_lag"]) + default_variant = next( + ( + variant + for variant in lag_variants + if int(variant["lag_ticks"]) == default_display_lag + ), + lag_variants[-1], + ) + checkpoints.append( + { + "kind": "phase_end", + "phase_name": phase_name, + "phase_kind": "phase_end", + "name": str(checkpoint["name"]), + "dir": cp_root, + "relative_dir": str(default_variant["relative_dir"]), + "meta": { + "tick": phase_end_tick, + "frame_index": int(checkpoint["frame_index"]), + "phase_end_tick": phase_end_tick, + "sim_time_secs": frame_sim_time_secs( + canonical_target_source_frame, control_frequency_hz + ), + "selection_reason": str(checkpoint["selection_reason"]), + "frame_source": frame_source, + "cameras": list(default_variant["cameras"]), + }, + "lag_options": list(checkpoint["lag_options"]), + "default_lag_ticks": default_display_lag, + "lag_variants": lag_variants, + "target_lag_options": list(checkpoint["target_lag_options"]), + "default_target_lag_ticks": int(checkpoint["default_target_display_lag"]), + "target_lag_variants": [], + } + ) + + target_lag_variants: list[dict[str, Any]] = [] + for target_lag_ticks in checkpoint["target_lag_options"]: + target_source_frame = frames[phase_end_tick + int(target_lag_ticks)] + target_pose_frame = _target_pose_frame(target_source_frame) + if target_pose_frame is None: + raise SystemExit( + f"phase-aware target checkpoint at tick {target_source_frame.get('tick')} is missing target_positions" + ) + target_variant_dir = cp_root / f"target_lag_{int(target_lag_ticks)}" + target_variant_dir.mkdir(parents=True, exist_ok=True) + cameras = capture_overlay_images( + model=model, + data=data, + renderer=renderer, + actual_frame=target_pose_frame, + target_frame=None, + joint_names=joint_names, + default_pose=default_pose, + joint_qpos_map=joint_qpos_map, + joint_qvel_map=joint_qvel_map, + floating_base_state=floating_base_state, + camera_configs=camera_configs, + output_dir=target_variant_dir, + require_target=False, + ) + sim_time_secs = frame_sim_time_secs(target_source_frame, control_frequency_hz) + write_checkpoint_metadata( + checkpoint_dir=target_variant_dir, + tick=int(target_source_frame["tick"]), + sim_time_secs=sim_time_secs, + cameras=cameras, + selection_reason=( + f"{phase_name} target pose sampled at +{int(target_lag_ticks)} ticks from phase end" + ), + frame_source=frame_source, + comparison_mode="target_pose_only", + phase_name=phase_name, + phase_end_tick=phase_end_tick, + lag_ticks=int(target_lag_ticks), + ) + target_lag_variants.append( + { + "lag_ticks": int(target_lag_ticks), + "lag_ms": lag_ticks_to_ms(int(target_lag_ticks), control_frequency_hz), + "tick": int(target_source_frame["tick"]), + "frame_index": phase_end_tick + int(target_lag_ticks), + "sim_time_secs": sim_time_secs, + "selection_reason": ( + f"{phase_name} target pose sampled at +{int(target_lag_ticks)} ticks from phase end" + ), + "frame_source": frame_source, + "relative_dir": target_variant_dir.relative_to(output_dir).as_posix(), + "cameras": cameras, + } + ) + + checkpoints[-1]["target_lag_variants"] = target_lag_variants + + return checkpoints + except Exception as exc: + if not is_headless_render_backend_error(exc): + raise + warning = frame_capture_warning(exc) + report["_proof_pack_capture"] = { + "status": "skipped", + "backend": os.environ.get("MUJOCO_GL", "auto"), + "warning": warning, + "error": f"{type(exc).__name__}: {exc}", + } + print(f"warning: {warning}", file=sys.stderr) + print(f"warning: frame capture error detail: {type(exc).__name__}: {exc}", file=sys.stderr) + return [] + finally: + if renderer is not None: + close = getattr(renderer, "close", None) + if callable(close): + close() + + +def _track_camera() -> Any: + """Return a default tracking camera configuration.""" + import mujoco + + cam = mujoco.MjvCamera() + cam.type = mujoco.mjtCamera.mjCAMERA_TRACKING + cam.trackbodyid = 0 + cam.distance = 2.5 + cam.azimuth = 135.0 + cam.elevation = -20.0 + cam.lookat[:] = [0.0, 0.0, 0.8] + return cam + + +def _phase_review_track_camera() -> Any: + """Return a more locomotion-informative chase view for staged velocity demos.""" + import mujoco + + cam = mujoco.MjvCamera() + cam.type = mujoco.mjtCamera.mjCAMERA_TRACKING + cam.trackbodyid = 0 + cam.distance = 3.1 + cam.azimuth = 155.0 + cam.elevation = -14.0 + cam.lookat[:] = [0.15, 0.0, 0.88] + return cam + + +def _side_camera() -> Any: + """Return a side-view camera configuration.""" + import mujoco + + cam = mujoco.MjvCamera() + cam.type = mujoco.mjtCamera.mjCAMERA_TRACKING + cam.trackbodyid = 0 + cam.distance = 2.5 + cam.azimuth = 90.0 + cam.elevation = -10.0 + cam.lookat[:] = [0.0, 0.0, 0.8] + return cam + + +def _top_camera() -> Any: + """Return a top-down camera configuration.""" + import mujoco + + cam = mujoco.MjvCamera() + cam.type = mujoco.mjtCamera.mjCAMERA_TRACKING + cam.trackbodyid = 0 + cam.distance = 3.0 + cam.azimuth = 0.0 + cam.elevation = -80.0 + cam.lookat[:] = [0.0, 0.0, 0.8] + return cam + + +def _phase_review_top_camera() -> Any: + """Return a path-oriented top view for staged velocity demos.""" + import mujoco + + cam = mujoco.MjvCamera() + cam.type = mujoco.mjtCamera.mjCAMERA_TRACKING + cam.trackbodyid = 0 + cam.distance = 4.8 + cam.azimuth = 22.0 + cam.elevation = -84.0 + cam.lookat[:] = [0.35, 0.0, 0.8] + return cam + + +def _save_png(path: Path, rgb: Any) -> None: + """Save a MuJoCo RGB render to PNG using Pillow if available, otherwise warn.""" + try: + from PIL import Image + except ImportError: + raise SystemExit( + "Pillow is required to save captured frames. Install with: pip install Pillow" + ) + + # MuJoCo renders as RGB uint8 array + img = Image.fromarray(rgb) + img.save(path) + + +def _target_pose_frame(frame: dict[str, Any]) -> dict[str, Any] | None: + target_positions = frame.get("target_positions") + if not isinstance(target_positions, list) or not target_positions: + return None + + target_frame = dict(frame) + target_frame.pop("mujoco_qpos", None) + target_frame.pop("mujoco_qvel", None) + target_frame["actual_positions"] = list(target_positions) + target_frame["actual_velocities"] = [0.0] * len(target_positions) + return target_frame + + +def _save_comparison_png(path: Path, actual_rgb: Any, target_rgb: Any) -> None: + """Save a target-vs-actual comparison image with a colored overlay.""" + try: + from PIL import Image, ImageDraw, ImageEnhance, ImageFilter, ImageOps + except ImportError: + raise SystemExit( + "Pillow is required to save captured frames. Install with: pip install Pillow" + ) + + actual = Image.fromarray(actual_rgb).convert("RGB") + target = Image.fromarray(target_rgb).convert("RGB") + + def backdrop(img: Image.Image) -> Image.Image: + gray = ImageOps.grayscale(img) + gray = ImageOps.autocontrast(gray, cutoff=1) + gray = ImageEnhance.Brightness(gray).enhance(1.15) + gray_rgb = Image.merge("RGB", (gray, gray, gray)) + return Image.blend(Image.new("RGB", img.size, (246, 248, 251)), gray_rgb, 0.16) + + def mask(img: Image.Image) -> Image.Image: + gray = ImageOps.grayscale(img) + gray = ImageOps.autocontrast(gray, cutoff=1) + binary = gray.point(lambda value: 255 if value > 8 else 0) + return binary.filter(ImageFilter.MaxFilter(5)) + + def colored_layer(img: Image.Image, color: tuple[int, int, int], alpha: float) -> Image.Image: + mask_img = mask(img).point(lambda value: int(value * alpha)) + layer = Image.new("RGBA", img.size, color + (255,)) + layer.putalpha(mask_img) + return layer + + canvas = backdrop(actual).convert("RGBA") + canvas.alpha_composite(colored_layer(target, (59, 130, 246), 0.50)) + canvas.alpha_composite(colored_layer(actual, (249, 115, 22), 0.62)) + + draw = ImageDraw.Draw(canvas) + draw.rounded_rectangle((12, 12, 264, 60), radius=8, fill=(255, 255, 255, 230)) + draw.text((24, 22), "Blue = target Orange = actual", fill=(23, 23, 23, 255)) + + canvas.convert("RGB").save(path) + + +# --------------------------------------------------------------------------- +# HTML report generation +# --------------------------------------------------------------------------- + + +def generate_report( + output_dir: Path, + report: dict[str, Any], + checkpoints: list[dict[str, Any]], +) -> Path: + from roboharness.reporting import generate_html_report + + metrics = report.get("metrics", {}) + policy_name = report.get("policy_name", "unknown") + robot_name = report.get("robot_name", "unknown") + command_kind = report.get("command_kind", "unknown") + command_data = report.get("command_data", []) + + summary_html = "\n".join( + [ + f"

    Policy: {policy_name}

    ", + f"

    Robot: {robot_name}

    ", + f"

    Command: {command_kind} {command_data}

    ", + "

    Image mode: each camera view is a single comparison image with blue target pose overlaid against orange actual pose.

    ", + "
    ", + f"
    Ticks{metrics.get('ticks', '-')}
    ", + f"
    Avg inference{metrics.get('average_inference_ms', 0.0):.3f} ms
    ", + f"
    Achieved rate{metrics.get('achieved_frequency_hz', 0.0):.2f} Hz
    ", + f"
    Dropped frames{metrics.get('dropped_frames', '-')}
    ", + "
    ", + ] + ) + + report_path = generate_html_report( + output_dir=output_dir, + task_name="roboharness_run", + title=f"Roboharness Report — {policy_name}", + subtitle=f"MuJoCo simulation visual report for {policy_name} on {robot_name}", + summary_html=summary_html, + footer_text=f"Generated by scripts/reports/roboharness_report.py at {dt.datetime.now(dt.timezone.utc).strftime('%Y-%m-%d %H:%M:%SZ')}", + trial_name="trial_001", + meshcat_mode="none", + evaluation_result=None, + ) + return report_path + + +def relative_output_artifact(output_dir: Path, artifact_path: str | None) -> str | None: + if not artifact_path: + return None + + path = Path(artifact_path) + try: + return path.relative_to(output_dir).as_posix() + except ValueError: + if path.is_absolute(): + return path.name if path.parent == output_dir else str(path) + return path.as_posix() + + +def manifest_checkpoint_entry(checkpoint: dict[str, Any]) -> dict[str, Any]: + entry = { + "name": checkpoint["name"], + "relative_dir": checkpoint["relative_dir"], + "tick": checkpoint["meta"]["tick"], + "frame_index": checkpoint["meta"]["frame_index"], + "sim_time_secs": checkpoint["meta"]["sim_time_secs"], + "selection_reason": checkpoint["meta"]["selection_reason"], + "frame_source": checkpoint["meta"]["frame_source"], + "cameras": checkpoint["meta"]["cameras"], + } + if isinstance(checkpoint.get("phase_name"), str): + entry["phase_name"] = checkpoint["phase_name"] + if isinstance(checkpoint.get("phase_kind"), str): + entry["phase_kind"] = checkpoint["phase_kind"] + if checkpoint.get("meta", {}).get("phase_end_tick") is not None: + entry["phase_end_tick"] = checkpoint["meta"]["phase_end_tick"] + if checkpoint.get("default_lag_ticks") is not None: + entry["default_lag_ticks"] = checkpoint["default_lag_ticks"] + if isinstance(checkpoint.get("lag_options"), list): + entry["lag_options"] = checkpoint["lag_options"] + if isinstance(checkpoint.get("lag_variants"), list): + entry["lag_variants"] = checkpoint["lag_variants"] + if checkpoint.get("default_target_lag_ticks") is not None: + entry["default_target_lag_ticks"] = checkpoint["default_target_lag_ticks"] + if isinstance(checkpoint.get("target_lag_options"), list): + entry["target_lag_options"] = checkpoint["target_lag_options"] + if isinstance(checkpoint.get("target_lag_variants"), list): + entry["target_lag_variants"] = checkpoint["target_lag_variants"] + return entry + + +def build_proof_pack_manifest_payload( + output_dir: Path, + report: dict[str, Any], + checkpoints: list[dict[str, Any]], + *, + html_entrypoint: str, +) -> dict[str, Any]: + replay_root, _, frame_source = replay_payload(report) + meta = report.get("_meta", {}) + if not isinstance(meta, dict): + meta = {} + + payload: dict[str, Any] = { + "schema_version": 1, + "generated_at": dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%d %H:%M:%SZ"), + "policy_name": report.get("policy_name"), + "robot_name": report.get("robot_name"), + "command_kind": report.get("command_kind"), + "command_data": report.get("command_data", []), + "control_frequency_hz": replay_root.get("control_frequency_hz"), + "html_entrypoint": html_entrypoint, + "metrics_source": "run_report.json", + "frame_source": frame_source, + "transport": replay_root.get("transport"), + "raw_artifacts": { + "run_report": relative_output_artifact(output_dir, meta.get("json_path")), + "replay_trace": relative_output_artifact(output_dir, meta.get("replay_trace_path")), + "rerun_recording": relative_output_artifact(output_dir, meta.get("rrd_path")), + "run_log": relative_output_artifact(output_dir, meta.get("log_path")), + "temp_config": relative_output_artifact(output_dir, meta.get("temp_config")), + }, + } + + legacy_checkpoints = [ + manifest_checkpoint_entry(checkpoint) + for checkpoint in checkpoints + ] + payload["checkpoints"] = legacy_checkpoints + + phase_review = report.get("_phase_review") + if isinstance(phase_review, dict): + phase_checkpoints = [ + manifest_checkpoint_entry(checkpoint) + for checkpoint in checkpoints + if checkpoint.get("kind") in {"phase_midpoint", "phase_end"} + ] + diagnostic_checkpoints = [ + manifest_checkpoint_entry(checkpoint) + for checkpoint in checkpoints + if checkpoint.get("kind") == "diagnostic" + ] + payload["phase_review"] = { + "enabled": True, + "version": PHASE_REVIEW_VERSION, + "source": phase_review.get("source"), + } + payload["phase_timeline"] = phase_review.get("phase_timeline", []) + payload["phase_checkpoints"] = phase_checkpoints + payload["diagnostic_checkpoints"] = diagnostic_checkpoints + payload["lag_options"] = phase_review.get("lag_options", []) + payload["default_lag_ticks"] = phase_review.get("default_lag_ticks") + payload["default_lag_ms"] = phase_review.get("default_lag_ms") + payload["target_lag_options"] = phase_review.get("target_lag_options", []) + payload["default_target_lag_ticks"] = phase_review.get("default_target_lag_ticks") + payload["default_target_lag_ms"] = phase_review.get("default_target_lag_ms") + + capture_meta = report.get("_proof_pack_capture") + if checkpoints: + payload["capture_status"] = "ok" + elif isinstance(capture_meta, dict): + payload["capture_status"] = str(capture_meta.get("status", "skipped")) + if capture_meta.get("backend"): + payload["capture_backend"] = capture_meta["backend"] + if capture_meta.get("warning"): + payload["capture_warning"] = capture_meta["warning"] + if capture_meta.get("error"): + payload["capture_error"] = capture_meta["error"] + + return payload + + +def write_proof_pack_manifest( + output_dir: Path, + report: dict[str, Any], + checkpoints: list[dict[str, Any]], + report_path: Path, +) -> Path: + manifest_path = output_dir / "proof_pack_manifest.json" + payload = build_proof_pack_manifest_payload( + output_dir, + report, + checkpoints, + html_entrypoint=str(relative_output_artifact(output_dir, str(report_path))), + ) + manifest_path.write_text(json.dumps(payload, indent=2), encoding="utf-8") + return manifest_path + + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- + + +def main() -> int: + args = parse_args() + repo_root = Path(args.repo_root).resolve() + output_dir = Path(args.output_dir).resolve() + output_dir.mkdir(parents=True, exist_ok=True) + binary = Path(args.robowbc_binary).resolve() + config_path = (repo_root / args.config).resolve() + + env = preflight_checks(repo_root, binary, config_path) + + print(f"Running robowbc with config: {config_path}") + report = run_robowbc( + repo_root, + binary, + output_dir, + config_path, + env, + max_ticks=args.max_ticks, + ) + print( + f"Run complete: ticks={report['metrics']['ticks']}, " + f"avg_inference_ms={report['metrics']['average_inference_ms']:.3f}" + ) + + print("Capturing frames from MuJoCo replay...") + checkpoints = capture_frames_from_report(repo_root, report, output_dir) + print(f"Captured {len(checkpoints)} checkpoints") + + print("Generating HTML report...") + report_path = generate_report(output_dir, report, checkpoints) + print(f"Report written to: {report_path}") + manifest_path = write_proof_pack_manifest(output_dir, report, checkpoints, report_path) + print(f"Proof-pack manifest written to: {manifest_path}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/roboharness_report.py b/scripts/roboharness_report.py index 88048eb..c2a0087 100755 --- a/scripts/roboharness_report.py +++ b/scripts/roboharness_report.py @@ -1,1949 +1,4 @@ #!/usr/bin/env python3 -"""Generate a roboharness-style visual report for a RoboWBC MuJoCo run. +from _compat import run_legacy_script -This script orchestrates: - 1. Pre-flight dependency checks (binary, models, roboharness, mujoco) - 2. RoboWBC CLI run with [report] and [vis] config injection - 3. Post-run frame capture by replaying the saved joint trajectory in MuJoCo - 4. HTML report generation using roboharness's report builder - -Example: - python scripts/roboharness_report.py \\ - --robowbc-binary target/release/robowbc \\ - --config configs/sonic_g1.toml \\ - --output-dir artifacts/roboharness-reports/sonic_g1 -""" - -from __future__ import annotations - -import argparse -import datetime as dt -import json -import math -import os -import re -import subprocess -import sys -import tempfile -import tomllib -from pathlib import Path -from typing import Any - -# Reuse runtime-env helpers from the existing showcase generator. -# We import them lazily so the script can still run --help even if -# generate_policy_showcase.py has heavy dependencies. -_SCRIPT_DIR = Path(__file__).parent.resolve() -PHASE_REVIEW_VERSION = 1 -DEFAULT_PHASE_REVIEW_LAG_TICKS = 3 -DEFAULT_PHASE_REVIEW_TARGET_LAG_TICKS = 0 -MAX_PHASE_REVIEW_LAG_TICKS = 5 -TRACKING_COMMAND_KINDS = { - "kinematic_pose", - "motion_tokens", - "reference_motion_tracking", - "standing_placeholder_tracking", -} - - -def _import_showcase_helpers(): - import importlib.util - - spec = importlib.util.spec_from_file_location( - "generate_policy_showcase", _SCRIPT_DIR / "generate_policy_showcase.py" - ) - module = importlib.util.module_from_spec(spec) # type: ignore[arg-type] - spec.loader.exec_module(module) # type: ignore[union-attr] - return module - - -# --------------------------------------------------------------------------- -# CLI -# --------------------------------------------------------------------------- - - -def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser( - description="Run RoboWBC policy in MuJoCo and produce a roboharness-style visual report." - ) - parser.add_argument("--repo-root", default=".", help="Repository root directory") - parser.add_argument( - "--robowbc-binary", - default="target/release/robowbc", - help="Path to the robowbc binary", - ) - parser.add_argument( - "--config", - default="configs/sonic_g1.toml", - help="Base TOML config for the policy run", - ) - parser.add_argument( - "--output-dir", - required=True, - help="Directory where the HTML report and artifacts will be written", - ) - parser.add_argument( - "--max-ticks", - type=int, - default=None, - help="Override max_ticks in the runtime config", - ) - return parser.parse_args() - - -# --------------------------------------------------------------------------- -# Pre-flight checks -# --------------------------------------------------------------------------- - - -def ensure_headless_mujoco_env() -> None: - """Default MuJoCo to an explicit offscreen backend for headless Linux runs. - - CI and most developer shells that render proof-pack screenshots do not have - a windowing session. MuJoCo's Python renderer needs an explicit offscreen - backend there or `mujoco.Renderer(...)` fails while creating the GL - context. - """ - backend = os.environ.get("MUJOCO_GL") - if backend: - backend = backend.strip().lower() - if backend in {"egl", "osmesa"} and not os.environ.get("PYOPENGL_PLATFORM"): - os.environ["PYOPENGL_PLATFORM"] = backend - return - - if sys.platform != "linux": - return - if os.environ.get("DISPLAY") or os.environ.get("WAYLAND_DISPLAY"): - return - - os.environ["MUJOCO_GL"] = "egl" - os.environ.setdefault("PYOPENGL_PLATFORM", "egl") - - -def _iter_exception_chain(exc: BaseException) -> list[BaseException]: - chain: list[BaseException] = [] - current: BaseException | None = exc - while current is not None and current not in chain: - chain.append(current) - if current.__cause__ is not None: - current = current.__cause__ - elif current.__context__ is not None and not current.__suppress_context__: - current = current.__context__ - else: - current = None - return chain - - -def is_headless_render_backend_error(exc: BaseException) -> bool: - """Return True for environment-specific GL bootstrap failures. - - These failures happen on some headless runners when MuJoCo's Python - renderer cannot initialize the configured offscreen backend. They are - different from deterministic logic bugs in the replay or image-writing - code, which should still fail loudly. - """ - - markers = ( - "eglquerystring", - "pyopengl_platform", - "opengl platform", - "failed to initialize glfw", - "glfwerror", - "glcontext", - "osmesa", - "no display name and no $display", - "cannot create opengl context", - "failed to create opengl context", - ) - for candidate in _iter_exception_chain(exc): - text = f"{type(candidate).__name__}: {candidate}".lower() - if any(marker in text for marker in markers): - return True - return False - - -def frame_capture_warning(exc: BaseException) -> str: - backend = os.environ.get("MUJOCO_GL", "auto") - return ( - "Skipped proof-pack screenshots because the MuJoCo offscreen renderer " - f"could not initialize the configured backend ({backend}). The raw run " - "report, replay trace, and Rerun recording are still available." - ) - - -def preflight_checks(repo_root: Path, binary: Path, config_path: Path) -> dict[str, str]: - """Verify that all required dependencies are present before running. - - Returns the runtime environment dict (with any necessary vars set). - """ - errors: list[str] = [] - ensure_headless_mujoco_env() - - # 1. Binary exists - if not binary.exists(): - errors.append(f"robowbc binary not found: {binary}") - - # 2. Config exists - if not config_path.exists(): - errors.append(f"config file not found: {config_path}") - - # 3. roboharness importable - try: - import roboharness # noqa: F401 - except ImportError: - errors.append( - "roboharness Python package is not installed. " - "Install it (e.g. pip install /path/to/roboharness) before running this script." - ) - - # 4. mujoco importable - try: - import mujoco # noqa: F401 - except ImportError: - errors.append( - "mujoco Python package is not installed. " - "Install it (e.g. pip install mujoco) before running this script." - ) - - # 5. Required model paths from config - if config_path.exists(): - config_text = config_path.read_text(encoding="utf-8") - config = tomllib.loads(config_text) - for key in ("encoder", "decoder", "planner"): - section = config.get("policy", {}).get("config", {}) - if isinstance(section, dict) and key in section: - model_path = section[key].get("model_path") - if model_path: - full_path = repo_root / model_path - if not full_path.exists(): - errors.append(f"required model missing: {model_path}") - - if errors: - raise SystemExit("Pre-flight checks failed:\n - " + "\n - ".join(errors)) - - # Configure runtime environment (ORT dylib, MuJoCo libdir, etc.) - showcase = _import_showcase_helpers() - env = os.environ.copy() - dylib = showcase.resolve_ort_dylib(repo_root) - if dylib: - env.setdefault("ROBOWBC_ORT_DYLIB_PATH", dylib) - env = showcase.configure_binary_runtime_env(env) - return env - - -# --------------------------------------------------------------------------- -# Config composition -# --------------------------------------------------------------------------- - - -def resolve_showcase_context(repo_root: Path, config_path: Path) -> dict[str, Any]: - """Extract simulation context from the base config.""" - app_config = tomllib.loads(config_path.read_text(encoding="utf-8")) - comm_cfg = app_config.get("comm") or app_config.get("communication") or {} - frequency_hz = int(comm_cfg.get("frequency_hz", 50) or 50) - existing_sim = app_config.get("sim") - - robot_cfg_path = app_config.get("robot", {}).get("config_path") - robot_model_path = None - if robot_cfg_path: - robot_cfg = tomllib.loads((repo_root / str(robot_cfg_path)).read_text(encoding="utf-8")) - robot_model_path = robot_cfg.get("model_path") - - timestep = 0.002 - default_substeps = round(1.0 / (max(frequency_hz, 1) * timestep)) - default_gain_profile = "simulation_pd" - - if isinstance(existing_sim, dict): - model_path = str(existing_sim.get("model_path") or robot_model_path or "") - timestep = float(existing_sim.get("timestep", timestep)) - substeps = int(existing_sim.get("substeps", default_substeps)) - gain_profile = str(existing_sim.get("gain_profile") or default_gain_profile) - return { - "transport": "mujoco" if model_path else "synthetic", - "model_path": model_path or None, - "timestep": timestep, - "substeps": substeps, - "gain_profile": gain_profile if model_path else None, - "robot_config_path": str(robot_cfg_path) if robot_cfg_path else None, - "config_has_sim_section": True, - } - - if robot_model_path is None: - return { - "transport": "synthetic", - "model_path": None, - "timestep": None, - "substeps": None, - "gain_profile": None, - "robot_config_path": str(robot_cfg_path) if robot_cfg_path else None, - "config_has_sim_section": False, - } - - return { - "transport": "mujoco", - "model_path": str(robot_model_path), - "timestep": timestep, - "substeps": default_substeps, - "gain_profile": default_gain_profile, - "robot_config_path": str(robot_cfg_path) if robot_cfg_path else None, - "config_has_sim_section": False, - } - - -def ensure_showcase_sim_section(base_toml: str, showcase_context: dict[str, Any]) -> str: - if showcase_context["transport"] != "mujoco": - return base_toml.rstrip() - - required_lines = [ - (r"^model_path\s*=", f'model_path = "{showcase_context["model_path"]}"'), - (r"^timestep\s*=", f'timestep = {showcase_context["timestep"]:g}'), - (r"^substeps\s*=", f'substeps = {showcase_context["substeps"]}'), - ( - r"^gain_profile\s*=", - f'gain_profile = "{showcase_context["gain_profile"]}"', - ), - ] - sim_section_pattern = re.compile(r"^(\[sim\].*?)(?=^\[|\Z)", re.MULTILINE | re.DOTALL) - match = sim_section_pattern.search(base_toml) - if match is None: - sim_lines = ["[sim]"] - sim_lines.extend(line for _, line in required_lines) - return "\n\n".join([base_toml.rstrip(), "\n".join(sim_lines)]) - - sim_section = match.group(1).rstrip() - for pattern, line in required_lines: - if re.search(pattern, sim_section, re.MULTILINE) is None: - sim_section += "\n" + line - sim_section += "\n" - return base_toml[: match.start()] + sim_section + base_toml[match.end() :] - - -def compose_run_config( - base_toml: str, - json_path: Path, - replay_path: Path, - rrd_path: Path, - showcase_context: dict[str, Any], - max_ticks: int | None = None, -) -> str: - sections = [ensure_showcase_sim_section(base_toml, showcase_context).rstrip()] - - # Inject report and vis sections - sections.append( - "\n".join( - [ - "[vis]", - 'app_id = "robowbc-roboharness"', - "spawn_viewer = false", - f'save_path = "{rrd_path.as_posix()}"', - "", - "[report]", - f'output_path = "{json_path.as_posix()}"', - "max_frames = 200", - f'replay_output_path = "{replay_path.as_posix()}"', - ] - ) - ) - - composed = "\n\n".join(sections) + "\n" - - if max_ticks is not None: - # Patch existing [runtime] max_ticks if present, else append a runtime block - runtime_pattern = re.compile(r"^(\[runtime\].*?)(?=^\[|\Z)", re.MULTILINE | re.DOTALL) - max_ticks_pattern = re.compile(r"^(max_ticks\s*=\s*)\d+", re.MULTILINE) - - match = runtime_pattern.search(composed) - if match: - runtime_section = match.group(1) - if max_ticks_pattern.search(runtime_section): - patched = max_ticks_pattern.sub(r"\g<1>" + str(max_ticks), runtime_section) - else: - patched = runtime_section.rstrip() + f"\nmax_ticks = {max_ticks}\n" - composed = composed[: match.start()] + patched + composed[match.end() :] - else: - composed = composed.rstrip() + f"\n\n[runtime]\nmax_ticks = {max_ticks}\n\n" - - return composed - - -# --------------------------------------------------------------------------- -# Run robowbc CLI -# --------------------------------------------------------------------------- - - -def run_robowbc( - repo_root: Path, - binary: Path, - output_dir: Path, - config_path: Path, - env: dict[str, str], - max_ticks: int | None = None, -) -> dict[str, Any]: - showcase_context = resolve_showcase_context(repo_root, config_path) - if showcase_context["transport"] != "mujoco": - raise SystemExit( - "roboharness_report requires a MuJoCo simulation config. " - f"Config {config_path} does not specify a model_path." - ) - - base_config = config_path.read_text(encoding="utf-8") - temp_config = output_dir / "roboharness_run.toml" - json_path = output_dir / "run_report.json" - replay_path = derive_replay_trace_path(json_path) - rrd_path = output_dir / "run_recording.rrd" - log_path = output_dir / "run.log" - - temp_config.write_text( - compose_run_config( - base_config, - json_path, - replay_path, - rrd_path, - showcase_context, - max_ticks, - ), - encoding="utf-8", - ) - - proc = subprocess.run( - [str(binary), "run", "--config", str(temp_config)], - cwd=repo_root, - env=env, - capture_output=True, - text=True, - check=False, - ) - log_text = proc.stdout + "\n--- STDERR ---\n" + proc.stderr - log_path.write_text(log_text, encoding="utf-8") - - if proc.returncode != 0: - raise SystemExit( - f"robowbc run failed with exit code {proc.returncode}; see {log_path}" - ) - if not json_path.exists(): - raise SystemExit( - f"robowbc run did not write the expected report JSON at {json_path}; see {log_path}" - ) - - report = json.loads(json_path.read_text(encoding="utf-8")) - replay_trace = None - if replay_path.exists(): - replay_trace = json.loads(replay_path.read_text(encoding="utf-8")) - - report["_meta"] = { - "config_path": str(config_path), - "log_path": str(log_path), - "rrd_path": str(rrd_path), - "json_path": str(json_path), - "replay_trace_path": str(replay_path), - "replay_trace_present": replay_trace is not None, - "temp_config": str(temp_config), - "showcase_context": showcase_context, - } - if replay_trace is not None: - report["_replay_trace"] = replay_trace - return report - - -# --------------------------------------------------------------------------- -# MuJoCo frame capture (replay trajectory) -# --------------------------------------------------------------------------- - - -def derive_replay_trace_path(report_path: Path) -> Path: - stem = report_path.stem or "run_report" - suffix = report_path.suffix or ".json" - return report_path.with_name(f"{stem}_replay_trace{suffix}") - - -def load_meshless_mujoco_model(model_path: Path) -> tuple[Any, Any]: - """Load a MuJoCo model, falling back to visible proxy geoms if meshes are missing.""" - import mujoco - import xml.etree.ElementTree as ET - - try: - model = mujoco.MjModel.from_xml_path(str(model_path)) - data = mujoco.MjData(model) - return model, data - except Exception: - pass - - # Build meshless fallback by stripping mesh assets and replacing mesh geoms - # with simple proxy spheres so proof-pack capture still shows a readable body. - xml_text = model_path.read_text(encoding="utf-8") - root = ET.fromstring(xml_text) - - # Remove all elements from - for asset in root.findall("asset"): - for mesh in asset.findall("mesh"): - asset.remove(mesh) - - for geom in root.iter("geom"): - mesh_name = geom.attrib.pop("mesh", None) - if mesh_name is None: - continue - geom.attrib["type"] = "sphere" - geom.attrib["size"] = _meshless_proxy_size(mesh_name) - geom.attrib["rgba"] = _meshless_proxy_rgba(geom.attrib.get("rgba")) - - # Serialize back to string - meshless_xml = ET.tostring(root, encoding="unicode") - model = mujoco.MjModel.from_xml_string(meshless_xml) - data = mujoco.MjData(model) - return model, data - - -def _meshless_proxy_size(mesh_name: str) -> str: - lowered = mesh_name.lower() - if any(token in lowered for token in ("pelvis", "torso", "waist", "head")): - return "0.055" - if any(token in lowered for token in ("thigh", "hip", "knee")): - return "0.05" - if any(token in lowered for token in ("ankle", "foot")): - return "0.045" - if any(token in lowered for token in ("shoulder", "upper_arm", "lower_arm", "elbow")): - return "0.042" - if any(token in lowered for token in ("wrist", "hand", "finger")): - return "0.028" - return "0.04" - - -def _meshless_proxy_rgba(original_rgba: str | None) -> str: - if not original_rgba: - return "0.82 0.82 0.86 1" - - parts = original_rgba.split() - if len(parts) != 4: - return "0.82 0.82 0.86 1" - - try: - rgb = [float(parts[0]), float(parts[1]), float(parts[2])] - alpha = float(parts[3]) - except ValueError: - return "0.82 0.82 0.86 1" - - lifted = [min(1.0, 0.45 + channel * 0.55) for channel in rgb] - return f"{lifted[0]:.3f} {lifted[1]:.3f} {lifted[2]:.3f} {max(alpha, 1.0):.3f}" - - -def replay_payload(report: dict[str, Any]) -> tuple[dict[str, Any], list[dict[str, Any]], str]: - replay_trace = report.get("_replay_trace") - if isinstance(replay_trace, dict): - frames = replay_trace.get("frames") - if isinstance(frames, list) and frames: - return replay_trace, frames, "canonical_replay_trace" - - frames = report.get("frames", []) - if isinstance(frames, list): - return report, frames, "run_report_frames" - - return report, [], "run_report_frames" - - -def frame_sim_time_secs(frame: dict[str, Any], control_frequency_hz: int) -> float: - sim_time_secs = frame.get("sim_time_secs") - if isinstance(sim_time_secs, (int, float)): - return float(sim_time_secs) - - tick = int(frame.get("tick", 0) or 0) - if control_frequency_hz <= 0: - return float(tick) - return tick / float(control_frequency_hz) - - -def planar_position(frame: dict[str, Any]) -> tuple[float, float] | None: - base_pose = frame.get("base_pose") - if isinstance(base_pose, dict): - position_world = base_pose.get("position_world") - if isinstance(position_world, list) and len(position_world) >= 2: - return float(position_world[0]), float(position_world[1]) - - qpos = frame.get("mujoco_qpos") - if isinstance(qpos, list) and len(qpos) >= 2: - return float(qpos[0]), float(qpos[1]) - - return None - - -def planar_displacement(reference: dict[str, Any], frame: dict[str, Any]) -> float | None: - reference_position = planar_position(reference) - current_position = planar_position(frame) - if reference_position is None or current_position is None: - return None - - return math.hypot( - current_position[0] - reference_position[0], - current_position[1] - reference_position[1], - ) - - -def mean_joint_delta(reference: dict[str, Any], frame: dict[str, Any]) -> float | None: - reference_positions = reference.get("actual_positions") - current_positions = frame.get("actual_positions") - if not ( - isinstance(reference_positions, list) - and isinstance(current_positions, list) - and reference_positions - and len(reference_positions) == len(current_positions) - ): - return None - - total = 0.0 - for reference_position, current_position in zip(reference_positions, current_positions): - total += abs(float(current_position) - float(reference_position)) - return total / len(reference_positions) - - -def lag_ticks_to_ms(lag_ticks: int, control_frequency_hz: int) -> float: - if control_frequency_hz <= 0: - return float(lag_ticks) - return (float(lag_ticks) * 1_000.0) / float(control_frequency_hz) - - -def normalize_phase_name(name: object, *, source: str, index: int) -> str: - if not isinstance(name, str): - raise SystemExit(f"{source}: phase #{index + 1} is missing a string phase name") - - trimmed = name.strip() - if not trimmed: - raise SystemExit(f"{source}: phase #{index + 1} has an empty phase name") - if "/" in trimmed or "\\" in trimmed or ".." in trimmed: - raise SystemExit( - f"{source}: phase {trimmed!r} must not contain path separators or '..'" - ) - if any(ord(ch) < 32 for ch in trimmed): - raise SystemExit(f"{source}: phase {trimmed!r} contains control characters") - return trimmed - - -def slugify_phase_name(name: str) -> str: - slug = re.sub(r"[^a-z0-9]+", "-", name.lower()).strip("-") - if not slug: - raise SystemExit(f"phase {name!r} did not produce a safe directory slug") - return slug - - -def normalize_phase_timeline_entries( - entries: object, - *, - frame_count: int, - control_frequency_hz: int, - source: str, -) -> list[dict[str, object]]: - if not isinstance(entries, list) or not entries: - raise SystemExit(f"{source}: phase timeline must be a non-empty list") - - normalized: list[dict[str, object]] = [] - seen_names: set[str] = set() - previous_end = -1 - for index, raw_entry in enumerate(entries): - if not isinstance(raw_entry, dict): - raise SystemExit(f"{source}: phase #{index + 1} must be a TOML/JSON object") - - phase_name = normalize_phase_name( - raw_entry.get("phase_name", raw_entry.get("name")), - source=source, - index=index, - ) - if phase_name in seen_names: - raise SystemExit(f"{source}: duplicate phase name {phase_name!r}") - seen_names.add(phase_name) - - start_tick = raw_entry.get("start_tick") - end_tick = raw_entry.get("end_tick") - if not isinstance(start_tick, int) or start_tick < 0: - raise SystemExit(f"{source}: phase {phase_name!r} must define start_tick >= 0") - if not isinstance(end_tick, int) or end_tick < start_tick: - raise SystemExit( - f"{source}: phase {phase_name!r} must define end_tick >= start_tick" - ) - if start_tick <= previous_end: - raise SystemExit( - f"{source}: phase {phase_name!r} overlaps or is out of order relative to the previous phase" - ) - if end_tick >= frame_count: - raise SystemExit( - f"{source}: phase {phase_name!r} ends at tick {end_tick}, but only {frame_count} frames were recorded" - ) - - midpoint_tick = raw_entry.get("midpoint_tick") - expected_midpoint = start_tick + ((end_tick - start_tick) // 2) - if midpoint_tick is None: - midpoint_tick = expected_midpoint - elif not isinstance(midpoint_tick, int) or midpoint_tick != expected_midpoint: - raise SystemExit( - f"{source}: phase {phase_name!r} midpoint_tick must equal the canonical midpoint {expected_midpoint}" - ) - - duration_ticks = end_tick - start_tick + 1 - raw_duration_ticks = raw_entry.get("duration_ticks") - if raw_duration_ticks is not None and raw_duration_ticks != duration_ticks: - raise SystemExit( - f"{source}: phase {phase_name!r} duration_ticks must equal {duration_ticks}" - ) - - duration_secs = raw_entry.get("duration_secs") - if duration_secs is None: - duration_secs = duration_ticks / float(max(control_frequency_hz, 1)) - elif not isinstance(duration_secs, (int, float)) or float(duration_secs) <= 0.0: - raise SystemExit( - f"{source}: phase {phase_name!r} must define a positive duration_secs when provided" - ) - - normalized.append( - { - "phase_name": phase_name, - "phase_slug": slugify_phase_name(phase_name), - "start_tick": start_tick, - "midpoint_tick": midpoint_tick, - "end_tick": end_tick, - "duration_ticks": duration_ticks, - "duration_secs": float(duration_secs), - } - ) - previous_end = end_tick - - return normalized - - -def resolve_report_config_path(repo_root: Path, report: dict[str, Any]) -> Path | None: - meta = report.get("_meta") - if not isinstance(meta, dict): - return None - - config_path = meta.get("config_path") - if not isinstance(config_path, str) or not config_path: - return None - - candidate = Path(config_path) - if not candidate.is_absolute(): - candidate = (repo_root / candidate).resolve() - else: - candidate = candidate.resolve() - - try: - candidate.relative_to(repo_root.resolve()) - except ValueError as exc: - raise SystemExit( - f"tracking phase sidecars must stay inside the repo root; got config path {candidate}" - ) from exc - return candidate - - -def load_tracking_phase_review_contract( - sidecar_path: Path, - *, - frame_count: int, - control_frequency_hz: int, -) -> dict[str, object]: - raw = tomllib.loads(sidecar_path.read_text(encoding="utf-8")) - default_lag_ticks = raw.get("default_lag_ticks", DEFAULT_PHASE_REVIEW_LAG_TICKS) - if ( - not isinstance(default_lag_ticks, int) - or default_lag_ticks < 0 - or default_lag_ticks > MAX_PHASE_REVIEW_LAG_TICKS - ): - raise SystemExit( - f"{sidecar_path}: default_lag_ticks must be an integer between 0 and {MAX_PHASE_REVIEW_LAG_TICKS}" - ) - - normalized_timeline = normalize_phase_timeline_entries( - raw.get("phases"), - frame_count=frame_count, - control_frequency_hz=control_frequency_hz, - source=str(sidecar_path), - ) - return { - "source": "tracking_sidecar", - "default_lag_ticks": default_lag_ticks, - "default_lag_ms": lag_ticks_to_ms(default_lag_ticks, control_frequency_hz), - "lag_options": list(range(MAX_PHASE_REVIEW_LAG_TICKS + 1)), - "default_target_lag_ticks": DEFAULT_PHASE_REVIEW_TARGET_LAG_TICKS, - "default_target_lag_ms": lag_ticks_to_ms( - DEFAULT_PHASE_REVIEW_TARGET_LAG_TICKS, control_frequency_hz - ), - "target_lag_options": list(range(MAX_PHASE_REVIEW_LAG_TICKS + 1)), - "phase_timeline": normalized_timeline, - } - - -def resolve_phase_review_contract(repo_root: Path, report: dict[str, Any]) -> dict[str, object] | None: - replay_root, frames, _ = replay_payload(report) - if not frames: - return None - - control_frequency_hz = int(replay_root.get("control_frequency_hz", 50) or 50) - command_kind = str(report.get("command_kind") or replay_root.get("command_kind") or "") - - if command_kind == "velocity_schedule": - raw_timeline = report.get("phase_timeline", replay_root.get("phase_timeline")) - if raw_timeline is None: - return None - normalized_timeline = normalize_phase_timeline_entries( - raw_timeline, - frame_count=len(frames), - control_frequency_hz=control_frequency_hz, - source="run artifact phase_timeline", - ) - return { - "source": "velocity_schedule", - "default_lag_ticks": DEFAULT_PHASE_REVIEW_LAG_TICKS, - "default_lag_ms": lag_ticks_to_ms( - DEFAULT_PHASE_REVIEW_LAG_TICKS, control_frequency_hz - ), - "lag_options": list(range(MAX_PHASE_REVIEW_LAG_TICKS + 1)), - "default_target_lag_ticks": DEFAULT_PHASE_REVIEW_TARGET_LAG_TICKS, - "default_target_lag_ms": lag_ticks_to_ms( - DEFAULT_PHASE_REVIEW_TARGET_LAG_TICKS, control_frequency_hz - ), - "target_lag_options": list(range(MAX_PHASE_REVIEW_LAG_TICKS + 1)), - "phase_timeline": normalized_timeline, - } - - if command_kind not in TRACKING_COMMAND_KINDS: - return None - - config_path = resolve_report_config_path(repo_root, report) - if config_path is None: - return None - - sidecar_path = config_path.with_suffix(".phases.toml") - if not sidecar_path.exists(): - return None - if sidecar_path.parent != config_path.parent: - raise SystemExit( - f"tracking phase sidecar must be a sibling of {config_path}; got {sidecar_path}" - ) - - return load_tracking_phase_review_contract( - sidecar_path.resolve(), - frame_count=len(frames), - control_frequency_hz=control_frequency_hz, - ) - - -def build_phase_review_capture_plan( - frames: list[dict[str, Any]], - phase_review: dict[str, object], - *, - frame_source: str, - control_frequency_hz: int, -) -> list[dict[str, object]]: - plan: list[dict[str, object]] = [] - timeline = phase_review["phase_timeline"] - assert isinstance(timeline, list) - lag_options = phase_review["lag_options"] - assert isinstance(lag_options, list) - default_lag_ticks = int(phase_review["default_lag_ticks"]) - target_lag_options = phase_review.get("target_lag_options", lag_options) - assert isinstance(target_lag_options, list) - default_target_lag_ticks = int( - phase_review.get("default_target_lag_ticks", DEFAULT_PHASE_REVIEW_TARGET_LAG_TICKS) - ) - - for phase_entry in timeline: - assert isinstance(phase_entry, dict) - phase_name = str(phase_entry["phase_name"]) - phase_slug = str(phase_entry["phase_slug"]) - midpoint_tick = int(phase_entry["midpoint_tick"]) - end_tick = int(phase_entry["end_tick"]) - - plan.append( - { - "kind": "phase_midpoint", - "name": f"{phase_name}_midpoint", - "phase_name": phase_name, - "phase_slug": phase_slug, - "tick": midpoint_tick, - "frame_index": midpoint_tick, - "selection_reason": f"{phase_name} midpoint from the explicit phase timeline", - "frame_source": frame_source, - } - ) - - available_lag_options = [ - int(lag) - for lag in lag_options - if isinstance(lag, int) and 0 <= lag <= MAX_PHASE_REVIEW_LAG_TICKS - and end_tick + lag < len(frames) - ] - if not available_lag_options: - raise SystemExit( - f"phase {phase_name!r} has no recorded frames at or after phase end tick {end_tick}" - ) - default_display_lag = ( - default_lag_ticks - if default_lag_ticks in available_lag_options - else available_lag_options[-1] - ) - available_target_lag_options = [ - int(lag) - for lag in target_lag_options - if isinstance(lag, int) and 0 <= lag <= MAX_PHASE_REVIEW_LAG_TICKS - and end_tick + lag < len(frames) - ] - if not available_target_lag_options: - raise SystemExit( - f"phase {phase_name!r} has no recorded target frames at or after phase end tick {end_tick}" - ) - default_target_display_lag = ( - default_target_lag_ticks - if default_target_lag_ticks in available_target_lag_options - else available_target_lag_options[-1] - ) - variants = [ - { - "lag_ticks": lag_ticks, - "lag_ms": lag_ticks_to_ms(lag_ticks, control_frequency_hz), - "frame_index": end_tick + lag_ticks, - "tick": int(frames[end_tick + lag_ticks]["tick"]), - "selection_reason": ( - f"{phase_name} phase end actual response at +{lag_ticks} ticks" - ), - "frame_source": frame_source, - } - for lag_ticks in available_lag_options - ] - plan.append( - { - "kind": "phase_end", - "name": f"{phase_name}_end", - "phase_name": phase_name, - "phase_slug": phase_slug, - "phase_end_tick": end_tick, - "frame_index": end_tick, - "selection_reason": ( - f"{phase_name} phase end with positive-lag actual-response review" - ), - "frame_source": frame_source, - "lag_options": available_lag_options, - "default_display_lag": default_display_lag, - "target_lag_options": available_target_lag_options, - "default_target_display_lag": default_target_display_lag, - "variants": variants, - } - ) - - return plan - - -def select_checkpoint_specs( - frames: list[dict[str, Any]], -) -> list[dict[str, Any]]: - if not frames: - return [] - - target_checkpoint_count = 5 - first_frame = frames[0] - last_index = len(frames) - 1 - candidates: list[dict[str, Any]] = [ - { - "name": "start", - "index": 0, - "reason": "initial state before meaningful motion", - } - ] - - first_motion: dict[str, Any] | None = None - for index, frame in enumerate(frames[1:], start=1): - displacement = planar_displacement(first_frame, frame) - if displacement is not None and displacement >= 0.05: - first_motion = { - "name": "first_motion", - "index": index, - "reason": f"first floating-base displacement >= 0.05 m ({displacement:.3f} m)", - } - break - - joint_delta = mean_joint_delta(first_frame, frame) - if joint_delta is not None and joint_delta >= 0.08: - first_motion = { - "name": "first_motion", - "index": index, - "reason": f"first mean joint delta >= 0.08 rad ({joint_delta:.3f} rad)", - } - break - - if first_motion is not None: - candidates.append(first_motion) - - peak_latency_index = max( - range(len(frames)), - key=lambda index: float(frames[index].get("inference_latency_ms", 0.0) or 0.0), - ) - peak_latency_ms = float( - frames[peak_latency_index].get("inference_latency_ms", 0.0) or 0.0 - ) - candidates.append( - { - "name": "peak_latency", - "index": peak_latency_index, - "reason": f"highest inference latency ({peak_latency_ms:.3f} ms)", - } - ) - - furthest_progress: dict[str, Any] | None = None - best_planar_displacement = -1.0 - for index, frame in enumerate(frames[1:], start=1): - displacement = planar_displacement(first_frame, frame) - if displacement is not None and displacement > best_planar_displacement: - best_planar_displacement = displacement - furthest_progress = { - "name": "furthest_progress", - "index": index, - "reason": f"largest planar displacement from start ({displacement:.3f} m)", - } - - if furthest_progress is None: - best_joint_delta = -1.0 - for index, frame in enumerate(frames[1:], start=1): - joint_delta = mean_joint_delta(first_frame, frame) - if joint_delta is not None and joint_delta > best_joint_delta: - best_joint_delta = joint_delta - furthest_progress = { - "name": "furthest_progress", - "index": index, - "reason": f"largest mean joint delta from start ({joint_delta:.3f} rad)", - } - - if furthest_progress is not None: - candidates.append(furthest_progress) - - candidates.append( - { - "name": "final", - "index": last_index, - "reason": "final recorded simulator state", - } - ) - - selected: list[dict[str, Any]] = [] - used_indices: set[int] = set() - for candidate in candidates: - index = int(candidate["index"]) - if index in used_indices: - continue - selected.append(candidate) - used_indices.add(index) - - fallback_specs = [ - ("fallback_mid_25", 0.25), - ("fallback_mid_50", 0.50), - ("fallback_mid_75", 0.75), - ] - for name, fraction in fallback_specs: - if len(selected) >= target_checkpoint_count or last_index <= 0: - break - index = int(round(fraction * last_index)) - if index in used_indices: - continue - selected.append( - { - "name": name, - "index": index, - "reason": ( - f"deterministic fallback at {int(fraction * 100)}% of the replay " - "because the evidence checkpoints collapsed to the same frame" - ), - } - ) - used_indices.add(index) - - return sorted(selected, key=lambda checkpoint: (int(checkpoint["index"]), checkpoint["name"])) - - -def restore_frame_state( - model: Any, - data: Any, - frame: dict[str, Any], - joint_names: list[str], - default_pose: list[float], - joint_qpos_map: dict[str, int], - joint_qvel_map: dict[str, int], - floating_base_state: dict[str, int] | None, -) -> None: - import mujoco - - mujoco.mj_resetData(model, data) - - qpos = frame.get("mujoco_qpos") - qvel = frame.get("mujoco_qvel") - if ( - isinstance(qpos, list) - and isinstance(qvel, list) - and len(qpos) == model.nq - and len(qvel) == model.nv - ): - data.qpos[:] = qpos - data.qvel[:] = qvel - else: - for jnt_id in range(model.njnt): - qpos_adr = int(model.jnt_qposadr[jnt_id]) - name = mujoco.mj_id2name(model, mujoco.mjtObj.mjOBJ_JOINT, jnt_id) - if name in joint_names: - default_index = joint_names.index(name) - data.qpos[qpos_adr] = default_pose[default_index] - - if floating_base_state is not None: - base_pose = frame.get("base_pose") - if isinstance(base_pose, dict): - position_world = base_pose.get("position_world") - rotation_xyzw = base_pose.get("rotation_xyzw") - if ( - isinstance(position_world, list) - and len(position_world) >= 3 - and isinstance(rotation_xyzw, list) - and len(rotation_xyzw) == 4 - ): - base_qpos = [ - float(position_world[0]), - float(position_world[1]), - float(position_world[2]), - float(rotation_xyzw[3]), - float(rotation_xyzw[0]), - float(rotation_xyzw[1]), - float(rotation_xyzw[2]), - ] - qpos_adr = floating_base_state["qpos_adr"] - data.qpos[qpos_adr : qpos_adr + 7] = base_qpos - qvel_adr = floating_base_state["qvel_adr"] - data.qvel[qvel_adr : qvel_adr + 6] = [0.0] * 6 - - positions = frame.get("actual_positions", []) - if isinstance(positions, list): - for joint_name, joint_position in zip(joint_names, positions): - if joint_name in joint_qpos_map: - data.qpos[joint_qpos_map[joint_name]] = float(joint_position) - - velocities = frame.get("actual_velocities", []) - if isinstance(velocities, list): - for joint_name, joint_velocity in zip(joint_names, velocities): - if joint_name in joint_qvel_map: - data.qvel[joint_qvel_map[joint_name]] = float(joint_velocity) - - sim_time_secs = frame.get("sim_time_secs") - if isinstance(sim_time_secs, (int, float)): - data.time = float(sim_time_secs) - - mujoco.mj_forward(model, data) - - -def phase_review_camera_configs(phase_review: dict[str, object] | None) -> list[tuple[str, Any]]: - if isinstance(phase_review, dict) and phase_review.get("source") == "velocity_schedule": - return [ - ("track", _phase_review_track_camera()), - ("side", _side_camera()), - ("top", _phase_review_top_camera()), - ] - return [ - ("track", _track_camera()), - ("side", _side_camera()), - ("top", _top_camera()), - ] - - -def capture_overlay_images( - *, - model: Any, - data: Any, - renderer: Any, - actual_frame: dict[str, Any], - target_frame: dict[str, Any] | None, - joint_names: list[str], - default_pose: list[float], - joint_qpos_map: dict[str, int], - joint_qvel_map: dict[str, int], - floating_base_state: dict[str, int] | None, - camera_configs: list[tuple[str, Any]], - output_dir: Path, - require_target: bool, -) -> list[str]: - if require_target and target_frame is None: - raise SystemExit( - f"phase-aware checkpoint at tick {actual_frame.get('tick')} is missing target_positions" - ) - - cameras: list[str] = [] - for cam_name, cam_obj in camera_configs: - restore_frame_state( - model, - data, - actual_frame, - joint_names, - default_pose, - joint_qpos_map, - joint_qvel_map, - floating_base_state, - ) - renderer.update_scene(data, camera=cam_obj) - actual_rgb = renderer.render() - - image_path = output_dir / f"{cam_name}_rgb.png" - _save_png(output_dir / f"{cam_name}_actual_rgb.png", actual_rgb) - if target_frame is None: - _save_png(image_path, actual_rgb) - else: - restore_frame_state( - model, - data, - target_frame, - joint_names, - default_pose, - joint_qpos_map, - joint_qvel_map, - floating_base_state, - ) - renderer.update_scene(data, camera=cam_obj) - target_rgb = renderer.render() - _save_png(output_dir / f"{cam_name}_target_rgb.png", target_rgb) - _save_comparison_png(image_path, actual_rgb, target_rgb) - cameras.append(cam_name) - - return cameras - - -def write_checkpoint_metadata( - *, - checkpoint_dir: Path, - tick: int, - sim_time_secs: float, - cameras: list[str], - selection_reason: str, - frame_source: str, - comparison_mode: str, - phase_name: str | None = None, - phase_end_tick: int | None = None, - lag_ticks: int | None = None, -) -> None: - metadata = { - "step": tick, - "sim_time": sim_time_secs, - "cameras": cameras, - "camera_capability": "rgb", - "comparison_mode": comparison_mode, - "selection_reason": selection_reason, - "frame_source": frame_source, - } - if phase_name is not None: - metadata["phase_name"] = phase_name - if phase_end_tick is not None: - metadata["phase_end_tick"] = phase_end_tick - if lag_ticks is not None: - metadata["lag_ticks"] = lag_ticks - - (checkpoint_dir / "metadata.json").write_text( - json.dumps(metadata), - encoding="utf-8", - ) - - -def capture_frames_from_report( - repo_root: Path, - report: dict[str, Any], - output_dir: Path, -) -> list[dict[str, Any]]: - """Replay the recorded trajectory in MuJoCo and capture screenshots. - - Returns a list of checkpoint metadata dicts for the HTML report. - """ - ensure_headless_mujoco_env() - renderer = None - try: - import mujoco - - showcase_context = report["_meta"]["showcase_context"] - model_path = repo_root / showcase_context["model_path"] - robot_cfg_path = repo_root / showcase_context["robot_config_path"] - robot_cfg = tomllib.loads(robot_cfg_path.read_text(encoding="utf-8")) - joint_names = robot_cfg.get("joint_names", []) - default_pose = robot_cfg.get("default_pose", [0.0] * len(joint_names)) - - model, data = load_meshless_mujoco_model(model_path) - renderer = mujoco.Renderer(model, height=480, width=640) - replay_root, frames, frame_source = replay_payload(report) - control_frequency_hz = int(replay_root.get("control_frequency_hz", 50) or 50) - phase_review = resolve_phase_review_contract(repo_root, report) - if phase_review is not None: - report["_phase_review"] = { - "source": phase_review["source"], - "default_lag_ticks": phase_review["default_lag_ticks"], - "default_lag_ms": phase_review["default_lag_ms"], - "lag_options": list(phase_review["lag_options"]), - "default_target_lag_ticks": phase_review["default_target_lag_ticks"], - "default_target_lag_ms": phase_review["default_target_lag_ms"], - "target_lag_options": list(phase_review["target_lag_options"]), - "phase_timeline": list(phase_review["phase_timeline"]), - } - - # Build joint name -> qpos address mapping - joint_qpos_map: dict[str, int] = {} - joint_qvel_map: dict[str, int] = {} - floating_base_state: dict[str, int] | None = None - for jnt_id in range(model.njnt): - name = mujoco.mj_id2name(model, mujoco.mjtObj.mjOBJ_JOINT, jnt_id) - qpos_adr = model.jnt_qposadr[jnt_id] - qvel_adr = model.jnt_dofadr[jnt_id] - joint_qpos_map[name] = int(qpos_adr) - joint_qvel_map[name] = int(qvel_adr) - if ( - floating_base_state is None - and model.jnt_type[jnt_id] == mujoco.mjtJoint.mjJNT_FREE - ): - floating_base_state = { - "qpos_adr": int(qpos_adr), - "qvel_adr": int(qvel_adr), - } - - if not frames: - return [] - - capture_plan: list[dict[str, Any]] = [] - primary_ticks: set[int] = set() - if phase_review is not None: - capture_plan.extend( - build_phase_review_capture_plan( - frames, - phase_review, - frame_source=frame_source, - control_frequency_hz=control_frequency_hz, - ) - ) - for checkpoint in capture_plan: - if checkpoint["kind"] == "phase_midpoint": - primary_ticks.add(int(checkpoint["tick"])) - elif checkpoint["kind"] == "phase_end": - primary_ticks.add(int(checkpoint["phase_end_tick"])) - - for checkpoint in select_checkpoint_specs(frames): - frame_index = int(checkpoint["index"]) - if frame_index in primary_ticks: - continue - frame = frames[frame_index] - capture_plan.append( - { - "kind": "diagnostic", - "name": str(checkpoint["name"]), - "tick": int(frame["tick"]), - "frame_index": frame_index, - "selection_reason": str(checkpoint["reason"]), - "frame_source": frame_source, - } - ) - - # roboharness expects: output_dir / task_name / trial_name / checkpoint_name - capture_dir = output_dir / "roboharness_run" / "trial_001" - capture_dir.mkdir(parents=True, exist_ok=True) - - checkpoints: list[dict[str, Any]] = [] - camera_configs = phase_review_camera_configs(phase_review) - - for checkpoint in capture_plan: - kind = str(checkpoint["kind"]) - if kind in {"diagnostic", "phase_midpoint"}: - frame_index = int(checkpoint["frame_index"]) - frame = frames[frame_index] - target_frame = _target_pose_frame(frame) - phase_name = checkpoint.get("phase_name") - if kind == "phase_midpoint": - cp_name = f"{checkpoint['phase_slug']}_midpoint_tick_{int(frame['tick']):04d}" - else: - cp_name = f"{checkpoint['name']}_tick_{int(frame['tick']):04d}" - cp_dir = capture_dir / cp_name - cp_dir.mkdir(parents=True, exist_ok=True) - sim_time_secs = frame_sim_time_secs(frame, control_frequency_hz) - cameras = capture_overlay_images( - model=model, - data=data, - renderer=renderer, - actual_frame=frame, - target_frame=target_frame, - joint_names=joint_names, - default_pose=default_pose, - joint_qpos_map=joint_qpos_map, - joint_qvel_map=joint_qvel_map, - floating_base_state=floating_base_state, - camera_configs=camera_configs, - output_dir=cp_dir, - require_target=kind == "phase_midpoint", - ) - write_checkpoint_metadata( - checkpoint_dir=cp_dir, - tick=int(frame["tick"]), - sim_time_secs=sim_time_secs, - cameras=cameras, - selection_reason=str(checkpoint["selection_reason"]), - frame_source=frame_source, - comparison_mode="target_vs_actual_overlay" - if target_frame is not None - else "actual_only", - phase_name=str(phase_name) if isinstance(phase_name, str) else None, - ) - checkpoints.append( - { - "kind": kind, - "phase_name": phase_name if isinstance(phase_name, str) else None, - "phase_kind": "midpoint" if kind == "phase_midpoint" else None, - "name": str(checkpoint["name"]), - "dir": cp_dir, - "relative_dir": cp_dir.relative_to(output_dir).as_posix(), - "meta": { - "tick": int(frame["tick"]), - "frame_index": frame_index, - "sim_time_secs": sim_time_secs, - "inference_latency_ms": float( - frame.get("inference_latency_ms", 0.0) or 0.0 - ), - "selection_reason": str(checkpoint["selection_reason"]), - "frame_source": frame_source, - "cameras": cameras, - }, - } - ) - continue - - if kind != "phase_end": - raise SystemExit(f"unknown checkpoint kind: {kind}") - - phase_name = str(checkpoint["phase_name"]) - phase_slug = str(checkpoint["phase_slug"]) - phase_end_tick = int(checkpoint["phase_end_tick"]) - canonical_target_source_frame = frames[int(checkpoint["frame_index"])] - target_frame = _target_pose_frame(canonical_target_source_frame) - cp_root = capture_dir / f"{phase_slug}_end_tick_{phase_end_tick:04d}" - cp_root.mkdir(parents=True, exist_ok=True) - lag_variants: list[dict[str, Any]] = [] - for variant in checkpoint["variants"]: - assert isinstance(variant, dict) - lag_ticks = int(variant["lag_ticks"]) - actual_frame = frames[int(variant["frame_index"])] - variant_dir = cp_root / f"lag_{lag_ticks}" - variant_dir.mkdir(parents=True, exist_ok=True) - cameras = capture_overlay_images( - model=model, - data=data, - renderer=renderer, - actual_frame=actual_frame, - target_frame=target_frame, - joint_names=joint_names, - default_pose=default_pose, - joint_qpos_map=joint_qpos_map, - joint_qvel_map=joint_qvel_map, - floating_base_state=floating_base_state, - camera_configs=camera_configs, - output_dir=variant_dir, - require_target=True, - ) - sim_time_secs = frame_sim_time_secs(actual_frame, control_frequency_hz) - write_checkpoint_metadata( - checkpoint_dir=variant_dir, - tick=int(actual_frame["tick"]), - sim_time_secs=sim_time_secs, - cameras=cameras, - selection_reason=str(variant["selection_reason"]), - frame_source=frame_source, - comparison_mode="target_vs_actual_overlay", - phase_name=phase_name, - phase_end_tick=phase_end_tick, - lag_ticks=lag_ticks, - ) - lag_variants.append( - { - "lag_ticks": lag_ticks, - "lag_ms": float(variant["lag_ms"]), - "tick": int(actual_frame["tick"]), - "frame_index": int(variant["frame_index"]), - "sim_time_secs": sim_time_secs, - "selection_reason": str(variant["selection_reason"]), - "frame_source": frame_source, - "relative_dir": variant_dir.relative_to(output_dir).as_posix(), - "cameras": cameras, - } - ) - - default_display_lag = int(checkpoint["default_display_lag"]) - default_variant = next( - ( - variant - for variant in lag_variants - if int(variant["lag_ticks"]) == default_display_lag - ), - lag_variants[-1], - ) - checkpoints.append( - { - "kind": "phase_end", - "phase_name": phase_name, - "phase_kind": "phase_end", - "name": str(checkpoint["name"]), - "dir": cp_root, - "relative_dir": str(default_variant["relative_dir"]), - "meta": { - "tick": phase_end_tick, - "frame_index": int(checkpoint["frame_index"]), - "phase_end_tick": phase_end_tick, - "sim_time_secs": frame_sim_time_secs( - canonical_target_source_frame, control_frequency_hz - ), - "selection_reason": str(checkpoint["selection_reason"]), - "frame_source": frame_source, - "cameras": list(default_variant["cameras"]), - }, - "lag_options": list(checkpoint["lag_options"]), - "default_lag_ticks": default_display_lag, - "lag_variants": lag_variants, - "target_lag_options": list(checkpoint["target_lag_options"]), - "default_target_lag_ticks": int(checkpoint["default_target_display_lag"]), - "target_lag_variants": [], - } - ) - - target_lag_variants: list[dict[str, Any]] = [] - for target_lag_ticks in checkpoint["target_lag_options"]: - target_source_frame = frames[phase_end_tick + int(target_lag_ticks)] - target_pose_frame = _target_pose_frame(target_source_frame) - if target_pose_frame is None: - raise SystemExit( - f"phase-aware target checkpoint at tick {target_source_frame.get('tick')} is missing target_positions" - ) - target_variant_dir = cp_root / f"target_lag_{int(target_lag_ticks)}" - target_variant_dir.mkdir(parents=True, exist_ok=True) - cameras = capture_overlay_images( - model=model, - data=data, - renderer=renderer, - actual_frame=target_pose_frame, - target_frame=None, - joint_names=joint_names, - default_pose=default_pose, - joint_qpos_map=joint_qpos_map, - joint_qvel_map=joint_qvel_map, - floating_base_state=floating_base_state, - camera_configs=camera_configs, - output_dir=target_variant_dir, - require_target=False, - ) - sim_time_secs = frame_sim_time_secs(target_source_frame, control_frequency_hz) - write_checkpoint_metadata( - checkpoint_dir=target_variant_dir, - tick=int(target_source_frame["tick"]), - sim_time_secs=sim_time_secs, - cameras=cameras, - selection_reason=( - f"{phase_name} target pose sampled at +{int(target_lag_ticks)} ticks from phase end" - ), - frame_source=frame_source, - comparison_mode="target_pose_only", - phase_name=phase_name, - phase_end_tick=phase_end_tick, - lag_ticks=int(target_lag_ticks), - ) - target_lag_variants.append( - { - "lag_ticks": int(target_lag_ticks), - "lag_ms": lag_ticks_to_ms(int(target_lag_ticks), control_frequency_hz), - "tick": int(target_source_frame["tick"]), - "frame_index": phase_end_tick + int(target_lag_ticks), - "sim_time_secs": sim_time_secs, - "selection_reason": ( - f"{phase_name} target pose sampled at +{int(target_lag_ticks)} ticks from phase end" - ), - "frame_source": frame_source, - "relative_dir": target_variant_dir.relative_to(output_dir).as_posix(), - "cameras": cameras, - } - ) - - checkpoints[-1]["target_lag_variants"] = target_lag_variants - - return checkpoints - except Exception as exc: - if not is_headless_render_backend_error(exc): - raise - warning = frame_capture_warning(exc) - report["_proof_pack_capture"] = { - "status": "skipped", - "backend": os.environ.get("MUJOCO_GL", "auto"), - "warning": warning, - "error": f"{type(exc).__name__}: {exc}", - } - print(f"warning: {warning}", file=sys.stderr) - print(f"warning: frame capture error detail: {type(exc).__name__}: {exc}", file=sys.stderr) - return [] - finally: - if renderer is not None: - close = getattr(renderer, "close", None) - if callable(close): - close() - - -def _track_camera() -> Any: - """Return a default tracking camera configuration.""" - import mujoco - - cam = mujoco.MjvCamera() - cam.type = mujoco.mjtCamera.mjCAMERA_TRACKING - cam.trackbodyid = 0 - cam.distance = 2.5 - cam.azimuth = 135.0 - cam.elevation = -20.0 - cam.lookat[:] = [0.0, 0.0, 0.8] - return cam - - -def _phase_review_track_camera() -> Any: - """Return a more locomotion-informative chase view for staged velocity demos.""" - import mujoco - - cam = mujoco.MjvCamera() - cam.type = mujoco.mjtCamera.mjCAMERA_TRACKING - cam.trackbodyid = 0 - cam.distance = 3.1 - cam.azimuth = 155.0 - cam.elevation = -14.0 - cam.lookat[:] = [0.15, 0.0, 0.88] - return cam - - -def _side_camera() -> Any: - """Return a side-view camera configuration.""" - import mujoco - - cam = mujoco.MjvCamera() - cam.type = mujoco.mjtCamera.mjCAMERA_TRACKING - cam.trackbodyid = 0 - cam.distance = 2.5 - cam.azimuth = 90.0 - cam.elevation = -10.0 - cam.lookat[:] = [0.0, 0.0, 0.8] - return cam - - -def _top_camera() -> Any: - """Return a top-down camera configuration.""" - import mujoco - - cam = mujoco.MjvCamera() - cam.type = mujoco.mjtCamera.mjCAMERA_TRACKING - cam.trackbodyid = 0 - cam.distance = 3.0 - cam.azimuth = 0.0 - cam.elevation = -80.0 - cam.lookat[:] = [0.0, 0.0, 0.8] - return cam - - -def _phase_review_top_camera() -> Any: - """Return a path-oriented top view for staged velocity demos.""" - import mujoco - - cam = mujoco.MjvCamera() - cam.type = mujoco.mjtCamera.mjCAMERA_TRACKING - cam.trackbodyid = 0 - cam.distance = 4.8 - cam.azimuth = 22.0 - cam.elevation = -84.0 - cam.lookat[:] = [0.35, 0.0, 0.8] - return cam - - -def _save_png(path: Path, rgb: Any) -> None: - """Save a MuJoCo RGB render to PNG using Pillow if available, otherwise warn.""" - try: - from PIL import Image - except ImportError: - raise SystemExit( - "Pillow is required to save captured frames. Install with: pip install Pillow" - ) - - # MuJoCo renders as RGB uint8 array - img = Image.fromarray(rgb) - img.save(path) - - -def _target_pose_frame(frame: dict[str, Any]) -> dict[str, Any] | None: - target_positions = frame.get("target_positions") - if not isinstance(target_positions, list) or not target_positions: - return None - - target_frame = dict(frame) - target_frame.pop("mujoco_qpos", None) - target_frame.pop("mujoco_qvel", None) - target_frame["actual_positions"] = list(target_positions) - target_frame["actual_velocities"] = [0.0] * len(target_positions) - return target_frame - - -def _save_comparison_png(path: Path, actual_rgb: Any, target_rgb: Any) -> None: - """Save a target-vs-actual comparison image with a colored overlay.""" - try: - from PIL import Image, ImageDraw, ImageEnhance, ImageFilter, ImageOps - except ImportError: - raise SystemExit( - "Pillow is required to save captured frames. Install with: pip install Pillow" - ) - - actual = Image.fromarray(actual_rgb).convert("RGB") - target = Image.fromarray(target_rgb).convert("RGB") - - def backdrop(img: Image.Image) -> Image.Image: - gray = ImageOps.grayscale(img) - gray = ImageOps.autocontrast(gray, cutoff=1) - gray = ImageEnhance.Brightness(gray).enhance(1.15) - gray_rgb = Image.merge("RGB", (gray, gray, gray)) - return Image.blend(Image.new("RGB", img.size, (246, 248, 251)), gray_rgb, 0.16) - - def mask(img: Image.Image) -> Image.Image: - gray = ImageOps.grayscale(img) - gray = ImageOps.autocontrast(gray, cutoff=1) - binary = gray.point(lambda value: 255 if value > 8 else 0) - return binary.filter(ImageFilter.MaxFilter(5)) - - def colored_layer(img: Image.Image, color: tuple[int, int, int], alpha: float) -> Image.Image: - mask_img = mask(img).point(lambda value: int(value * alpha)) - layer = Image.new("RGBA", img.size, color + (255,)) - layer.putalpha(mask_img) - return layer - - canvas = backdrop(actual).convert("RGBA") - canvas.alpha_composite(colored_layer(target, (59, 130, 246), 0.50)) - canvas.alpha_composite(colored_layer(actual, (249, 115, 22), 0.62)) - - draw = ImageDraw.Draw(canvas) - draw.rounded_rectangle((12, 12, 264, 60), radius=8, fill=(255, 255, 255, 230)) - draw.text((24, 22), "Blue = target Orange = actual", fill=(23, 23, 23, 255)) - - canvas.convert("RGB").save(path) - - -# --------------------------------------------------------------------------- -# HTML report generation -# --------------------------------------------------------------------------- - - -def generate_report( - output_dir: Path, - report: dict[str, Any], - checkpoints: list[dict[str, Any]], -) -> Path: - from roboharness.reporting import generate_html_report - - metrics = report.get("metrics", {}) - policy_name = report.get("policy_name", "unknown") - robot_name = report.get("robot_name", "unknown") - command_kind = report.get("command_kind", "unknown") - command_data = report.get("command_data", []) - - summary_html = "\n".join( - [ - f"

    Policy: {policy_name}

    ", - f"

    Robot: {robot_name}

    ", - f"

    Command: {command_kind} {command_data}

    ", - "

    Image mode: each camera view is a single comparison image with blue target pose overlaid against orange actual pose.

    ", - "
    ", - f"
    Ticks{metrics.get('ticks', '-')}
    ", - f"
    Avg inference{metrics.get('average_inference_ms', 0.0):.3f} ms
    ", - f"
    Achieved rate{metrics.get('achieved_frequency_hz', 0.0):.2f} Hz
    ", - f"
    Dropped frames{metrics.get('dropped_frames', '-')}
    ", - "
    ", - ] - ) - - report_path = generate_html_report( - output_dir=output_dir, - task_name="roboharness_run", - title=f"Roboharness Report — {policy_name}", - subtitle=f"MuJoCo simulation visual report for {policy_name} on {robot_name}", - summary_html=summary_html, - footer_text=f"Generated by scripts/roboharness_report.py at {dt.datetime.now(dt.timezone.utc).strftime('%Y-%m-%d %H:%M:%SZ')}", - trial_name="trial_001", - meshcat_mode="none", - evaluation_result=None, - ) - return report_path - - -def relative_output_artifact(output_dir: Path, artifact_path: str | None) -> str | None: - if not artifact_path: - return None - - path = Path(artifact_path) - try: - return path.relative_to(output_dir).as_posix() - except ValueError: - if path.is_absolute(): - return path.name if path.parent == output_dir else str(path) - return path.as_posix() - - -def manifest_checkpoint_entry(checkpoint: dict[str, Any]) -> dict[str, Any]: - entry = { - "name": checkpoint["name"], - "relative_dir": checkpoint["relative_dir"], - "tick": checkpoint["meta"]["tick"], - "frame_index": checkpoint["meta"]["frame_index"], - "sim_time_secs": checkpoint["meta"]["sim_time_secs"], - "selection_reason": checkpoint["meta"]["selection_reason"], - "frame_source": checkpoint["meta"]["frame_source"], - "cameras": checkpoint["meta"]["cameras"], - } - if isinstance(checkpoint.get("phase_name"), str): - entry["phase_name"] = checkpoint["phase_name"] - if isinstance(checkpoint.get("phase_kind"), str): - entry["phase_kind"] = checkpoint["phase_kind"] - if checkpoint.get("meta", {}).get("phase_end_tick") is not None: - entry["phase_end_tick"] = checkpoint["meta"]["phase_end_tick"] - if checkpoint.get("default_lag_ticks") is not None: - entry["default_lag_ticks"] = checkpoint["default_lag_ticks"] - if isinstance(checkpoint.get("lag_options"), list): - entry["lag_options"] = checkpoint["lag_options"] - if isinstance(checkpoint.get("lag_variants"), list): - entry["lag_variants"] = checkpoint["lag_variants"] - if checkpoint.get("default_target_lag_ticks") is not None: - entry["default_target_lag_ticks"] = checkpoint["default_target_lag_ticks"] - if isinstance(checkpoint.get("target_lag_options"), list): - entry["target_lag_options"] = checkpoint["target_lag_options"] - if isinstance(checkpoint.get("target_lag_variants"), list): - entry["target_lag_variants"] = checkpoint["target_lag_variants"] - return entry - - -def build_proof_pack_manifest_payload( - output_dir: Path, - report: dict[str, Any], - checkpoints: list[dict[str, Any]], - *, - html_entrypoint: str, -) -> dict[str, Any]: - replay_root, _, frame_source = replay_payload(report) - meta = report.get("_meta", {}) - if not isinstance(meta, dict): - meta = {} - - payload: dict[str, Any] = { - "schema_version": 1, - "generated_at": dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%d %H:%M:%SZ"), - "policy_name": report.get("policy_name"), - "robot_name": report.get("robot_name"), - "command_kind": report.get("command_kind"), - "command_data": report.get("command_data", []), - "control_frequency_hz": replay_root.get("control_frequency_hz"), - "html_entrypoint": html_entrypoint, - "metrics_source": "run_report.json", - "frame_source": frame_source, - "transport": replay_root.get("transport"), - "raw_artifacts": { - "run_report": relative_output_artifact(output_dir, meta.get("json_path")), - "replay_trace": relative_output_artifact(output_dir, meta.get("replay_trace_path")), - "rerun_recording": relative_output_artifact(output_dir, meta.get("rrd_path")), - "run_log": relative_output_artifact(output_dir, meta.get("log_path")), - "temp_config": relative_output_artifact(output_dir, meta.get("temp_config")), - }, - } - - legacy_checkpoints = [ - manifest_checkpoint_entry(checkpoint) - for checkpoint in checkpoints - ] - payload["checkpoints"] = legacy_checkpoints - - phase_review = report.get("_phase_review") - if isinstance(phase_review, dict): - phase_checkpoints = [ - manifest_checkpoint_entry(checkpoint) - for checkpoint in checkpoints - if checkpoint.get("kind") in {"phase_midpoint", "phase_end"} - ] - diagnostic_checkpoints = [ - manifest_checkpoint_entry(checkpoint) - for checkpoint in checkpoints - if checkpoint.get("kind") == "diagnostic" - ] - payload["phase_review"] = { - "enabled": True, - "version": PHASE_REVIEW_VERSION, - "source": phase_review.get("source"), - } - payload["phase_timeline"] = phase_review.get("phase_timeline", []) - payload["phase_checkpoints"] = phase_checkpoints - payload["diagnostic_checkpoints"] = diagnostic_checkpoints - payload["lag_options"] = phase_review.get("lag_options", []) - payload["default_lag_ticks"] = phase_review.get("default_lag_ticks") - payload["default_lag_ms"] = phase_review.get("default_lag_ms") - payload["target_lag_options"] = phase_review.get("target_lag_options", []) - payload["default_target_lag_ticks"] = phase_review.get("default_target_lag_ticks") - payload["default_target_lag_ms"] = phase_review.get("default_target_lag_ms") - - capture_meta = report.get("_proof_pack_capture") - if checkpoints: - payload["capture_status"] = "ok" - elif isinstance(capture_meta, dict): - payload["capture_status"] = str(capture_meta.get("status", "skipped")) - if capture_meta.get("backend"): - payload["capture_backend"] = capture_meta["backend"] - if capture_meta.get("warning"): - payload["capture_warning"] = capture_meta["warning"] - if capture_meta.get("error"): - payload["capture_error"] = capture_meta["error"] - - return payload - - -def write_proof_pack_manifest( - output_dir: Path, - report: dict[str, Any], - checkpoints: list[dict[str, Any]], - report_path: Path, -) -> Path: - manifest_path = output_dir / "proof_pack_manifest.json" - payload = build_proof_pack_manifest_payload( - output_dir, - report, - checkpoints, - html_entrypoint=str(relative_output_artifact(output_dir, str(report_path))), - ) - manifest_path.write_text(json.dumps(payload, indent=2), encoding="utf-8") - return manifest_path - - -# --------------------------------------------------------------------------- -# Main -# --------------------------------------------------------------------------- - - -def main() -> int: - args = parse_args() - repo_root = Path(args.repo_root).resolve() - output_dir = Path(args.output_dir).resolve() - output_dir.mkdir(parents=True, exist_ok=True) - binary = Path(args.robowbc_binary).resolve() - config_path = (repo_root / args.config).resolve() - - env = preflight_checks(repo_root, binary, config_path) - - print(f"Running robowbc with config: {config_path}") - report = run_robowbc( - repo_root, - binary, - output_dir, - config_path, - env, - max_ticks=args.max_ticks, - ) - print( - f"Run complete: ticks={report['metrics']['ticks']}, " - f"avg_inference_ms={report['metrics']['average_inference_ms']:.3f}" - ) - - print("Capturing frames from MuJoCo replay...") - checkpoints = capture_frames_from_report(repo_root, report, output_dir) - print(f"Captured {len(checkpoints)} checkpoints") - - print("Generating HTML report...") - report_path = generate_report(output_dir, report, checkpoints) - print(f"Report written to: {report_path}") - manifest_path = write_proof_pack_manifest(output_dir, report, checkpoints, report_path) - print(f"Proof-pack manifest written to: {manifest_path}") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) +run_legacy_script("reports/roboharness_report.py") diff --git a/scripts/sdk/python_sdk_smoke.py b/scripts/sdk/python_sdk_smoke.py new file mode 100755 index 0000000..de076f5 --- /dev/null +++ b/scripts/sdk/python_sdk_smoke.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python3 +"""Smoke-test the installed RoboWBC Python SDK.""" + +from __future__ import annotations + +from pathlib import Path + +from robowbc import ( + KinematicPoseCommand, + LinkPose, + Observation, + Registry, + VelocityCommand, +) + + +def main() -> int: + names = Registry.list_policies() + assert names, f"expected at least one policy, got: {names}" + print("Registered policies:", names) + + repo_root = Path(__file__).resolve().parents[2] + config_path = repo_root / "configs" / "decoupled_smoke.toml" + + policy = Registry.build("decoupled_wbc", str(config_path)) + capabilities = policy.capabilities() + assert hasattr(capabilities, "supported_commands"), capabilities + assert "velocity" in capabilities.supported_commands, capabilities + print("Policy capabilities:", capabilities.supported_commands) + + structured_obs = Observation( + joint_positions=[0.0] * 4, + joint_velocities=[0.0] * 4, + gravity_vector=[0.0, 0.0, -1.0], + command=VelocityCommand( + linear=[0.2, 0.0, 0.0], + angular=[0.0, 0.0, 0.1], + ), + ) + targets = policy.predict(structured_obs) + assert len(targets.positions) == 4, targets.positions + print("Structured velocity targets:", targets.positions) + + legacy_obs = Observation( + joint_positions=[0.0] * 4, + joint_velocities=[0.0] * 4, + gravity_vector=[0.0, 0.0, -1.0], + command_type="motion_tokens", + command_data=[0.1, 0.2], + ) + assert legacy_obs.joint_positions == [0.0] * 4 + assert legacy_obs.command_type == "motion_tokens" + assert len(legacy_obs.command_data) == 2 + assert all( + abs(actual - expected) < 1e-6 + for actual, expected in zip(legacy_obs.command_data, [0.1, 0.2], strict=True) + ), legacy_obs.command_data + print("Legacy observation:", legacy_obs) + + manipulation_obs = Observation( + joint_positions=[0.0] * 4, + joint_velocities=[0.0] * 4, + gravity_vector=[0.0, 0.0, -1.0], + command=KinematicPoseCommand( + [ + LinkPose( + name="left_wrist", + translation=[0.35, 0.20, 0.95], + rotation_xyzw=[0.0, 0.0, 0.0, 1.0], + ) + ] + ), + ) + assert manipulation_obs.command_type == "kinematic_pose" + assert manipulation_obs.command.links[0].name == "left_wrist" + try: + _ = manipulation_obs.command_data + except ValueError: + pass + else: + raise AssertionError("kinematic_pose should not expose flat command_data") + print("Structured manipulation observation:", manipulation_obs) + + try: + Registry.build_from_str('[policy]\nname = "no_such_policy"') + except RuntimeError as exc: + assert "unknown policy" in str(exc).lower(), f"unexpected error: {exc}" + else: + raise AssertionError("Registry.build_from_str should have raised") + + print("Python SDK smoke tests passed.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/serve_showcase.py b/scripts/serve_showcase.py index f876e04..84499a5 100755 --- a/scripts/serve_showcase.py +++ b/scripts/serve_showcase.py @@ -1,68 +1,4 @@ #!/usr/bin/env python3 -"""Serve a generated RoboWBC site bundle over HTTP for local debugging.""" +from _compat import run_legacy_script -from __future__ import annotations - -import argparse -import functools -import http.server -import socketserver -import webbrowser -from pathlib import Path - - -class ShowcaseRequestHandler(http.server.SimpleHTTPRequestHandler): - extensions_map = { - **http.server.SimpleHTTPRequestHandler.extensions_map, - ".wasm": "application/wasm", - ".rrd": "application/octet-stream", - } - - def end_headers(self) -> None: - self.send_header("Cache-Control", "no-store") - self.send_header("Access-Control-Allow-Origin", "*") - super().end_headers() - - -class ReusableTCPServer(socketserver.TCPServer): - allow_reuse_address = True - - -def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser() - parser.add_argument( - "--dir", - default=".", - help="Directory containing the generated site bundle and index.html", - ) - parser.add_argument("--bind", default="127.0.0.1", help="Address to bind the local server") - parser.add_argument("--port", type=int, default=8000, help="Port to bind the local server") - parser.add_argument("--open", action="store_true", help="Open the served page in the default browser") - return parser.parse_args() - - -def main() -> int: - args = parse_args() - root = Path(args.dir).resolve() - if not root.is_dir(): - raise SystemExit(f"site directory not found: {root}") - if not (root / "index.html").is_file(): - raise SystemExit(f"expected {root / 'index.html'} to exist") - - handler = functools.partial(ShowcaseRequestHandler, directory=str(root)) - with ReusableTCPServer((args.bind, args.port), handler) as httpd: - url = f"http://{args.bind}:{args.port}/" - print(f"Serving RoboWBC site from {root}") - print(f"Open {url}") - print("Press Ctrl-C to stop.") - if args.open: - webbrowser.open(url) - try: - httpd.serve_forever() - except KeyboardInterrupt: - print("\nStopping showcase server.") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) +run_legacy_script("site/serve_showcase.py") diff --git a/scripts/site/build_site.py b/scripts/site/build_site.py new file mode 100755 index 0000000..dedf7cd --- /dev/null +++ b/scripts/site/build_site.py @@ -0,0 +1,303 @@ +#!/usr/bin/env python3 +"""Build the full RoboWBC static site bundle in one command.""" + +from __future__ import annotations + +import argparse +import hashlib +import os +import platform +import shutil +import subprocess +import sys +import tarfile +import urllib.request +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "benchmarks")) +import normalize_nvidia_benchmarks as benchmark_schema + +MUJOCO_VERSION = "3.6.0" +BENCHMARK_PROVIDERS = benchmark_schema.PROVIDER_ORDER +BENCHMARK_IMPLEMENTATIONS = benchmark_schema.IMPLEMENTATION_ORDER + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--repo-root", default=".", help="Repository root directory") + parser.add_argument( + "--output-dir", + default="/tmp/robowbc-site", + help="Output directory for the generated static site bundle. The directory is recreated on each run.", + ) + parser.add_argument( + "--robowbc-binary", + default="./target/debug/robowbc", + help="Path to the robowbc binary used for policy runs", + ) + parser.add_argument( + "--skip-benchmarks", + action="store_true", + help="Build only the policy site and skip benchmark generation", + ) + return parser.parse_args() + + +def run(argv: list[str], *, cwd: Path, env: dict[str, str] | None = None) -> None: + subprocess.run(argv, cwd=cwd, check=True, text=True, env=env) + + +def recreate_dir(path: Path) -> None: + if path.exists(): + shutil.rmtree(path) + path.mkdir(parents=True, exist_ok=True) + + +def sync_benchmark_metadata(repo_root: Path) -> None: + source_root = repo_root / "benchmarks" / "nvidia" + if not source_root.is_dir(): + raise SystemExit(f"benchmark source root not found: {source_root}") + + artifact_root = repo_root / "artifacts" / "benchmarks" / "nvidia" + artifact_root.mkdir(parents=True, exist_ok=True) + shutil.copy2(source_root / "cases.json", artifact_root / "cases.json") + shutil.copy2(source_root / "README.md", artifact_root / "README.md") + + source_patches = source_root / "patches" + if source_patches.is_dir(): + shutil.copytree(source_patches, artifact_root / "patches", dirs_exist_ok=True) + + +def reset_benchmark_artifacts(repo_root: Path) -> Path: + artifact_root = repo_root / "artifacts" / "benchmarks" / "nvidia" + for implementation in BENCHMARK_IMPLEMENTATIONS: + impl_root = artifact_root / implementation + if impl_root.exists(): + shutil.rmtree(impl_root) + impl_root.mkdir(parents=True, exist_ok=True) + for legacy_dir in benchmark_schema.LEGACY_ARTIFACT_DIRS.values(): + legacy_root = artifact_root / legacy_dir + if legacy_root.exists(): + shutil.rmtree(legacy_root) + return artifact_root + + +def resolve_build_env(repo_root: Path) -> tuple[dict[str, str], Path]: + env = os.environ.copy() + configured = env.get("MUJOCO_DOWNLOAD_DIR") + if configured: + download_dir = Path(configured).expanduser().resolve() + else: + download_dir = (repo_root / ".cache" / "mujoco").resolve() + download_dir.mkdir(parents=True, exist_ok=True) + env["MUJOCO_DOWNLOAD_DIR"] = str(download_dir) + return env, download_dir + + +def prepend_env_path(env: dict[str, str], key: str, value: Path) -> None: + current = env.get(key) + env[key] = f"{value}{os.pathsep}{current}" if current else str(value) + + +def mujoco_archive_name() -> str: + if sys.platform != "linux": + raise SystemExit("scripts/site/build_site.py currently supports Linux-only MuJoCo site builds") + + machine = platform.machine().lower() + arch_map = { + "x86_64": "x86_64", + "amd64": "x86_64", + "aarch64": "aarch64", + "arm64": "aarch64", + } + try: + arch = arch_map[machine] + except KeyError as exc: + raise SystemExit(f"unsupported architecture for MuJoCo site build: {machine}") from exc + return f"mujoco-{MUJOCO_VERSION}-linux-{arch}.tar.gz" + + +def sha256sum(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def download_file(url: str, destination: Path) -> None: + with urllib.request.urlopen(url) as response, destination.open("wb") as handle: + shutil.copyfileobj(response, handle) + + +def mujoco_runtime_library(download_dir: Path) -> Path: + return download_dir / f"mujoco-{MUJOCO_VERSION}" / "lib" / "libmujoco.so" + + +def ensure_mujoco_runtime(download_dir: Path) -> None: + library_path = mujoco_runtime_library(download_dir) + if library_path.is_file(): + return + + archive = mujoco_archive_name() + base_url = f"https://github.com/google-deepmind/mujoco/releases/download/{MUJOCO_VERSION}" + archive_path = download_dir / archive + checksum_path = download_dir / f"{archive}.sha256" + + print(f"MuJoCo runtime missing; downloading {archive} into {download_dir}") + download_file(f"{base_url}/{archive}", archive_path) + download_file(f"{base_url}/{archive}.sha256", checksum_path) + + expected_sha = checksum_path.read_text(encoding="utf-8").split()[0] + actual_sha = sha256sum(archive_path) + if actual_sha != expected_sha: + raise SystemExit( + "MuJoCo archive checksum mismatch: " + f"expected {expected_sha}, got {actual_sha} for {archive_path}" + ) + + with tarfile.open(archive_path) as tar: + try: + tar.extractall(download_dir, filter="data") + except TypeError: + tar.extractall(download_dir) + + if not library_path.is_file(): + raise SystemExit(f"MuJoCo runtime library not found after extraction: {library_path}") + + +def configure_mujoco_runtime_env(env: dict[str, str], download_dir: Path) -> dict[str, str]: + library_dir = mujoco_runtime_library(download_dir).parent + if library_dir.is_dir(): + if os.name == "nt": + prepend_env_path(env, "PATH", library_dir) + elif sys.platform == "darwin": + prepend_env_path(env, "DYLD_LIBRARY_PATH", library_dir) + else: + prepend_env_path(env, "LD_LIBRARY_PATH", library_dir) + return env + + +def build_binary(repo_root: Path, binary: Path, env: dict[str, str]) -> None: + run( + [ + "cargo", + "build", + "--bin", + "robowbc", + "--features", + "robowbc-cli/sim-auto-download,robowbc-cli/vis", + ], + cwd=repo_root, + env=env, + ) + if not binary.exists(): + raise SystemExit(f"robowbc binary not found after build: {binary}") + + +def build_benchmarks(repo_root: Path, output_dir: Path, env: dict[str, str]) -> None: + sync_benchmark_metadata(repo_root) + artifact_root = reset_benchmark_artifacts(repo_root) + python = sys.executable + for provider in BENCHMARK_PROVIDERS: + run( + [ + python, + "scripts/benchmarks/bench_robowbc_compare.py", + "--all", + "--provider", + provider, + "--output-root", + str(artifact_root / "ort-rs"), + ], + cwd=repo_root, + env=env, + ) + run( + [ + python, + "scripts/benchmarks/bench_nvidia_official.py", + "--all", + "--provider", + provider, + "--output-root", + str(artifact_root / "ort-cpp-sonic"), + ], + cwd=repo_root, + env=env, + ) + + source_root = repo_root / "artifacts" / "benchmarks" / "nvidia" + if not source_root.is_dir(): + raise SystemExit(f"benchmark artifact root not found: {source_root}") + + run( + [ + python, + "scripts/benchmarks/render_nvidia_benchmark_summary.py", + "--root", + str(source_root), + "--output", + str(source_root / "SUMMARY.md"), + "--html-output", + str(source_root / "index.html"), + ], + cwd=repo_root, + env=env, + ) + + dest_root = output_dir / "benchmarks" / "nvidia" + dest_root.parent.mkdir(parents=True, exist_ok=True) + shutil.copytree(source_root, dest_root, dirs_exist_ok=True) + + +def build_policy_site( + repo_root: Path, + output_dir: Path, + binary: Path, + env: dict[str, str], +) -> None: + run( + [ + sys.executable, + "scripts/site/generate_policy_showcase.py", + "--repo-root", + str(repo_root), + "--robowbc-binary", + str(binary), + "--output-dir", + str(output_dir), + ], + cwd=repo_root, + env=env, + ) + + +def main() -> int: + args = parse_args() + repo_root = Path(args.repo_root).resolve() + output_dir = Path(args.output_dir).resolve() + binary = (repo_root / args.robowbc_binary).resolve() + env, mujoco_download_dir = resolve_build_env(repo_root) + + recreate_dir(output_dir) + ensure_mujoco_runtime(mujoco_download_dir) + env = configure_mujoco_runtime_env(env, mujoco_download_dir) + build_binary(repo_root, binary, env) + + if not args.skip_benchmarks: + build_benchmarks(repo_root, output_dir, env) + + # Benchmark helpers can invoke `cargo run -p robowbc-cli`, so rebuild the + # final CLI binary with the site feature set before generating policy pages. + build_binary(repo_root, binary, env) + build_policy_site(repo_root, output_dir, binary, env) + + print(f"Built RoboWBC site at {output_dir}") + print(f"Using MUJOCO_DOWNLOAD_DIR={env['MUJOCO_DOWNLOAD_DIR']}") + print(f"Open the home page via: python scripts/site/serve_showcase.py --dir {output_dir} --open") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/site/generate_policy_showcase.py b/scripts/site/generate_policy_showcase.py new file mode 100755 index 0000000..41760e6 --- /dev/null +++ b/scripts/site/generate_policy_showcase.py @@ -0,0 +1,2899 @@ +#!/usr/bin/env python3 +"""Generate a mixed-source RoboWBC policy showcase as a static HTML artifact.""" + +from __future__ import annotations + +import argparse +import datetime as dt +import html +import importlib.util +import json +import math +import os +from pathlib import Path +import re +import shutil +import subprocess +import tarfile +import tempfile +import sys +import tomllib +from typing import Iterable + +POLICIES = [ + { + "id": "gear_sonic", + "policy_family": "gear_sonic", + "title": "GEAR-SONIC", + "config": "configs/showcase/gear_sonic_real.toml", + "source": "NVIDIA GR00T", + "summary": "Real CPU planner_sonic.onnx run inside the MuJoCo-backed G1 showcase, driven by an explicit staged velocity-tracking script instead of a single constant command.", + "coverage": "Planner-only velocity tracking on the published G1 planner contract", + "execution_kind": "real", + "checkpoint_source": "Published GEAR-SONIC ONNX checkpoints", + "command_source": "runtime.velocity_schedule", + "demo_family": "Velocity tracking", + "demo_sequence": "Stand, accelerate from 0.0 to 0.6 m/s over 2 s, command a 90 degree right turn over 1 s, accelerate into a 1.0 m/s run over 3 s, then settle back to stand.", + "showcase_gain_profile": "default_pd", + "model_artifact": "models/gear-sonic/planner_sonic.onnx", + "required_paths": [ + "models/gear-sonic/model_encoder.onnx", + "models/gear-sonic/model_decoder.onnx", + "models/gear-sonic/planner_sonic.onnx", + ], + "blocked_reason": "Requires downloaded GEAR-SONIC checkpoints. Run scripts/models/download_gear_sonic_models.sh or let CI warm the cache first.", + }, + { + "id": "gear_sonic_tracking", + "policy_family": "gear_sonic", + "title": "GEAR-SONIC Reference Motion", + "config": "configs/showcase/gear_sonic_tracking_real.toml", + "source": "NVIDIA GR00T", + "summary": "Real published GEAR-Sonic encoder+decoder tracking on the official `macarena_001__A545` clip, running inside the MuJoCo-backed G1 showcase with the upstream heading-corrected reference orientation contract.", + "coverage": "Published G1 reference-motion tracking with explicit upper-body motion from the official example clip", + "execution_kind": "real", + "checkpoint_source": "Published GEAR-Sonic ONNX checkpoints plus pinned official reference-motion CSVs", + "command_source": "runtime.reference_motion_tracking", + "demo_family": "Reference / pose tracking", + "demo_sequence": "Autoplays the official `macarena_001__A545` reference clip to showcase clip-backed upper-body tracking instead of a standing placeholder.", + "showcase_gain_profile": "simulation_pd", + "model_artifact": "models/gear-sonic/reference/example/macarena_001__A545", + "required_paths": [ + "models/gear-sonic/model_encoder.onnx", + "models/gear-sonic/model_decoder.onnx", + "models/gear-sonic/planner_sonic.onnx", + "models/gear-sonic/reference/example/macarena_001__A545/joint_pos.csv", + "models/gear-sonic/reference/example/macarena_001__A545/joint_vel.csv", + "models/gear-sonic/reference/example/macarena_001__A545/body_quat.csv", + ], + "blocked_reason": "Requires the published GEAR-Sonic checkpoints and the official reference clip payloads. Run scripts/models/download_gear_sonic_models.sh and scripts/models/download_gear_sonic_reference_motions.sh first.", + }, + { + "id": "decoupled_wbc", + "title": "Decoupled WBC", + "config": "configs/showcase/decoupled_wbc_real.toml", + "source": "NVIDIA GR00T", + "summary": "Real public GR00T WholeBodyControl run inside the MuJoCo-backed G1 showcase, driven by the same staged locomotion script used for the velocity-only cards.", + "coverage": "Lower-body RL locomotion with default upper-body posture", + "execution_kind": "real", + "checkpoint_source": "Published GR00T WholeBodyControl ONNX checkpoints", + "command_source": "runtime.velocity_schedule", + "demo_family": "Velocity tracking", + "demo_sequence": "Stand, accelerate from 0.0 to 0.6 m/s over 2 s, command a 90 degree right turn over 1 s, accelerate into a 1.0 m/s run over 3 s, then settle back to stand.", + "showcase_gain_profile": "simulation_pd", + "model_artifact": "models/decoupled-wbc/GR00T-WholeBodyControl-Walk.onnx", + "required_paths": [ + "models/decoupled-wbc/GR00T-WholeBodyControl-Balance.onnx", + "models/decoupled-wbc/GR00T-WholeBodyControl-Walk.onnx", + ], + "blocked_reason": "Requires downloaded GR00T WholeBodyControl checkpoints. Run scripts/models/download_decoupled_wbc_models.sh or let CI warm the cache first.", + }, + { + "id": "bfm_zero", + "title": "BFM-Zero", + "config": "configs/bfm_zero_g1.toml", + "source": "CMU", + "summary": "Real public G1 tracking contract running inside the MuJoCo-backed showcase with a 721D prompt-conditioned observation, IMU gyro/history features, and the shipped walking latent context.", + "coverage": "Reference/context walking tracking", + "execution_kind": "real", + "checkpoint_source": "Prepared BFM-Zero ONNX checkpoint plus tracking context assets", + "command_source": "runtime.motion_tokens", + "demo_family": "Reference / pose tracking", + "demo_sequence": "Replays the shipped `zs_walking.npy` latent walking context. No verified public waving or upper-body mocap clip is bundled in this repo today.", + "showcase_gain_profile": "simulation_pd", + "model_artifact": "models/bfm_zero/bfm_zero_g1.onnx", + "required_paths": [ + "models/bfm_zero/bfm_zero_g1.onnx", + "models/bfm_zero/zs_walking.npy", + ], + "blocked_reason": "Requires public BFM-Zero assets. Run scripts/models/download_bfm_zero_models.sh or warm the CI cache to fetch the ONNX checkpoint and zs_walking.npy context automatically.", + }, + { + "id": "hover", + "title": "HOVER", + "config": "configs/hover_h1.toml", + "source": "NVIDIA", + "summary": "Real H1 multi-modal masked policy wrapper for locomotion and body-pose commands, enabled when a user-exported checkpoint is available.", + "coverage": "Multi-modal masked H1 controller", + "execution_kind": "real", + "checkpoint_source": "User-exported HOVER ONNX checkpoint", + "command_source": "runtime.velocity", + "demo_family": "Velocity tracking", + "demo_sequence": "Blocked until a compatible public checkpoint exists; intended to use an explicit locomotion command rather than a fabricated upper-body demo.", + "model_artifact": "models/hover/hover_h1.onnx", + "required_paths": [ + "models/hover/hover_h1.onnx", + ], + "blocked_reason": "HOVER ships public code and deployment tooling, but the public repo/releases do not include pretrained checkpoints. Provide your own exported ONNX model to enable this card.", + }, + { + "id": "wbc_agile", + "title": "WBC-AGILE", + "config": "configs/showcase/wbc_agile_real.toml", + "source": "NVIDIA Isaac", + "summary": "Real public G1 checkpoint using the published recurrent history tensors and lower-body target mapping, driven by the staged velocity-tracking script rather than a single constant command.", + "coverage": "Published G1 locomotion checkpoint on the public 29-DOF embodiment", + "execution_kind": "real", + "checkpoint_source": "Published NVIDIA Isaac G1 ONNX checkpoint", + "command_source": "runtime.velocity_schedule", + "demo_family": "Velocity tracking", + "demo_sequence": "Stand, accelerate from 0.0 to 0.6 m/s over 2 s, command a 90 degree right turn over 1 s, accelerate into a 1.0 m/s run over 3 s, then settle back to stand.", + "showcase_gain_profile": "simulation_pd", + "model_artifact": "models/wbc-agile/unitree_g1_velocity_e2e.onnx", + "required_paths": [ + "models/wbc-agile/unitree_g1_velocity_e2e.onnx", + ], + "blocked_reason": "Requires downloaded WBC-AGILE G1 checkpoint. Run scripts/models/download_wbc_agile_models.sh or let CI warm the cache first.", + }, + { + "id": "wholebody_vla", + "title": "WholeBodyVLA", + "config": "configs/wholebody_vla_x2.toml", + "source": "OpenDriveLab", + "summary": "Experimental AGIBOT X2 kinematic-pose contract wrapper for WholeBodyVLA. The public upstream project does not yet expose a runnable inference release, so this card documents the expected handoff shape and stays blocked until a compatible local model exists.", + "coverage": "Experimental KinematicPose contract placeholder", + "execution_kind": "experimental", + "checkpoint_source": "Local/private WholeBodyVLA ONNX checkpoint", + "command_source": "runtime.kinematic_pose", + "demo_family": "Reference / pose tracking", + "demo_sequence": "Pose-target handoff only. This remains blocked until a runnable upstream model exists; the showcase does not invent a fake upper-body clip.", + "model_artifact": "models/wholebody_vla/wholebody_vla_x2.onnx", + "required_paths": [ + "models/wholebody_vla/wholebody_vla_x2.onnx", + ], + "blocked_reason": "The public WholeBodyVLA repo does not currently provide runnable code or ONNX checkpoints. This wrapper remains blocked until a compatible local model is available.", + }, +] + +NOT_YET_SHOWCASED = [ + { + "name": "wbc_agile_t1", + "reason": "The Booster T1 path exists, but the public upstream release still does not match the ONNX contract used by the Rust CLI today.", + }, + { + "name": "py_model", + "reason": "The showcase job is focused on compiled ORT-backed policies inside the Rust CLI.", + }, +] + +COLORS = ["#0f766e", "#dc2626", "#2563eb", "#d97706", "#7c3aed", "#0891b2"] +RERUN_WEB_VIEWER_DIR = "assets/rerun-web-viewer" +DISPLAY_ORDER = { + "gear_sonic": 0, + "gear_sonic_tracking": 1, + "decoupled_wbc": 2, + "wbc_agile": 3, + "bfm_zero": 4, + "hover": 5, + "wholebody_vla": 6, +} + +DEMO_FAMILY_DESCRIPTIONS = { + "Velocity tracking": "Policies driven by an explicit locomotion command profile. These cards now use the same staged sequence instead of a single constant velocity.", + "Reference / pose tracking": "Policies driven by pose targets, motion references, or latent tracking context. If no verified official clip is wired, the card stays blocked instead of inventing a demo.", +} + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--repo-root", default=".") + parser.add_argument("--robowbc-binary", required=True) + parser.add_argument("--output-dir", required=True) + return parser.parse_args() + + +def site_relative_path(root: Path, path: Path) -> str: + return path.relative_to(root).as_posix() + + +def policy_output_dir(output_dir: Path, policy_id: str) -> Path: + return output_dir / "policies" / policy_id + + +def detail_page_path(output_dir: Path, card_id: str) -> Path: + return policy_output_dir(output_dir, card_id) / "index.html" + + +def resolve_ort_dylib(repo_root: Path) -> str | None: + explicit = os.environ.get("ROBOWBC_ORT_DYLIB_PATH") + if explicit: + return explicit + + candidates = sorted( + repo_root.glob( + "target/debug/build/robowbc-ort-*/out/onnxruntime-linux-x64-1.24.2/lib/libonnxruntime.so.1.24.2" + ), + key=lambda path: path.stat().st_mtime, + reverse=True, + ) + for path in candidates: + providers = path.parent / "libonnxruntime_providers_shared.so" + if providers.exists(): + return str(path) + return str(candidates[0]) if candidates else None + + +def resolve_mujoco_runtime_libdir(env: dict[str, str]) -> Path | None: + explicit = env.get("MUJOCO_DYNAMIC_LINK_DIR") + if explicit: + return Path(explicit) + + download_dir = env.get("MUJOCO_DOWNLOAD_DIR") + if not download_dir: + return None + + if os.name == "nt": + library_name = "mujoco.dll" + elif sys.platform == "darwin": + library_name = "libmujoco.dylib" + else: + library_name = "libmujoco.so" + + candidates = sorted( + Path(download_dir).glob(f"mujoco-*/lib/{library_name}"), + key=lambda path: path.stat().st_mtime, + reverse=True, + ) + return candidates[0].parent if candidates else None + + +def prepend_env_path(env: dict[str, str], key: str, value: Path) -> None: + current = env.get(key) + env[key] = f"{value}{os.pathsep}{current}" if current else str(value) + + +def configure_binary_runtime_env(env: dict[str, str]) -> dict[str, str]: + libdir = resolve_mujoco_runtime_libdir(env) + if libdir is None: + return env + + if os.name == "nt": + prepend_env_path(env, "PATH", libdir) + elif sys.platform == "darwin": + prepend_env_path(env, "DYLD_LIBRARY_PATH", libdir) + else: + prepend_env_path(env, "LD_LIBRARY_PATH", libdir) + return env + + +def resolve_rerun_web_viewer_version(repo_root: Path) -> str: + lock_text = (repo_root / "Cargo.lock").read_text(encoding="utf-8") + match = re.search(r'name = "rerun"\nversion = "([^"]+)"', lock_text) + if match is None: + raise SystemExit("failed to resolve rerun version from Cargo.lock") + return match.group(1) + + +def vendor_rerun_web_viewer(repo_root: Path, output_dir: Path) -> dict[str, str]: + version = resolve_rerun_web_viewer_version(repo_root) + viewer_dir = output_dir / RERUN_WEB_VIEWER_DIR + viewer_dir.mkdir(parents=True, exist_ok=True) + version_file = viewer_dir / "VERSION" + + target_files = { + "index_js": viewer_dir / "index.js", + "viewer_js": viewer_dir / "re_viewer.js", + "viewer_wasm": viewer_dir / "re_viewer_bg.wasm", + } + if ( + all(path.exists() for path in target_files.values()) + and version_file.exists() + and version_file.read_text(encoding="utf-8").strip() == version + ): + return { + "version": version, + "module_path": f"./{RERUN_WEB_VIEWER_DIR}/index.js", + } + + if shutil.which("npm") is None: + raise SystemExit( + "npm is required to vendor the embedded Rerun web viewer assets for the policy showcase" + ) + + with tempfile.TemporaryDirectory(prefix="robowbc-rerun-web-viewer-") as tempdir: + temp_path = Path(tempdir) + proc = subprocess.run( + ["npm", "pack", f"@rerun-io/web-viewer@{version}"], + cwd=temp_path, + capture_output=True, + text=True, + check=False, + ) + if proc.returncode != 0: + raise SystemExit( + "failed to fetch @rerun-io/web-viewer via npm pack:\n" + f"{proc.stdout}\n--- STDERR ---\n{proc.stderr}" + ) + + tgz_files = list(temp_path.glob("rerun-io-web-viewer-*.tgz")) + if len(tgz_files) != 1: + raise SystemExit("unexpected npm pack output for @rerun-io/web-viewer") + + with tarfile.open(tgz_files[0]) as tar: + try: + tar.extractall(temp_path, filter="data") + except TypeError: + tar.extractall(temp_path) + + package_dir = temp_path / "package" + index_text = (package_dir / "index.js").read_text(encoding="utf-8") + index_text = index_text.replace('import("./re_viewer")', 'import("./re_viewer.js")') + target_files["index_js"].write_text(index_text, encoding="utf-8") + shutil.copy2(package_dir / "re_viewer.js", target_files["viewer_js"]) + shutil.copy2(package_dir / "re_viewer_bg.wasm", target_files["viewer_wasm"]) + version_file.write_text(version, encoding="utf-8") + + return { + "version": version, + "module_path": f"./{RERUN_WEB_VIEWER_DIR}/index.js", + } + + +def derive_replay_trace_path(report_path: Path) -> Path: + stem = report_path.stem or "run" + suffix = report_path.suffix or ".json" + return report_path.with_name(f"{stem}_replay_trace{suffix}") + + +def load_roboharness_report_module(): + script_path = Path(__file__).resolve().parents[1] / "reports" / "roboharness_report.py" + spec = importlib.util.spec_from_file_location("roboharness_report", script_path) + if spec is None or spec.loader is None: + raise RuntimeError(f"failed to load roboharness report helpers from {script_path}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def resolve_showcase_context(repo_root: Path, policy: dict[str, object]) -> dict[str, object]: + config_path = repo_root / str(policy["config"]) + app_config = tomllib.loads(config_path.read_text(encoding="utf-8")) + comm_cfg = app_config.get("comm") or app_config.get("communication") or {} + frequency_hz = int(comm_cfg.get("frequency_hz", 50) or 50) + runtime_cfg = app_config.get("runtime") or {} + configured_max_ticks = runtime_cfg.get("max_ticks") + report_max_frames = int(policy.get("report_max_frames") or configured_max_ticks or 120) + existing_sim = app_config.get("sim") + + robot_cfg_path = app_config.get("robot", {}).get("config_path") + robot_model_path = None + if robot_cfg_path: + robot_cfg = tomllib.loads((repo_root / str(robot_cfg_path)).read_text(encoding="utf-8")) + robot_model_path = robot_cfg.get("model_path") + + timestep = float(policy.get("showcase_timestep", 0.002)) + derived_substeps = round(1.0 / (max(frequency_hz, 1) * timestep)) + default_substeps = int(policy.get("showcase_substeps", max(derived_substeps, 1))) + default_gain_profile = str(policy.get("showcase_gain_profile", "simulation_pd")) + + if isinstance(existing_sim, dict): + model_path = str(existing_sim.get("model_path") or robot_model_path or "") + timestep = float(existing_sim.get("timestep", timestep)) + substeps = int(existing_sim.get("substeps", default_substeps)) + gain_profile = str(existing_sim.get("gain_profile") or default_gain_profile) + return { + "transport": "mujoco" if model_path else "synthetic", + "model_path": model_path or None, + "timestep": timestep, + "substeps": substeps, + "gain_profile": gain_profile if model_path else None, + "robot_config_path": str(robot_cfg_path) if robot_cfg_path else None, + "config_has_sim_section": True, + "report_max_frames": report_max_frames, + } + + if robot_model_path is None: + return { + "transport": "synthetic", + "model_path": None, + "timestep": None, + "substeps": None, + "gain_profile": None, + "robot_config_path": str(robot_cfg_path) if robot_cfg_path else None, + "config_has_sim_section": False, + "report_max_frames": report_max_frames, + } + + return { + "transport": "mujoco", + "model_path": str(robot_model_path), + "timestep": timestep, + "substeps": default_substeps, + "gain_profile": default_gain_profile, + "robot_config_path": str(robot_cfg_path) if robot_cfg_path else None, + "config_has_sim_section": False, + "report_max_frames": report_max_frames, + } + + +def ensure_showcase_sim_section(base_toml: str, showcase_context: dict[str, object]) -> str: + if showcase_context["transport"] != "mujoco": + return base_toml.rstrip() + + required_lines = [ + (r"^model_path\s*=", f'model_path = "{showcase_context["model_path"]}"'), + (r"^timestep\s*=", f'timestep = {showcase_context["timestep"]:g}'), + (r"^substeps\s*=", f'substeps = {showcase_context["substeps"]}'), + ( + r"^gain_profile\s*=", + f'gain_profile = "{showcase_context["gain_profile"]}"', + ), + ] + sim_section_pattern = re.compile(r"^(\[sim\].*?)(?=^\[|\Z)", re.MULTILINE | re.DOTALL) + match = sim_section_pattern.search(base_toml) + if match is None: + sim_lines = ["[sim]"] + sim_lines.extend(line for _, line in required_lines) + return "\n\n".join([base_toml.rstrip(), "\n".join(sim_lines)]) + + sim_section = match.group(1).rstrip() + for pattern, line in required_lines: + if re.search(pattern, sim_section, re.MULTILINE) is None: + sim_section += "\n" + line + sim_section += "\n" + return base_toml[: match.start()] + sim_section + base_toml[match.end() :] + + +def compose_showcase_config( + base_toml: str, + policy_id: str, + json_path: Path, + rrd_path: Path, + showcase_context: dict[str, object], +) -> str: + sections = [ensure_showcase_sim_section(base_toml, showcase_context).rstrip()] + sections.append( + "\n".join( + [ + "[vis]", + f'app_id = "robowbc-showcase-{policy_id}"', + "spawn_viewer = false", + f'save_path = "{rrd_path.as_posix()}"', + "", + "[report]", + f'output_path = "{json_path.as_posix()}"', + f'max_frames = {int(showcase_context["report_max_frames"])}', + ] + ) + ) + return "\n\n".join(sections) + "\n" + + +def missing_required_paths(repo_root: Path, policy: dict[str, object]) -> list[str]: + required = policy.get("required_paths", []) + assert isinstance(required, list) + missing: list[str] = [] + for rel_path in required: + candidate = repo_root / str(rel_path) + if not candidate.exists(): + missing.append(str(rel_path)) + return missing + + +def detect_transport(log_text: str) -> str: + if "mujoco simulation transport active" in log_text: + return "mujoco" + if "unitree g1 hardware transport active" in log_text: + return "hardware" + return "synthetic" + + +def detect_mujoco_model_variant(log_text: str) -> str | None: + match = re.search(r"model_variant=([^,\s)]+)", log_text) + return match.group(1) if match else None + + +def build_proof_pack_manifest_payload( + policy_dir: Path, + report: dict[str, object], + checkpoints: list[dict[str, object]], +) -> dict[str, object]: + helpers = load_roboharness_report_module() + return helpers.build_proof_pack_manifest_payload( + policy_dir, + report, + checkpoints, + html_entrypoint="index.html", + ) + + +def generate_policy_proof_pack( + repo_root: Path, + policy_dir: Path, + report: dict[str, object], + site_root: Path, +) -> tuple[dict[str, str], dict[str, object]] | None: + report_meta = report.get("_meta") + if not isinstance(report_meta, dict): + return None + + showcase_context = report_meta.get("showcase_context") + if not isinstance(showcase_context, dict) or showcase_context.get("transport") != "mujoco": + return None + + helpers = load_roboharness_report_module() + checkpoints = helpers.capture_frames_from_report(repo_root, report, policy_dir) + manifest_payload = build_proof_pack_manifest_payload(policy_dir, report, checkpoints) + manifest_path = policy_dir / "proof_pack_manifest.json" + manifest_path.write_text(json.dumps(manifest_payload, indent=2), encoding="utf-8") + return ( + { + "proof_pack_manifest_file": site_relative_path(site_root, manifest_path), + }, + manifest_payload, + ) + + +def policy_meta( + policy: dict[str, object], + site_root: Path, + showcase_context: dict[str, object], + actual_transport: str | None = None, + actual_model_variant: str | None = None, + json_path: Path | None = None, + rrd_path: Path | None = None, + log_path: Path | None = None, + proof_pack_artifacts: dict[str, str] | None = None, +) -> dict[str, object]: + meta = { + "card_id": policy["id"], + "policy_family": policy.get("policy_family", policy["id"]), + "title": policy["title"], + "source": policy["source"], + "summary": policy["summary"], + "coverage": policy["coverage"], + "execution_kind": policy["execution_kind"], + "checkpoint_source": policy["checkpoint_source"], + "command_source": policy["command_source"], + "demo_family": policy["demo_family"], + "demo_sequence": policy["demo_sequence"], + "model_artifact": policy.get("model_artifact", ""), + "config_path": policy["config"], + "required_paths": list(policy.get("required_paths", [])), + "blocked_reason": policy.get("blocked_reason"), + "showcase_transport": actual_transport or str(showcase_context["transport"]), + "showcase_model_path": showcase_context.get("model_path"), + "showcase_gain_profile": showcase_context.get("gain_profile"), + "showcase_model_variant": actual_model_variant, + "robot_config_path": showcase_context.get("robot_config_path"), + } + if json_path is not None: + meta["json_file"] = site_relative_path(site_root, json_path) + if rrd_path is not None: + meta["rrd_file"] = site_relative_path(site_root, rrd_path) + if log_path is not None: + meta["log_file"] = site_relative_path(site_root, log_path) + if proof_pack_artifacts is not None: + meta.update(proof_pack_artifacts) + return meta + + + +def blocked_entry(repo_root: Path, policy: dict[str, object]) -> dict[str, object]: + missing = missing_required_paths(repo_root, policy) + showcase_context = resolve_showcase_context(repo_root, policy) + return { + "card_id": policy["id"], + "policy_name": policy.get("policy_family", policy["id"]), + "status": "blocked", + "metrics": None, + "frames": [], + "joint_names": [], + "command_kind": str(policy["command_source"]).removeprefix("runtime."), + "command_data": [], + "_meta": { + **policy_meta(policy, repo_root, showcase_context), + "missing_paths": missing, + }, + } + + + +def run_policy( + repo_root: Path, + binary: Path, + output_dir: Path, + policy: dict[str, object], + env: dict[str, str], +) -> dict[str, object]: + missing = missing_required_paths(repo_root, policy) + if missing: + return blocked_entry(repo_root, policy) + + policy_id = str(policy["id"]) + policy_dir = policy_output_dir(output_dir, policy_id) + policy_dir.mkdir(parents=True, exist_ok=True) + showcase_context = resolve_showcase_context(repo_root, policy) + base_config = (repo_root / str(policy["config"])).read_text(encoding="utf-8") + temp_config = policy_dir / "run.toml" + json_path = policy_dir / "run.json" + replay_path = derive_replay_trace_path(json_path) + rrd_path = policy_dir / "run.rrd" + log_path = policy_dir / "run.log" + temp_config.write_text( + compose_showcase_config( + base_config, + policy_id, + json_path, + rrd_path, + showcase_context, + ), + encoding="utf-8", + ) + + proc = subprocess.run( + [str(binary), "run", "--config", str(temp_config)], + cwd=repo_root, + env=env, + capture_output=True, + text=True, + check=False, + ) + log_text = proc.stdout + "\n--- STDERR ---\n" + proc.stderr + log_path.write_text(log_text, encoding="utf-8") + if proc.returncode != 0: + raise SystemExit( + f"policy showcase run failed for {policy_id} with exit code {proc.returncode}; see {log_path}" + ) + if not json_path.exists(): + raise SystemExit( + f"policy showcase run for {policy_id} did not write the expected report JSON at {json_path}; see {log_path}" + ) + if not rrd_path.exists(): + raise SystemExit( + f"policy showcase run for {policy_id} did not write the expected Rerun recording at {rrd_path}; " + "build robowbc with --features robowbc-cli/sim,robowbc-cli/vis before running the showcase generator" + ) + + actual_transport = detect_transport(log_text) + actual_model_variant = detect_mujoco_model_variant(log_text) + if showcase_context["transport"] == "mujoco" and actual_transport != "mujoco": + raise SystemExit( + f"policy showcase run for {policy_id} did not activate MuJoCo transport; " + f"see {log_path} and build robowbc with sim + vis support before regenerating the showcase" + ) + + report = json.loads(json_path.read_text(encoding="utf-8")) + replay_trace = None + if replay_path.exists(): + replay_trace = json.loads(replay_path.read_text(encoding="utf-8")) + + report["_meta"] = { + "log_path": str(log_path), + "rrd_path": str(rrd_path), + "json_path": str(json_path), + "replay_trace_path": str(replay_path), + "replay_trace_present": replay_trace is not None, + "temp_config": str(temp_config), + "showcase_context": showcase_context, + } + if replay_trace is not None: + report["_replay_trace"] = replay_trace + + proof_pack_artifacts: dict[str, str] | None = None + proof_pack_manifest: dict[str, object] | None = None + proof_pack_result = generate_policy_proof_pack(repo_root, policy_dir, report, output_dir) + if proof_pack_result is not None: + proof_pack_artifacts, proof_pack_manifest = proof_pack_result + + report["card_id"] = policy_id + report.setdefault("policy_name", str(policy.get("policy_family", policy_id))) + report["status"] = "ok" + report["_meta"] = policy_meta( + policy, + output_dir, + showcase_context, + actual_transport=actual_transport, + actual_model_variant=actual_model_variant, + json_path=json_path, + rrd_path=rrd_path, + log_path=log_path, + proof_pack_artifacts=proof_pack_artifacts, + ) + if proof_pack_manifest is not None: + report["_proof_pack_manifest"] = proof_pack_manifest + return report + + +def series_from_frames(frames: list[dict[str, object]], field: str, joint_idx: int) -> list[float]: + values: list[float] = [] + for frame in frames: + data = frame.get(field, []) + if isinstance(data, list) and joint_idx < len(data): + values.append(float(data[joint_idx])) + return values + + +def yaw_from_rotation_xyzw(rotation_xyzw: list[float]) -> float: + x, y, z, w = [float(value) for value in rotation_xyzw] + siny_cosp = 2.0 * (w * z + x * y) + cosy_cosp = 1.0 - 2.0 * (y * y + z * z) + return math.atan2(siny_cosp, cosy_cosp) + + +def wrap_angle_rad(angle_rad: float) -> float: + wrapped = angle_rad + while wrapped > math.pi: + wrapped -= 2.0 * math.pi + while wrapped < -math.pi: + wrapped += 2.0 * math.pi + return wrapped + + +def derive_velocity_tracking_series( + frames: list[dict[str, object]], control_frequency_hz: int +) -> dict[str, list[float]] | None: + if control_frequency_hz <= 0 or len(frames) < 2: + return None + + dt_secs = 1.0 / control_frequency_hz + vx_cmd: list[float] = [] + vx_actual: list[float] = [] + yaw_cmd: list[float] = [] + yaw_actual: list[float] = [] + + for previous, current in zip(frames, frames[1:]): + command_data = previous.get("command_data") + previous_pose = previous.get("base_pose") + current_pose = current.get("base_pose") + if not ( + isinstance(command_data, list) + and len(command_data) >= 3 + and isinstance(previous_pose, dict) + and isinstance(current_pose, dict) + ): + continue + + previous_position = previous_pose.get("position_world") + previous_rotation = previous_pose.get("rotation_xyzw") + current_position = current_pose.get("position_world") + current_rotation = current_pose.get("rotation_xyzw") + if not ( + isinstance(previous_position, list) + and len(previous_position) >= 2 + and isinstance(previous_rotation, list) + and len(previous_rotation) == 4 + and isinstance(current_position, list) + and len(current_position) >= 2 + and isinstance(current_rotation, list) + and len(current_rotation) == 4 + ): + continue + + yaw_prev = yaw_from_rotation_xyzw(previous_rotation) + yaw_curr = yaw_from_rotation_xyzw(current_rotation) + dx_world = float(current_position[0]) - float(previous_position[0]) + dy_world = float(current_position[1]) - float(previous_position[1]) + cos_yaw = math.cos(yaw_prev) + sin_yaw = math.sin(yaw_prev) + dx_body = cos_yaw * dx_world + sin_yaw * dy_world + vx_cmd.append(float(command_data[0])) + vx_actual.append(dx_body / dt_secs) + yaw_cmd.append(float(command_data[2])) + yaw_actual.append(wrap_angle_rad(yaw_curr - yaw_prev) / dt_secs) + + if not vx_cmd: + return None + + return { + "vx_cmd": vx_cmd, + "vx_actual": vx_actual, + "yaw_cmd": yaw_cmd, + "yaw_actual": yaw_actual, + } + + +def derive_target_tracking_metrics( + frames: list[dict[str, object]], +) -> dict[str, float | int | None] | None: + joint_abs_errors: list[float] = [] + joint_sq_error_sum = 0.0 + matched_frame_count = 0 + base_heights_m: list[float] = [] + + for frame in frames: + actual_positions = frame.get("actual_positions") + target_positions = frame.get("target_positions") + if ( + isinstance(actual_positions, list) + and isinstance(target_positions, list) + and actual_positions + and len(actual_positions) == len(target_positions) + ): + matched_frame_count += 1 + for actual_position, target_position in zip(actual_positions, target_positions): + abs_error = abs(float(actual_position) - float(target_position)) + joint_abs_errors.append(abs_error) + joint_sq_error_sum += abs_error * abs_error + + base_pose = frame.get("base_pose") + if isinstance(base_pose, dict): + position_world = base_pose.get("position_world") + if isinstance(position_world, list) and len(position_world) >= 3: + base_heights_m.append(float(position_world[2])) + + if not joint_abs_errors: + return None + + joint_abs_errors_sorted = sorted(joint_abs_errors) + p95_index = max(0, math.ceil(0.95 * len(joint_abs_errors_sorted)) - 1) + joint_sample_count = len(joint_abs_errors) + mean_joint_abs_error_rad = sum(joint_abs_errors) / joint_sample_count + joint_rmse_rad = math.sqrt(joint_sq_error_sum / joint_sample_count) + base_height_min_m = min(base_heights_m) if base_heights_m else None + base_height_max_m = max(base_heights_m) if base_heights_m else None + + return { + "matched_frame_count": matched_frame_count, + "joint_sample_count": joint_sample_count, + "mean_joint_abs_error_rad": mean_joint_abs_error_rad, + "p95_joint_abs_error_rad": joint_abs_errors_sorted[p95_index], + "peak_joint_abs_error_rad": max(joint_abs_errors), + "joint_rmse_rad": joint_rmse_rad, + "base_height_min_m": base_height_min_m, + "base_height_max_m": base_height_max_m, + "frames_below_base_height_0_4m": sum(height < 0.4 for height in base_heights_m), + "frames_below_base_height_0_2m": sum(height < 0.2 for height in base_heights_m), + } + + +def classify_quality_verdict( + status: str, + command_kind: str, + metrics: dict[str, object] | None, +) -> dict[str, str] | None: + if status != "ok": + return None + if not isinstance(metrics, dict): + return {"label": "??", "css_class": "unknown", "summary": "runtime metrics unavailable"} + + dropped_frames = int(metrics.get("dropped_frames", 0) or 0) + achieved_frequency_hz = float(metrics.get("achieved_frequency_hz", 0.0) or 0.0) + target_tracking = metrics.get("target_tracking") + target_tracking_dict = target_tracking if isinstance(target_tracking, dict) else None + + if command_kind in {"velocity", "velocity_schedule"}: + velocity_tracking = metrics.get("velocity_tracking") + if not isinstance(velocity_tracking, dict): + return { + "label": "??", + "css_class": "unknown", + "summary": "velocity-tracking metrics unavailable", + } + + vx_rmse_mps = float(velocity_tracking.get("vx_rmse_mps", math.inf)) + yaw_rate_rmse_rad_s = float( + velocity_tracking.get("yaw_rate_rmse_rad_s", math.inf) + ) + forward_distance_m = float(velocity_tracking.get("forward_distance_m", 0.0)) + heading_change_deg = float(velocity_tracking.get("heading_change_deg", math.nan)) + collapse_frames = ( + int(target_tracking_dict.get("frames_below_base_height_0_4m", 0)) + if target_tracking_dict is not None + else 0 + ) + + if ( + dropped_frames <= 1 + and achieved_frequency_hz >= 47.0 + and vx_rmse_mps < 0.4 + and yaw_rate_rmse_rad_s < 1.5 + and forward_distance_m > 2.5 + and collapse_frames == 0 + ): + return { + "label": "GOOD", + "css_class": "good", + "summary": "meets the showcase velocity gates", + } + + bad_reasons: list[str] = [] + if dropped_frames > 5: + bad_reasons.append(f"dropped frames {dropped_frames} > 5") + if achieved_frequency_hz < 45.0: + bad_reasons.append(f"achieved rate {achieved_frequency_hz:.1f} Hz < 45") + if vx_rmse_mps >= 0.6: + bad_reasons.append(f"vx RMSE {vx_rmse_mps:.3f} >= 0.6") + if yaw_rate_rmse_rad_s >= 1.5: + bad_reasons.append(f"yaw RMSE {yaw_rate_rmse_rad_s:.3f} >= 1.5") + if forward_distance_m < 0.5: + bad_reasons.append(f"forward distance {forward_distance_m:.3f} m < 0.5") + if collapse_frames > 20: + bad_reasons.append(f"collapse frames {collapse_frames} > 20") + if bad_reasons: + return { + "label": "BAD", + "css_class": "bad", + "summary": "; ".join(bad_reasons[:3]), + } + + mixed_reasons: list[str] = [] + if vx_rmse_mps >= 0.4: + mixed_reasons.append(f"vx RMSE {vx_rmse_mps:.3f} above target") + if forward_distance_m <= 2.5: + mixed_reasons.append(f"forward distance {forward_distance_m:.3f} m below target") + if collapse_frames > 0: + mixed_reasons.append(f"collapse frames {collapse_frames} > 0") + + return { + "label": "??", + "css_class": "unknown", + "summary": "; ".join(mixed_reasons[:3]) or "mixed velocity metrics", + } + + if target_tracking_dict is None: + return { + "label": "??", + "css_class": "unknown", + "summary": "joint-tracking metrics unavailable", + } + + mean_joint_abs_error_rad = float(target_tracking_dict["mean_joint_abs_error_rad"]) + p95_joint_abs_error_rad = float(target_tracking_dict["p95_joint_abs_error_rad"]) + base_height_min_m = target_tracking_dict.get("base_height_min_m") + frames_below_base_height_0_4m = int( + target_tracking_dict["frames_below_base_height_0_4m"] + ) + frames_below_base_height_0_2m = int( + target_tracking_dict["frames_below_base_height_0_2m"] + ) + + bad_reasons = [] + if mean_joint_abs_error_rad > 0.35: + bad_reasons.append(f"mean joint error {mean_joint_abs_error_rad:.3f} rad > 0.35") + if p95_joint_abs_error_rad > 1.0: + bad_reasons.append(f"joint error p95 {p95_joint_abs_error_rad:.3f} rad > 1.0") + if frames_below_base_height_0_4m > 20: + bad_reasons.append( + f"collapse frames {frames_below_base_height_0_4m} > 20" + ) + if frames_below_base_height_0_2m > 5: + bad_reasons.append( + f"deep-collapse frames {frames_below_base_height_0_2m} > 5" + ) + if ( + base_height_min_m is not None + and float(base_height_min_m) < 0.4 + ): + bad_reasons.append(f"min base height {float(base_height_min_m):.3f} m < 0.4") + if dropped_frames > 5: + bad_reasons.append(f"dropped frames {dropped_frames} > 5") + if achieved_frequency_hz < 45.0: + bad_reasons.append(f"achieved rate {achieved_frequency_hz:.1f} Hz < 45") + + if bad_reasons: + return { + "label": "BAD", + "css_class": "bad", + "summary": "; ".join(bad_reasons[:3]), + } + + if ( + mean_joint_abs_error_rad <= 0.15 + and p95_joint_abs_error_rad <= 0.45 + and frames_below_base_height_0_4m == 0 + and frames_below_base_height_0_2m == 0 + and (base_height_min_m is None or float(base_height_min_m) >= 0.6) + and dropped_frames == 0 + and achieved_frequency_hz >= 47.0 + ): + return { + "label": "GOOD", + "css_class": "good", + "summary": "stable run with tight joint-target tracking", + } + + return { + "label": "??", + "css_class": "unknown", + "summary": "stable run, but only generic tracking heuristics are available", + } + + +def spark_svg(series_list: list[dict[str, object]], width: int = 360, height: int = 140) -> str: + if not series_list: + return '' + + all_values = [value for series in series_list for value in series["values"]] + if not all_values: + return '' + + min_v = min(all_values) + max_v = max(all_values) + if math.isclose(min_v, max_v): + min_v -= 1.0 + max_v += 1.0 + + pad = 12 + inner_w = width - 2 * pad + inner_h = height - 2 * pad + + def point_x(idx: int, total: int) -> float: + if total <= 1: + return pad + inner_w / 2 + return pad + inner_w * idx / (total - 1) + + def point_y(val: float) -> float: + return pad + inner_h * (1.0 - (val - min_v) / (max_v - min_v)) + + paths = [] + for series in series_list: + values = series["values"] + pts = " ".join( + f"{point_x(idx, len(values)):.2f},{point_y(val):.2f}" + for idx, val in enumerate(values) + ) + dash = ' stroke-dasharray="5 4"' if series.get("dashed") else "" + paths.append( + f'' + ) + + baseline = point_y(0.0) + return ( + f'' + f'' + f'' + + "".join(paths) + + "" + ) + + +def format_vector(values: Iterable[float], limit: int = 6) -> str: + items = list(values) + head = ", ".join(f"{value:.3f}" for value in items[:limit]) + if len(items) > limit: + head += ", ..." + return f"[{head}]" + + +def pill(label: str, css_class: str) -> str: + return f'{html.escape(label)}' + + +def showcase_transport_badge_label(transport: str) -> str: + if transport == "mujoco": + return "MUJOCO SIM" + if transport == "hardware": + return "HARDWARE" + return transport.upper() + + +def showcase_transport_text(transport: str) -> str: + if transport == "mujoco": + return "MuJoCo sim" + if transport == "hardware": + return "Hardware" + return "Synthetic fallback" + + +def showcase_model_variant_text(variant: str | None) -> str: + if variant == "meshless-public-mjcf": + return "Meshless public MJCF" + if variant == "upstream-mjcf": + return "Upstream MJCF" + return variant or "-" + + +def display_sort_key(index: int, entry: dict[str, object]) -> tuple[int, int, int]: + meta = entry["_meta"] + status = entry.get("status", "ok") + execution_kind = str(meta["execution_kind"]) + card_id = entry_card_id(entry) + return ( + 0 if status == "ok" else 1, + 0 if execution_kind == "real" else 1, + DISPLAY_ORDER.get(card_id, index), + ) + + +def entry_card_id(entry: dict[str, object]) -> str: + explicit = entry.get("card_id") + if explicit: + return str(explicit) + + meta = entry.get("_meta") + if isinstance(meta, dict): + for key in ("card_id", "json_file", "rrd_file"): + value = meta.get(key) + if value: + return Path(str(value)).stem + + return str(entry.get("policy_name") or "") + + +def entry_policy_family(entry: dict[str, object]) -> str: + explicit = entry.get("policy_name") + if explicit: + return str(explicit) + + meta = entry.get("_meta") + if isinstance(meta, dict) and meta.get("policy_family"): + return str(meta["policy_family"]) + + return entry_card_id(entry) + + +def entry_identity_label(entry: dict[str, object]) -> str: + card_id = entry_card_id(entry) + policy_family = entry_policy_family(entry) + if policy_family == card_id: + return card_id + return f"{card_id} · family {policy_family}" + + + +def render_demo_section(title: str, cards: list[str]) -> str: + if not cards: + return "" + + description = DEMO_FAMILY_DESCRIPTIONS.get(title, "") + return f'''
    +
    +

    {html.escape(title)}

    +

    {html.escape(description)}

    +
    +
    + {''.join(cards)} +
    +
    ''' + + +def relative_href(from_path: Path, target_path: Path) -> str: + return os.path.relpath(target_path, start=from_path.parent).replace(os.sep, "/") + +def rebase_meta_artifact_paths( + meta: dict[str, object], + from_page: Path, + output_dir: Path, +) -> dict[str, object]: + rebased = dict(meta) + for key in ( + "json_file", + "rrd_file", + "log_file", + "proof_pack_manifest_file", + ): + value = meta.get(key) + if isinstance(value, str) and value: + rebased[key] = relative_href(from_page, output_dir / value) + return rebased + + +def render_overview_links(meta: dict[str, object], detail_href: str) -> str: + links = [f'Detail page'] + if meta.get("proof_pack_manifest_file"): + links.append(f'Visual checkpoints') + for key, label in ( + ("rrd_file", "Rerun"), + ("json_file", "JSON"), + ("log_file", "log"), + ("proof_pack_manifest_file", "Proof-pack manifest"), + ): + value = meta.get(key) + if isinstance(value, str) and value: + links.append(f'{html.escape(label)}') + return "
    ".join(links) + + +def discover_benchmark_pages(output_dir: Path, index_path: Path) -> list[dict[str, str]]: + benchmark_specs = [ + ( + output_dir / "benchmarks" / "nvidia" / "index.html", + "NVIDIA Comparison", + "RoboWBC-vs-official benchmark matrix built from the normalized NVIDIA artifacts.", + ) + ] + pages: list[dict[str, str]] = [] + for page_path, title, summary in benchmark_specs: + if page_path.is_file(): + pages.append( + { + "title": title, + "summary": summary, + "href": relative_href(index_path, page_path), + } + ) + return pages + + +def render_benchmark_section(pages: list[dict[str, str]]) -> str: + if not pages: + return "" + + cards = [] + for page in pages: + cards.append( + f'''''' + ) + + return f'''
    +
    +

    Benchmarks

    +

    Normalized benchmark packages that sit beside the policy pages in the same site bundle.

    +
    +
    + {''.join(cards)} +
    +
    ''' + + +def render_generic_proof_pack_section(proof_pack_manifest: dict[str, object]) -> str: + checkpoints = proof_pack_manifest.get("checkpoints") + if not isinstance(checkpoints, list) or not checkpoints: + capture_warning = proof_pack_manifest.get("capture_warning") + if isinstance(capture_warning, str) and capture_warning: + capture_backend = proof_pack_manifest.get("capture_backend") + backend_html = ( + f'

    Configured offscreen backend: {html.escape(str(capture_backend))}.

    ' + if capture_backend + else "" + ) + return f'''
    +

    Visual Checkpoints

    +
    +

    Screenshots unavailable for this build.

    +

    {html.escape(capture_warning)}

    + {backend_html} +

    The raw run report, replay trace, and Rerun recording are still published above so the result remains reviewable.

    +
    +
    ''' + return "" + + checkpoint_cards: list[str] = [] + for checkpoint in checkpoints: + if not isinstance(checkpoint, dict): + continue + relative_dir = checkpoint.get("relative_dir") + cameras = checkpoint.get("cameras") + if not isinstance(relative_dir, str) or not isinstance(cameras, list): + continue + + camera_cards = [] + for camera in cameras: + if not isinstance(camera, str): + continue + image_href = f"{relative_dir}/{camera}_rgb.png" + camera_cards.append( + f'''
    + {html.escape(str(checkpoint.get( +
    {html.escape(camera)}
    +
    ''' + ) + + if not camera_cards: + continue + + tick = checkpoint.get("tick", "-") + sim_time_secs = checkpoint.get("sim_time_secs") + sim_time_text = ( + f"{float(sim_time_secs):.2f} s" + if isinstance(sim_time_secs, (int, float)) + else "n/a" + ) + selection_reason = str(checkpoint.get("selection_reason", "")) + checkpoint_cards.append( + f'''
    +
    +
    +

    {html.escape(str(checkpoint.get("name", "checkpoint")))}

    +

    {html.escape(selection_reason)}

    +
    +
    + {html.escape(f"tick {tick}")} + {html.escape(sim_time_text)} +
    +
    +
    + {''.join(camera_cards)} +
    +
    ''' + ) + + if not checkpoint_cards: + return "" + + return f'''
    +

    Visual checkpoints

    +

    Each image overlays the target pose in blue against the actual replayed pose in orange. The checkpoints are selected from the replay trace so you can cross-check startup, motion onset, peak latency, furthest progress, and final state without opening a second report.

    +
    + {''.join(checkpoint_cards)} +
    +
    ''' + + +def render_proof_pack_section(proof_pack_manifest: dict[str, object] | None) -> str: + if not isinstance(proof_pack_manifest, dict): + return "" + + phase_review = proof_pack_manifest.get("phase_review") + if not (isinstance(phase_review, dict) and phase_review.get("enabled") is True): + return render_generic_proof_pack_section(proof_pack_manifest) + + phase_timeline = proof_pack_manifest.get("phase_timeline") + phase_checkpoints = proof_pack_manifest.get("phase_checkpoints") + diagnostic_checkpoints = proof_pack_manifest.get("diagnostic_checkpoints") + lag_options = proof_pack_manifest.get("lag_options") + default_lag_ticks = proof_pack_manifest.get("default_lag_ticks") + default_lag_ms = proof_pack_manifest.get("default_lag_ms") + target_lag_options = proof_pack_manifest.get("target_lag_options") + default_target_lag_ticks = proof_pack_manifest.get("default_target_lag_ticks") + default_target_lag_ms = proof_pack_manifest.get("default_target_lag_ms") + if not isinstance(phase_timeline, list) or not isinstance(phase_checkpoints, list): + raise SystemExit("phase-aware proof-pack manifest is missing phase timeline data") + if not isinstance(diagnostic_checkpoints, list): + diagnostic_checkpoints = [] + if not isinstance(lag_options, list) or not all(isinstance(item, int) for item in lag_options): + raise SystemExit("phase-aware proof-pack manifest is missing integer lag_options") + if not isinstance(default_lag_ticks, int): + raise SystemExit("phase-aware proof-pack manifest is missing default_lag_ticks") + if not isinstance(target_lag_options, list) or not all( + isinstance(item, int) for item in target_lag_options + ): + target_lag_options = [0] + if not isinstance(default_target_lag_ticks, int): + default_target_lag_ticks = 0 + + checkpoint_map: dict[str, dict[str, dict[str, object]]] = {} + for checkpoint in phase_checkpoints: + if not isinstance(checkpoint, dict): + continue + phase_name = checkpoint.get("phase_name") + phase_kind = checkpoint.get("phase_kind") + if not isinstance(phase_name, str) or not isinstance(phase_kind, str): + continue + checkpoint_map.setdefault(phase_name, {})[phase_kind] = checkpoint + + def debug_variant_summary(variant: dict[str, object]) -> dict[str, object]: + return { + "lag_ticks": int(variant.get("lag_ticks", 0) or 0), + "lag_ms": float(variant.get("lag_ms", 0.0) or 0.0), + "tick": int(variant.get("tick", 0) or 0), + "frame_index": int(variant.get("frame_index", 0) or 0), + "sim_time_secs": float(variant.get("sim_time_secs", 0.0) or 0.0), + "relative_dir": str(variant.get("relative_dir", "")), + "frame_source": str(variant.get("frame_source", "")), + "selection_reason": str(variant.get("selection_reason", "")), + "cameras": [str(camera) for camera in variant.get("cameras", []) if isinstance(camera, str)], + } + + def render_lag_buttons(lags: list[int], selected_lag: int) -> str: + return "".join( + f'' + for lag in lags + ) + + timeline_cards: list[str] = [] + phase_cards: list[str] = [] + for entry in phase_timeline: + if not isinstance(entry, dict): + continue + phase_name = entry.get("phase_name") + if not isinstance(phase_name, str): + continue + midpoint_checkpoint = checkpoint_map.get(phase_name, {}).get("midpoint") + phase_end_checkpoint = checkpoint_map.get(phase_name, {}).get("phase_end") + if not isinstance(midpoint_checkpoint, dict) or not isinstance(phase_end_checkpoint, dict): + raise SystemExit(f"phase-aware manifest is missing midpoint/end checkpoints for {phase_name}") + + start_tick = int(entry.get("start_tick", 0) or 0) + midpoint_tick = int(entry.get("midpoint_tick", 0) or 0) + end_tick = int(entry.get("end_tick", 0) or 0) + duration_ticks = int(entry.get("duration_ticks", 0) or 0) + duration_secs = float(entry.get("duration_secs", 0.0) or 0.0) + timeline_cards.append( + f'''
    +

    {html.escape(phase_name)}

    +

    ticks {start_tick}–{end_tick} · midpoint {midpoint_tick} · {duration_ticks} ticks / {duration_secs:.2f} s

    +
    ''' + ) + + midpoint_relative_dir = midpoint_checkpoint.get("relative_dir") + midpoint_cameras = midpoint_checkpoint.get("cameras") + if not isinstance(midpoint_relative_dir, str) or not isinstance(midpoint_cameras, list): + raise SystemExit(f"midpoint checkpoint for {phase_name} is malformed") + midpoint_views = "".join( + f'''
    + {html.escape(phase_name)} midpoint {html.escape(camera)} overlay +
    {html.escape(camera)}
    +
    ''' + for camera in midpoint_cameras + if isinstance(camera, str) + ) + + lag_variants = phase_end_checkpoint.get("lag_variants") + if not isinstance(lag_variants, list) or not lag_variants: + raise SystemExit(f"phase-end checkpoint for {phase_name} is missing lag_variants") + lag_variants_by_tick = { + int(variant["lag_ticks"]): variant + for variant in lag_variants + if isinstance(variant, dict) + and isinstance(variant.get("lag_ticks"), int) + and isinstance(variant.get("relative_dir"), str) + } + available_lags = sorted(lag_variants_by_tick) + if not available_lags: + raise SystemExit(f"phase-end checkpoint for {phase_name} has no usable lag variants") + display_lag = default_lag_ticks if default_lag_ticks in lag_variants_by_tick else available_lags[-1] + default_variant = lag_variants_by_tick[display_lag] + default_variant_dir = str(default_variant["relative_dir"]) + default_variant_ms = float(default_variant.get("lag_ms", 0.0) or 0.0) + target_phase_lag_options = phase_end_checkpoint.get("target_lag_options") + if not isinstance(target_phase_lag_options, list) or not all( + isinstance(item, int) for item in target_phase_lag_options + ): + target_phase_lag_options = [0] + target_lag_variants = phase_end_checkpoint.get("target_lag_variants") + target_variant_fallback_dir = str(default_variant["relative_dir"]) + target_variants_by_tick: dict[int, dict[str, object]] = {} + if isinstance(target_lag_variants, list) and target_lag_variants: + target_variants_by_tick = { + int(variant["lag_ticks"]): variant + for variant in target_lag_variants + if isinstance(variant, dict) + and isinstance(variant.get("lag_ticks"), int) + and isinstance(variant.get("relative_dir"), str) + } + if not target_variants_by_tick: + target_variants_by_tick = { + 0: { + "lag_ticks": 0, + "lag_ms": 0.0, + "tick": phase_end_checkpoint.get("phase_end_tick", end_tick), + "frame_index": phase_end_checkpoint.get("frame_index", end_tick), + "sim_time_secs": phase_end_checkpoint.get("sim_time_secs", 0.0), + "selection_reason": f"{phase_name} target pose at canonical phase end", + "frame_source": phase_end_checkpoint.get("frame_source", "canonical_replay_trace"), + "relative_dir": target_variant_fallback_dir, + "cameras": default_variant.get("cameras", []), + "_raw_suffix": "_target_rgb.png", + } + } + available_target_lags = sorted(target_variants_by_tick) + display_target_lag = ( + default_target_lag_ticks + if default_target_lag_ticks in target_variants_by_tick + else available_target_lags[-1] + ) + default_target_variant = target_variants_by_tick[display_target_lag] + default_target_variant_ms = float(default_target_variant.get("lag_ms", 0.0) or 0.0) + default_actual_tick = int(default_variant.get("tick", end_tick) or end_tick) + default_actual_frame_index = int(default_variant.get("frame_index", end_tick) or end_tick) + default_target_tick = int(default_target_variant.get("tick", end_tick) or end_tick) + default_target_frame_index = int( + default_target_variant.get("frame_index", end_tick) or end_tick + ) + end_cameras = default_variant.get("cameras") + if not isinstance(end_cameras, list): + raise SystemExit(f"phase-end checkpoint for {phase_name} is missing camera data") + phase_end_views = [] + for camera in end_cameras: + if not isinstance(camera, str): + continue + overlay_lag_map = { + str(lag): f"{str(variant['relative_dir'])}/{camera}_rgb.png" + for lag, variant in lag_variants_by_tick.items() + if isinstance(variant.get("cameras"), list) and camera in variant["cameras"] + } + actual_lag_map = { + str(lag): f"{str(variant['relative_dir'])}/{camera}_actual_rgb.png" + for lag, variant in lag_variants_by_tick.items() + if isinstance(variant.get("cameras"), list) and camera in variant["cameras"] + } + actual_lag_ms_map = { + str(lag): float(variant.get("lag_ms", 0.0) or 0.0) + for lag, variant in lag_variants_by_tick.items() + if isinstance(variant.get("cameras"), list) and camera in variant["cameras"] + } + target_lag_map = {} + target_lag_ms_map = {} + for lag, variant in target_variants_by_tick.items(): + cameras = variant.get("cameras") + if not isinstance(cameras, list) or camera not in cameras: + continue + suffix = str(variant.get("_raw_suffix") or "_rgb.png") + target_lag_map[str(lag)] = f"{str(variant['relative_dir'])}/{camera}{suffix}" + target_lag_ms_map[str(lag)] = float(variant.get("lag_ms", 0.0) or 0.0) + phase_end_views.append( + f'''
    + {html.escape(phase_name)} phase-end {html.escape(camera)} overlay +
    {html.escape(camera)} · T+{display_target_lag} ({default_target_variant_ms:.0f} ms) · A+{display_lag} ({default_variant_ms:.0f} ms)
    +
    ''' + ) + + phase_end_anchor = debug_variant_summary(phase_end_checkpoint) + phase_end_anchor["phase_end_tick"] = int( + phase_end_checkpoint.get("phase_end_tick", end_tick) or end_tick + ) + phase_debug_payload = { + "phase_name": phase_name, + "timeline": { + "start_tick": start_tick, + "midpoint_tick": midpoint_tick, + "end_tick": end_tick, + "duration_ticks": duration_ticks, + "duration_secs": duration_secs, + }, + "midpoint": debug_variant_summary(midpoint_checkpoint), + "phase_end_anchor": phase_end_anchor, + "default_review": { + "target_lag_ticks": display_target_lag, + "target_lag_ms": default_target_variant_ms, + "target_tick": default_target_tick, + "target_frame_index": default_target_frame_index, + "target_relative_dir": str(default_target_variant.get("relative_dir", "")), + "actual_lag_ticks": display_lag, + "actual_lag_ms": default_variant_ms, + "actual_tick": default_actual_tick, + "actual_frame_index": default_actual_frame_index, + "actual_relative_dir": default_variant_dir, + }, + "actual_variants": [ + debug_variant_summary(lag_variants_by_tick[lag]) for lag in available_lags + ], + "target_variants": [ + debug_variant_summary(target_variants_by_tick[lag]) + for lag in available_target_lags + ], + } + phase_debug_json = html.escape( + json.dumps(phase_debug_payload, indent=2, sort_keys=True) + ) + + phase_cards.append( + f'''
    +
    +
    +

    {html.escape(phase_name)}

    +

    Midpoint capture at tick {midpoint_tick}; phase-end review anchored at tick {end_tick}.

    +
    +
    + {html.escape(f"start {start_tick}")} + {html.escape(f"end {end_tick}")} +
    +
    +
    +
    +

    Midpoint

    +
    + {midpoint_views} +
    +
    +
    +

    Phase End

    +
    + {''.join(phase_end_views)} +
    +
    +
    +
    + Debug metadata +

    Static tick and asset-path contract for manual debugging without driving the browser controls.

    + 
    {phase_debug_json}
    +
    +
    ''' + ) + + diagnostic_cards: list[str] = [] + for checkpoint in diagnostic_checkpoints: + if not isinstance(checkpoint, dict): + continue + relative_dir = checkpoint.get("relative_dir") + cameras = checkpoint.get("cameras") + if not isinstance(relative_dir, str) or not isinstance(cameras, list): + continue + diagnostic_cards.append( + f'''
    +
    +
    +

    {html.escape(str(checkpoint.get("name", "diagnostic")))}

    +

    {html.escape(str(checkpoint.get("selection_reason", "")))}

    +
    +
    + {html.escape(f"tick {checkpoint.get('tick', '-')}")} +
    +
    +
    + {''.join( + f'
    {html.escape(str(checkpoint.get(
    {html.escape(camera)}
    ' + for camera in cameras + if isinstance(camera, str) + )} +
    +
    ''' + ) + + lag_button_html = render_lag_buttons(lag_options, default_lag_ticks) + target_lag_button_html = render_lag_buttons( + target_lag_options, default_target_lag_ticks + ) + default_lag_ms_text = ( + f"{float(default_lag_ms):.0f} ms" + if isinstance(default_lag_ms, (int, float)) + else "n/a" + ) + default_target_lag_ms_text = ( + f"{float(default_target_lag_ms):.0f} ms" + if isinstance(default_target_lag_ms, (int, float)) + else "n/a" + ) + + diagnostics_html = ( + f'''
    +

    Diagnostics

    +

    Generic evidence checkpoints stay available as secondary diagnostics instead of the primary story.

    +
    + {''.join(diagnostic_cards)} +
    +
    ''' + if diagnostic_cards + else "" + ) + + return f'''
    +
    +
    +

    Phase review

    +

    The proof pack follows the authored phase timeline directly, so the staged locomotion story reads as stand → accelerate → turn → run → settle instead of generic checkpoint archaeology.

    +
    +
    + {html.escape(f"default target +{default_target_lag_ticks}")} + {html.escape(default_target_lag_ms_text)} + {html.escape(f"default actual +{default_lag_ticks}")} + {html.escape(default_lag_ms_text)} +
    +
    +
    + {''.join(timeline_cards)} +
    +
    +
    + Target timestamp selector +
    + {target_lag_button_html} +
    +
    +
    + Actual / robot timestamp selector +
    + {lag_button_html} +
    +
    +
    +
    + {''.join(phase_cards)} +
    +
    +{diagnostics_html}''' + + +def render_policy_link_card( + entry: dict[str, object], + detail_href: str, + quality_html: str, +) -> str: + meta = dict(entry["_meta"]) + status = str(entry.get("status", "ok")) + metrics = entry.get("metrics") or {} + badge_bits = [ + quality_html if status == "ok" else pill("BLOCKED", "blocked"), + pill(str(meta["execution_kind"]).upper(), str(meta["execution_kind"])), + pill(showcase_transport_badge_label(str(meta.get("showcase_transport", "synthetic"))), "transport"), + pill(str(entry.get("command_kind", "")).upper(), "command"), + ] + metric_line = ( + f'{metrics.get("ticks", "-")} ticks · ' + f'{float(metrics.get("average_inference_ms", 0.0)):.3f} ms avg · ' + f'{float(metrics.get("achieved_frequency_hz", 0.0)):.2f} Hz' + if status == "ok" and isinstance(metrics, dict) + else f'{html.escape(str(meta.get("blocked_reason", "Blocked")))}' + "" + ) + links = render_overview_links(meta, detail_href) + return f'''
    +
    +
    +

    {html.escape(str(meta["title"]))}

    +

    {html.escape(str(meta["source"]))} · {html.escape(str(meta["coverage"]))}

    +

    {html.escape(entry_identity_label(entry))}

    +
    +
    {" ".join(badge_bits)}
    +
    +

    {html.escape(str(meta["summary"]))}

    +
    + {metric_line} + {html.escape(str(meta["demo_family"]))} +
    +

    {links}

    +
    ''' + + +def showcase_styles() -> str: + return """\ + :root { + color-scheme: light; + --bg: #f5f7fb; + --panel: #ffffff; + --text: #142033; + --muted: #5f6f85; + --border: #d9e0ea; + --shadow: 0 18px 50px rgba(20, 32, 51, 0.08); + --real-bg: #e7f7ef; + --real-fg: #11643a; + --experimental-bg: #fff4e5; + --experimental-fg: #9a3412; + --fixture-bg: #e7f0ff; + --fixture-fg: #1146a6; + --blocked-bg: #fff1f2; + --blocked-fg: #b42318; + --command-bg: #fff6db; + --command-fg: #8a5b00; + --meta-bg: #edf2f7; + --meta-fg: #334155; + --transport-bg: #e8f1ff; + --transport-fg: #1d4ed8; + --good-bg: #e7f7ef; + --good-fg: #11643a; + --bad-bg: #fff1f2; + --bad-fg: #b42318; + --unknown-bg: #fff6db; + --unknown-fg: #8a5b00; + } + * { box-sizing: border-box; } + body { margin: 0; font-family: "IBM Plex Sans", "Segoe UI", sans-serif; background: radial-gradient(circle at top, #eef7ff, var(--bg) 45%); color: var(--text); } + main { width: min(1180px, calc(100% - 32px)); margin: 0 auto; padding: 40px 0 64px; } + h1, h2, h3, p { margin-top: 0; } + a { color: #0f5bd3; } + .hero { background: linear-gradient(135deg, #ffffff, #ecf4ff); border: 1px solid var(--border); border-radius: 28px; box-shadow: var(--shadow); padding: 32px; margin-bottom: 28px; } + .hero p { max-width: 80ch; line-height: 1.6; } + .meta-row { display: flex; gap: 16px; flex-wrap: wrap; color: var(--muted); font-size: 0.95rem; } + .breadcrumbs { margin-bottom: 12px; font-weight: 700; } + .overview, .footer-panel { background: var(--panel); border: 1px solid var(--border); border-radius: 24px; box-shadow: var(--shadow); padding: 24px; margin-bottom: 28px; } + .demo-section { margin-bottom: 28px; } + .section-header { margin-bottom: 16px; } + .section-header p { max-width: 80ch; } + table { width: 100%; border-collapse: collapse; } + th, td { text-align: left; padding: 12px 10px; border-bottom: 1px solid var(--border); vertical-align: top; } + th { font-size: 0.82rem; text-transform: uppercase; letter-spacing: 0.06em; color: var(--muted); } + .cards { display: grid; gap: 20px; } + .card { background: var(--panel); border: 1px solid var(--border); border-radius: 24px; box-shadow: var(--shadow); padding: 24px; } + .policy-link-card { display: block; text-decoration: none; background: var(--panel); border: 1px solid var(--border); border-radius: 24px; box-shadow: var(--shadow); padding: 24px; color: inherit; } + .policy-link-card:hover { border-color: #b9cae3; box-shadow: 0 20px 55px rgba(20, 32, 51, 0.12); } + .policy-link-header { display: flex; align-items: flex-start; justify-content: space-between; gap: 16px; } + .policy-link-meta { display: flex; gap: 16px; flex-wrap: wrap; color: var(--muted); margin-bottom: 14px; } + .blocked-card { border-color: #f5c2c7; } + .card-header { display: flex; align-items: flex-start; justify-content: space-between; gap: 16px; } + .badge-row { display: flex; flex-wrap: wrap; gap: 8px; justify-content: flex-end; } + .pill { border-radius: 999px; padding: 8px 12px; font-size: 0.82rem; font-weight: 700; text-transform: uppercase; letter-spacing: 0.04em; display: inline-flex; align-items: center; } + .pill.real { background: var(--real-bg); color: var(--real-fg); } + .pill.experimental { background: var(--experimental-bg); color: var(--experimental-fg); } + .pill.fixture { background: var(--fixture-bg); color: var(--fixture-fg); } + .pill.blocked { background: var(--blocked-bg); color: var(--blocked-fg); } + .pill.ok { background: var(--real-bg); color: var(--real-fg); } + .pill.good { background: var(--good-bg); color: var(--good-fg); } + .pill.bad { background: var(--bad-bg); color: var(--bad-fg); } + .pill.unknown { background: var(--unknown-bg); color: var(--unknown-fg); } + .pill.command { background: var(--command-bg); color: var(--command-fg); } + .pill.meta { background: var(--meta-bg); color: var(--meta-fg); text-transform: none; } + .pill.transport { background: var(--transport-bg); color: var(--transport-fg); } + .muted { color: var(--muted); } + .stats { display: grid; grid-template-columns: repeat(auto-fit, minmax(160px, 1fr)); gap: 12px; margin: 16px 0 20px; } + .stats div, .details-grid div { background: #f7f9fc; border: 1px solid var(--border); border-radius: 16px; padding: 12px 14px; } + .stats span, .details-grid span, .blocked-paths span { display: block; color: var(--muted); font-size: 0.82rem; margin-bottom: 6px; text-transform: uppercase; letter-spacing: 0.05em; } + .stats strong { font-size: 1.05rem; } + .charts-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(240px, 1fr)); gap: 16px; margin-bottom: 18px; } + figure { margin: 0; } + figcaption { margin-bottom: 10px; font-weight: 700; } + .chart { width: 100%; height: auto; display: block; } + .chart rect { fill: #fbfdff; stroke: var(--border); } + .chart .baseline { stroke: #d4dae3; stroke-width: 1; } + .details-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(220px, 1fr)); gap: 12px; } + .rerun-block { margin: 0 0 18px; } + .rerun-block-header { display: flex; justify-content: space-between; gap: 12px; align-items: baseline; margin-bottom: 10px; flex-wrap: wrap; } + .rerun-stage { min-height: 420px; border-radius: 18px; border: 1px solid var(--border); background: linear-gradient(180deg, #0f172a, #111827); overflow: hidden; position: relative; } + .rerun-stage canvas { display: block; width: 100%; height: 420px; } + .rerun-stage-placeholder, .rerun-stage-error { position: absolute; inset: 0; display: flex; flex-direction: column; align-items: center; justify-content: center; gap: 8px; padding: 20px; text-align: center; color: #e5edf8; font-size: 0.95rem; } + .rerun-stage-placeholder strong, .rerun-stage-error strong { color: #ffffff; } + .rerun-stage-error { background: linear-gradient(180deg, rgba(127, 29, 29, 0.95), rgba(69, 10, 10, 0.96)); } + .blocked-reason, .blocked-paths { margin-top: 18px; background: #fff7f7; border: 1px solid #f5c2c7; border-radius: 16px; padding: 14px; } + code { font-family: "IBM Plex Mono", "SFMono-Regular", monospace; font-size: 0.9rem; word-break: break-word; } + .links { margin-top: 16px; line-height: 1.35; } + .proof-checkpoint-grid { display: grid; gap: 18px; } + .proof-checkpoint-card { border: 1px solid var(--border); border-radius: 20px; padding: 18px; background: #fbfdff; } + .proof-checkpoint-head { display: flex; justify-content: space-between; gap: 12px; align-items: flex-start; flex-wrap: wrap; margin-bottom: 14px; } + .proof-checkpoint-head h3 { margin-bottom: 6px; } + .proof-checkpoint-meta { display: flex; flex-wrap: wrap; gap: 10px; color: var(--muted); font-size: 0.92rem; } + .proof-view-grid { display: grid; gap: 14px; grid-template-columns: repeat(auto-fit, minmax(220px, 1fr)); } + .proof-view { margin: 0; } + .proof-view img { width: 100%; height: auto; display: block; border-radius: 14px; border: 1px solid var(--border); background: #f8fafc; } + .proof-view figcaption { margin-top: 8px; color: var(--muted); font-weight: 600; text-transform: uppercase; letter-spacing: 0.04em; font-size: 0.78rem; } + .phase-review-header { display: flex; justify-content: space-between; align-items: flex-start; gap: 16px; flex-wrap: wrap; margin-bottom: 18px; } + .phase-timeline-grid { display: grid; gap: 12px; grid-template-columns: repeat(auto-fit, minmax(170px, 1fr)); margin-bottom: 18px; } + .phase-timeline-card { border: 1px solid var(--border); border-radius: 18px; padding: 14px 16px; background: linear-gradient(180deg, #fbfdff, #f2f7ff); } + .phase-timeline-card h3 { margin-bottom: 6px; } + .phase-lag-controls { display: grid; gap: 12px; margin-bottom: 18px; } + .phase-lag-selector { display: flex; justify-content: space-between; align-items: center; gap: 14px; flex-wrap: wrap; padding: 14px 16px; border: 1px solid var(--border); border-radius: 18px; background: #f7f9fc; margin-bottom: 0; } + .phase-lag-buttons { display: flex; gap: 10px; flex-wrap: wrap; } + .phase-lag-button { border: 1px solid var(--border); background: white; color: #0f172a; border-radius: 999px; padding: 8px 12px; font: inherit; font-weight: 700; cursor: pointer; } + .phase-lag-button[data-active="true"] { background: #0f766e; border-color: #0f766e; color: white; } + .phase-checkpoint-stack { display: grid; gap: 18px; } + .phase-checkpoint-stack h4 { margin-bottom: 10px; font-size: 0.96rem; letter-spacing: 0.02em; } + .phase-debug-panel { margin-top: 16px; border: 1px solid var(--border); border-radius: 16px; background: #ffffff; padding: 12px 14px; } + .phase-debug-panel summary { cursor: pointer; font-weight: 700; } + .phase-debug-panel p { margin: 10px 0 0; } + .phase-debug-panel pre { margin: 10px 0 0; padding: 12px; border-radius: 12px; background: #0f172a; color: #e2e8f0; overflow-x: auto; font-size: 0.82rem; line-height: 1.45; } + .diagnostics-section { margin-top: 18px; } + .diagnostic-grid .diagnostic-card { background: #ffffff; border: 1px solid var(--border); border-radius: 18px; padding: 16px; } + ul { margin-bottom: 0; } + @media (max-width: 720px) { + main { width: min(100% - 20px, 1180px); padding-top: 20px; } + .hero, .overview, .card, .footer-panel, .policy-link-card { padding: 20px; border-radius: 20px; } + .card-header, .policy-link-header { flex-direction: column; } + .badge-row { justify-content: flex-start; } + } + """ + + +def viewer_loader_script(viewer_module_path: str) -> str: + return f"""""" + + +def render_policy_detail_page( + page_title: str, + page_summary: str, + body_html: str, + back_href: str, + generated_at: str, + commit_html: str, + run_html: str, + viewer_module_path: str, +) -> str: + return f''' + + + + + {html.escape(page_title)} · RoboWBC Site + + + +
    +
    +

    ← Back to site home

    +

    {html.escape(page_title)}

    +

    {html.escape(page_summary)}

    +
    + Generated: {html.escape(generated_at)} + Commit: {commit_html or 'local'} + {run_html} +
    +
    + {body_html} +
    + {viewer_loader_script(viewer_module_path)} + +''' + + +def render_html(entries: list[dict[str, object]], output_dir: Path, repo_root: Path) -> None: + generated_at = dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%d %H:%M:%SZ") + sha = os.environ.get("GITHUB_SHA", "") + repo = os.environ.get("GITHUB_REPOSITORY", "") + server = os.environ.get("GITHUB_SERVER_URL", "https://github.com") + run_id = os.environ.get("GITHUB_RUN_ID", "") + commit_link = f"{server}/{repo}/commit/{sha}" if sha and repo else "" + run_link = f"{server}/{repo}/actions/runs/{run_id}" if run_id and repo else "" + vendor_rerun_web_viewer(repo_root, output_dir) + index_path = output_dir / "index.html" + (output_dir / "policies").mkdir(parents=True, exist_ok=True) + commit_html = ( + f'{html.escape(sha[:12])}' + if commit_link + else html.escape(sha[:12]) + ) + run_html = f'Actions run' if run_link else "" + benchmark_pages = discover_benchmark_pages(output_dir, index_path) + + overview_rows: list[str] = [] + velocity_cards: list[str] = [] + tracking_cards: list[str] = [] + normalized_entries: list[dict[str, object]] = [] + + sorted_entries = [ + entry + for index, entry in sorted( + enumerate(entries), + key=lambda item: display_sort_key(item[0], item[1]), + ) + ] + + for entry in sorted_entries: + card_id = entry_card_id(entry) + policy_family = entry_policy_family(entry) + detail_path = detail_page_path(output_dir, card_id) + detail_path.parent.mkdir(parents=True, exist_ok=True) + detail_href = f"{relative_href(index_path, detail_path.parent)}/" + back_href = relative_href(detail_path, index_path) + viewer_module_path = relative_href( + detail_path, + output_dir / RERUN_WEB_VIEWER_DIR / "index.js", + ) + meta = dict(entry["_meta"]) + meta.setdefault("card_id", card_id) + meta.setdefault("policy_family", policy_family) + meta["detail_page"] = detail_href + normalized_entry = dict(entry) + normalized_entry["card_id"] = card_id + normalized_entry["policy_name"] = policy_family + normalized_entry["_meta"] = meta + normalized_entry["detail_page"] = detail_href + normalized_entries.append(normalized_entry) + + status = normalized_entry.get("status", "ok") + execution_kind = str(meta["execution_kind"]) + identity_label = entry_identity_label(entry) + command_kind = str(normalized_entry.get("command_kind", "")) + transport = str(meta.get("showcase_transport", "synthetic")) + model_variant = showcase_model_variant_text(meta.get("showcase_model_variant")) + transport_html = pill(showcase_transport_badge_label(transport), "transport") + status_html = pill( + "OK" if status == "ok" else "BLOCKED", + "ok" if status == "ok" else "blocked", + ) + provenance_html = " ".join([pill(execution_kind.upper(), execution_kind), transport_html]) + + frames = normalized_entry.get("frames", []) + metrics = normalized_entry.get("metrics") or {} + quality_verdict = None + if status == "ok" and isinstance(metrics, dict): + metrics.setdefault("target_tracking", derive_target_tracking_metrics(frames)) + quality_verdict = classify_quality_verdict(status, command_kind, metrics) + normalized_entry["quality_verdict"] = quality_verdict + quality_html = ( + pill(str(quality_verdict["label"]), str(quality_verdict["css_class"])) + if isinstance(quality_verdict, dict) + else 'n/a' + ) + ticks = metrics.get("ticks", "-") + avg_inference = ( + f"{metrics['average_inference_ms']:.3f} ms" if metrics else "-" + ) + achieved_hz = ( + f"{metrics['achieved_frequency_hz']:.2f} Hz" if metrics else "-" + ) + dropped_frames = metrics.get("dropped_frames", "-") + overview_links = render_overview_links(meta, detail_href) + + overview_rows.append( + f"{html.escape(str(meta['title']))}" + f"
    {html.escape(identity_label)}
    " + f"

    {overview_links}

    " + f"{status_html}" + f"{quality_html}" + f"{provenance_html}" + f"{html.escape(str(meta['demo_family']))}" + f"{html.escape(str(meta['coverage']))}" + f"{ticks}" + f"{avg_inference}" + f"{achieved_hz}" + f"{dropped_frames}" + ) + + badge_bits = [] + if isinstance(quality_verdict, dict): + badge_bits.append( + pill(str(quality_verdict["label"]), str(quality_verdict["css_class"])) + ) + elif status != "ok": + badge_bits.append(pill("BLOCKED", "blocked")) + badge_bits.extend( + [ + pill(execution_kind.upper(), execution_kind), + transport_html, + pill(command_kind.upper(), "command"), + pill(str(meta["command_source"]), "meta"), + ] + ) + badge_row = " ".join(badge_bits) + detail_meta = rebase_meta_artifact_paths(meta, detail_path, output_dir) + detail_summary = ( + f"{str(meta['summary'])} This page records the exact blocker and any missing assets." + if status != "ok" + else f"{str(meta['summary'])} This page keeps the full charts, playback, and downloadable artifacts." + ) + overview_card = render_policy_link_card(normalized_entry, detail_href, quality_html) + + if status != "ok": + missing_paths = meta.get("missing_paths", []) + missing_html = "
    ".join(f"{html.escape(path)}" for path in missing_paths) + detail_body_html = f'''
    +
    +
    +

    {html.escape(str(detail_meta['title']))}

    +

    {html.escape(str(detail_meta['source']))} · {html.escape(str(detail_meta['coverage']))}

    +

    {html.escape(identity_label)}

    +
    +
    {badge_row}
    +
    +

    {html.escape(str(detail_meta['summary']))}

    +
    +
    + Case key + {html.escape(card_id)} +
    +
    + Policy family + {html.escape(policy_family)} +
    +
    + Command kind + {html.escape(command_kind)} +
    +
    + Expected behavior + {html.escape(str(detail_meta['coverage']))} +
    +
    + Status + Blocked +
    +
    + Showcase transport + {html.escape(showcase_transport_text(transport))} +
    +
    + Embodiment + {html.escape(str(detail_meta['showcase_model_path'] or '-'))} +
    +
    + MuJoCo model variant + {html.escape(model_variant)} +
    +
    + Checkpoint source + {html.escape(str(detail_meta['checkpoint_source']))} +
    +
    + Demo family + {html.escape(str(detail_meta['demo_family']))} +
    +
    + Demo sequence + {html.escape(str(detail_meta['demo_sequence']))} +
    +
    + Model artifact + {html.escape(str(detail_meta['model_artifact']))} +
    +
    + Config + {html.escape(str(detail_meta['config_path']))} +
    +
    +
    + Why blocked: {html.escape(str(detail_meta['blocked_reason']))} +
    +
    + Missing required paths +
    {missing_html or 'None'}
    +
    +
    ''' + detail_path.write_text( + render_policy_detail_page( + page_title=str(detail_meta["title"]), + page_summary=detail_summary, + body_html=detail_body_html, + back_href=back_href, + generated_at=generated_at, + commit_html=commit_html, + run_html=run_html, + viewer_module_path=viewer_module_path, + ), + encoding="utf-8", + ) + if str(meta["demo_family"]) == "Velocity tracking": + velocity_cards.append(overview_card) + else: + tracking_cards.append(overview_card) + continue + + metrics = normalized_entry["metrics"] + joint_names = normalized_entry.get("joint_names", []) + velocity_tracking_metrics = None + target_tracking_metrics = None + if isinstance(metrics, dict): + velocity_tracking_metrics = metrics.get("velocity_tracking") + target_tracking_metrics = metrics.get("target_tracking") + + target_series = [] + for idx, joint_name in enumerate(joint_names[:4]): + values = series_from_frames(frames, "target_positions", idx) + target_series.append( + { + "label": joint_name, + "values": values, + "color": COLORS[idx % len(COLORS)], + } + ) + + actual_vs_target = [] + if joint_names: + actual_vs_target = [ + { + "label": f"{joint_names[0]} actual", + "values": series_from_frames(frames, "actual_positions", 0), + "color": COLORS[0], + "dashed": True, + }, + { + "label": f"{joint_names[0]} target", + "values": series_from_frames(frames, "target_positions", 0), + "color": COLORS[1], + }, + ] + + latency_series = [ + { + "label": "latency_ms", + "values": [frame["inference_latency_ms"] for frame in frames], + "color": COLORS[4], + } + ] + + velocity_tracking_series = None + if command_kind in {"velocity", "velocity_schedule"}: + velocity_tracking_series = derive_velocity_tracking_series( + frames, + int(normalized_entry.get("control_frequency_hz", 0) or 0), + ) + command_series = [ + { + "label": "vx_cmd", + "values": series_from_frames(frames, "command_data", 0), + "color": COLORS[2], + }, + { + "label": "yaw_cmd", + "values": series_from_frames(frames, "command_data", 2), + "color": COLORS[3], + }, + ] + command_chart_title = "Velocity command profile" + else: + command_series = [ + { + "label": f"{joint_names[0]} velocity" if joint_names else "joint0_velocity", + "values": series_from_frames(frames, "actual_velocities", 0), + "color": COLORS[2], + } + ] + command_chart_title = "Observed joint velocity" + + if velocity_tracking_series is not None: + second_chart_title = "Body vx command vs actual" + second_chart_series = [ + { + "label": "vx_cmd", + "values": velocity_tracking_series["vx_cmd"], + "color": COLORS[2], + }, + { + "label": "vx_actual", + "values": velocity_tracking_series["vx_actual"], + "color": COLORS[0], + "dashed": True, + }, + ] + third_chart_title = "Yaw rate command vs actual" + third_chart_series = [ + { + "label": "yaw_cmd", + "values": velocity_tracking_series["yaw_cmd"], + "color": COLORS[3], + }, + { + "label": "yaw_actual", + "values": velocity_tracking_series["yaw_actual"], + "color": COLORS[1], + "dashed": True, + }, + ] + else: + second_chart_title = "Joint 0 actual vs target" + second_chart_series = actual_vs_target + third_chart_title = command_chart_title + third_chart_series = command_series + + if isinstance(velocity_tracking_metrics, dict): + velocity_tracking_details = f''' +
    + VX RMSE + {float(velocity_tracking_metrics["vx_rmse_mps"]):.3f} m/s +
    +
    + Yaw RMSE + {float(velocity_tracking_metrics["yaw_rate_rmse_rad_s"]):.3f} rad/s +
    +
    + Heading change + {float(velocity_tracking_metrics["heading_change_deg"]):.1f} deg +
    +
    + Forward distance + {float(velocity_tracking_metrics["forward_distance_m"]):.3f} m +
    ''' + else: + velocity_tracking_details = "" + + if isinstance(target_tracking_metrics, dict): + min_base_height = target_tracking_metrics.get("base_height_min_m") + min_base_height_text = ( + f"{float(min_base_height):.3f} m" + if min_base_height is not None + else "n/a" + ) + target_tracking_details = f''' +
    + Mean joint error + {float(target_tracking_metrics["mean_joint_abs_error_rad"]):.3f} rad +
    +
    + Joint error p95 + {float(target_tracking_metrics["p95_joint_abs_error_rad"]):.3f} rad +
    +
    + Peak joint error + {float(target_tracking_metrics["peak_joint_abs_error_rad"]):.3f} rad +
    +
    + Min base height + {html.escape(min_base_height_text)} +
    +
    + Frames height < 0.4 m + {int(target_tracking_metrics["frames_below_base_height_0_4m"])} +
    ''' + else: + target_tracking_details = "" + + if isinstance(quality_verdict, dict): + verdict_details = f''' +
    + Quality verdict + {html.escape(str(quality_verdict["label"]))} +
    +
    + Verdict basis + {html.escape(str(quality_verdict["summary"]))} +
    ''' + else: + verdict_details = "" + + proof_pack_links: list[str] = [] + proof_pack_manifest_file = detail_meta.get("proof_pack_manifest_file") + if proof_pack_manifest_file: + proof_pack_links.append( + f'Proof-pack manifest' + ) + proof_pack_links_html = ( + " · " + " · ".join(proof_pack_links) if proof_pack_links else "" + ) + proof_pack_section = render_proof_pack_section( + normalized_entry.get("_proof_pack_manifest") + if isinstance(normalized_entry.get("_proof_pack_manifest"), dict) + else None + ) + + detail_body_html = f'''
    +
    +
    +

    {html.escape(str(detail_meta['title']))}

    +

    {html.escape(str(detail_meta['source']))} · {html.escape(str(detail_meta['coverage']))}

    +

    {html.escape(identity_label)}

    +
    +
    {badge_row}
    +
    +

    {html.escape(str(detail_meta['summary']))}

    +
    +
    Robot{html.escape(str(normalized_entry['robot_name']))}
    +
    Ticks{metrics['ticks']}
    +
    Avg inference{metrics['average_inference_ms']:.3f} ms
    +
    Achieved rate{metrics['achieved_frequency_hz']:.2f} Hz
    +
    +
    +
    +
    Target positions
    + {spark_svg(target_series)} +
    +
    +
    {html.escape(second_chart_title)}
    + {spark_svg(second_chart_series)} +
    +
    +
    {html.escape(third_chart_title)}
    + {spark_svg(third_chart_series)} +
    +
    +
    Inference latency
    + {spark_svg(latency_series)} +
    +
    +
    +
    + Embedded Rerun viewer + Fetches {html.escape(str(detail_meta['rrd_file']))} lazily when the viewer enters the viewport. +
    +
    +
    + Preparing interactive view + Loads the viewer runtime and recording on demand when visible. +
    +
    +
    +
    +
    + Case key + {html.escape(card_id)} +
    +
    + Policy family + {html.escape(policy_family)} +
    +
    + Command kind + {html.escape(command_kind)} +
    +
    + Expected behavior + {html.escape(str(detail_meta['coverage']))} +
    +
    + Showcase transport + {html.escape(showcase_transport_text(transport))} +
    +
    + Embodiment + {html.escape(str(detail_meta['showcase_model_path'] or '-'))} +
    +
    + MuJoCo model variant + {html.escape(model_variant)} +
    +
    + Command data + {html.escape(format_vector(normalized_entry.get('command_data', [])))} +
    +
    + Checkpoint source + {html.escape(str(detail_meta['checkpoint_source']))} +
    +
    + Demo family + {html.escape(str(detail_meta['demo_family']))} +
    +
    + Demo sequence + {html.escape(str(detail_meta['demo_sequence']))} +
    +
    + Model artifact + {html.escape(str(detail_meta['model_artifact']))} +
    +
    + Command source + {html.escape(str(detail_meta['command_source']))} +
    +
    + First target frame + {html.escape(format_vector(frames[0]['target_positions'] if frames else []))} +
    +
    + Last target frame + {html.escape(format_vector(frames[-1]['target_positions'] if frames else []))} +
    + {verdict_details} + {target_tracking_details} + {velocity_tracking_details} +
    +

    Rerun recording · JSON summary · run log{proof_pack_links_html} · {html.escape(str(detail_meta['config_path']))}

    +
    +{proof_pack_section}''' + detail_path.write_text( + render_policy_detail_page( + page_title=str(detail_meta["title"]), + page_summary=detail_summary, + body_html=detail_body_html, + back_href=back_href, + generated_at=generated_at, + commit_html=commit_html, + run_html=run_html, + viewer_module_path=viewer_module_path, + ), + encoding="utf-8", + ) + if str(meta["demo_family"]) == "Velocity tracking": + velocity_cards.append(overview_card) + else: + tracking_cards.append(overview_card) + + excluded = "".join( + f"
  • {html.escape(item['name'])}: {html.escape(item['reason'])}
    • " + for item in NOT_YET_SHOWCASED + ) + + html_doc = f''' + + + + + RoboWBC Site + + + +
    +
    +

    RoboWBC Site

    +

    This site is generated automatically in CI from the runnable policy integrations and benchmark packages that exist today. The home page is comparison-first: use it to cross-check policy status, quality, provenance, and demo coverage quickly, then open a policy folder for charts, Rerun playback, logs, and visual checkpoints.

    +

    Each policy now owns its own folder under policies/<policy>/, so the HTML page and raw artifacts live together instead of being scattered at the site root. The same bundle also carries benchmark families under benchmarks/.

    +

    Velocity runs use staged locomotion command profiles instead of a single constant command. Reference or pose-tracking cases stay explicitly blocked unless a verified official asset and runtime path actually exist, so the site does not silently drift into mock output.

    +

    The public G1 cards currently load a meshless MuJoCo MJCF variant because this repository does not redistribute Unitree's upstream STL mesh bundle. The dynamics stay MuJoCo-backed, while the Rerun robot scene is reconstructed from the same open MJCF kinematic tree.

    +

    Serve the generated folder over HTTP for reliable playback. Each policy page lazy-loads the saved .rrd recording and the visual checkpoints inline, so you do not need a second proof-pack HTML flow for normal review.

    +
    + Generated: {html.escape(generated_at)} + Commit: {commit_html or 'local'} + {run_html} +
    +
    + +
    +

    Policy runs

    +

    Successful entries use real checkpoints or public asset bundles cached by CI and must activate the requested MuJoCo transport. The links in each row jump straight to the per-policy folder and raw artifacts. Blocked entries surface the exact missing files or unavailable upstream artifacts instead of falling back to mock output.

    + + + + + + {''.join(overview_rows)} + +
    PolicyStatusQualityRun pathDemo familyCoverageTicksAvg inferenceAchieved rateDropped frames
    +
    + + {render_demo_section("Velocity tracking", velocity_cards)} + {render_demo_section("Reference / pose tracking", tracking_cards)} + {render_benchmark_section(benchmark_pages)} + +
    +

    Not Yet In This Site

    +
      {excluded}
    +
    +
    + +''' + + index_path.write_text(html_doc, encoding="utf-8") + (output_dir / "manifest.json").write_text( + json.dumps(normalized_entries, indent=2), + encoding="utf-8", + ) + + +def main() -> int: + args = parse_args() + repo_root = Path(args.repo_root).resolve() + output_dir = Path(args.output_dir).resolve() + output_dir.mkdir(parents=True, exist_ok=True) + binary = Path(args.robowbc_binary).resolve() + if not binary.exists(): + raise SystemExit(f"robowbc binary not found: {binary}") + + env = os.environ.copy() + dylib = resolve_ort_dylib(repo_root) + if dylib: + env.setdefault("ROBOWBC_ORT_DYLIB_PATH", dylib) + env = configure_binary_runtime_env(env) + + entries = [run_policy(repo_root, binary, output_dir, policy, env) for policy in POLICIES] + render_html(entries, output_dir, repo_root) + print(f"wrote site home to {output_dir / 'index.html'}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/site/serve_showcase.py b/scripts/site/serve_showcase.py new file mode 100755 index 0000000..f876e04 --- /dev/null +++ b/scripts/site/serve_showcase.py @@ -0,0 +1,68 @@ +#!/usr/bin/env python3 +"""Serve a generated RoboWBC site bundle over HTTP for local debugging.""" + +from __future__ import annotations + +import argparse +import functools +import http.server +import socketserver +import webbrowser +from pathlib import Path + + +class ShowcaseRequestHandler(http.server.SimpleHTTPRequestHandler): + extensions_map = { + **http.server.SimpleHTTPRequestHandler.extensions_map, + ".wasm": "application/wasm", + ".rrd": "application/octet-stream", + } + + def end_headers(self) -> None: + self.send_header("Cache-Control", "no-store") + self.send_header("Access-Control-Allow-Origin", "*") + super().end_headers() + + +class ReusableTCPServer(socketserver.TCPServer): + allow_reuse_address = True + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument( + "--dir", + default=".", + help="Directory containing the generated site bundle and index.html", + ) + parser.add_argument("--bind", default="127.0.0.1", help="Address to bind the local server") + parser.add_argument("--port", type=int, default=8000, help="Port to bind the local server") + parser.add_argument("--open", action="store_true", help="Open the served page in the default browser") + return parser.parse_args() + + +def main() -> int: + args = parse_args() + root = Path(args.dir).resolve() + if not root.is_dir(): + raise SystemExit(f"site directory not found: {root}") + if not (root / "index.html").is_file(): + raise SystemExit(f"expected {root / 'index.html'} to exist") + + handler = functools.partial(ShowcaseRequestHandler, directory=str(root)) + with ReusableTCPServer((args.bind, args.port), handler) as httpd: + url = f"http://{args.bind}:{args.port}/" + print(f"Serving RoboWBC site from {root}") + print(f"Open {url}") + print("Press Ctrl-C to stop.") + if args.open: + webbrowser.open(url) + try: + httpd.serve_forever() + except KeyboardInterrupt: + print("\nStopping showcase server.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/site/site_browser_smoke.py b/scripts/site/site_browser_smoke.py new file mode 100755 index 0000000..acabbf6 --- /dev/null +++ b/scripts/site/site_browser_smoke.py @@ -0,0 +1,570 @@ +#!/usr/bin/env python3 +"""Run a reusable headless browser smoke check against a built RoboWBC site.""" + +from __future__ import annotations + +import argparse +import contextlib +import html +import http.server +import json +import os +import re +import shutil +import subprocess +import threading +import time +import urllib.error +import urllib.request +import uuid +from pathlib import Path + +PROBE_RESULT_PATTERN = re.compile( + r'
    ]*>(?P.*?)
    ', + re.DOTALL, +) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--root", + type=Path, + required=True, + help="Root directory of the generated site bundle", + ) + parser.add_argument( + "--policy", + default="gear_sonic", + help="Policy detail page to inspect under policies//", + ) + parser.add_argument( + "--bind", + default="127.0.0.1", + help="Bind address for the temporary HTTP server", + ) + parser.add_argument( + "--port", + type=int, + default=0, + help="Port for the temporary HTTP server; use 0 for an ephemeral port", + ) + parser.add_argument( + "--chrome-binary", + default="", + help="Optional path to the Chrome/Chromium binary to use", + ) + parser.add_argument( + "--timeout-secs", + type=int, + default=30, + help="Timeout for the browser probe command", + ) + parser.add_argument( + "--keep-probe", + action="store_true", + help="Keep the generated probe HTML file in the site root for manual debugging", + ) + return parser.parse_args() + + +def validate_policy_name(policy: str) -> str: + candidate = Path(policy) + if candidate.is_absolute() or any(part in {"", ".", ".."} for part in candidate.parts): + raise SystemExit(f"policy id must stay within policies//, got {policy!r}") + if len(candidate.parts) != 1: + raise SystemExit(f"policy id must be a single path component, got {policy!r}") + return candidate.parts[0] + + +def find_chrome_binary(requested: str) -> str: + if requested: + resolved = shutil.which(requested) or requested + if Path(resolved).is_file(): + return str(Path(resolved)) + raise SystemExit(f"chrome binary not found: {requested}") + + for candidate in ( + "google-chrome", + "google-chrome-stable", + "chromium", + "chromium-browser", + ): + resolved = shutil.which(candidate) + if resolved: + return resolved + raise SystemExit( + "could not find a Chrome/Chromium binary; pass --chrome-binary explicitly" + ) + + +def build_probe_html(policy: str) -> str: + policy_path = f"/policies/{policy}/" + return f""" + + + + RoboWBC Site Browser Smoke + + + +

    RoboWBC Site Browser Smoke

    +

    Probing {html.escape(policy_path)}.

    + + 
    {{"status": "pending"}}
    + + + +""" + + +def extract_probe_result(dumped_dom: str) -> dict[str, object]: + match = PROBE_RESULT_PATTERN.search(dumped_dom) + if match is None: + raise SystemExit("browser probe output missing #smoke-result payload") + payload_text = html.unescape(match.group("payload")).strip() + if not payload_text: + raise SystemExit("browser probe emitted an empty result payload") + payload = json.loads(payload_text) + if not isinstance(payload, dict): + raise SystemExit("browser probe payload must be a JSON object") + return payload + + +def validate_probe_result(result: dict[str, object], policy: str) -> None: + if result.get("policy") != policy: + raise SystemExit( + f"browser probe returned result for {result.get('policy')!r}, expected {policy!r}" + ) + if result.get("error"): + raise SystemExit(f"browser probe failed: {result['error']}") + + plan = result.get("plan") + if not isinstance(plan, dict): + raise SystemExit("browser probe result missing plan payload") + expected_plan_keys = { + "initial_target_lag", + "initial_actual_lag", + "target_probe_lag", + "actual_probe_lag", + "reset_target_lag", + "final_actual_lag", + } + if not expected_plan_keys.issubset(plan): + raise SystemExit(f"browser probe plan is incomplete: {sorted(plan.keys())}") + + results = result.get("results") + if not isinstance(results, list): + raise SystemExit("browser probe result missing transition snapshots") + snapshot_map = { + snapshot.get("tag"): snapshot + for snapshot in results + if isinstance(snapshot, dict) and isinstance(snapshot.get("tag"), str) + } + expected_snapshots = { + "initial": { + "srcKind": "asset", + "selectedTarget": str(plan["initial_target_lag"]), + "selectedActual": str(plan["initial_actual_lag"]), + }, + "target-2": { + "srcKind": "data-url", + "selectedTarget": str(plan["target_probe_lag"]), + "selectedActual": str(plan["initial_actual_lag"]), + }, + "actual-1": { + "srcKind": "data-url", + "selectedTarget": str(plan["target_probe_lag"]), + "selectedActual": str(plan["actual_probe_lag"]), + }, + "target-reset": { + "srcKind": "asset", + "selectedTarget": str(plan["reset_target_lag"]), + "selectedActual": str(plan["actual_probe_lag"]), + }, + "actual-5": { + "srcKind": "asset", + "selectedTarget": str(plan["reset_target_lag"]), + "selectedActual": str(plan["final_actual_lag"]), + }, + } + + for tag, expected in expected_snapshots.items(): + snapshot = snapshot_map.get(tag) + if not isinstance(snapshot, dict): + raise SystemExit(f"browser probe missing snapshot {tag!r}") + if snapshot.get("srcKind") != expected["srcKind"]: + raise SystemExit( + f"browser probe snapshot {tag!r} expected srcKind={expected['srcKind']!r}, " + f"found {snapshot.get('srcKind')!r}" + ) + if snapshot.get("selectedTarget") != expected["selectedTarget"]: + raise SystemExit( + f"browser probe snapshot {tag!r} expected selectedTarget={expected['selectedTarget']!r}, " + f"found {snapshot.get('selectedTarget')!r}" + ) + if snapshot.get("selectedActual") != expected["selectedActual"]: + raise SystemExit( + f"browser probe snapshot {tag!r} expected selectedActual={expected['selectedActual']!r}, " + f"found {snapshot.get('selectedActual')!r}" + ) + label = snapshot.get("label") + if not isinstance(label, str): + raise SystemExit(f"browser probe snapshot {tag!r} missing label text") + if f"T+{expected['selectedTarget']}" not in label: + raise SystemExit( + f"browser probe snapshot {tag!r} label missing target lag {expected['selectedTarget']!r}: {label!r}" + ) + if f"A+{expected['selectedActual']}" not in label: + raise SystemExit( + f"browser probe snapshot {tag!r} label missing actual lag {expected['selectedActual']!r}: {label!r}" + ) + + sample_labels = result.get("sampleLabels") + if not isinstance(sample_labels, list) or not sample_labels: + raise SystemExit("browser probe missing sample label coverage") + final_label = snapshot_map["actual-5"]["label"] + if not all(label == final_label for label in sample_labels if isinstance(label, str)): + raise SystemExit( + "browser probe sample labels did not converge on the final phase lag label" + ) + + +def wait_for_url(url: str, timeout_secs: float) -> None: + deadline = time.time() + timeout_secs + while time.time() < deadline: + try: + with urllib.request.urlopen(url, timeout=1.0) as response: + if 200 <= response.status < 500: + return + except (OSError, urllib.error.URLError): + time.sleep(0.1) + raise SystemExit(f"temporary site server did not become ready: {url}") + + +@contextlib.contextmanager +def serve_directory(root: Path, bind: str, port: int) -> tuple[str, int]: + class QuietHandler(http.server.SimpleHTTPRequestHandler): + def __init__(self, *args, **kwargs): + super().__init__(*args, directory=str(root), **kwargs) + + def log_message(self, format: str, *args) -> None: + return + + server = http.server.ThreadingHTTPServer((bind, port), QuietHandler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + yield bind, int(server.server_address[1]) + finally: + server.shutdown() + thread.join(timeout=5) + server.server_close() + + +def run_browser_probe(chrome_binary: str, url: str, timeout_secs: int) -> str: + command = [ + chrome_binary, + "--headless=new", + "--disable-gpu", + "--no-sandbox", + "--disable-dev-shm-usage", + "--run-all-compositor-stages-before-draw", + f"--virtual-time-budget={timeout_secs * 1000}", + "--dump-dom", + url, + ] + try: + completed = subprocess.run( + command, + check=False, + capture_output=True, + text=True, + timeout=timeout_secs, + env={**os.environ, "HOME": os.environ.get("HOME", str(Path.home()))}, + ) + except subprocess.TimeoutExpired as exc: + raise SystemExit( + f"chrome browser probe timed out after {timeout_secs}s while loading {url}" + ) from exc + if completed.returncode != 0: + raise SystemExit( + "chrome browser probe failed with " + f"exit code {completed.returncode}\nstdout:\n{completed.stdout}\nstderr:\n{completed.stderr}" + ) + return completed.stdout + + +def main() -> int: + args = parse_args() + root = args.root.resolve() + if not root.is_dir(): + raise SystemExit(f"site root does not exist: {root}") + + policy = validate_policy_name(args.policy) + detail_page = root / "policies" / policy / "index.html" + if not detail_page.is_file(): + raise SystemExit(f"missing policy detail page: {detail_page}") + + chrome_binary = find_chrome_binary(args.chrome_binary) + probe_name = f".site-browser-smoke-{uuid.uuid4().hex}.html" + probe_path = root / probe_name + probe_path.write_text(build_probe_html(policy), encoding="utf-8") + + try: + with serve_directory(root, args.bind, args.port) as (bind, port): + probe_url = f"http://{bind}:{port}/{probe_name}" + wait_for_url(probe_url, timeout_secs=5.0) + dumped_dom = run_browser_probe( + chrome_binary=chrome_binary, + url=probe_url, + timeout_secs=args.timeout_secs, + ) + result = extract_probe_result(dumped_dom) + validate_probe_result(result, policy) + finally: + if not args.keep_probe and probe_path.exists(): + probe_path.unlink() + + final_snapshot = next( + snapshot + for snapshot in result["results"] + if isinstance(snapshot, dict) and snapshot.get("tag") == "actual-5" + ) + print( + "site browser smoke check passed: " + f"policy={policy} final_label={final_snapshot['label']} src_kind={final_snapshot['srcKind']}" + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/site/validate_site_bundle.py b/scripts/site/validate_site_bundle.py new file mode 100755 index 0000000..c958c43 --- /dev/null +++ b/scripts/site/validate_site_bundle.py @@ -0,0 +1,323 @@ +#!/usr/bin/env python3 +"""Smoke-check a generated RoboWBC static site bundle.""" + +from __future__ import annotations + +import argparse +import html +import json +from pathlib import Path + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--root", + type=Path, + required=True, + help="Root directory of the generated site bundle", + ) + return parser.parse_args() + + +def load_json(path: Path) -> object: + return json.loads(path.read_text(encoding="utf-8")) + + +def resolve_relative_child(root: Path, relative_path: str, *, label: str) -> Path: + candidate = Path(relative_path) + if candidate.is_absolute(): + raise SystemExit(f"{label} must be repo-relative, got absolute path {candidate}") + resolved = (root / candidate).resolve() + try: + resolved.relative_to(root.resolve()) + except ValueError as exc: + raise SystemExit(f"{label} escapes the site bundle root: {relative_path}") from exc + return resolved + + +def detail_dir(root: Path, entry: dict[str, object]) -> Path: + detail_page = entry.get("detail_page") + if not isinstance(detail_page, str) or not detail_page: + raise SystemExit("detail_page missing from manifest entry") + return resolve_relative_child(root, detail_page, label="detail_page") + + +def validate_detail_page(detail_path: Path) -> str: + detail_index = detail_path / "index.html" + if not detail_index.is_file(): + raise SystemExit(f"missing policy detail page: {detail_index}") + return detail_index.read_text(encoding="utf-8") + + +def validate_ok_detail_page(detail_path: Path, detail_text: str) -> None: + detail_index = detail_path / "index.html" + if "../../assets/rerun-web-viewer/index.js" not in detail_text: + raise SystemExit(f"detail page missing rebased viewer path: {detail_index}") + if 'data-rrd-file="run.rrd"' not in detail_text: + raise SystemExit(f"detail page missing local run recording path: {detail_index}") + if "proof_pack_manifest.json" not in detail_text: + raise SystemExit(f"detail page missing proof-pack manifest reference: {detail_index}") + + +def validate_proof_pack_capture(root: Path, entry: dict[str, object], detail_text: str) -> None: + card_id = str(entry.get("card_id", "")) + meta = entry.get("_meta") + if not isinstance(meta, dict): + raise SystemExit(f"{card_id}: manifest entry missing _meta payload") + + if meta.get("showcase_transport") != "mujoco": + return + + manifest_rel = meta.get("proof_pack_manifest_file") + if not isinstance(manifest_rel, str) or not manifest_rel: + raise SystemExit(f"{card_id}: missing proof_pack_manifest_file") + + manifest_path = resolve_relative_child( + root, manifest_rel, label=f"{card_id}: proof_pack_manifest_file" + ) + if not manifest_path.is_file(): + raise SystemExit(f"{card_id}: missing proof-pack manifest: {manifest_path}") + + proof_pack_manifest = load_json(manifest_path) + if not isinstance(proof_pack_manifest, dict): + raise SystemExit(f"{card_id}: invalid proof-pack manifest payload: {manifest_path}") + + capture_status = proof_pack_manifest.get("capture_status") + if capture_status != "ok": + raise SystemExit( + f"{card_id}: expected proof-pack capture_status=ok, found {capture_status!r}" + ) + + if "Screenshots unavailable for this build" in detail_text: + raise SystemExit(f"{card_id}: detail page still shows screenshot fallback copy") + + proof_pack_root = manifest_path.parent + phase_review = proof_pack_manifest.get("phase_review") + if isinstance(phase_review, dict) and phase_review.get("enabled") is True: + if 'id="phase-timeline"' not in detail_text: + raise SystemExit(f"{card_id}: phase-aware detail page missing phase timeline section") + if 'id="phase-lag-selector"' not in detail_text: + raise SystemExit(f"{card_id}: phase-aware detail page missing actual timestamp selector") + if 'id="phase-target-lag-selector"' not in detail_text: + raise SystemExit(f"{card_id}: phase-aware detail page missing target timestamp selector") + if f'data-selected-lag="{proof_pack_manifest.get("default_lag_ticks")}"' not in detail_text: + raise SystemExit(f"{card_id}: phase-aware detail page missing static actual selected lag") + if ( + f'data-selected-lag="{proof_pack_manifest.get("default_target_lag_ticks")}"' + not in detail_text + ): + raise SystemExit(f"{card_id}: phase-aware detail page missing static target selected lag") + + phase_timeline = proof_pack_manifest.get("phase_timeline") + phase_checkpoints = proof_pack_manifest.get("phase_checkpoints") + lag_options = proof_pack_manifest.get("lag_options") + default_lag_ticks = proof_pack_manifest.get("default_lag_ticks") + target_lag_options = proof_pack_manifest.get("target_lag_options") + default_target_lag_ticks = proof_pack_manifest.get("default_target_lag_ticks") + if not isinstance(phase_timeline, list) or not phase_timeline: + raise SystemExit(f"{card_id}: phase-aware proof-pack manifest missing phase_timeline") + if not isinstance(phase_checkpoints, list) or not phase_checkpoints: + raise SystemExit(f"{card_id}: phase-aware proof-pack manifest missing phase_checkpoints") + if not isinstance(lag_options, list) or not lag_options: + raise SystemExit(f"{card_id}: phase-aware proof-pack manifest missing lag_options") + if not isinstance(default_lag_ticks, int): + raise SystemExit(f"{card_id}: phase-aware proof-pack manifest missing default_lag_ticks") + if not isinstance(target_lag_options, list) or not target_lag_options: + raise SystemExit(f"{card_id}: phase-aware proof-pack manifest missing target_lag_options") + if not isinstance(default_target_lag_ticks, int): + raise SystemExit( + f"{card_id}: phase-aware proof-pack manifest missing default_target_lag_ticks" + ) + for phase_entry in phase_timeline: + if not isinstance(phase_entry, dict): + raise SystemExit(f"{card_id}: phase-aware timeline entry is not an object") + phase_name = phase_entry.get("phase_name") + if not isinstance(phase_name, str) or not phase_name: + raise SystemExit(f"{card_id}: phase-aware timeline entry missing phase_name") + debug_marker = ( + f'data-phase-debug-phase="{html.escape(phase_name, quote=True)}"' + ) + if debug_marker not in detail_text: + raise SystemExit( + f"{card_id}: phase-aware detail page missing debug metadata for {phase_name}" + ) + + for checkpoint in phase_checkpoints: + if not isinstance(checkpoint, dict): + raise SystemExit(f"{card_id}: phase-aware checkpoint payload is not an object") + phase_kind = checkpoint.get("phase_kind") + cameras = checkpoint.get("cameras") + if phase_kind == "phase_end": + lag_variants = checkpoint.get("lag_variants") + checkpoint_lag_options = checkpoint.get("lag_options") + target_variants = checkpoint.get("target_lag_variants") + checkpoint_target_lag_options = checkpoint.get("target_lag_options") + if not isinstance(lag_variants, list) or not lag_variants: + raise SystemExit(f"{card_id}: phase-end checkpoint missing lag_variants") + if not isinstance(checkpoint_lag_options, list) or not checkpoint_lag_options: + raise SystemExit(f"{card_id}: phase-end checkpoint missing lag_options") + if not isinstance(target_variants, list) or not target_variants: + raise SystemExit(f"{card_id}: phase-end checkpoint missing target_lag_variants") + if not isinstance(checkpoint_target_lag_options, list) or not checkpoint_target_lag_options: + raise SystemExit(f"{card_id}: phase-end checkpoint missing target_lag_options") + for variant in lag_variants: + if not isinstance(variant, dict): + raise SystemExit(f"{card_id}: lag variant payload is not an object") + variant_dir = variant.get("relative_dir") + variant_cameras = variant.get("cameras") + if not isinstance(variant_dir, str) or not variant_dir: + raise SystemExit(f"{card_id}: lag variant missing relative_dir") + if not isinstance(variant_cameras, list) or not variant_cameras: + raise SystemExit(f"{card_id}: lag variant missing camera list") + resolved_variant_dir = resolve_relative_child( + proof_pack_root, + variant_dir, + label=f"{card_id}: lag variant relative_dir", + ) + if not resolved_variant_dir.is_dir(): + raise SystemExit( + f"{card_id}: missing lag checkpoint directory: {resolved_variant_dir}" + ) + for camera in variant_cameras: + if not isinstance(camera, str) or not camera: + raise SystemExit(f"{card_id}: invalid lag camera entry: {camera!r}") + image_path = resolved_variant_dir / f"{camera}_rgb.png" + if not image_path.is_file(): + raise SystemExit( + f"{card_id}: missing lag screenshot for +{variant.get('lag_ticks')}: {image_path}" + ) + actual_image_path = resolved_variant_dir / f"{camera}_actual_rgb.png" + if not actual_image_path.is_file(): + raise SystemExit( + f"{card_id}: missing raw actual lag screenshot for +{variant.get('lag_ticks')}: {actual_image_path}" + ) + target_image_path = resolved_variant_dir / f"{camera}_target_rgb.png" + if not target_image_path.is_file(): + raise SystemExit( + f"{card_id}: missing raw target lag screenshot for +{variant.get('lag_ticks')}: {target_image_path}" + ) + for variant in target_variants: + if not isinstance(variant, dict): + raise SystemExit(f"{card_id}: target lag variant payload is not an object") + variant_dir = variant.get("relative_dir") + variant_cameras = variant.get("cameras") + if not isinstance(variant_dir, str) or not variant_dir: + raise SystemExit(f"{card_id}: target lag variant missing relative_dir") + if not isinstance(variant_cameras, list) or not variant_cameras: + raise SystemExit(f"{card_id}: target lag variant missing camera list") + resolved_variant_dir = resolve_relative_child( + proof_pack_root, + variant_dir, + label=f"{card_id}: target lag variant relative_dir", + ) + if not resolved_variant_dir.is_dir(): + raise SystemExit( + f"{card_id}: missing target lag checkpoint directory: {resolved_variant_dir}" + ) + for camera in variant_cameras: + if not isinstance(camera, str) or not camera: + raise SystemExit( + f"{card_id}: invalid target lag camera entry: {camera!r}" + ) + image_path = resolved_variant_dir / f"{camera}_rgb.png" + if not image_path.is_file(): + raise SystemExit( + f"{card_id}: missing target lag screenshot for +{variant.get('lag_ticks')}: {image_path}" + ) + continue + + relative_dir = checkpoint.get("relative_dir") + if not isinstance(relative_dir, str) or not relative_dir: + raise SystemExit(f"{card_id}: phase midpoint checkpoint missing relative_dir") + if not isinstance(cameras, list) or not cameras: + raise SystemExit(f"{card_id}: phase midpoint checkpoint missing camera list") + checkpoint_dir = resolve_relative_child( + proof_pack_root, + relative_dir, + label=f"{card_id}: checkpoint relative_dir", + ) + if not checkpoint_dir.is_dir(): + raise SystemExit(f"{card_id}: missing proof-pack checkpoint directory: {checkpoint_dir}") + for camera in cameras: + if not isinstance(camera, str) or not camera: + raise SystemExit(f"{card_id}: invalid proof-pack camera entry: {camera!r}") + image_path = checkpoint_dir / f"{camera}_rgb.png" + if not image_path.is_file(): + raise SystemExit(f"{card_id}: missing proof-pack screenshot: {image_path}") + return + + checkpoints = proof_pack_manifest.get("checkpoints") + if not isinstance(checkpoints, list) or not checkpoints: + raise SystemExit(f"{card_id}: proof-pack manifest has no checkpoints") + for checkpoint in checkpoints: + if not isinstance(checkpoint, dict): + raise SystemExit(f"{card_id}: proof-pack checkpoint payload is not an object") + relative_dir = checkpoint.get("relative_dir") + cameras = checkpoint.get("cameras") + if not isinstance(relative_dir, str) or not relative_dir: + raise SystemExit(f"{card_id}: proof-pack checkpoint missing relative_dir") + if not isinstance(cameras, list) or not cameras: + raise SystemExit(f"{card_id}: proof-pack checkpoint missing camera list") + + checkpoint_dir = resolve_relative_child( + proof_pack_root, + relative_dir, + label=f"{card_id}: checkpoint relative_dir", + ) + if not checkpoint_dir.is_dir(): + raise SystemExit(f"{card_id}: missing proof-pack checkpoint directory: {checkpoint_dir}") + + for camera in cameras: + if not isinstance(camera, str) or not camera: + raise SystemExit(f"{card_id}: invalid proof-pack camera entry: {camera!r}") + image_path = checkpoint_dir / f"{camera}_rgb.png" + if not image_path.is_file(): + raise SystemExit(f"{card_id}: missing proof-pack screenshot: {image_path}") + + +def validate_site_bundle(root: Path) -> None: + if not (root / "index.html").is_file(): + raise SystemExit(f"missing site home page: {root / 'index.html'}") + if not (root / "manifest.json").is_file(): + raise SystemExit(f"missing site manifest: {root / 'manifest.json'}") + if not (root / "assets" / "rerun-web-viewer" / "index.js").is_file(): + raise SystemExit("missing Rerun web viewer bundle") + if not (root / "benchmarks" / "nvidia" / "index.html").is_file(): + raise SystemExit("missing benchmark page") + + manifest = load_json(root / "manifest.json") + if not isinstance(manifest, list): + raise SystemExit("site manifest must be a JSON array") + if not manifest: + raise SystemExit("site manifest is empty") + if not all("detail_page" in entry for entry in manifest): + raise SystemExit("detail_page missing from manifest") + + ok_entries = [entry for entry in manifest if entry.get("status") == "ok"] + if not ok_entries: + raise SystemExit("expected at least one successful policy entry") + + for entry in manifest: + if not isinstance(entry, dict): + raise SystemExit("site manifest entries must be JSON objects") + path = detail_dir(root, entry) + detail_text = validate_detail_page(path) + if entry.get("status") == "ok": + validate_ok_detail_page(path, detail_text) + validate_proof_pack_capture(root, entry, detail_text) + + index_text = (root / "index.html").read_text(encoding="utf-8") + if "benchmarks/nvidia/index.html" not in index_text: + raise SystemExit("site home missing benchmark link") + +def main() -> int: + args = parse_args() + root = args.root.resolve() + validate_site_bundle(root) + print(f"site bundle smoke check passed: {root}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/site_browser_smoke.py b/scripts/site_browser_smoke.py old mode 100644 new mode 100755 index acabbf6..43bcc14 --- a/scripts/site_browser_smoke.py +++ b/scripts/site_browser_smoke.py @@ -1,570 +1,4 @@ #!/usr/bin/env python3 -"""Run a reusable headless browser smoke check against a built RoboWBC site.""" +from _compat import run_legacy_script -from __future__ import annotations - -import argparse -import contextlib -import html -import http.server -import json -import os -import re -import shutil -import subprocess -import threading -import time -import urllib.error -import urllib.request -import uuid -from pathlib import Path - -PROBE_RESULT_PATTERN = re.compile( - r'
    ]*>(?P.*?)
    ', - re.DOTALL, -) - - -def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument( - "--root", - type=Path, - required=True, - help="Root directory of the generated site bundle", - ) - parser.add_argument( - "--policy", - default="gear_sonic", - help="Policy detail page to inspect under policies//", - ) - parser.add_argument( - "--bind", - default="127.0.0.1", - help="Bind address for the temporary HTTP server", - ) - parser.add_argument( - "--port", - type=int, - default=0, - help="Port for the temporary HTTP server; use 0 for an ephemeral port", - ) - parser.add_argument( - "--chrome-binary", - default="", - help="Optional path to the Chrome/Chromium binary to use", - ) - parser.add_argument( - "--timeout-secs", - type=int, - default=30, - help="Timeout for the browser probe command", - ) - parser.add_argument( - "--keep-probe", - action="store_true", - help="Keep the generated probe HTML file in the site root for manual debugging", - ) - return parser.parse_args() - - -def validate_policy_name(policy: str) -> str: - candidate = Path(policy) - if candidate.is_absolute() or any(part in {"", ".", ".."} for part in candidate.parts): - raise SystemExit(f"policy id must stay within policies//, got {policy!r}") - if len(candidate.parts) != 1: - raise SystemExit(f"policy id must be a single path component, got {policy!r}") - return candidate.parts[0] - - -def find_chrome_binary(requested: str) -> str: - if requested: - resolved = shutil.which(requested) or requested - if Path(resolved).is_file(): - return str(Path(resolved)) - raise SystemExit(f"chrome binary not found: {requested}") - - for candidate in ( - "google-chrome", - "google-chrome-stable", - "chromium", - "chromium-browser", - ): - resolved = shutil.which(candidate) - if resolved: - return resolved - raise SystemExit( - "could not find a Chrome/Chromium binary; pass --chrome-binary explicitly" - ) - - -def build_probe_html(policy: str) -> str: - policy_path = f"/policies/{policy}/" - return f""" - - - - RoboWBC Site Browser Smoke - - - -

    RoboWBC Site Browser Smoke

    -

    Probing {html.escape(policy_path)}.

    - - 
    {{"status": "pending"}}
    - - - -""" - - -def extract_probe_result(dumped_dom: str) -> dict[str, object]: - match = PROBE_RESULT_PATTERN.search(dumped_dom) - if match is None: - raise SystemExit("browser probe output missing #smoke-result payload") - payload_text = html.unescape(match.group("payload")).strip() - if not payload_text: - raise SystemExit("browser probe emitted an empty result payload") - payload = json.loads(payload_text) - if not isinstance(payload, dict): - raise SystemExit("browser probe payload must be a JSON object") - return payload - - -def validate_probe_result(result: dict[str, object], policy: str) -> None: - if result.get("policy") != policy: - raise SystemExit( - f"browser probe returned result for {result.get('policy')!r}, expected {policy!r}" - ) - if result.get("error"): - raise SystemExit(f"browser probe failed: {result['error']}") - - plan = result.get("plan") - if not isinstance(plan, dict): - raise SystemExit("browser probe result missing plan payload") - expected_plan_keys = { - "initial_target_lag", - "initial_actual_lag", - "target_probe_lag", - "actual_probe_lag", - "reset_target_lag", - "final_actual_lag", - } - if not expected_plan_keys.issubset(plan): - raise SystemExit(f"browser probe plan is incomplete: {sorted(plan.keys())}") - - results = result.get("results") - if not isinstance(results, list): - raise SystemExit("browser probe result missing transition snapshots") - snapshot_map = { - snapshot.get("tag"): snapshot - for snapshot in results - if isinstance(snapshot, dict) and isinstance(snapshot.get("tag"), str) - } - expected_snapshots = { - "initial": { - "srcKind": "asset", - "selectedTarget": str(plan["initial_target_lag"]), - "selectedActual": str(plan["initial_actual_lag"]), - }, - "target-2": { - "srcKind": "data-url", - "selectedTarget": str(plan["target_probe_lag"]), - "selectedActual": str(plan["initial_actual_lag"]), - }, - "actual-1": { - "srcKind": "data-url", - "selectedTarget": str(plan["target_probe_lag"]), - "selectedActual": str(plan["actual_probe_lag"]), - }, - "target-reset": { - "srcKind": "asset", - "selectedTarget": str(plan["reset_target_lag"]), - "selectedActual": str(plan["actual_probe_lag"]), - }, - "actual-5": { - "srcKind": "asset", - "selectedTarget": str(plan["reset_target_lag"]), - "selectedActual": str(plan["final_actual_lag"]), - }, - } - - for tag, expected in expected_snapshots.items(): - snapshot = snapshot_map.get(tag) - if not isinstance(snapshot, dict): - raise SystemExit(f"browser probe missing snapshot {tag!r}") - if snapshot.get("srcKind") != expected["srcKind"]: - raise SystemExit( - f"browser probe snapshot {tag!r} expected srcKind={expected['srcKind']!r}, " - f"found {snapshot.get('srcKind')!r}" - ) - if snapshot.get("selectedTarget") != expected["selectedTarget"]: - raise SystemExit( - f"browser probe snapshot {tag!r} expected selectedTarget={expected['selectedTarget']!r}, " - f"found {snapshot.get('selectedTarget')!r}" - ) - if snapshot.get("selectedActual") != expected["selectedActual"]: - raise SystemExit( - f"browser probe snapshot {tag!r} expected selectedActual={expected['selectedActual']!r}, " - f"found {snapshot.get('selectedActual')!r}" - ) - label = snapshot.get("label") - if not isinstance(label, str): - raise SystemExit(f"browser probe snapshot {tag!r} missing label text") - if f"T+{expected['selectedTarget']}" not in label: - raise SystemExit( - f"browser probe snapshot {tag!r} label missing target lag {expected['selectedTarget']!r}: {label!r}" - ) - if f"A+{expected['selectedActual']}" not in label: - raise SystemExit( - f"browser probe snapshot {tag!r} label missing actual lag {expected['selectedActual']!r}: {label!r}" - ) - - sample_labels = result.get("sampleLabels") - if not isinstance(sample_labels, list) or not sample_labels: - raise SystemExit("browser probe missing sample label coverage") - final_label = snapshot_map["actual-5"]["label"] - if not all(label == final_label for label in sample_labels if isinstance(label, str)): - raise SystemExit( - "browser probe sample labels did not converge on the final phase lag label" - ) - - -def wait_for_url(url: str, timeout_secs: float) -> None: - deadline = time.time() + timeout_secs - while time.time() < deadline: - try: - with urllib.request.urlopen(url, timeout=1.0) as response: - if 200 <= response.status < 500: - return - except (OSError, urllib.error.URLError): - time.sleep(0.1) - raise SystemExit(f"temporary site server did not become ready: {url}") - - -@contextlib.contextmanager -def serve_directory(root: Path, bind: str, port: int) -> tuple[str, int]: - class QuietHandler(http.server.SimpleHTTPRequestHandler): - def __init__(self, *args, **kwargs): - super().__init__(*args, directory=str(root), **kwargs) - - def log_message(self, format: str, *args) -> None: - return - - server = http.server.ThreadingHTTPServer((bind, port), QuietHandler) - thread = threading.Thread(target=server.serve_forever, daemon=True) - thread.start() - try: - yield bind, int(server.server_address[1]) - finally: - server.shutdown() - thread.join(timeout=5) - server.server_close() - - -def run_browser_probe(chrome_binary: str, url: str, timeout_secs: int) -> str: - command = [ - chrome_binary, - "--headless=new", - "--disable-gpu", - "--no-sandbox", - "--disable-dev-shm-usage", - "--run-all-compositor-stages-before-draw", - f"--virtual-time-budget={timeout_secs * 1000}", - "--dump-dom", - url, - ] - try: - completed = subprocess.run( - command, - check=False, - capture_output=True, - text=True, - timeout=timeout_secs, - env={**os.environ, "HOME": os.environ.get("HOME", str(Path.home()))}, - ) - except subprocess.TimeoutExpired as exc: - raise SystemExit( - f"chrome browser probe timed out after {timeout_secs}s while loading {url}" - ) from exc - if completed.returncode != 0: - raise SystemExit( - "chrome browser probe failed with " - f"exit code {completed.returncode}\nstdout:\n{completed.stdout}\nstderr:\n{completed.stderr}" - ) - return completed.stdout - - -def main() -> int: - args = parse_args() - root = args.root.resolve() - if not root.is_dir(): - raise SystemExit(f"site root does not exist: {root}") - - policy = validate_policy_name(args.policy) - detail_page = root / "policies" / policy / "index.html" - if not detail_page.is_file(): - raise SystemExit(f"missing policy detail page: {detail_page}") - - chrome_binary = find_chrome_binary(args.chrome_binary) - probe_name = f".site-browser-smoke-{uuid.uuid4().hex}.html" - probe_path = root / probe_name - probe_path.write_text(build_probe_html(policy), encoding="utf-8") - - try: - with serve_directory(root, args.bind, args.port) as (bind, port): - probe_url = f"http://{bind}:{port}/{probe_name}" - wait_for_url(probe_url, timeout_secs=5.0) - dumped_dom = run_browser_probe( - chrome_binary=chrome_binary, - url=probe_url, - timeout_secs=args.timeout_secs, - ) - result = extract_probe_result(dumped_dom) - validate_probe_result(result, policy) - finally: - if not args.keep_probe and probe_path.exists(): - probe_path.unlink() - - final_snapshot = next( - snapshot - for snapshot in result["results"] - if isinstance(snapshot, dict) and snapshot.get("tag") == "actual-5" - ) - print( - "site browser smoke check passed: " - f"policy={policy} final_label={final_snapshot['label']} src_kind={final_snapshot['srcKind']}" - ) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) +run_legacy_script("site/site_browser_smoke.py") diff --git a/scripts/validate_site_bundle.py b/scripts/validate_site_bundle.py old mode 100644 new mode 100755 index c958c43..4f1c946 --- a/scripts/validate_site_bundle.py +++ b/scripts/validate_site_bundle.py @@ -1,323 +1,4 @@ #!/usr/bin/env python3 -"""Smoke-check a generated RoboWBC static site bundle.""" +from _compat import run_legacy_script -from __future__ import annotations - -import argparse -import html -import json -from pathlib import Path - - -def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument( - "--root", - type=Path, - required=True, - help="Root directory of the generated site bundle", - ) - return parser.parse_args() - - -def load_json(path: Path) -> object: - return json.loads(path.read_text(encoding="utf-8")) - - -def resolve_relative_child(root: Path, relative_path: str, *, label: str) -> Path: - candidate = Path(relative_path) - if candidate.is_absolute(): - raise SystemExit(f"{label} must be repo-relative, got absolute path {candidate}") - resolved = (root / candidate).resolve() - try: - resolved.relative_to(root.resolve()) - except ValueError as exc: - raise SystemExit(f"{label} escapes the site bundle root: {relative_path}") from exc - return resolved - - -def detail_dir(root: Path, entry: dict[str, object]) -> Path: - detail_page = entry.get("detail_page") - if not isinstance(detail_page, str) or not detail_page: - raise SystemExit("detail_page missing from manifest entry") - return resolve_relative_child(root, detail_page, label="detail_page") - - -def validate_detail_page(detail_path: Path) -> str: - detail_index = detail_path / "index.html" - if not detail_index.is_file(): - raise SystemExit(f"missing policy detail page: {detail_index}") - return detail_index.read_text(encoding="utf-8") - - -def validate_ok_detail_page(detail_path: Path, detail_text: str) -> None: - detail_index = detail_path / "index.html" - if "../../assets/rerun-web-viewer/index.js" not in detail_text: - raise SystemExit(f"detail page missing rebased viewer path: {detail_index}") - if 'data-rrd-file="run.rrd"' not in detail_text: - raise SystemExit(f"detail page missing local run recording path: {detail_index}") - if "proof_pack_manifest.json" not in detail_text: - raise SystemExit(f"detail page missing proof-pack manifest reference: {detail_index}") - - -def validate_proof_pack_capture(root: Path, entry: dict[str, object], detail_text: str) -> None: - card_id = str(entry.get("card_id", "")) - meta = entry.get("_meta") - if not isinstance(meta, dict): - raise SystemExit(f"{card_id}: manifest entry missing _meta payload") - - if meta.get("showcase_transport") != "mujoco": - return - - manifest_rel = meta.get("proof_pack_manifest_file") - if not isinstance(manifest_rel, str) or not manifest_rel: - raise SystemExit(f"{card_id}: missing proof_pack_manifest_file") - - manifest_path = resolve_relative_child( - root, manifest_rel, label=f"{card_id}: proof_pack_manifest_file" - ) - if not manifest_path.is_file(): - raise SystemExit(f"{card_id}: missing proof-pack manifest: {manifest_path}") - - proof_pack_manifest = load_json(manifest_path) - if not isinstance(proof_pack_manifest, dict): - raise SystemExit(f"{card_id}: invalid proof-pack manifest payload: {manifest_path}") - - capture_status = proof_pack_manifest.get("capture_status") - if capture_status != "ok": - raise SystemExit( - f"{card_id}: expected proof-pack capture_status=ok, found {capture_status!r}" - ) - - if "Screenshots unavailable for this build" in detail_text: - raise SystemExit(f"{card_id}: detail page still shows screenshot fallback copy") - - proof_pack_root = manifest_path.parent - phase_review = proof_pack_manifest.get("phase_review") - if isinstance(phase_review, dict) and phase_review.get("enabled") is True: - if 'id="phase-timeline"' not in detail_text: - raise SystemExit(f"{card_id}: phase-aware detail page missing phase timeline section") - if 'id="phase-lag-selector"' not in detail_text: - raise SystemExit(f"{card_id}: phase-aware detail page missing actual timestamp selector") - if 'id="phase-target-lag-selector"' not in detail_text: - raise SystemExit(f"{card_id}: phase-aware detail page missing target timestamp selector") - if f'data-selected-lag="{proof_pack_manifest.get("default_lag_ticks")}"' not in detail_text: - raise SystemExit(f"{card_id}: phase-aware detail page missing static actual selected lag") - if ( - f'data-selected-lag="{proof_pack_manifest.get("default_target_lag_ticks")}"' - not in detail_text - ): - raise SystemExit(f"{card_id}: phase-aware detail page missing static target selected lag") - - phase_timeline = proof_pack_manifest.get("phase_timeline") - phase_checkpoints = proof_pack_manifest.get("phase_checkpoints") - lag_options = proof_pack_manifest.get("lag_options") - default_lag_ticks = proof_pack_manifest.get("default_lag_ticks") - target_lag_options = proof_pack_manifest.get("target_lag_options") - default_target_lag_ticks = proof_pack_manifest.get("default_target_lag_ticks") - if not isinstance(phase_timeline, list) or not phase_timeline: - raise SystemExit(f"{card_id}: phase-aware proof-pack manifest missing phase_timeline") - if not isinstance(phase_checkpoints, list) or not phase_checkpoints: - raise SystemExit(f"{card_id}: phase-aware proof-pack manifest missing phase_checkpoints") - if not isinstance(lag_options, list) or not lag_options: - raise SystemExit(f"{card_id}: phase-aware proof-pack manifest missing lag_options") - if not isinstance(default_lag_ticks, int): - raise SystemExit(f"{card_id}: phase-aware proof-pack manifest missing default_lag_ticks") - if not isinstance(target_lag_options, list) or not target_lag_options: - raise SystemExit(f"{card_id}: phase-aware proof-pack manifest missing target_lag_options") - if not isinstance(default_target_lag_ticks, int): - raise SystemExit( - f"{card_id}: phase-aware proof-pack manifest missing default_target_lag_ticks" - ) - for phase_entry in phase_timeline: - if not isinstance(phase_entry, dict): - raise SystemExit(f"{card_id}: phase-aware timeline entry is not an object") - phase_name = phase_entry.get("phase_name") - if not isinstance(phase_name, str) or not phase_name: - raise SystemExit(f"{card_id}: phase-aware timeline entry missing phase_name") - debug_marker = ( - f'data-phase-debug-phase="{html.escape(phase_name, quote=True)}"' - ) - if debug_marker not in detail_text: - raise SystemExit( - f"{card_id}: phase-aware detail page missing debug metadata for {phase_name}" - ) - - for checkpoint in phase_checkpoints: - if not isinstance(checkpoint, dict): - raise SystemExit(f"{card_id}: phase-aware checkpoint payload is not an object") - phase_kind = checkpoint.get("phase_kind") - cameras = checkpoint.get("cameras") - if phase_kind == "phase_end": - lag_variants = checkpoint.get("lag_variants") - checkpoint_lag_options = checkpoint.get("lag_options") - target_variants = checkpoint.get("target_lag_variants") - checkpoint_target_lag_options = checkpoint.get("target_lag_options") - if not isinstance(lag_variants, list) or not lag_variants: - raise SystemExit(f"{card_id}: phase-end checkpoint missing lag_variants") - if not isinstance(checkpoint_lag_options, list) or not checkpoint_lag_options: - raise SystemExit(f"{card_id}: phase-end checkpoint missing lag_options") - if not isinstance(target_variants, list) or not target_variants: - raise SystemExit(f"{card_id}: phase-end checkpoint missing target_lag_variants") - if not isinstance(checkpoint_target_lag_options, list) or not checkpoint_target_lag_options: - raise SystemExit(f"{card_id}: phase-end checkpoint missing target_lag_options") - for variant in lag_variants: - if not isinstance(variant, dict): - raise SystemExit(f"{card_id}: lag variant payload is not an object") - variant_dir = variant.get("relative_dir") - variant_cameras = variant.get("cameras") - if not isinstance(variant_dir, str) or not variant_dir: - raise SystemExit(f"{card_id}: lag variant missing relative_dir") - if not isinstance(variant_cameras, list) or not variant_cameras: - raise SystemExit(f"{card_id}: lag variant missing camera list") - resolved_variant_dir = resolve_relative_child( - proof_pack_root, - variant_dir, - label=f"{card_id}: lag variant relative_dir", - ) - if not resolved_variant_dir.is_dir(): - raise SystemExit( - f"{card_id}: missing lag checkpoint directory: {resolved_variant_dir}" - ) - for camera in variant_cameras: - if not isinstance(camera, str) or not camera: - raise SystemExit(f"{card_id}: invalid lag camera entry: {camera!r}") - image_path = resolved_variant_dir / f"{camera}_rgb.png" - if not image_path.is_file(): - raise SystemExit( - f"{card_id}: missing lag screenshot for +{variant.get('lag_ticks')}: {image_path}" - ) - actual_image_path = resolved_variant_dir / f"{camera}_actual_rgb.png" - if not actual_image_path.is_file(): - raise SystemExit( - f"{card_id}: missing raw actual lag screenshot for +{variant.get('lag_ticks')}: {actual_image_path}" - ) - target_image_path = resolved_variant_dir / f"{camera}_target_rgb.png" - if not target_image_path.is_file(): - raise SystemExit( - f"{card_id}: missing raw target lag screenshot for +{variant.get('lag_ticks')}: {target_image_path}" - ) - for variant in target_variants: - if not isinstance(variant, dict): - raise SystemExit(f"{card_id}: target lag variant payload is not an object") - variant_dir = variant.get("relative_dir") - variant_cameras = variant.get("cameras") - if not isinstance(variant_dir, str) or not variant_dir: - raise SystemExit(f"{card_id}: target lag variant missing relative_dir") - if not isinstance(variant_cameras, list) or not variant_cameras: - raise SystemExit(f"{card_id}: target lag variant missing camera list") - resolved_variant_dir = resolve_relative_child( - proof_pack_root, - variant_dir, - label=f"{card_id}: target lag variant relative_dir", - ) - if not resolved_variant_dir.is_dir(): - raise SystemExit( - f"{card_id}: missing target lag checkpoint directory: {resolved_variant_dir}" - ) - for camera in variant_cameras: - if not isinstance(camera, str) or not camera: - raise SystemExit( - f"{card_id}: invalid target lag camera entry: {camera!r}" - ) - image_path = resolved_variant_dir / f"{camera}_rgb.png" - if not image_path.is_file(): - raise SystemExit( - f"{card_id}: missing target lag screenshot for +{variant.get('lag_ticks')}: {image_path}" - ) - continue - - relative_dir = checkpoint.get("relative_dir") - if not isinstance(relative_dir, str) or not relative_dir: - raise SystemExit(f"{card_id}: phase midpoint checkpoint missing relative_dir") - if not isinstance(cameras, list) or not cameras: - raise SystemExit(f"{card_id}: phase midpoint checkpoint missing camera list") - checkpoint_dir = resolve_relative_child( - proof_pack_root, - relative_dir, - label=f"{card_id}: checkpoint relative_dir", - ) - if not checkpoint_dir.is_dir(): - raise SystemExit(f"{card_id}: missing proof-pack checkpoint directory: {checkpoint_dir}") - for camera in cameras: - if not isinstance(camera, str) or not camera: - raise SystemExit(f"{card_id}: invalid proof-pack camera entry: {camera!r}") - image_path = checkpoint_dir / f"{camera}_rgb.png" - if not image_path.is_file(): - raise SystemExit(f"{card_id}: missing proof-pack screenshot: {image_path}") - return - - checkpoints = proof_pack_manifest.get("checkpoints") - if not isinstance(checkpoints, list) or not checkpoints: - raise SystemExit(f"{card_id}: proof-pack manifest has no checkpoints") - for checkpoint in checkpoints: - if not isinstance(checkpoint, dict): - raise SystemExit(f"{card_id}: proof-pack checkpoint payload is not an object") - relative_dir = checkpoint.get("relative_dir") - cameras = checkpoint.get("cameras") - if not isinstance(relative_dir, str) or not relative_dir: - raise SystemExit(f"{card_id}: proof-pack checkpoint missing relative_dir") - if not isinstance(cameras, list) or not cameras: - raise SystemExit(f"{card_id}: proof-pack checkpoint missing camera list") - - checkpoint_dir = resolve_relative_child( - proof_pack_root, - relative_dir, - label=f"{card_id}: checkpoint relative_dir", - ) - if not checkpoint_dir.is_dir(): - raise SystemExit(f"{card_id}: missing proof-pack checkpoint directory: {checkpoint_dir}") - - for camera in cameras: - if not isinstance(camera, str) or not camera: - raise SystemExit(f"{card_id}: invalid proof-pack camera entry: {camera!r}") - image_path = checkpoint_dir / f"{camera}_rgb.png" - if not image_path.is_file(): - raise SystemExit(f"{card_id}: missing proof-pack screenshot: {image_path}") - - -def validate_site_bundle(root: Path) -> None: - if not (root / "index.html").is_file(): - raise SystemExit(f"missing site home page: {root / 'index.html'}") - if not (root / "manifest.json").is_file(): - raise SystemExit(f"missing site manifest: {root / 'manifest.json'}") - if not (root / "assets" / "rerun-web-viewer" / "index.js").is_file(): - raise SystemExit("missing Rerun web viewer bundle") - if not (root / "benchmarks" / "nvidia" / "index.html").is_file(): - raise SystemExit("missing benchmark page") - - manifest = load_json(root / "manifest.json") - if not isinstance(manifest, list): - raise SystemExit("site manifest must be a JSON array") - if not manifest: - raise SystemExit("site manifest is empty") - if not all("detail_page" in entry for entry in manifest): - raise SystemExit("detail_page missing from manifest") - - ok_entries = [entry for entry in manifest if entry.get("status") == "ok"] - if not ok_entries: - raise SystemExit("expected at least one successful policy entry") - - for entry in manifest: - if not isinstance(entry, dict): - raise SystemExit("site manifest entries must be JSON objects") - path = detail_dir(root, entry) - detail_text = validate_detail_page(path) - if entry.get("status") == "ok": - validate_ok_detail_page(path, detail_text) - validate_proof_pack_capture(root, entry, detail_text) - - index_text = (root / "index.html").read_text(encoding="utf-8") - if "benchmarks/nvidia/index.html" not in index_text: - raise SystemExit("site home missing benchmark link") - -def main() -> int: - args = parse_args() - root = args.root.resolve() - validate_site_bundle(root) - print(f"site bundle smoke check passed: {root}") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) +run_legacy_script("site/validate_site_bundle.py") diff --git a/tests/README.md b/tests/README.md new file mode 100644 index 0000000..2f50930 --- /dev/null +++ b/tests/README.md @@ -0,0 +1,30 @@ +# Tests + +This folder contains Python tests for benchmark helpers, site generation, +proof-pack validation, browser-smoke helper logic, and RoboHarness report +contracts. + +Run all Python tests: + +```bash +make python-test +``` + +Run by marker: + +```bash +python3 -m pytest tests -m contract +python3 -m pytest tests -m integration +``` + +## Layout + +```text +tests/ + contract/ public script behavior, generated JSON/HTML shape, bundle validation, report contracts + integration/ subprocess, temporary repository, wrapper command, and external-boundary coverage +``` + +`tests/conftest.py` marks tests from these folders automatically. Future +regression, slow, or local-only tests should get explicit folders or markers +when they first appear. diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 0000000..7d1171d --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,13 @@ +from __future__ import annotations + +from pathlib import Path + + +def pytest_collection_modifyitems(config, items): + del config + for item in items: + parts = set(Path(str(item.path)).parts) + if "contract" in parts: + item.add_marker("contract") + if "integration" in parts: + item.add_marker("integration") diff --git a/tests/test_policy_showcase.py b/tests/contract/test_policy_showcase.py similarity index 99% rename from tests/test_policy_showcase.py rename to tests/contract/test_policy_showcase.py index 61b624e..18536bf 100644 --- a/tests/test_policy_showcase.py +++ b/tests/contract/test_policy_showcase.py @@ -6,8 +6,8 @@ from pathlib import Path -ROOT = Path(__file__).resolve().parents[1] -SHOWCASE_PATH = ROOT / "scripts" / "generate_policy_showcase.py" +ROOT = Path(__file__).resolve().parents[2] +SHOWCASE_PATH = ROOT / "scripts" / "site" / "generate_policy_showcase.py" SHOWCASE_SPEC = importlib.util.spec_from_file_location( "generate_policy_showcase", SHOWCASE_PATH diff --git a/tests/test_roboharness_report.py b/tests/contract/test_roboharness_report.py similarity index 99% rename from tests/test_roboharness_report.py rename to tests/contract/test_roboharness_report.py index 191ad2d..a13b28c 100644 --- a/tests/test_roboharness_report.py +++ b/tests/contract/test_roboharness_report.py @@ -4,8 +4,8 @@ from pathlib import Path -ROOT = Path(__file__).resolve().parents[1] -REPORT_PATH = ROOT / "scripts" / "roboharness_report.py" +ROOT = Path(__file__).resolve().parents[2] +REPORT_PATH = ROOT / "scripts" / "reports" / "roboharness_report.py" SPEC = importlib.util.spec_from_file_location("roboharness_report", REPORT_PATH) REPORT = importlib.util.module_from_spec(SPEC) diff --git a/tests/test_site_browser_smoke.py b/tests/contract/test_site_browser_smoke.py similarity index 96% rename from tests/test_site_browser_smoke.py rename to tests/contract/test_site_browser_smoke.py index 648cfeb..ae24a59 100644 --- a/tests/test_site_browser_smoke.py +++ b/tests/contract/test_site_browser_smoke.py @@ -3,8 +3,8 @@ from pathlib import Path -ROOT = Path(__file__).resolve().parents[1] -SMOKE_PATH = ROOT / "scripts" / "site_browser_smoke.py" +ROOT = Path(__file__).resolve().parents[2] +SMOKE_PATH = ROOT / "scripts" / "site" / "site_browser_smoke.py" SPEC = importlib.util.spec_from_file_location("site_browser_smoke", SMOKE_PATH) SMOKE = importlib.util.module_from_spec(SPEC) diff --git a/tests/test_validate_site_bundle.py b/tests/contract/test_validate_site_bundle.py similarity index 99% rename from tests/test_validate_site_bundle.py rename to tests/contract/test_validate_site_bundle.py index c141b1b..1db7168 100644 --- a/tests/test_validate_site_bundle.py +++ b/tests/contract/test_validate_site_bundle.py @@ -5,8 +5,8 @@ from pathlib import Path -ROOT = Path(__file__).resolve().parents[1] -VALIDATOR_PATH = ROOT / "scripts" / "validate_site_bundle.py" +ROOT = Path(__file__).resolve().parents[2] +VALIDATOR_PATH = ROOT / "scripts" / "site" / "validate_site_bundle.py" SPEC = importlib.util.spec_from_file_location("validate_site_bundle", VALIDATOR_PATH) VALIDATOR = importlib.util.module_from_spec(SPEC) diff --git a/tests/test_nvidia_benchmarks.py b/tests/integration/test_nvidia_benchmarks.py similarity index 97% rename from tests/test_nvidia_benchmarks.py rename to tests/integration/test_nvidia_benchmarks.py index 837b06a..9c1d3fd 100644 --- a/tests/test_nvidia_benchmarks.py +++ b/tests/integration/test_nvidia_benchmarks.py @@ -12,13 +12,13 @@ from pathlib import Path -ROOT = Path(__file__).resolve().parents[1] +ROOT = Path(__file__).resolve().parents[2] REGISTRY_PATH = ROOT / "benchmarks/nvidia/cases.json" -NORMALIZER_PATH = ROOT / "scripts/normalize_nvidia_benchmarks.py" -RENDER_PATH = ROOT / "scripts/render_nvidia_benchmark_summary.py" -ROBOHARNESS_REPORT_PATH = ROOT / "scripts/roboharness_report.py" -ROBOWBC_COMPARE_PATH = ROOT / "scripts/bench_robowbc_compare.py" -OFFICIAL_COMPARE_PATH = ROOT / "scripts/bench_nvidia_official.py" +NORMALIZER_PATH = ROOT / "scripts/benchmarks/normalize_nvidia_benchmarks.py" +RENDER_PATH = ROOT / "scripts/benchmarks/render_nvidia_benchmark_summary.py" +ROBOHARNESS_REPORT_PATH = ROOT / "scripts/reports/roboharness_report.py" +ROBOWBC_COMPARE_PATH = ROOT / "scripts/benchmarks/bench_robowbc_compare.py" +OFFICIAL_COMPARE_PATH = ROOT / "scripts/benchmarks/bench_nvidia_official.py" SPEC = importlib.util.spec_from_file_location("normalize_nvidia_benchmarks", NORMALIZER_PATH) NORMALIZER = importlib.util.module_from_spec(SPEC) @@ -406,7 +406,7 @@ def test_robowbc_source_command_for_case_includes_provider(self) -> None: ) self.assertEqual( command, - "python3 scripts/bench_robowbc_compare.py --case " + "python3 scripts/benchmarks/bench_robowbc_compare.py --case " "gear_sonic/full_velocity_tick_steady_state --provider cuda " "--output-root /tmp/robowbc-bench-out", ) @@ -425,7 +425,7 @@ def test_official_source_command_for_case_includes_provider_and_overrides(self) ) self.assertEqual( command, - "python3 scripts/bench_nvidia_official.py --case " + "python3 scripts/benchmarks/bench_nvidia_official.py --case " "gear_sonic/planner_only_cold_start --provider tensor_rt " "--repo-dir /tmp/official-src --output-root /tmp/official-bench-out " "--samples 9 --ticks 17 --control-frequency-hz 60", @@ -647,7 +647,7 @@ def test_official_wrapper_blocks_decoupled_case_when_models_are_missing(self) -> subprocess.run( [ "python3", - str(ROOT / "scripts/bench_nvidia_official.py"), + str(ROOT / "scripts/benchmarks/bench_nvidia_official.py"), "--case", "decoupled_wbc/walk_predict", "--repo-dir", @@ -667,7 +667,7 @@ def test_official_wrapper_blocks_decoupled_case_when_models_are_missing(self) -> self.assertEqual(artifact["status"], "blocked") self.assertEqual(artifact["implementation"], "ort-cpp-sonic") self.assertEqual(artifact["variant_label"], "cpu-baseline") - self.assertIn("scripts/download_decoupled_wbc_models.sh", artifact["notes"]) + self.assertIn("scripts/models/download_decoupled_wbc_models.sh", artifact["notes"]) def test_official_wrapper_blocks_decoupled_non_cpu_provider_without_relabeling(self) -> None: with tempfile.TemporaryDirectory() as tmpdir: @@ -677,7 +677,7 @@ def test_official_wrapper_blocks_decoupled_non_cpu_provider_without_relabeling(s subprocess.run( [ "python3", - str(ROOT / "scripts/bench_nvidia_official.py"), + str(ROOT / "scripts/benchmarks/bench_nvidia_official.py"), "--case", "decoupled_wbc/walk_predict", "--provider", @@ -714,7 +714,7 @@ def test_official_wrapper_runs_decoupled_cases_and_blocks_gear_sonic_cases(self) subprocess.run( [ "python3", - str(ROOT / "scripts/bench_nvidia_official.py"), + str(ROOT / "scripts/benchmarks/bench_nvidia_official.py"), "--all", "--repo-dir", str(repo_dir), @@ -779,7 +779,7 @@ def test_robowbc_cli_wrapper_records_actual_case_command(self) -> None: subprocess.run( [ "python3", - str(ROOT / "scripts/bench_robowbc_compare.py"), + str(ROOT / "scripts/benchmarks/bench_robowbc_compare.py"), "--case", case_id, "--output-root", @@ -808,7 +808,7 @@ def test_robowbc_wrapper_blocks_decoupled_non_cpu_provider_without_relabeling(se subprocess.run( [ "python3", - str(ROOT / "scripts/bench_robowbc_compare.py"), + str(ROOT / "scripts/benchmarks/bench_robowbc_compare.py"), "--case", "decoupled_wbc/walk_predict", "--provider", @@ -843,7 +843,7 @@ def test_robowbc_microbench_provider_failure_emits_blocked_artifact(self) -> Non subprocess.run( [ "python3", - str(ROOT / "scripts/bench_robowbc_compare.py"), + str(ROOT / "scripts/benchmarks/bench_robowbc_compare.py"), "--case", "gear_sonic/planner_only_cold_start", "--provider", @@ -887,7 +887,7 @@ def test_official_gear_sonic_provider_failure_emits_blocked_artifact(self) -> No subprocess.run( [ "python3", - str(ROOT / "scripts/bench_nvidia_official.py"), + str(ROOT / "scripts/benchmarks/bench_nvidia_official.py"), "--case", "gear_sonic/planner_only_cold_start", "--provider",