Skip to content

CV: overhead ArUco localization panel#2

Merged
jonasneves merged 8 commits into
mainfrom
cv
May 13, 2026
Merged

CV: overhead ArUco localization panel#2
jonasneves merged 8 commits into
mainfrom
cv

Conversation

@JaredBaileyDuke
Copy link
Copy Markdown
Collaborator

@JaredBaileyDuke JaredBaileyDuke commented May 12, 2026

Summary

Overhead ArUco localization is now a headless detection service that consumes whichever camera the helpers card designates, instead of a dedicated CV panel with its own video element and decoder. Every camera is a helper; each helper carries a single "Camera role" picker (Operator / Overhead / Mount on robot for phones; Idle / Overhead for local cams), mutually exclusive across the helpers list.

What's in

  • aruco.js: headless service exposing setOverheadSource(videoEl, {onResult}) / clearOverheadSource(). No DOM ownership, no second decoder, single source of truth for the marker→robot binding.
  • Helpers card (helpers.js): single role picker per helper. Local cameras enumerate via getUserMedia; empty deviceIds (pre-permission) fall back to {video:true} and the resolved track's getSettings().deviceId migrates the entry. Auto-resume on reload when camera permission is already granted.
  • SVG overlay on the helper's existing preview tile — revives the patchArucoOverlay shape from the deleted phone-on-robot tracker, retargeted at the right surface (overhead, not robot-mounted).
  • Per-robot binding via entry.arucoMarkerId (persisted in localStorage; set via window.bindArucoMarker(robotId, markerId)). Detection prefers explicit binding; positional fallback only when no entry has claimed a marker id. Substrate for the multi-robot orchestration direction in .claude/CLAUDE.md — N markers in frame → N robots updated per scan.
  • Print-marker affordance is permanent under the role row whenever overhead is designated.
  • Staleness contract: producer writes updatedAt; consumer (motion-planner PR) is responsible for gating. Documented in aruco.js header and .claude/notes.md.

What's deleted

  • public/aruco.js (phone-on-robot tracker — wired but unproven against real hardware).
  • public/cv-localize.js (dedicated CV panel with separate video element).
  • <section id="aruco-panel"> and its controls in index.html.
  • sendArucoStatus + the WebRTC aruco-status channel + the phone-side aruco-lock box (phone-on-robot lifecycle).

Bugs fixed along the way

  • posit1.js does require('./svd') in browser context — loading svd.js first satisfies the this.SVD || short-circuit so the require branch never runs.
  • Dictionary name corrected: ARUCO_4X4_50 doesn't exist in js-aruco2; switched to ARUCO_4X4_1000 and explicitly loaded its dictionary file. First 50 codes match the OpenCV DICT_4X4_50 PDFs in /assets/.
  • getUserMedia({video:{deviceId:{exact:""}}}) failed pre-permission (empty deviceIds); now falls back to {video:true} and migrates the entry to the resolved id.
  • Race protection: in-flight getUserMedia no longer leaks a stream if the user switches role mid-await.
  • Track-ended: OS-level revoke / USB unplug flips the role back to Idle instead of leaving the detector chewing on dead frames.
  • Dropdown-honest after reload: currentRole requires both isOverhead && live, matching the phone variant.

UI polish

  • Helpers section gets a tight heading; helper cards have proper card chrome matching the robot cards above, WITHOUT reusing the green .robot.status-connected left-stripe (different semantic; would have stolen robot identity).
  • Camera-role select gets a custom chevron via inline SVG so it doesn't read as a debug field.
  • Print-marker links (Sheet 1 / Sheet 2) are permanent under the role row when overhead is designated.

Test plan

  • make smoke passes (22/22).
  • Helper card shows local cameras only when a robot or phone is already paired (no orphan local-cam card on first visit).
  • Camera role picker on a phone helper: Operator / Overhead / Mount on <robot> are mutually exclusive.
  • Local-camera helper: Idle / Overhead; getUserMedia prompt fires on first overhead selection.
  • Designating either as overhead starts detection on the same preview tile (no duplicate video element).
  • Detector loads cleanly (ARUCO_4X4_1000); no "dictionary not recognized" error.
  • Auto-resume on reload when permission is granted; silent failure when origin changes (no console noise).
  • Multi-robot end-to-end: print sheets 0 + 1, tape one marker on each robot, designate phone as overhead, confirm both robots' arucoPosition update simultaneously per scan.

Followups for sibling PRs

🤖 Generated with Claude Code

JaredBaileyDuke and others added 3 commits May 11, 2026 20:02
Global dashboard panel for detecting DICT_4X4_50 markers via an external
overhead camera. Uses getUserMedia + js-aruco2 (ARUCO_4X4_50 dictionary)
to match the object-tracking project PDFs. One-shot scan captures a frame,
detects all markers, and overlays polygon + heading arrow annotations on
the result canvas with a per-marker results table (id, pixel position,
heading). Phase 1 — pixel-space only; metric pose is Phase 2.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Captures chessboard frames interactively (SPACE to grab, Q to calibrate)
and writes camera_matrix + dist_coeffs to scripts/camera_calibration.json.
Board size and square size are configurable at the top of the file.
Output feeds Phase 2 metric pose in the overhead localization panel.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Drop the chessboard calibration script. Instead, the user enters the
printed marker size in mm and POS.Posit estimates 3D pose using a focal
length derived from the image dimensions (no calibration file needed).
Scan results now show approximate mm floor position alongside heading.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@JaredBaileyDuke JaredBaileyDuke self-assigned this May 12, 2026
JaredBaileyDuke and others added 5 commits May 11, 2026 20:12
Copy DICT_4X4_50 marker sheets into public/assets/ and surface them as
Sheet 1 / Sheet 2 links in the overhead localization panel. Also register
application/pdf in the dev server MIME table so they open correctly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add an interval selector (manual / 5s / 10s / 30s / 60s) that auto-scans
on a timer when the camera is running. Each scan maps marker ID to robot by
index (marker 0 → first paired robot, etc.), writes the ground-truth pose to
entry.cvPosition {x, y, headingDeg, updatedAt}, and shows the robot name in
the results table. Robots not in frame are left at their last known position;
out-of-range marker IDs are silently ignored.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Reframe: ArUco detection is a consumer of camera frames, not a UI
surface of its own. The original PR added a dedicated card with its own
camera selector, video element, and decode loop — duplicating the
helpers card whenever a paired phone was the source. This collapses the
two into one: every camera (phone or local) is a helper, and a "Camera
role" picker on each helper card designates one as the overhead
localizer.

  - `aruco.js` is now headless: `setOverheadSource(videoEl, {onResult})`
    + `clearOverheadSource()`. No DOM ownership, no second decoder.
  - Helper cards carry a single role select (Operator / Overhead / Mount
    on robot for phones; Idle / Overhead for local cams). Mutually
    exclusive across the whole helpers list.
  - SVG overlay paints detected markers on the helper's existing preview
    tile. The "patchArucoOverlay" shape from the deleted phone-on-robot
    aruco.js was correct — only the geometry was wrong. Pattern revived
    against the right surface.
  - Local cameras enumerate via `getUserMedia` and migrate empty
    deviceIds to real ones after permission grant; auto-resume on
    reload when permission is already granted, silent fail otherwise.
  - Multi-robot ready: `entry.arucoMarkerId` persists per-robot binding
    (settable via `window.bindArucoMarker`); positional fallback only
    when no entry has claimed a marker id. The detection loop iterates
    over all visible markers per scan, so two robots in frame get two
    `arucoPosition` updates in the same tick — the substrate for the
    multi-robot orchestration direction in `.claude/CLAUDE.md`.
  - Staleness contract documented: producer writes `updatedAt`,
    consumer is responsible for gating. The motion-planner PR will
    need to honor this when it rebases onto `arucoPosition`.

Deletions:

  - `public/aruco.js` (the phone-on-robot tracker, wired but unproven).
  - `public/cv-localize.js` (the dedicated CV panel).
  - `<section id="aruco-panel">` and its controls in `index.html`.
  - `sendArucoStatus` + the `aruco-status` WebRTC channel + the
    phone-side `aruco-lock` box (phone-on-robot lifecycle).

Bug fixes folded in:

  - js-aruco2's `posit1.js` does `require('./svd')` in browser context;
    loading `svd.js` first satisfies the `this.SVD ||` short-circuit so
    the require branch never runs.
  - Dictionary name: `ARUCO_4X4_50` doesn't exist in js-aruco2; switched
    to `ARUCO_4X4_1000` and explicitly loaded its dictionary file. First
    50 codes match the OpenCV `DICT_4X4_50` PDFs in `/assets/`.
  - `getUserMedia({video:{deviceId:{exact:""}}})` fails when permission
    hasn't been granted yet (`enumerateDevices` returns empty deviceIds
    pre-permission). Fall back to `{video:true}` and use the resolved
    track's `getSettings().deviceId` to migrate the entry.
  - Race: in-flight `getUserMedia` no longer leaks a stream if the user
    switches role mid-await.
  - Track-ended: OS-level revoke / USB unplug now flips the role back
    to Idle instead of leaving the detector chewing on dead frames.
  - Stale dropdown after reload: `currentRole` now requires `live`,
    matching the phone variant's honesty.

UI polish:

  - Helper cards drop the phone-on-robot SVG/help/status divs.
  - Helpers section has a heading; cards have proper card chrome
    matching the robot cards above, without re-using the green
    `.robot.status-connected` left-stripe (different semantic, would
    have stolen robot identity).
  - Camera-role select gets a custom chevron via inline SVG so it
    doesn't read as a debug field.
  - Print-marker affordance ("Sheet 1 / Sheet 2") is permanent under
    the role row when overhead is designated.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@jonasneves jonasneves merged commit 7347121 into main May 13, 2026
@jonasneves jonasneves deleted the cv branch May 13, 2026 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants