CV: overhead ArUco localization panel#2
Merged
Merged
Conversation
Global dashboard panel for detecting DICT_4X4_50 markers via an external overhead camera. Uses getUserMedia + js-aruco2 (ARUCO_4X4_50 dictionary) to match the object-tracking project PDFs. One-shot scan captures a frame, detects all markers, and overlays polygon + heading arrow annotations on the result canvas with a per-marker results table (id, pixel position, heading). Phase 1 — pixel-space only; metric pose is Phase 2. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Captures chessboard frames interactively (SPACE to grab, Q to calibrate) and writes camera_matrix + dist_coeffs to scripts/camera_calibration.json. Board size and square size are configurable at the top of the file. Output feeds Phase 2 metric pose in the overhead localization panel. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Drop the chessboard calibration script. Instead, the user enters the printed marker size in mm and POS.Posit estimates 3D pose using a focal length derived from the image dimensions (no calibration file needed). Scan results now show approximate mm floor position alongside heading. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copy DICT_4X4_50 marker sheets into public/assets/ and surface them as Sheet 1 / Sheet 2 links in the overhead localization panel. Also register application/pdf in the dev server MIME table so they open correctly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add an interval selector (manual / 5s / 10s / 30s / 60s) that auto-scans
on a timer when the camera is running. Each scan maps marker ID to robot by
index (marker 0 → first paired robot, etc.), writes the ground-truth pose to
entry.cvPosition {x, y, headingDeg, updatedAt}, and shows the robot name in
the results table. Robots not in frame are left at their last known position;
out-of-range marker IDs are silently ignored.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Reframe: ArUco detection is a consumer of camera frames, not a UI
surface of its own. The original PR added a dedicated card with its own
camera selector, video element, and decode loop — duplicating the
helpers card whenever a paired phone was the source. This collapses the
two into one: every camera (phone or local) is a helper, and a "Camera
role" picker on each helper card designates one as the overhead
localizer.
- `aruco.js` is now headless: `setOverheadSource(videoEl, {onResult})`
+ `clearOverheadSource()`. No DOM ownership, no second decoder.
- Helper cards carry a single role select (Operator / Overhead / Mount
on robot for phones; Idle / Overhead for local cams). Mutually
exclusive across the whole helpers list.
- SVG overlay paints detected markers on the helper's existing preview
tile. The "patchArucoOverlay" shape from the deleted phone-on-robot
aruco.js was correct — only the geometry was wrong. Pattern revived
against the right surface.
- Local cameras enumerate via `getUserMedia` and migrate empty
deviceIds to real ones after permission grant; auto-resume on
reload when permission is already granted, silent fail otherwise.
- Multi-robot ready: `entry.arucoMarkerId` persists per-robot binding
(settable via `window.bindArucoMarker`); positional fallback only
when no entry has claimed a marker id. The detection loop iterates
over all visible markers per scan, so two robots in frame get two
`arucoPosition` updates in the same tick — the substrate for the
multi-robot orchestration direction in `.claude/CLAUDE.md`.
- Staleness contract documented: producer writes `updatedAt`,
consumer is responsible for gating. The motion-planner PR will
need to honor this when it rebases onto `arucoPosition`.
Deletions:
- `public/aruco.js` (the phone-on-robot tracker, wired but unproven).
- `public/cv-localize.js` (the dedicated CV panel).
- `<section id="aruco-panel">` and its controls in `index.html`.
- `sendArucoStatus` + the `aruco-status` WebRTC channel + the
phone-side `aruco-lock` box (phone-on-robot lifecycle).
Bug fixes folded in:
- js-aruco2's `posit1.js` does `require('./svd')` in browser context;
loading `svd.js` first satisfies the `this.SVD ||` short-circuit so
the require branch never runs.
- Dictionary name: `ARUCO_4X4_50` doesn't exist in js-aruco2; switched
to `ARUCO_4X4_1000` and explicitly loaded its dictionary file. First
50 codes match the OpenCV `DICT_4X4_50` PDFs in `/assets/`.
- `getUserMedia({video:{deviceId:{exact:""}}})` fails when permission
hasn't been granted yet (`enumerateDevices` returns empty deviceIds
pre-permission). Fall back to `{video:true}` and use the resolved
track's `getSettings().deviceId` to migrate the entry.
- Race: in-flight `getUserMedia` no longer leaks a stream if the user
switches role mid-await.
- Track-ended: OS-level revoke / USB unplug now flips the role back
to Idle instead of leaving the detector chewing on dead frames.
- Stale dropdown after reload: `currentRole` now requires `live`,
matching the phone variant's honesty.
UI polish:
- Helper cards drop the phone-on-robot SVG/help/status divs.
- Helpers section has a heading; cards have proper card chrome
matching the robot cards above, without re-using the green
`.robot.status-connected` left-stripe (different semantic, would
have stolen robot identity).
- Camera-role select gets a custom chevron via inline SVG so it
doesn't read as a debug field.
- Print-marker affordance ("Sheet 1 / Sheet 2") is permanent under
the role row when overhead is designated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
# Conflicts: # public/sw.js
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Overhead ArUco localization is now a headless detection service that consumes whichever camera the helpers card designates, instead of a dedicated CV panel with its own video element and decoder. Every camera is a helper; each helper carries a single "Camera role" picker (Operator / Overhead / Mount on robot for phones; Idle / Overhead for local cams), mutually exclusive across the helpers list.
What's in
aruco.js: headless service exposingsetOverheadSource(videoEl, {onResult})/clearOverheadSource(). No DOM ownership, no second decoder, single source of truth for the marker→robot binding.helpers.js): single role picker per helper. Local cameras enumerate viagetUserMedia; emptydeviceIds (pre-permission) fall back to{video:true}and the resolved track'sgetSettings().deviceIdmigrates the entry. Auto-resume on reload when camera permission is already granted.patchArucoOverlayshape from the deleted phone-on-robot tracker, retargeted at the right surface (overhead, not robot-mounted).entry.arucoMarkerId(persisted in localStorage; set viawindow.bindArucoMarker(robotId, markerId)). Detection prefers explicit binding; positional fallback only when no entry has claimed a marker id. Substrate for the multi-robot orchestration direction in.claude/CLAUDE.md— N markers in frame → N robots updated per scan.updatedAt; consumer (motion-planner PR) is responsible for gating. Documented inaruco.jsheader and.claude/notes.md.What's deleted
public/aruco.js(phone-on-robot tracker — wired but unproven against real hardware).public/cv-localize.js(dedicated CV panel with separate video element).<section id="aruco-panel">and its controls inindex.html.sendArucoStatus+ the WebRTCaruco-statuschannel + the phone-sidearuco-lockbox (phone-on-robot lifecycle).Bugs fixed along the way
posit1.jsdoesrequire('./svd')in browser context — loadingsvd.jsfirst satisfies thethis.SVD ||short-circuit so the require branch never runs.ARUCO_4X4_50doesn't exist in js-aruco2; switched toARUCO_4X4_1000and explicitly loaded its dictionary file. First 50 codes match the OpenCVDICT_4X4_50PDFs in/assets/.getUserMedia({video:{deviceId:{exact:""}}})failed pre-permission (empty deviceIds); now falls back to{video:true}and migrates the entry to the resolved id.getUserMediano longer leaks a stream if the user switches role mid-await.currentRolerequires bothisOverhead && live, matching the phone variant.UI polish
.robot.status-connectedleft-stripe (different semantic; would have stolen robot identity).Sheet 1/Sheet 2) are permanent under the role row when overhead is designated.Test plan
make smokepasses (22/22).<robot>are mutually exclusive.ARUCO_4X4_1000); no "dictionary not recognized" error.arucoPositionupdate simultaneously per scan.Followups for sibling PRs
entry.arucoPosition(renamed fromcvPosition) and add aDate.now() - updatedAtstaleness gate before steering. Both flagged in.claude/notes.mdunder "Wired but unproven."🤖 Generated with Claude Code