fix: use full-screen reference frame for recording touch overlays#765
Merged
Conversation
Coordinate presses during recording resolved their touch reference frame from an interactive-only snapshot that was also captured in compact mode. Compact pruning drops Application/Window containers, so viewport inference fell back to a bounding box of leaf elements (e.g. 386x208 instead of the 440x956 screen) and the overlay burn-in scaled tap markers with that garbage frame: x landed nearly right by coincidence, y blew past the canvas and clamped to the bottom edge. Capture the reference-frame snapshot with compact disabled so the Application node survives and the frame is the real screen point size. Verified on an iPhone 17 simulator (402x874pt screen, 220x480 video): gesture telemetry now carries 402x874 and the burned-in marker lands at (121, 329) vs expected (120.4, 329.5); previously it rendered clamped at the bottom edge.
Size Report
Startup median (7 runs, lower is better):
Top changed chunks: no changes in the largest emitted chunks. |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
iOS simulator screen recordings with touch overlays rendered tap markers at the wrong position — y clamped to the video's bottom edge (repro:
record start,press 220 600on a 440×956pt device, marker burned in at bottom-center instead of ~(110, 301) in the 220×480 video).Why
For coordinate presses during recording with no prior snapshot,
resolveDirectTouchReferenceFramecaptured the reference-frame snapshot withinteractiveOnly: true, whichcaptureSnapshotForSessioncoupled to compact mode. Compact pruning dropsApplication/Windowcontainer nodes, soinferViewportRectfell back to a bounding box of leaf elements (e.g. 386×208.33 in the Settings app instead of the screen size). The overlay renderer (recording-overlay.swift) then scaled marker coordinates with that garbage frame: x looked nearly right by coincidence (bbox width ≈ screen width), while y scaled ~2× past the canvas and clamped.How
captureSnapshotForSessionaccepts an optionalcompactoverride (default unchanged:compact = interactiveOnly).resolveDirectTouchReferenceFramerequestscompact: false, so the Application node survives and the inferred frame is the real full-screen point size — matching the coordinate space of the recorded touch events and the recorded video.The leaf-bbox fallback in
inferViewportRectis left as is — Android snapshots have no Application/Window-typed nodes and rely on it (there the bbox approximates the screen).Validation (iPhone 17 simulator, iOS 27, 402×874pt screen, 220×480 video)
<video>.gesture-telemetry.jsoncarriedreferenceWidth: 386, referenceHeight: 208.33; marker clamped at the bottom edge.402×874; the burned-in marker measures at (121, 329) in the frame vs expected (120.4, 329.5) — within a pixel.pnpm typecheckclean; daemon unit suites pass (734 tests).