fix: wire cursorCaptureMode into getUserMedia constraints for Windows web fallback#579
Closed
makaradam wants to merge 65 commits into
Closed
fix: wire cursorCaptureMode into getUserMedia constraints for Windows web fallback#579makaradam wants to merge 65 commits into
makaradam wants to merge 65 commits into
Conversation
…mport - Add Clapperboard icon button to HUD toolbar that calls switchToEditor(), opening the editor without a recording (Open Studio) - Add EditorEmptyState component shown when no video is loaded, featuring: - Drag-and-drop zone for .openscreen project files - Import Video File button (MP4, MOV, WebM, MKV, AVI, M4V, WMV, FLV, TS) - Load Project button - Add File menu items: New Project (Ctrl+N) and Import Video File (Ctrl+I) - Add loadProjectFileFromPath to the full native-bridge chain so drag-dropped .openscreen files can be loaded without a file-picker dialog - Expand ALLOWED_IMPORT_VIDEO_EXTENSIONS and file picker filters for all common video formats from any screen recorder - Add i18n keys: tooltips.openStudio, unsavedChanges.newProject/importVideo
When switchToEditor() is called from the HUD (Open Studio), there is no current video/session/project. Previously this set an error state causing a white/broken screen. Now we simply leave videoPath as null, which lets the EditorEmptyState component render correctly.
Using the CSS 'hidden' class kept VideoPlayback mounted with an empty src, which fired an error event and triggered setError(). Switch to proper conditional rendering so VideoPlayback only mounts when a video is actually loaded.
… video
- Add missing ipcMain.handle('start-new-recording') that calls switchToHud,
fixing the broken Return to Recorder confirm button
- Skip the Return to Recorder confirmation dialog when no video is loaded
(Open Studio with nothing imported) — just switch back immediately
Replace light bg-background with dark #09090b on the loading and error screens so they match the editor theme. Add a green spinning SVG loader and muted text instead of the jarring white flash.
- Call clearCurrentVideoPath before startNewRecording so the session is properly dropped; next editor open starts fresh instead of reloading the previous video - Set body background to #09090b in index.html so the pre-React paint is dark, eliminating the white flash before the spinner appears
The handler was closing mainWindow (the HUD) directly, then calling createEditorWindowWrapper which closes it again. This double-close left a ghost transparent window on each open/close cycle, causing the HUD shadow to visually compound darker on every return-to-recorder. createEditorWindowWrapper already handles closing the current window cleanly, so the redundant close is removed.
The HUD pill uses box-shadow with a 60px blur radius. The previous 600x160 window was a tight fit, causing the shadow to be hard-clipped at the transparent window edge. Increase to 800x260 to give the shadow room — the pill's visual screen position is unchanged since it uses CSS 'fixed bottom-5' relative to the window bottom.
- Use show:false + ready-to-show event so window only appears once content is painted, eliminating the black rectangle flash - Increase window height to 320px and shift y down 55px so the pill (now bottom-20 = 80px from window bottom) has 80px of transparent space below it for the downward shadow to render unclipped
VideoPlayback's pointer drag handler was calling clampFocusToStage with region.depth, which ignored customScale entirely. When dragging the zoom indicator with a custom scale set, the drag was clamped to the preset depth boundaries instead of the actual scale — making it impossible to drag beyond those boundaries. Switch to clampFocusToScale(focus, getZoomScale(region)) so the drag respects customScale the same way the overlay indicator and export do.
The Open Studio button (Clapperboard) now handles opening the editor where users can import any video. The separate open-video-file icon and its handler are redundant and have been removed.
Open Studio already leads to the editor empty state which has a Load Project button. The separate HUD toolbar shortcut is redundant. Removed the button, its handler, and the now-unused folder icon import.
Use show:false + ready-to-show on the editor window so it stays hidden until the first paint. Also set backgroundColor to #09090b so even if the window becomes visible before React mounts, it shows the correct dark background rather than white.
…nsertCSS - Remove File > Import Video File (Ctrl+I) — importing lives in the editor empty state only, avoiding confusion with 'add on top of existing video' semantics - Inject 'background: #09090b' on dom-ready so html/body/#root are dark before React mounts, eliminating the white sub-titlebar flash even on a cold first Vite load
The flash was caused by two compounding issues: 1. The Tailwind/shadcn CSS uses `.dark` class to switch from light (--background: white) to dark variables. Since no code applied `.dark` to <html> before React mounted, every `bg-background` usage resolved to white. 2. The lazy-loaded VideoEditor Suspense fallback (`bg-background`) was the visible culprit — it rendered white for the ~50-100ms while VideoEditor was being imported. Fix: add `class="dark"` directly to the <html> element in index.html. The attribute is parsed before any CSS or JS runs, so dark CSS variables are in effect from frame 0 — no flash. HUD/source-selector windows are unaffected because they override `background: transparent` via inline styles (which beat stylesheet rules). Also harden the Suspense fallback to use an explicit `bg-[#09090b]` instead of `bg-background` as a belt-and-suspenders guard.
- Add `loadingEditor` translation key to all 11 locale editor.json files - Suspense fallback in App.tsx now shows "Loading editor..." since at that point we don't yet know whether there is a video to load - VideoEditor loading spinner starts as "Loading editor..." and flips to "Loading video..." only once loadInitialData discovers a recording session or video path to load — no video means it stays as "Loading editor..." throughout
…no project file hasProjectUnsavedChanges only fires when the current snapshot differs from the baseline — but for a fresh import or recording, the baseline IS the current state (nothing edited yet), so the app considered it "saved" even though no .openscreen file exists on disk. Extend the hasUnsavedChanges check: if a video is loaded but currentProjectPath is null (no file saved yet), treat it as unsaved regardless of the snapshot diff. After a successful Save, currentProjectPath is set and this clause becomes false, handing control back to the normal snapshot comparison.
When File > New Project is triggered with an unsaved video/project loaded,
show the UnsavedChangesDialog with a "newProject" variant instead of
immediately clearing state. The variant swaps the copy to:
- "Do you want to save your project before creating a new one?"
- "Save & New Project" / "Discard & New Project" / Cancel
Implementation:
- UnsavedChangesDialog accepts an optional variant prop ("close"|"newProject")
that switches detail text and button labels via translation keys
- showCloseConfirmDialog boolean replaced by confirmDialogVariant state
("close"|"newProject"|null) — one dialog handles both cases
- handleNewProject checks hasUnsavedChanges before clearing state;
new doNewProject helper resets videoPath, currentProjectPath and
lastSavedSnapshot cleanly
- Three new translation keys added to all 11 locale dialogs.json files:
detailNewProject, saveAndNewProject, discardAndNewProject
The countdown-overlay-show handler was calling showInactive() before waiting for the page to load — Chromium showed a black rectangle while still painting the first frame. Fix: wait for the ready-to-show event (fires after first paint) before calling showInactive(), so the window only becomes visible once its content is fully rendered. The redundant did-finish-load wait is replaced by ready-to-show which is the correct Electron signal for this purpose.
…sing When isNativeWindowsCaptureAvailable returns reason="missing-helper" (helper binary not installed — always the case in dev mode), the code was throwing instead of returning false, which killed the recording entirely instead of falling through to the standard web MediaRecorder path. Treat missing-helper the same as unsupported-os: silently return false and let the web recorder take over.
…dia on Windows The Windows code path used navigator.mediaDevices.getDisplayMedia() which requires setDisplayMediaRequestHandler to be registered in the main process. That handler was never implemented, causing a "Not supported" error and preventing recording from starting on Windows when falling back from the native helper. Replace the platform-branched capture logic with a single getUserMedia + chromeMediaSource: "desktop" path that works on both macOS and Windows. Cursor capture mode is already handled separately via setRecordingState.
Use a ref to keep the last non-null confirmDialogVariant stable while the dialog animates out. Previously, setting variant to null caused the fallback ??\"close\" to kick in mid-animation, briefly showing \"Save & Close\" content and firing a spurious sendCloseConfirmResponse(\"cancel\") IPC call.
The mic/webcam selectors were positioned at bottom-[68px] but the HUD bar sits at bottom-20 (80px), causing selectors to render behind the bar. Moved to bottom-[136px] (bar bottom 80px + bar height ~46px + 10px gap).
doNewProject() was not resetting webcamVideoPath/webcamVideoSourcePath, causing the webcam track from a previous recording to bleed into the next project. Also clear them in handleImportVideo since imported external videos never carry a webcam track.
The drop handler called onProjectLoaded() which was wired to handleLoadProject, causing it to re-open a file picker dialog after the file was already loaded. Added onProjectFileDropped callback so the loaded project data is passed directly to applyLoadedProject without a second dialog. Also improved the empty state UI: moved the drag-and-drop hint below the supported formats line with a small upload icon for visual clarity.
- Non-.openscreen files now show "Unsupported Format" dialog instead of silently doing nothing - Failed project loads (e.g. video path no longer accessible) show a "Could Not Open File" dialog with a helpful message - Both use the same design language as the rest of the app (dark dialog, app icon, close button)
- Unified EditorEmptyState to a single onProjectOpened(project, path) callback used by both Load Project button and drag-and-drop, eliminating the double file-picker that was opening (EditorEmptyState + VideoEditor each calling loadProjectFile independently) - Made loadProjectFileFromPath resilient to getApprovedProjectSession failures: path approval errors no longer block the project from loading, they just skip setting the recording session (video player handles the "file not found" case gracefully)
…Path via preload - Import webUtils from electron in preload and expose getPathForFile(file) via contextBridge — replaces the removed File.path property (Electron 32+) - Expose loadProjectFileFromPath(filePath) as a direct ipcRenderer.invoke call, bypassing the native-bridge routing layer for drag-drop use - Add matching TypeScript declarations to electron-env.d.ts so the renderer can call window.electronAPI.getPathForFile / window.electronAPI.loadProjectFileFromPath with full type safety Files changed: electron/preload.ts — webUtils import + two new contextBridge entries electron/electron-env.d.ts — type declarations for both new API methods
…w project clearCurrentVideoPath() previously only nulled currentVideoPath, leaving currentProjectPath set. On next editor open, loadCurrentProjectFile() would find a stale path and reload the discarded project. Fix by also clearing currentProjectPath and calling setCurrentRecordingSessionState(null) so the main-process state is fully reset when the user chooses New Project. Files changed: electron/ipc/handlers.ts — clearCurrentVideoPath clears all three state vars
…egression This fix has been reverted twice by parallel PRs touching VideoPlayback.tsx. The comment explains why getZoomScale(region) must be used instead of region.depth in the drag handler, and explicitly warns against switching it back. Files changed: src/components/video-editor/VideoPlayback.tsx — protective comment on clampFocusForRegion
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Tracking issue: siddharthvaddem#578 Branch to wire cursorCaptureMode into getUserMedia constraints so the cursor toggle is respected when falling back from native Windows capture.
…ecording
getUserMedia on this Chromium version always bakes the OS cursor into raw
footage regardless of the optional cursor:"never" constraint (confirmed
non-functional). Implement the correct workaround:
• On recording start (editable-overlay, web getUserMedia path):
- Run a fire-and-forget PowerShell one-shot that calls SetSystemCursor
with a 32x32 fully-transparent bitmap for all 16 standard cursor IDs,
making the OS cursor invisible (and therefore absent from captured frames)
- Create a full-screen transparent BrowserWindow (cursor-overlay) at
screen-saver always-on-top level, excluded from capture via
setContentProtection(true), with setIgnoreMouseEvents(true,{forward:true})
so it is fully click-through but its renderer still receives mousemove
- The CursorOverlay React component tracks window.mousemove and renders
an SVG Windows-style arrow cursor via rAF-batched CSS transform,
giving the user visual cursor feedback that never appears in footage
- Start main-process 50ms HUD cursor polling so HUD controls remain
interactive even while the OS cursor is transparent
• On recording stop:
- Close cursor overlay window immediately
- Restore all system cursors via SystemParametersInfo(SPI_SETCURSORS=87)
- Stop HUD cursor polling
• before-quit safety hook: spawnSync restore script so cursor is always
recovered even if the user force-closes while recording
• WGC / native Windows recording path is unaffected -- WGC already excludes
the cursor from footage via captureCursor:false without needing OS-level
cursor hiding
…r SVG - restoreSystemCursor() is no longer awaited on recording stop — Add-Type C# compilation in PowerShell takes 3-5 s and was blocking the entire stop handler, preventing the editor from opening and making it appear the OS cursor was permanently lost. Restore now runs fire-and-forget in the background; cursor returns within a few seconds of stopping. - Replace rough SVG sketch with an accurate Windows 10/11 Aero arrow cursor path (M 3 2 L 3 23 L 7 19 L 10 27 L 13 25.5 L 10 17.5 L 17 17.5 Z). The shape now matches the standard arrow: straight left edge, diagonal body right edge, notch cut at lower-left, and the characteristic fishhook tail. Rendered with white fill + black stroke + feDropShadow filter to match the Windows shadow appearance.
…ommand
Passing multiline here-strings via powershell.exe -Command mangles embedded
newlines and double-quote literals through Node's CreateProcess argument
encoding on Windows. The [DllImport("user32.dll")] attribute string was
arriving as [DllImport(user32.dll)], causing Add-Type compilation to fail
silently — so neither the cursor hide nor restore actually ran.
Fix: write each script to a uniquely named .ps1 file in a temp directory
and invoke it with -File, exactly as WindowsNativeRecordingSession already
does for the cursor sampler. The temp file is cleaned up after the process
exits (or on kill/error). The before-quit sync path gets the same treatment.
Cursor style dropdown in editor:
- SettingsPanel shows "Native OS" / "Custom cursor" Select when native
recording data is present; custom cursor sliders hidden in native-os mode
- VideoPlayback respects cursorDisplayMode prop — native-os renders live
captured bitmaps, custom forces the stylised SVG/PixiJS overlay
- VideoEditor wires cursorDisplayMode state + passes it to both components
Real-time cursor shape relay (live OS bitmap — better than .cur files):
- PowerShell sampler already captures cursor bitmaps as base64 imageDataUrl;
onCursorTypeChange(type, asset) now carries the bitmap + hotspot so the
overlay renders the user's actual cursor at the correct DPI for any theme
- Re-fires when either cursor type OR asset handle changes
- handlers.ts forwards both via sendToCursorOverlay("cursor-type-change")
- New sendToCursorOverlay() helper in windows.ts
- preload.ts + electron-env.d.ts expose onCursorTypeChange with CursorOverlayAsset
CursorOverlay:
- Renders <img src={liveAsset.imageDataUrl}> offset by hotspot when available
- SVG shapes (arrow/pointer/text/crosshair/resize/not-allowed) as fallback
for the first ~50ms before the first bitmap arrives from the sampler
- Each SVG shape has correct hotspot offset so cursor tip aligns with mouse
Bug fix:
- Replace require("node:child_process") in before-quit ESM handler with
properly imported spawnSync
makaradam
force-pushed
the
feature/cursor-capture-web-fallback
branch
2 times, most recently
from
3478fbd to
338e0bc
Compare
…wn guard
GDI alpha channel fix (I-beam invisible):
- Get-CursorAsset now draws on an opaque #00DC00 background instead of
transparent. GDI's DrawIcon does not write the alpha channel when the
target surface is transparent, leaving all pixels with A=0 (invisible).
Drawing on a fully-opaque surface preserves A=255 for every drawn pixel.
A LockBits chroma-key pass then replaces the green background with alpha=0
so transparent cursor areas become transparent again.
I-beam detection in Get-CustomCursorType:
- Added 'text' cursor detection before the hand-detection path.
Narrow (< 35% bitmapWidth), tall (> 50% bitmapHeight), centred hotspot
cursors are now classified as 'text' rather than falling through to null.
The minimum-pixel threshold is lowered to 20 (from 90) for this early path
so thin I-beam shapes aren't rejected before type detection runs.
Taskbar z-order fix:
- Added startCursorOverlayZOrderPolling / stopCursorOverlayZOrderPolling in
windows.ts. Re-asserts setAlwaysOnTop("screen-saver") every 500 ms while
recording so the virtual cursor overlay stays above the Windows 11 taskbar,
which periodically re-asserts its own TOPMOST z-order and can push our
window below it. Polling starts on recording start, stops on stop/quit.
Native-OS dropdown guard:
- Cursor Style dropdown in SettingsPanel is now gated behind showCursorSettings
(true only when native bitmap data exists for the recording). Previously the
dropdown appeared for all recordings, causing confusion when native data
hadn't loaded yet and the PIXI custom cursor was shown instead.
… detection The LockBits/DrawIcon approach introduced in the previous commit was crashing the PowerShell sampler script (ErrorActionPreference=Stop). Any exception between LockBits and UnlockBits — e.g. a format mismatch or GDI+ surface conflict — left the bitmap locked, causing Save() to throw, killing the entire script. This meant zero cursor assets were captured, provider stayed "none", hasNativeCursorRecordingData returned false, and the editor could never switch to Native OS mode (PIXI custom cursor showed instead). Fix: replace the whole DrawIcon + LockBits chroma-key block with a single call to Icon.ToBitmap(). This .NET API correctly converts cursor icons to 32bppArgb with proper alpha for ALL cursor types including monochrome (I-beam, hourglass) where DrawIcon silently left A=0 for every pixel. No LockBits, no crash risk. Also add pointer/hand detection to Get-CustomCursorType: Chrome uses its own hand cursor resource (not IDC_HAND 32649) so Get-StandardCursorType returns null for link hovers. New check: hotspot near top of image (fingertip, Y<20%) with reasonable body size → classified as 'pointer'.
…r native cursor PS script: add $lastResolvedCursorType so custom cursors (e.g. Chrome hand) keep their classified type on every sample after the first handle change. Previously Get-StandardCursorType returned null for custom handles, reverting the type to null one frame after detection. VideoPlayback: add explicit z-index:10 to the PIXI canvas container and willChange:transform to the native cursor <img> so Chromium's GPU compositor orders them correctly (native cursor definitively above PIXI canvas). Also reset the PIXI cursor overlay when switching away from custom mode so no residual sprite state bleeds through on the next render tick.
…XI bleed-through Three targeted fixes for the cursor helper and editor cursor mode bugs: 1. CursorOverlay: remove liveAsset bitmap rendering entirely — always use SVG shapes. During editable-overlay recording SetSystemCursor replaces every standard Windows cursor handle with a transparent 32×32 bitmap; when Chrome triggers a wait/app-starting cursor (e.g. clicking a link) ToBitmap() produces a fully transparent PNG which made the helper cursor vanish. SVG shapes derived from the cursor TYPE are always opaque and never disappear. 2. PS script Get-CustomCursorType: add arrow detection (hotspot X < 15%, Y < 15%) placed before the pointer check. Chrome's private arrow cursor resource has its hotspot at ~(3, 2) out of a 32×32 bitmap (~9% X, 6% Y) which previously fell through all conditions and returned null, keeping the overlay stuck on the last resolved type or defaulting to arrow only because of the fallback — now it is explicitly classified as 'arrow' so type-change events fire correctly when hovering non-interactive areas. 3. VideoPlayback: promote cursorDisplayModeRef and hasNativeCursorRecordingRef sync from useEffect to useLayoutEffect. The PIXI ticker runs in RAF (before paint); useEffect fires after paint, leaving a one-frame window where the ticker sees stale ref values and renders the PIXI cursor on top of the native OS cursor when switching to "native-os" mode. useLayoutEffect is synchronous with the commit phase and guarantees the refs are current before the next RAF fires.
…r editable-overlay recordings Four files changed: .gitattributes (new): Added eol=lf rules for all TypeScript/JS/JSON source files. Without this, git core.autocrlf=true on Windows was converting LF→CRLF on every checkout/stash-restore, causing Biome (lineEnding: lf) to fail the pre-commit hook on otherwise correctly formatted files. RECORDING HELPER — CursorOverlay.tsx: Previously used hand-crafted SVG <path> elements that bore little resemblance to real Windows cursors. Replaced with the same cursor SVG files from @/assets/cursors/ (Cursor=Default.svg, Cursor=Hand-(Pointing).svg, etc.) that the editor already uses for its native-os preview. Hotspot values mirror PRETTY_NATIVE_CURSOR_ASSETS in nativeCursor.ts so the recording overlay and the editor preview share identical cursor geometry. EDITOR NATIVE-OS MODE — nativeCursor.ts + windowsNativeRecordingSession.ts: Editable-overlay recordings hide the OS cursor via SetSystemCursor, which replaces every standard Windows handle with a transparent 32x32 bitmap. Because no bitmap assets survived, stop() was returning provider=none, hasNativeCursorRecordingData() returned false, and the editor always fell through to the PIXI custom cursor regardless of the selected mode. - windowsNativeRecordingSession.ts: provider is now native whenever samples exist, not only when bitmap assets exist. Position+type samples are enough to drive the native-os cursor preview. - hasNativeCursorRecordingData: assets.length > 0 requirement removed. - resolveActiveNativeCursorFrame / resolveInterpolatedNativeCursorFrame: when a sample has no assetId (editable-overlay), a NativeCursorAsset is synthesised from PRETTY_NATIVE_CURSOR_ASSETS keyed on the sample cursorType. The existing resolveNativeCursorRenderAsset path already prefers pretty SVGs over raw bitmaps, so native-os mode now shows a clean Windows Aero cursor at the correct recorded position for ALL recording modes. - PrettyNativeCursorAsset and PRETTY_NATIVE_CURSOR_ASSETS are now exported.
makaradam
force-pushed
the
feature/cursor-capture-web-fallback
branch
from
d209490 to
8b98cf0
Compare
…detection - Replace Cursor=Default.svg (wrong-direction bottom-right arrow) with Cursor=AeroDefault.svg (proper Windows Aero top-left arrow) in both the recording helper (CursorOverlay.tsx) and the editor native-os preview (nativeCursor.ts). Hotspot updated to (1, 1). - resolveNativeCursorRenderAsset now falls back to the arrow SVG when the cursor type cannot be determined. Prevents the cursor becoming invisible for editable-overlay recordings where SetSystemCursor replaced every OS bitmap with a transparent one and the type-detection heuristic failed. - Get-CustomCursorType rewrites arrow and pointer detection to measure the hotspot distance from the opaque bounding-box corner rather than the hotspot absolute fraction of the full bitmap. The old thresholds (< 15%) were calibrated for 1x displays; on 2x HiDPI Chrome arrow cursors have their hotspot at ~25% of the bitmap, silently failing detection. The new logic is scale-independent and also requires significant rightward extent for arrow (diagonal body) vs. primarily downward for pointer.
…ng/blur
- CursorOverlay (recording helper): derive logical CSS size from IPC asset
(asset.width / asset.scaleFactor) so the SVG renders at the real OS cursor
size, not a hardcoded 32px. Falls back to 32/devicePixelRatio before the
first sample arrives. Hotspot is scaled proportionally.
- CursorOverlayAsset: add scaleFactor field so the recording session can
forward the display scale factor to the overlay renderer.
- windowsNativeRecordingSession: pass scaleFactor in the onCursorTypeChange
overlay asset payload.
- resolveNativeCursorRenderAsset: scale the pretty SVG asset to the actual
logical cursor size (asset.width / scaleFactor) instead of always returning
32px. Hotspot is scaled proportionally so VideoPlayback positions the
cursor correctly at any OS cursor size.
- VideoPlayback native-OS cursor path:
* Remove cursorSizeRef.current multiplier (was DEFAULT_CURSOR_SIZE=3.0) —
native cursor uses scale=1.0 so only camera zoom is applied.
* Remove click bounce (getNativeCursorClickBounceScale) — OS cursors do
not animate on click.
* Remove motion blur (getNativeCursorMotionBlurPx = 0) — OS cursors are
always sharp.
* Remove position smoothing (smoothing=0, forceSnap=true) — OS cursors
are crisp and immediate, zero interpolation.
- electron/windows.ts: reduce cursor overlay z-order re-assert polling from
500 ms → 50 ms so the virtual cursor reappears within 50 ms after the
Windows 11 taskbar pushes it below.
- Cursor=AeroDefault.svg: update to cleaner Adobe Illustrator version (white
fill + black stroke polygon, viewBox 0 0 36.7 56.2). Update arrow width/
height/hotspot constants in CursorOverlay.tsx and nativeCursor.ts to match
the new aspect ratio (32×49, hotspot 1,2).
Reverts the logical-pixel sizing (asset.width / asset.scaleFactor) that shrank every cursor except the arrow on HiDPI displays. Captured bitmap widths are physical pixels, so dividing by scaleFactor double- applied DPI and made pointer/text/crosshair tiny while leaving the already-too-tall arrow oversized. - resolveNativeCursorRenderAsset: return prettyAsset.width/height directly. The browser handles HiDPI natively (32 CSS px → 64 physical px on a 2× display) so no manual scaling is needed. - CursorOverlay: drop logicalSize state; render at the SVG's canonical width/height/hotspot. Matches the editor preview exactly. - Arrow (PRETTY_NATIVE_CURSOR_ASSETS.arrow and CursorOverlay ARROW): shrink from 32×49 to 20×31. The SVG fills its 36.7×56.2 viewBox tightly while a real Windows arrow occupies only the top-left of a 32×32 cursor box — rendering at 20×31 preserves the SVG aspect ratio while matching the visual size of a real Windows arrow and unifying perceived size with the other (box-filling) cursor SVGs. Tip at SVG (0.8, 1.8) → hotspot (1, 1) at 20px width. - Remove unused scaleFactor field from CursorOverlayAsset interface.
The frameRenderer exporter was applying cursorScale (the editor's cursor size slider, default 3.0) and cursorClickBounce/cursorMotionBlur/ cursorSmoothing to the native OS cursor. VideoPlayback.tsx ignores those for native cursors (scale=1.0, no smoothing/blur/bounce), so the exported video did not match the editor preview. The exporter only has a drawNativeCursor path — there is no custom- cursor rendering in the export pipeline, so cursorScale here is always applied to a native cursor. Mirror the preview's behavior so what the user sees in the editor is what ends up in the exported MP4/GIF: - smoothing=0, forceSnap=true (no interpolation; OS cursor is crisp) - scale=1.0 (no cursorSize multiplier, no click bounce animation) - blurPx removed (no motion blur) The early-return for `cursorScale <= 0` is preserved so the user can still hide the cursor entirely via the editor toggle.
makaradam
force-pushed
the
feature/cursor-capture-web-fallback
branch
from
7c6db49 to
70a5086
Compare
Three related cursor consistency fixes: 1. Cursor=AeroDefault.svg now ships at the same 32×33 viewBox as every other cursor SVG (pointer, text, crosshair, etc.). The arrow shape sits in the lower-middle of the canvas with its tip at SVG (13.38, 12.90). Update the per-asset metrics in PRETTY_NATIVE_CURSOR_ASSETS and CursorOverlay so the arrow renders at the same dimensions as the rest of the set and the hotspot lands on the tip. 2. resolvePrettyNativeCursorAsset no longer falls back to the heuristic resolveUntypedPrettyNativeCursorAsset for untyped cursors. That heuristic guessed open-hand for some Chromium grab cursors, so the editor preview / exported video would show a different icon than the recording helper had drawn during capture (the helper has no such heuristic — it just looks up cursorType and falls back to arrow). Direct type lookup only, matching CursorOverlay exactly. Removed the now-dead resolveUntypedPrettyNativeCursorAsset function. 3. Cursor-overlay z-order polling reworked. The 50 ms interval was still letting the Windows 11 taskbar paint over the virtual cursor because (a) the interval was too slow for visible refresh and (b) setAlwaysOnTop short-circuits when the flag is already true, so no actual SetWindowPos call fired and the taskbar kept winning the topmost activation race. Now poll at ~16 ms (≈60 fps), toggle alwaysOnTop off→on to force the underlying SetWindowPos, and call moveTop() to push above any other recently-activated topmost window.
Three issues fixed:
1. EDITOR PREVIEW SIZE — cursor was rendering at its SVG natural size
(32 CSS px) regardless of how the preview shrank the source video.
On a HiDPI source displayed inside the editor panel the cursor
looked roughly 2× too big relative to the video, because we were
missing two scaling factors:
a) maskRect.width / videoSize.width — how much the preview
shrinks the source video to fit the panel.
b) asset.scaleFactor — the source display's DPI (a 32-CSS-px
cursor on a 1.5× display was captured as 48 source-px in the
recorded video).
transformedScale now multiplies these together with cameraScale
for zoom regions. Editor preview now shows the cursor at the same
relative-to-video size the user saw during recording.
2. EXPORT SIZE — multiply cursor scale by asset.scaleFactor in the
frame renderer. On a 1.5× DPI source the cursor was captured at
48 source-px wide but we were drawing 32 px on the output canvas —
~33% smaller than the original. Same factor as the preview, so
editor preview and exported MP4/GIF now agree on cursor size.
3. EDGE FLICKER — getCroppedCursorPosition returned null whenever the
normalized cursor coordinate jittered just past 1.0 (e.g. when the
recorded cursor hovered over the right edge of the display or the
bottom taskbar). Next frame it would be back in bounds, so the
cursor flapped visible/hidden every other frame. Added a 5%
tolerance around [0, 1]: within slack we clamp to the edge (no
flicker), beyond slack we still return null so user-cropped regions
remain genuinely hidden.
Known limitation (not fixed): the Win11 taskbar can still occasionally
paint over the recording-helper virtual cursor because the shell tray
window re-asserts HWND_TOPMOST faster than our 16 ms toggle+moveTop
polling. A proper fix requires a native Win32 module (SetWindowPos
with SWP_NOSENDCHANGING + WS_EX_NOACTIVATE) — out of scope for this
patch but flagged for follow-up.
The custom PIXI cursor overlay never flickered because it pre-allocates
one sprite per cursor type and only toggles `visible`; the position
update is purely on the GPU compositor. Our native-OS cursor renders
through an HTML <img> instead, and every rAF tick was also re-setting
`style.width` and `style.height`. Setting either of those forces a
synchronous DOM layout pass, which on top of the simultaneous transform
update produced visible blinking during continuous playback.
Fix matches the custom cursor's strategy:
- width + height are mutated ONLY when the cursor type changes
(same time we change `src`). Intrinsic CSS-pixel size lives on
the <img> as a stable layout property.
- Size scaling moves into `transform: ... scale(s)` alongside the
existing translate3d. CSS transforms are GPU-composited with zero
layout cost, so per-frame size changes for zoom/camera scale no
longer trigger reflow.
Transform order is `translate3d(X, Y, 0) scale(s)` which CSS applies
right-to-left: scale around transformOrigin (0,0) first, then translate
the scaled top-left to (X, Y). Hotspot offset is pre-scaled in X, Y so
the math is identical to before.
…ffi FFI)
Electron's BrowserWindow.setAlwaysOnTop(true, screen-saver) short-
circuits when the alwaysOnTop flag is already set, so repeat calls
become no-ops and the Win11 shell tray (Shell_TrayWnd) — which also
sits at HWND_TOPMOST and re-asserts itself continuously — wins the
activation race and paints over our cursor overlay. Toggling false→
true forces a real call but still goes through window-state tracking
that lags the shell by a frame or two.
Fix: call SetWindowPos directly via koffi (runtime FFI, no native
compilation, no electron-rebuild dance) with HWND_TOPMOST and
SWP_NOACTIVATE | SWP_NOSENDCHANGING. SWP_NOSENDCHANGING is the
critical flag — it suppresses the WM_WINDOWPOSCHANGING broadcast that
the shell listens for to re-promote itself, so our promotion sticks.
Wiring:
- new electron/native-bridge/win32/topmost.ts: koffi binding plus
promoteAboveTaskbar(win) helper that's a safe no-op on non-Windows
and on FFI load failure.
- electron/windows.ts: call promoteAboveTaskbar on overlay
ready-to-show, then again inside the 60 fps polling loop. The old
setAlwaysOnTop toggle remains as a fallback for non-Windows and
FFI-failure paths so cross-platform behavior is unchanged.
Also: filled the empty insertCSS catch handler with an explanatory
comment (pre-existing biome lint warning that was blocking commit on
files touched here).
The virtual cursor now stays above the Win11 taskbar regardless of
taskbar position, auto-hide setting, or how aggressively Explorer
re-asserts its own z-order.
…-cursor)
Apps like Figma, Photoshop, and most games hide the OS cursor with
ShowCursor(false) and draw their own custom cursor instead. During
editable-overlay recording we were drawing our virtual SVG cursor on
top, so the user saw two cursors live (helper SVG + Figma's drawn one)
and the exported video also had two cursors (Figma's baked-in pixels +
our SVG re-rendered on top).
Detection (Win32):
GetCursorInfo's flags has CURSOR_SHOWING (0x1) set when the OS thinks
the cursor is visible. When an app calls ShowCursor(false) enough
times to drive the per-window counter negative, the flag flips to 0.
This signal is INDEPENDENT of SetSystemCursor(transparent) — that
only swaps the cursor bitmap, the visibility flag still reflects the
app's intent. So we can read it reliably even while our own capture
override is in effect.
Plumbing:
- PowerShell sampler: emit osCursorHidden = !($flags & CURSOR_SHOWING)
alongside the existing visible flag.
- WindowsCursorSampleEvent + CursorRecordingSample: optional
osCursorHidden boolean.
- recording session: forwards osCursorHidden in every sample, and
re-fires the onCursorTypeChange IPC whenever the hidden state
toggles so the helper window can react in real time.
- preload + electron-env.d.ts: add the third arg to the IPC bridge.
Honoring it:
- CursorOverlay (recording helper): when osCursorHidden, set the
<img>'s display:none so the app's own cursor is the only one visible.
- VideoPlayback (editor preview): when frame.sample.osCursorHidden,
call hideNativeCursorPreview() instead of rendering the SVG.
- frameRenderer (export): early-return from drawNativeCursor when
the sample has osCursorHidden so the exported MP4/GIF doesn't get
a duplicate cursor either.
Result: in apps that draw their own cursor, the recording helper, the
editor preview, and the exported video all show ONLY the app's own
cursor — no duplicate. In every other app the existing behavior is
unchanged.
The CI vite build failed when bundling electron/main.ts because Vite's
commonjs-resolver tried to read koffi's pre-built .node binary as
JavaScript:
[commonjs--resolver] node_modules/koffi/build/koffi/win32_ia32/
koffi.node (1:3): Unexpected character '\0'
koffi ships its native FFI binary as part of the package and resolves
it at runtime via require(). We don't want Vite to try to bundle that
binary — we want the bundled main.js to keep the require(koffi) call
intact and let Electron's runtime resolve it from node_modules.
Add koffi to the electron-main vite plugin's rollup `external` list so
the bundler leaves the require alone.
Packaging side is already covered:
- koffi is in `dependencies` (not devDependencies) so electron-
builder includes it in the bundled node_modules.
- electron-builder.json5 already has `asarUnpack: [**/*.node]`,
which extracts koffi's .node binary out of the asar archive so
dlopen can load it at runtime.
…ilure on x64)
The koffi binding declared SetWindowPos's HWND args as long. On
Windows x64, koffi's long is 32 bits (matches the Win32 LONG type),
but HWND is pointer-sized — 64 bits. Every HWND we passed got its
top 32 bits truncated to zero, SetWindowPos was being called on an
invalid handle and returning 0 (failure).
Worse, my error log was guarded by `if (!suppressErrorLog)` so only
the very first failure surfaced — every subsequent failed call at
60 fps stayed silent. Net effect: the overlay never actually got
promoted via the direct Win32 path, the fallback toggle code in
windows.ts also never ran (because promoteAboveTaskbar appeared to
succeed from its boolean return), and the taskbar kept winning the
z-order race exactly as before.
Fixes:
- Use intptr_t for both HWND args. Koffi sizes that to the
native pointer width on each platform.
- Add a one-time SUCCESS log when SetWindowPos actually lands so
we have positive confirmation in dev/prod that the fix worked.
- Include the HWND hex value in both success and failure logs so
we can tell whether the handle we extracted is plausible.
After this lands, [win32-topmost] SetWindowPos(HWND_TOPMOST) OK
should appear in the main-process console the first time the cursor
overlay is shown. If you instead see a `failed` line with a
GetLastError code, that tells us exactly which API check is rejecting
the call.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements cursor capture for
editable-overlayrecording mode (webgetUserMediapath):SetSystemCursorreplaces all 16 system cursor handles with a transparent 32×32 bitmap, so the OS cursor never appears in raw footageBrowserWindow(excluded from capture viasetContentProtection) renders the user's real-time cursor position as a software cursor they can see but the recorder can'tstoreRecordedSessionFilesnow awaitsstopCursorRecording()so cursor telemetry is always flushed before the session is writtenrequire()from ESMbefore-quithandler;spawnSyncis now a top-level importTest plan