Skip to content

feat: support simulator camera video files#752

Open
thymikee wants to merge 3 commits into
mainfrom
feat/android-camera-video-files
Open

feat: support simulator camera video files#752
thymikee wants to merge 3 commits into
mainfrom
feat/android-camera-video-files

Conversation

@thymikee

@thymikee thymikee commented Jun 10, 2026

Copy link
Copy Markdown
Member

Summary

Add sample-video camera inputs for both emulator families agents need to test:

  • Android emulator boot accepts --camera-front / --camera-back modes or video file paths, with running-emulator rejection when camera sources cannot be changed safely.
  • iOS simulator app launch accepts open <app> --camera-video <path> / cameraVideo, starts the vendored serve-sim helper, injects the AVFoundation dylib for that app process, and stops the helper on app close.
  • Vendored only the serve-sim runtime camera artifacts we need with Apache-2.0 attribution under third_party/serve-sim-camera: the helper binary, injector dylib, README, and license. The local binary filenames are neutral (camera-helper, camera-injector.dylib). Unpacked payload is ~600 KB; helper binaries are ~584 KB.
  • Updated CLI/client docs and typed command metadata for the new camera options.

Validation

  • pnpm format
  • pnpm exec vitest run src/utils/__tests__/args.test.ts src/__tests__/client.test.ts src/core/__tests__/dispatch-open.test.ts src/daemon/__tests__/context.test.ts src/platforms/ios/__tests__/simulator-camera.test.ts
  • pnpm exec vitest run src/platforms/ios/__tests__/simulator-camera.test.ts after trimming unused vendored source files and renaming local binary artifacts
  • pnpm check:quick
  • pnpm check:unit
  • pnpm build
  • pnpm check:fallow --base origin/main
  • Manual Android emulator proof: launched Pixel_9_Pro_XL_API_37 with -camera-back videofile:/private/tmp/agent-device-back-camera.mp4 -camera-front none -feature VideoPlayback; repeated front-camera playback.
  • Manual iOS simulator proof: launched throwaway AVFoundation camera app on iPhone 17, iOS 26.2, with open com.agentdevice.CameraFeed --camera-video /private/tmp/agent-device-camera-proof/camera-feed.mp4; captured /private/tmp/agent-device-camera-proof/simcam-proof-final.png showing the MP4 test pattern in the camera preview.

Touched files: 29. Scope expanded from Android boot camera files to iOS simulator app-launch video injection at reviewer request.

@github-actions

github-actions Bot commented Jun 10, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 1.2 MB 1.2 MB +2.3 kB
JS gzip 385.4 kB 386.1 kB +747 B
npm tarball 497.3 kB 498.0 kB +700 B
npm unpacked 1.7 MB 1.7 MB +2.4 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 28.3 ms 28.3 ms -0.0 ms
CLI --help 43.9 ms 42.4 ms -1.5 ms

Top changed chunks:

Chunk Raw diff Gzip diff
dist/src/8806.js +998 B +397 B
dist/src/1352.js +510 B +187 B
dist/src/session.js +537 B +96 B
dist/src/8173.js +168 B +50 B
dist/src/8699.js +100 B +17 B

@github-actions

Copy link
Copy Markdown
PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://callstack.github.io/agent-device/pr-preview/pr-752/

Built to branch gh-pages at 2026-06-10 18:06 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@thymikee thymikee changed the title feat: support Android emulator camera video files feat: support simulator camera video files Jun 10, 2026
@thymikee thymikee force-pushed the feat/android-camera-video-files branch from a382ae2 to 7faedef Compare June 10, 2026 19:56
@thymikee thymikee force-pushed the feat/android-camera-video-files branch from 7faedef to 79c8bc7 Compare June 11, 2026 05:55

Copy link
Copy Markdown
Member Author

Code review

Verdict: significant issues — feature plumbing and tests are solid, but the vendoring approach and helper-process lifecycle need work before merge.

Findings

  1. Majorthird_party/serve-sim-camera/README.md: no integrity pinning for the vendored binaries — the README records only serve-sim@0.1.34, with no SHA-256 of camera-helper or camera-injector.dylib, no upstream commit/tarball hash, and no verification script, so nobody can confirm the ~600 KB opaque Mach-O blobs actually match upstream.

  2. Major (repo convention break)third_party/serve-sim-camera/bin/*: these are the first compiled binaries committed to the repo; every existing helper (android-snapshot-helper, android-multitouch-helper, macos-helper, ios-runner) is vendored as source and built at package time. Committing unauditable executables that get DYLD-injected into user app processes is a meaningful supply-chain escalation.

  3. Majorsrc/platforms/ios/simulator-camera.ts:53-66 + src/platforms/ios/apps.ts:322,1174: the helper is spawned detached/unref'd and stopIosSimulatorCameraVideo is only called from closeIosApp and on launch failure. If the app crashes, the simulator is shut down (shutdown has no camera cleanup), the daemon dies, or the user re-opens the app without --camera-video, the helper loops the video forever with no cleanup path.

  4. Majorsrc/platforms/ios/simulator-camera.ts:53-66,110-121: only file existence is validated (no format check), and after runCmdDetached there's no check that the helper survived startup or created the shm segment. A corrupt/unsupported video makes the helper exit immediately; the app then launches "successfully" with a dead camera feed, and the only evidence is an unread log in os.tmpdir().

  5. Majorsrc/daemon/handlers/session-state.ts:205,243: iOS cameraVideo is expanded with SessionStore.expandHome(value, req.meta?.cwd), but Android flags.cameraFront/cameraBack are passed raw to ensureAndroidEmulatorBooted, where resolveAndroidEmulatorCameraMode (src/platforms/android/devices.ts:489-520) resolves against the daemon's cwd and never expands ~ — yet the flag help and docs advertise ./front.mp4, which will fail (or hit the wrong file) in daemon mode.

  6. Minorsrc/platforms/ios/simulator-camera.ts:86-101: stop SIGTERMs the pid from a tmp-dir state file without verifying the process is still the camera helper; after a host reboot or PID reuse it can kill an unrelated process.

  7. Minor — renaming upstream artifacts "to avoid exposing upstream internal artifact names" makes audit-by-diff against the upstream npm tarball harder, and the original filenames aren't recorded; combined with finding 1, the Apache-2.0 claim and version are effectively unverifiable from the repo (LICENSE itself does ship correctly via files).

  8. Minor (tests)simulator-camera.test.ts covers the happy path and non-simulator rejection only — nothing for stop-on-close, stale/missing state files, or missing video file; Android tests don't cover the explicit videofile: prefix branch or the invalid-mode error.

Verified clean

The Android running-emulator rejection (assertCameraInputsCanApplyToEmulator) is correct and tested for both paths; args pass via spawn(..., shell: false) arrays (no injection); shm names are per-launch hashed and within macOS limits, so concurrent sessions don't collide; deep-link paths correctly strip cameraVideo; docs honestly state the forced relaunch.

Overall

The TypeScript integration is well-plumbed and validated at every dispatch layer, but two structural weaknesses remain. Shipping renamed, checksum-less third-party Mach-O executables in git — in a repo where every other helper builds from source — is hard to audit and update safely; at minimum the README needs per-file SHA-256s, original artifact names, and the upstream tarball hash, and the better design is fetching the pinned serve-sim@0.1.34 artifact at install/build time with checksum verification. Separately, the detached helper needs a liveness check at startup and cleanup hooks on simulator shutdown/session close, or long-lived agent hosts will accumulate orphaned video-looping processes.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant