perf(android): improve emulator stream performance

Improve Android emulator WebRTC smoothness by preferring gRPC screenshots, sharing one source per UDID/codec, defaulting Android local streams to software H.264 at a 960px long edge, and exposing Android encoder/source metrics.

Author: DjDeveloperr

README.md

@@ -90,7 +90,11 @@ Use `simdeck service reset` only when you want to rotate the service token and
 restart the LaunchAgent.
 The service uses port 4310 unless you pass `-p` or `--port`, or set a default
 in `~/.simdeck/config.json`.
-SimDeck-owned Android emulator boots use host GPU rendering by default; use
+SimDeck-owned Android emulator boots use host GPU rendering by default; on macOS
+they start the emulator with `-qt-hide-window` so the Qt renderer stays active
+without showing the native emulator window. Managed boots also open the emulator
+gRPC endpoint for event-driven Android video capture, with shared-video polling
+kept as a fallback. Use
 `simdeck service restart --android-gpu auto` or
 `--android-gpu swiftshader_indirect` only as a machine-specific fallback.
 Managed Android boots also add `-no-audio` by default. Set

docs/api/health.md

@@ -61,18 +61,19 @@ GET /api/metrics
 
 Useful fields:
 
-| Field                              | What to look for                             |
-| ---------------------------------- | -------------------------------------------- |
-| `latest_first_frame_ms`            | First-frame startup time                     |
-| `frames_dropped_server`            | Server dropping stale frames to stay current |
-| `keyframe_requests`                | Stream refresh or recovery activity          |
-| `stream_pipeline_resets`           | Encoder resets after all viewers disconnect  |
-| `latest_accessibility_snapshot_ms` | Most recent native accessibility duration    |
-| `max_accessibility_snapshot_ms`    | Slowest native accessibility duration        |
-| `accessibility_snapshot_timeouts`  | Native accessibility calls that timed out    |
-| `active_streams`                   | Open browser streams                         |
-| `encoders[].encoder.overloadState` | `nominal`, `strained`, or `overloaded`       |
-| `client_streams`                   | Recent browser decoder and render reports    |
+| Field                              | What to look for                                           |
+| ---------------------------------- | ---------------------------------------------------------- |
+| `latest_first_frame_ms`            | First-frame startup time                                   |
+| `frames_dropped_server`            | Server dropping stale frames to stay current               |
+| `keyframe_requests`                | Stream refresh or recovery activity                        |
+| `stream_pipeline_resets`           | Encoder resets after all viewers disconnect                |
+| `latest_accessibility_snapshot_ms` | Most recent native accessibility duration                  |
+| `max_accessibility_snapshot_ms`    | Slowest native accessibility duration                      |
+| `accessibility_snapshot_timeouts`  | Native accessibility calls that timed out                  |
+| `active_streams`                   | Open browser streams                                       |
+| `encoders[].encoder.overloadState` | `nominal`, `strained`, or `overloaded`                     |
+| `androidEncoders[]`                | Android video source kind, `videoCodec`, and encoder stats |
+| `client_streams`                   | Recent browser decoder and render reports                  |
 
 If `overloadState` is `overloaded` or dropped frames keep increasing, lower stream quality or restart with software encoding:
 

docs/guide/video.md

@@ -1,9 +1,9 @@
 # Video and streaming
 
-SimDeck streams live device video to the browser. Local sessions default to full-resolution 60 fps. Remote or constrained sessions can trade detail for lower CPU and latency.
+SimDeck streams live device video to the browser. Local iOS sessions default to full-resolution 60 fps. Android emulator browser sessions default to software H.264 with the `balanced` profile capped at 960px on the long edge because the host reads and encodes emulator RGBA frames from the emulator gRPC screenshot stream. Remote or constrained sessions can trade detail for lower CPU and latency.
 
 iOS simulator H.264 uses VideoToolbox for hardware encoding and x264 for software encoding.
-Android emulator H.264 uses the emulator `-share-vid` display surface. SimDeck reads BGRA frames from the `videmulator<console-port>` shared memory region and encodes them on the Mac, so normal Android live video stays on the native shared display path.
+Android emulator H.264 uses the emulator gRPC `streamScreenshot` API when SimDeck owns the boot. SimDeck receives raw RGBA frames, pads odd dimensions for H.264, and encodes them on the Mac. If the gRPC endpoint is unavailable, SimDeck falls back to the emulator `-share-vid` display surface and reads BGRA frames from the `videmulator<console-port>` shared memory region.
 
 ## When encoding runs
 
@@ -13,7 +13,10 @@ shared refresh pump active while frame subscribers exist.
 For Android, SimDeck starts emulators with `-share-vid`, maps the shared display
 region, and feeds changed BGRA frames into the native host H.264 encoder.
 SimDeck-owned Android boots also default to `-gpu host`, matching the native
-emulator app's accelerated renderer while staying in headless shared-video mode.
+emulator app's accelerated renderer while staying hidden. On macOS, managed
+boots use `-qt-hide-window` instead of `-no-window` so the Qt render loop stays
+active without showing the emulator window. Managed Android boots also reserve a
+per-AVD `-grpc` port for event-driven screenshot streaming.
 
 The browser reports whether the page and stream canvas are foreground. When all
 known viewers are hidden or the last frame subscriber disconnects, the native
@@ -38,17 +41,17 @@ simdeck service restart --stream-quality ci-software
 
 Common profiles:
 
-| Profile       | Use it for                              |
-| ------------- | --------------------------------------- |
-| `full`        | Default local full-resolution 60 fps    |
-| `smooth`      | Full-size 60 fps with lower bitrate     |
-| `balanced`    | Good local quality with less bandwidth  |
-| `economy`     | Remote browser or busy machine          |
-| `low`         | Slower Wi-Fi or shared hosts            |
-| `tiny`        | Pull request previews and low bandwidth |
-| `ci-software` | Virtualized CI Macs                     |
-
-The browser also has stream controls for transport, resolution, FPS, and refresh.
+| Profile       | Use it for                                            |
+| ------------- | ----------------------------------------------------- |
+| `full`        | Default local full-resolution 60 fps                  |
+| `smooth`      | 60 fps with lower bitrate; Android caps this at 960px |
+| `balanced`    | Good local quality with less bandwidth                |
+| `economy`     | Remote browser or busy machine                        |
+| `low`         | Slower Wi-Fi or shared hosts                          |
+| `tiny`        | Pull request previews and low bandwidth               |
+| `ci-software` | Virtualized CI Macs                                   |
+
+The browser also has stream controls for transport, resolution, FPS, encoder mode, and refresh. Choosing `Full res` for an Android emulator keeps the native shared-video dimensions, which can be expensive on tall phone profiles. Set `SIMDECK_ANDROID_VIDEO_CODEC=hardware` or choose Hardware in the browser when you explicitly want VideoToolbox for Android.
 
 ## Pick an Android GPU mode
 

packages/client/src/app/AppShell.tsx

@@ -150,6 +150,11 @@ const LOCAL_STREAM_DEFAULTS: StreamConfig = {
   fps: 60,
   quality: "full",
 };
+const ANDROID_LOCAL_STREAM_DEFAULTS: StreamConfig = {
+  encoder: "software",
+  fps: 60,
+  quality: "balanced",
+};
 const REMOTE_STREAM_DEFAULTS: StreamConfig = {
   encoder: "software",
   fps: 30,
@@ -313,6 +318,34 @@ function defaultStreamConfigForTransport(
   return base;
 }
 
+function streamConfigForSelectedSimulator(
+  config: StreamConfig,
+  simulator: SimulatorMetadata | null,
+  remote: boolean,
+  userTouched: boolean,
+): StreamConfig {
+  if (remote || userTouched || !isAndroidSimulator(simulator)) {
+    return config;
+  }
+  const quality =
+    config.quality === "full" || config.quality === "quality"
+      ? ANDROID_LOCAL_STREAM_DEFAULTS.quality
+      : config.quality;
+  const encoder =
+    config.encoder === "auto"
+      ? ANDROID_LOCAL_STREAM_DEFAULTS.encoder
+      : config.encoder;
+  if (quality === config.quality && encoder === config.encoder) {
+    return config;
+  }
+  return {
+    ...config,
+    encoder,
+    maxEdge: undefined,
+    quality,
+  };
+}
+
 function shouldForceInitialFitMode(): boolean {
   if (typeof window === "undefined") {
     return false;
@@ -854,6 +887,37 @@ export function AppShell({
     };
   }, [remoteStream, syncStreamConfig]);
 
+  const effectiveStreamConfig = useMemo(
+    () =>
+      streamConfigForSelectedSimulator(
+        streamConfig,
+        selectedSimulator,
+        remoteStream,
+        streamConfigUserTouchedRef.current,
+      ),
+    [remoteStream, selectedSimulator, streamConfig],
+  );
+
+  useEffect(() => {
+    if (streamConfigUserTouchedRef.current) {
+      return;
+    }
+    setStreamConfig((current) => {
+      const next = streamConfigForSelectedSimulator(
+        current,
+        selectedSimulator,
+        remoteStream,
+        false,
+      );
+      return streamConfigsEqual(current, next) ? current : next;
+    });
+  }, [
+    remoteStream,
+    selectedSimulator?.platform,
+    selectedSimulator?.udid,
+    streamConfig.quality,
+  ]);
+
   const {
     deviceNaturalSize,
     error: streamError,
@@ -869,7 +933,7 @@ export function AppShell({
     paused: !streamConfigReady,
     remote: remoteStream,
     simulator: selectedSimulator,
-    streamConfig,
+    streamConfig: effectiveStreamConfig,
     streamConfigApplyKey,
     streamTransport,
   });
@@ -886,29 +950,42 @@ export function AppShell({
     void refreshRef.current();
   }, [streamStatus.error, streamStatus.state, syncStreamConfig]);
 
-  const updateStreamEncoder = useCallback((encoder: StreamEncoder) => {
-    streamConfigUserTouchedRef.current = true;
-    streamConfigUserChangeAtRef.current = Date.now();
-    setStreamConfigReady(true);
-    setStreamConfigApplyKey((current) => current + 1);
-    setStreamConfig((current) => ({ ...current, encoder }));
-  }, []);
+  const updateStreamEncoder = useCallback(
+    (encoder: StreamEncoder) => {
+      streamConfigUserTouchedRef.current = true;
+      streamConfigUserChangeAtRef.current = Date.now();
+      setStreamConfigReady(true);
+      setStreamConfigApplyKey((current) => current + 1);
+      setStreamConfig({ ...effectiveStreamConfig, encoder });
+    },
+    [effectiveStreamConfig],
+  );
 
-  const updateStreamFps = useCallback((fps: StreamFps) => {
-    streamConfigUserTouchedRef.current = true;
-    streamConfigUserChangeAtRef.current = Date.now();
-    setStreamConfigReady(true);
-    setStreamConfigApplyKey((current) => current + 1);
-    setStreamConfig((current) => ({ ...current, fps }));
-  }, []);
+  const updateStreamFps = useCallback(
+    (fps: StreamFps) => {
+      streamConfigUserTouchedRef.current = true;
+      streamConfigUserChangeAtRef.current = Date.now();
+      setStreamConfigReady(true);
+      setStreamConfigApplyKey((current) => current + 1);
+      setStreamConfig({ ...effectiveStreamConfig, fps });
+    },
+    [effectiveStreamConfig],
+  );
 
-  const updateStreamQuality = useCallback((quality: StreamQualityPreset) => {
-    streamConfigUserTouchedRef.current = true;
-    streamConfigUserChangeAtRef.current = Date.now();
-    setStreamConfigReady(true);
-    setStreamConfigApplyKey((current) => current + 1);
-    setStreamConfig((current) => ({ ...current, maxEdge: undefined, quality }));
-  }, []);
+  const updateStreamQuality = useCallback(
+    (quality: StreamQualityPreset) => {
+      streamConfigUserTouchedRef.current = true;
+      streamConfigUserChangeAtRef.current = Date.now();
+      setStreamConfigReady(true);
+      setStreamConfigApplyKey((current) => current + 1);
+      setStreamConfig({
+        ...effectiveStreamConfig,
+        maxEdge: undefined,
+        quality,
+      });
+    },
+    [effectiveStreamConfig],
+  );
 
   const updateStreamTransport = useCallback((transport: StreamTransport) => {
     setStreamTransport(transport);
@@ -3747,7 +3824,7 @@ export function AppShell({
           !selectedSimulator.isBooted &&
           !selectedSimulatorTransitionKind,
         )}
-        streamConfig={streamConfig}
+        streamConfig={effectiveStreamConfig}
         streamTransport={streamTransport}
         deviceChromeAvailable={selectedSupportsChrome}
         deviceChromeVisible={deviceChromeToggleActive}