fix(ios): recover sparse snapshot trees via element queries (React Native)

[alexzaetoro]

Problem

On some apps — most reliably React Native / Expo apps — agent-device snapshot on iOS returns a degenerate 2–3 node tree (application → window [→ other]) even though the screen is foregrounded and fully rendered. The same screen at the same instant exposes 300+ nodes (with all testIDs as accessibility identifiers) through Appium/WebDriverAgent's /source.

Everything that reads the tree is affected: selector click/find/is/get, and react-native dismiss-overlay (hangs to the daemon timeout when a LogBox toast is on screen). Coordinate taps are unaffected and work fine (thanks to the quiescence-skip fix in 0.17.0).

Root cause (verified)

I patched the shipped runner source locally to rule out the usual suspects:

• Idle/quiescence: setting waitForIdleTimeout = 0 (via KVC) around the capture still returns 2 nodes.
• Main-thread watchdog: raising mainThreadExecutionTimeout 30→120s — the snapshot completes ok=1 in runner.log, payload still 2 nodes.

So the public XCUIElement.snapshot() API itself returns the sparse tree for these apps. WDA sees the full tree because it bypasses the public snapshot with its own raw accessibility traversal. Notably, XCUIElementQuery still resolves the controls on the same screen — which this runner already relies on elsewhere (collapsedTabFallbackNodes uses descendants(matching:.any).allElementsBoundByIndex precisely because the snapshot tree omits tab children).

Change

Rather than introducing a private raw-AX traversal, this reuses the query path the codebase already trusts:

1. After snapshotFast builds its node list, detect the sparse-tree signature — only structural containers (Application/Window/Other/ScrollView) with no label/identifier/value and nothing hittable (snapshotNodesAreDegenerate).
2. When detected, recover via the existing query-based flat traversal (snapshotFlatInteractive with interactiveOnly: false) so on-screen controls and testIDs come back instead of an empty tree (snapshotQueryRecovery).
3. Recovery is conservative: if queries reveal nothing more than the sparse tree, the original result is kept. The recovered payload is marked truncated: true and carries an explanatory message, mirroring the existing snapshotDepthLimitedAccessibilityFallback pattern.

Public XCUITest APIs only; no new dependencies; the non-sparse path is unchanged (the detector returns early as soon as any real content/control is seen).

Tests

Added unit tests for the detector (testSnapshotNodesAreDegenerate*) covering: structural-only tree (degenerate), content present, hittable control present, typed control present, and empty input. swiftc -parse passes on the file.

Validation status

⚠️ Draft — the Swift unit tests and a full on-device XCUITest build/run against a real RN app are pending (I couldn't run the full UI-test build in my environment). I have side-by-side evidence (agent-device snapshot vs WDA /source from the same screen, runner.log excerpts, and the patch diffs used to rule out idle/watchdog causes) and a perfect repro app — happy to share, and to iterate on the detection heuristic or the recovered payload shape. Opening as draft so you can validate on-device first.

Closes the sparse-tree half of the iOS RN snapshot issue.

Made with Cursor

[thymikee]

Code review

First off — thank you for the excellent diagnosis. The side-by-side WDA comparison and the patches ruling out idle/watchdog causes are genuinely valuable, and reusing the trusted query path instead of private AX traversal is a reasonable instinct. That said, reviewing the implementation against the actual snapshotFast pipeline turned up problems that need addressing before this can land.

Findings

1. Major — the detector likely never fires on the real degenerate tree. snapshotNodesAreDegenerate requires every node to be non-hittable, but snapshotFast computes hittable via computedSnapshotHittable (main RunnerTests+Snapshot.swift:633-650), which returns true for any enabled, full-screen-framed node whose center is in the viewport with no later occluding nodes — exactly the Application/Window/Other nodes of a degenerate 2-3 node tree (each is the tail of the flattened order, so laterSnapshots is empty). The detector (RunnerTests+Snapshot.swift:396-407 in this branch) returns false on the first node and recovery never triggers in the targeted RN scenario. This is consistent with the on-device validation gap noted in the description — please verify against the repro app.

2. Major — unbounded query sweep when recovery fires. snapshotQueryRecovery (:410-432) calls snapshotFlatInteractive with interactiveOnly: false, which sets deadline = Date.distantFuture (main :321-323), so the 19 full-app allElementsBoundByIndex queries in flatInteractiveElements run with no time budget — unlike every existing fallback, which is capped by the 1 s flatInteractiveFallbackBudget. If the detector fires on a splash screen, loading state, or mid-transition (all of which legitimately produce structural-only trees), the snapshot can stall for many seconds, against the bounded-traversal design called out in Add iOS Simulator AX snapshot fallback for XCTest snapshot failures #701.

3. Minor — false-positive recovery on legitimately empty screens silently swaps payload shape: a hierarchical tree becomes a flat depth-1 list under a synthetic root whenever queries find ≥2 elements (:424), and elements rendered between snapshot() and the queries make the message assert "the screen has rendered content" — something the detector never verified.

4. Minor — the recovery hardcodes interactiveOnly: false (:417), dropping the caller's flag, so an interactive-only request recovers into a payload including all static texts/images — including off-screen ones (flatSnapshotNode skips the visibility filter when !interactiveOnly).

5. Minor — scope semantics silently change on recovery: snapshotFast resolves options.scope as a scope element, but the flat path applies it as a case-insensitive substring filter on label/identifier/value, so a scoped degenerate snapshot can recover unrelated elements that merely contain the scope text.

6. Minor — the explanatory message never reaches consumers: readAppleSnapshotResult (src/core/interactors/apple.ts:130-137) extracts only nodes and truncated, so the recovery is observable only as (truncated) plus an NSLog — which sits awkwardly with Add iOS Simulator AX snapshot fallback for XCTest snapshot failures #701's "no silent fallback without diagnostics" requirement (pre-existing pattern, but worth knowing the message won't surface).

7. Minor (tests) — the unit tests (:607-672) test only the pure detector with fabricated hittable: false nodes, encoding the assumption from finding 1; there's no test exercising the recovery integration, the nodes.count > 1 guard, or option propagation.

Relationship to #758

This directly overlaps and textually conflicts with #758, which modifies the same snapshotFast return site with its own sparse-tree detector (isSparseApplicationWindowTree, ≤2 Application/Window nodes ignoring hittability — notably sidestepping finding 1) plus a private-AX fallback, and additionally covers the compact-interactive (snapshot -i) path where #701 reports the degradation is actually observed — a path this PR doesn't touch. The maintainers will need to pick one approach; the public-API recovery idea here remains attractive long-term (it works on real devices and avoids private API), so even if #758 lands first, a fixed detector + budgeted query recovery could be a worthwhile follow-up.

Suggested next steps

• Relax the detector to ignore hittability (structural types + no label/identifier/value is probably sufficient), and validate it fires on your repro app.
• Cap the recovery with the existing flatInteractiveFallbackBudget (or a similar deadline) instead of distantFuture.
• Propagate the caller's interactiveOnly flag and add an integration-level test for the recovery path.

---

Generated by Claude Code