Log error on failed xpath lookup

[miguelg719]

why

To fix silent errors whenever an invalid ID is returned by the LLM (xpath lookup failed)

what changed

Add an error log when the LLM returns an element ID with no matching xpath and handle these cases gracefully in act and observe

• actHandler: log with category "action" when no xpath exists for the returned elementId; act() now fails gracefully with success: false and a clear message.
• observeHandler: log with category "observation" and filter out elements whose IDs are missing from the xpath map.

test plan

• Added xpath-lookup-failure-logging.test to verify logging, act() failure behavior, and observe() filtering (including multiple missing IDs).

[changeset-bot]

🦋 Changeset detected

Latest commit: 50bd18c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@browserbasehq/stagehand	Patch
@browserbasehq/stagehand-evals	Patch
@browserbasehq/stagehand-server	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[greptile-apps]

Greptile Overview

Greptile Summary

• Adds level-0 logging when the LLM returns an elementId that cannot be resolved to an xpath in both act and observe handlers.
• Makes ActHandler.act() fail gracefully (success=false with a clear message) when xpath resolution fails, instead of silently producing no action.
• Makes ObserveHandler.observe() filter out elements whose IDs are missing from the xpath map, while emitting a diagnostic log.
• Adds a unit test covering logging + act failure behavior + observe filtering for missing IDs.

Confidence Score: 4/5

• This PR is close to safe to merge, with one small but concrete inconsistency to fix.
• Core behavior changes are straightforward and covered by a focused unit test. The main issue is a definite log-level inconsistency for the same dragAndDrop target lookup failure between act vs observe, which can break log-based monitoring or expectations.
• packages/core/lib/v3/handlers/observeHandler.ts

Important Files Changed

Filename	Overview
.changeset/many-guests-collect.md	Adds a patch changeset entry describing new level-0 logging for xpath lookup failures.
packages/core/lib/v3/handlers/actHandler.ts	Logs level-0 message when LLM elementId has no xpath; act() already returns a graceful failure when no action is normalized.
packages/core/lib/v3/handlers/observeHandler.ts	Logs and filters out observed elements whose elementId has no xpath; introduces a log-level inconsistency for dragAndDrop target lookup failures vs ActHandler.
packages/core/tests/xpath-lookup-failure-logging.test.ts	Adds unit tests validating xpath lookup failure logging and act/observe behavior with missing element IDs (uses extensive mocking).

Sequence Diagram

sequenceDiagram
  autonumber
  actor User
  participant ActHandler
  participant ObserveHandler
  participant Snapshot as captureHybridSnapshot
  participant LLM as inference(act/observe)
  participant Log as v3Logger

  User->>ActHandler: act(instruction, page)
  ActHandler->>Snapshot: captureHybridSnapshot(page)
  Snapshot-->>ActHandler: combinedTree + combinedXpathMap
  ActHandler->>LLM: actInference(domElements, instruction)
  LLM-->>ActHandler: element.elementId
  ActHandler->>ActHandler: normalizeActInferenceElement(elementId)
  alt elementId missing from xpath map
    ActHandler->>Log: level 0 "LLM returned ID ... no xpath keyed ..."
    ActHandler-->>User: { success:false, message:"Failed to perform act: No action found" }
  else xpath found
    ActHandler-->>User: performUnderstudyMethod(...)
  end

  User->>ObserveHandler: observe(instruction, page)
  ObserveHandler->>Snapshot: captureHybridSnapshot(page)
  Snapshot-->>ObserveHandler: combinedTree + combinedXpathMap
  ObserveHandler->>LLM: observeInference(domElements, instruction)
  LLM-->>ObserveHandler: elements[] (elementId...)
  loop each element
    ObserveHandler->>ObserveHandler: lookup elementId in combinedXpathMap
    alt elementId missing from xpath map
      ObserveHandler->>Log: level 0 "LLM returned ID ... no xpath keyed ..."
      ObserveHandler->>ObserveHandler: drop element
    else xpath found
      ObserveHandler->>ObserveHandler: return Action with selector=xpath=...
    end
  end
  ObserveHandler-->>User: Action[] (filtered)

Loading

[greptile-apps]

_{4 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Additional Comments (1)

packages/core/lib/v3/handlers/observeHandler.ts
Inconsistent log severity

dragAndDrop target element lookup failed is logged at level: 0 here, but the same failure mode is logged at level: 1 in packages/core/lib/v3/handlers/actHandler.ts:494-503. If downstream log filtering/alerting depends on level, this makes observe vs act inconsistent for the same error condition. Align the log level between the two handlers.

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

[cubic-dev-ai]

No issues found across 4 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

Architecture diagram

sequenceDiagram
    participant C as Client
    participant H as Act/Observe Handler
    participant S as Snapshot (captureHybridSnapshot)
    participant LLM as Inference (LLM)
    participant L as v3Logger

    Note over C, L: Runtime Flow for Act/Observe with XPath Validation

    C->>H: act() or observe()
    H->>S: captureHybridSnapshot()
    S-->>H: { combinedXpathMap, combinedTree }
    
    H->>LLM: Request inference (instruction + tree)
    LLM-->>H: Return element(s) with elementId

    rect rgb(240, 240, 240)
    Note over H, L: NEW: XPath Lookup & Validation Logic
    
    loop For each returned elementId
        H->>H: Lookup elementId in combinedXpathMap
        
        alt elementId NOT in map (Failure Path)
            H->>L: NEW: v3Logger(level: 0, category: "action/observation")
            Note right of L: Logs missing ID for debugging
            
            alt actHandler flow
                H-->>C: CHANGED: Return { success: false, message: "No action found" }
            else observeHandler flow
                H->>H: CHANGED: Filter element out of results
            end
            
        else elementId found (Happy Path)
            H->>H: Normalize/Trim XPath
            opt actHandler flow
                H->>H: Perform browser action (click, type, etc.)
                H-->>C: Return { success: true }
            end
        end
    end
    end

    opt observeHandler flow
        H-->>C: Return list of validated elements
    end

Loading