feat(playwright): page perception for AI agents — human.outline() + human_outline MCP tool#47
Merged
Conversation
… MCP tool Gives AI agents a compact accessibility-tree outline of the page (ARIA role + accessible name, as YAML — Playwright's ariaSnapshot) so they can see what's actionable and pick role/accessible-name selectors without parsing HTML or burning tokens on a screenshot. - human.outline(target?) on @humanjs/playwright (inspection getter, no plugin events; scopes to a region when given a target). - human_outline MCP inspection tool mirroring it. - Documented in the skill's selector-strategy guide. - Bumps the playwright peer floor 1.40 -> 1.49 (ariaSnapshot's minimum).
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…iaSnapshot requirement)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
We've got the action half of the agent loop solid (click/type/form/etc.). This adds the perception half: a compact, structured view of what's on the page so an agent can decide what to act on and how to select it — without dumping raw HTML or spending tokens reasoning over a screenshot.
human.outline()returns the page's accessibility-tree outline (Playwright'sariaSnapshot): every interactive element and landmark by ARIA role + accessible name, as compact YAML —Those names map directly to the
getByRole/ accessible-name selectors HumanJS already favors, so it doubles as a selector finder. Pass atargetto scope it to a region.Surface
@humanjs/playwright:human.outline(target?)— inspection getter (no plugin events; sibling topageText()/content()).@humanjs/mcp:human_outlineinspection tool, alongsidehuman_page_text/human_get_html. Description steers agents to prefer it over HTML for "what can I click/fill".@humanjs/skill: added to the selector-strategy guide.Naming
Called it
outline(notsnapshot) deliberately —human.snapshot()/human_snapshotsits one letter from the existingscreenshotand would be easy for an agent to confuse.Compatibility
ariaSnapshotlanded in Playwright 1.49, so@humanjs/playwright'splaywrightpeer floor moves>=1.40.0→>=1.49.0. Dev/CI already run 1.60.Tests
3 colocated tests: default body outline, region scoping, and "no plugin events fire" (inspection-only). Gate green: lint · typecheck 9/9 · tests 8/8 · check:exports (publint) 11/11.
Changeset
minorfor@humanjs/playwright+@humanjs/mcp,patchfor@humanjs/skill.