Silent-pass trap audit — `waitForHuman` used for visual verification

[couimet]

Audit every [assisted] integration test that uses waitForHuman (not waitForHumanVerdict) to verify a visible outcome. Those tests are subject to a silent-pass trap: the human can be looking at a broken UI and the test will still report green.

Background

There are two helpers in assistedTestHelper.ts:

• waitForHuman(tcId, action, consoleSteps?) — shows a notification, returns when the human clicks Cancel. Captures no judgment. Used when the test's assertion is on logs/state, not on what the human saw.
• waitForHumanVerdict(tcId, action, consoleSteps?) — shows a notification + PASS/FAIL status bar buttons, returns 'pass' | 'fail'. Captures judgment. Used when the test's assertion IS what the human saw.

The contract matters because many [assisted] tests describe themselves as "verify X appears in the UI" but use waitForHuman. The verification is left to the human's intent, not encoded in the assertion. Result: if the UI is broken, the human can click Cancel by reflex, and the test passes against a broken state.

How this was discovered

While diagnosing the issues/547 paste-pipeline regression (see related bug issue), claude-code-004 was reported "passing" by pnpm test:release:with-extensions --grep "claude-code-004" — but the human watched Claude Code receive nothing. The log-based assertions (ComposablePasteDestination.pasteLink "Pasted link", VscodeAdapter.pasteTextFromClipboard "Clipboard paste succeeded") confirmed the code-side ran; the human's eye confirmed nothing arrived. The test reported green because waitForHuman doesn't capture a verdict — clicking Cancel resolves the promise regardless of what was observed.

The newly-written claude-code-002 uses waitForHumanVerdict and FAILED with the exact same code-side state. The contract switch made the silent-pass trap visible.

Scope of the audit

For every [assisted] test in packages/rangelink-vscode-extension/src/__integration-tests__/suite/*.test.ts:

1. Identify whether the test's contract includes verifying a visible outcome (e.g., "the link appears in Claude Code chat", "the chat panel receives content", "the toast appears").
2. Check which helper it uses (waitForHuman vs waitForHumanVerdict).
3. Flag any test that:
   • Uses waitForHuman
   • Has instructions like "verify X appears" / "confirm X" / "the link should appear in Y"
   • Has no other UI assertion that would fail if the visible outcome is broken (log assertions don't count if they only prove code-side execution)

Likely affected (rough scan, not authoritative):

• claude-code-004 (confirmed silent-pass against the issues/547 regression)
• claude-code-005 (warm step — same waitForHuman pattern, same risk)
• clipboard-preservation-001/004/005/009 — instruction asks "verify the link should appear in terminal/Dummy AI" but only requires Cancel; the clipboard assertion is partial coverage at best
• Most [assisted] tests in customAiAssistants.test.ts that verify content delivery to Dummy AI tiers
• Likely core-send-commands-r-l-003 and similar send tests

A proper audit would enumerate every match with its file:line, the assertion text, and a verdict on whether to convert.

Conversion pattern

For each affected test:

// Before — silent-pass trap
await waitForHuman(
  'cc-004',
  'Cold paste: select lines, press Cmd+R Cmd+L, verify link appears in Claude Code, then Cancel',
  [/* steps */],
);
// log assertions only — pass even when human saw nothing

// After — verdict captured
const verdict = await waitForHumanVerdict(
  'cc-004',
  'Cold paste: press Cmd+R Cmd+L. Did the link appear in Claude Code chat?',
  ['Click PASS if the link appeared, FAIL otherwise'],
);
assert.strictEqual(verdict, 'pass', 'Human reported the link did not appear');
// log assertions still run as belt-and-suspenders

For tests where the human's eye is the ENTIRE assertion (no log path even partially covers it), the conversion is mandatory. For tests where the log path covers most of the contract but the human's eye is the final confirmation, the conversion is still strongly recommended — it adds the verdict capture without removing any log-based coverage.

What this does NOT change

• Tests that use waitForHuman for non-visual contracts (e.g., "press a chord, then logs assert the side effect") are fine as-is. The human's eye isn't being relied on.
• Tests that already use waitForHumanVerdict — already correct, no change needed.
• The number of [assisted] tests stays the same. This isn't about reducing assisted count; it's about making the assisted tests actually verify what they claim to verify.

Implementation suggestion

Don't fix all in one PR. Suggested phasing:

1. Phase 1 — Audit: full enumeration of affected tests in a tracking comment/issue, with per-test verdicts.
2. Phase 2 — High-stakes tests first: convert claude-code-004/005 and the rest of builtInAiAssistants.test.ts (highest-risk because they're the canary for issues/547).
3. Phase 3 — Per-file batches: clipboardPreservation.test.ts, customAiAssistants.test.ts, coreSendCommands.test.ts in separate PRs.
4. Phase 4 — Update TESTING.md § "Assisted mode" with the contract distinction: use waitForHuman only when there's a non-visual machine-verifiable assertion; use waitForHumanVerdict whenever the human's eye is part of the assertion.

Definition of done

• Every [assisted] test with a "verify X appears in the UI" contract uses waitForHumanVerdict and asserts on the verdict.
• TESTING.md documents the contract rule.
• A future regression like the issues/547 paste pipeline would be caught by the test rather than silent-passed.

Related

• The issues/547 paste-pipeline regression that exposed this trap — see sibling issue.
• The boring-setup-steps refactor (Boring-setup-steps refactor — 7 assisted tests force the human through automatable setup #558) — overlaps with 7 specific tests; those should be converted as part of Boring-setup-steps refactor — 7 assisted tests force the human through automatable setup #558 because they touch the same set.
• The closeQuickOpen survey (closeQuickOpen automation survey — 41 assisted tests can become fully automated #557) — orthogonal; closeQuickOpen tests don't use waitForHuman for visual verification, they use it (or no human at all) for picker dismissal.

Priority

Medium-high. Not blocking any specific work, but until this is addressed, future regressions like #559 will continue to silent-pass against any visual-verification test. This is the systemic fix that would have caught #559 before it merged.

[couimet]

Status refresh — what #560 is about, and what changed since it was filed

Posting this because the context fragmented across PRs and issues. Reading this back cold should let anyone (or a future agent) pick up where we left off.

What this issue is, in one paragraph

[assisted] tests use one of two helpers in assistedTestHelper.ts:

• waitForHuman(tcId, action, consoleSteps?) — pauses, returns when the human clicks Cancel. No verdict captured.
• waitForHumanVerdict(tcId, action, consoleSteps?) — pauses, returns 'pass' | 'fail' based on which status-bar button the human clicks. Verdict captured.

Many existing [assisted] tests describe themselves as "verify X appears in the UI" but use waitForHuman. That means the human can be staring at a broken UI, click Cancel out of habit, and the test reports green. The log-based assertions in those tests only prove the code-side code path ran — not that anything reached the user-visible surface. This is the silent-pass trap.

This was discovered when claude-code-004 (existing test, uses waitForHuman) reported green on a branch where the human watched Claude Code receive nothing. The newly-written claude-code-002 (uses waitForHumanVerdict) failed against the exact same code-side state — the contract switch made the trap visible. The underlying root cause of the empty Claude Code chat is filed separately as #559 (a clipboard-write regression introduced by the issues/547 refactor — paste pipeline logs success while the AI webview gets nothing).

What changed since this issue was filed (2026-05-13 ~03:00 UTC)

Two PRs and one branch shifted the surrounding picture:

1. PR [issues/557] Automate 33 assisted integration tests via closeQuickOpen dismissal #561 (issues/557) merged — closeQuickOpen automation. 33 previously-[assisted] tests are now automated: true. None of them were silent-pass-trap candidates (they all just dismissed pickers; no visual verification was involved), so this PR did NOT reduce Silent-pass trap audit — waitForHuman used for visual verification #560's blast radius. But it did concentrate the remaining [assisted] tests onto the visual-verification subset — which is exactly the bucket Silent-pass trap audit — waitForHuman used for visual verification #560 targets.

2. issues/562 merged — new CI job test-with-extensions that runs pnpm test:release:with-extensions --no-assisted on ubuntu-latest under Xvfb on every PR. This is what introduces the infrastructure-level silent-pass concern described below.

3. Current QA tally (post-rebase of issues/483 onto origin/issues/562): 121 automated / 115 assisted / 12 false (of 248 TCs). Was 86 / 147 / 16 when Silent-pass trap audit — waitForHuman used for visual verification #560 was filed.

The green-because-blind aspect

The new CI job runs --no-assisted, which means it executes only the 121 automated: true tests, skipping all 115 [assisted] tests (including the silent-pass-trap candidates).

So today, the CI is green when:

• All 121 automated tests pass — which is what CI reports.

The CI is green regardless of whether:

• Any paste-to-AI delivery actually works (no automated: true test verifies actual paste content arrival in any AI webview surface — claude-code-001 and github-copilot-chat-001 only check picker contents).
• The 115 visual-verification tests would pass when a human actually runs them.

Concrete example: the #559 paste-pipeline regression is sitting in origin/issues/562 HEAD right now. CI is green. Running pnpm test:release:with-extensions --grep "claude-code-002" locally (a [assisted] test that uses waitForHumanVerdict) FAILS because the human sees nothing in Claude Code. CI doesn't fail because CI skips that test. The claude-code-004 test (existing, uses waitForHuman) would also fail if it used waitForHumanVerdict — but it doesn't, so when a human runs it locally it ALSO reports green, even with the same broken Claude Code chat. Silent-pass trap at both layers: CI skips the verdict tests, and the waitForHuman tests don't capture verdicts.

This is why the CI status badge is misleading: green-because-blind, not green-because-everything-works.

What it would take to make CI catch #559-class regressions

Fundamentally: the VS Code extension host has no API to read the content of a webview from a test. The Claude Code chat input, the Cursor chat panel, the Copilot Chat input — all are webviews. The paste action lands content there based on focus + clipboard timing, and we cannot inspect what arrived from a test.

Workarounds, in order of cost/feasibility:

1. Extend test-fixtures/dummy-ai-extension/ with an inspectable paste target. The fixture currently provides Dummy AI tiers — its paste target could be made readable (e.g., via a commandId that returns the current text). This would give us at least one automated: true paste-delivery test, covering the bind→send→deliver code path. Wouldn't catch built-in AI webview bugs but WOULD catch most regressions in the paste pipeline. Probably the highest-ROI option.

2. Convert paste-to-terminal [assisted] tests (clipboard-preservation-001/005, etc.). Terminal content IS inspectable via capturingPtyHelpers (assertTerminalBufferContains, assertTerminalBufferEquals). Several [assisted] tests today paste to terminals and have a human verify visually — but the verification could be automated. Estimated 5-8 conversions; worth its own issue. Notably these are NOT silent-pass-trap candidates (they're waitForHuman with logged assertions for the terminal target), so this is a different lever than Silent-pass trap audit — waitForHuman used for visual verification #560 per se, but it overlaps in spirit.

3. VS Code adds a webview-content-read API for test hosts. Out of our control.

Without one of the above, #559-style regressions stay in the manual-QA-only-catches-them bucket. CI catches detection regressions, picker regressions, status-bar regressions, clipboard preservation regressions — but not webview delivery.

What #560 still needs to do

The audit + conversion work is still scoped as written in this issue:

1. Enumerate every [assisted] test that uses waitForHuman for visual verification.
2. Convert to waitForHumanVerdict so the verdict is captured.
3. Update TESTING.md § "Assisted mode" with the contract distinction.

That's still the right work. What's new since filing:

• The pool to audit is now slightly different (PR [issues/557] Automate 33 assisted integration tests via closeQuickOpen dismissal #561 converted 33 tests, none of which used waitForHuman for visual verification — so the pool is unchanged in absolute terms but is now a larger fraction of the remaining [assisted] set).
• Once converted, these tests will FAIL on any branch carrying the issues/547 regression — paste-to-AI-assistant logs success but webview receives nothing #559 regression. That's the intended behavior — silent-pass trap closed — but it means converting them is a forcing function for issues/547 regression — paste-to-AI-assistant logs success but webview receives nothing #559 being fixed first. Or the conversion can land with xit/skip for the specific cc-004/cc-005/etc. tests until issues/547 regression — paste-to-AI-assistant logs success but webview receives nothing #559 is resolved.

Where the related artifacts live

• The full reasoning that uncovered the silent-pass trap: .claude-work/issues/483/notes/20260513-000115-issues-547-paste-regression.txt (local, contains the diagnostic narrative).
• The post-rebase reassessment with the green-because-blind framing: .claude-work/issues/483/notes/20260513-150843-post-rebase-reassessment.txt (local).
• Sibling issue issues/547 regression — paste-to-AI-assistant logs success but webview receives nothing #559 — the actual regression that silent-passed.
• Sibling issue Boring-setup-steps refactor — 7 assisted tests force the human through automatable setup #558 — boring-setup audit; overlapped with 7 specific tests that should also flip to waitForHumanVerdict (Implementer's checklist step 4 there).
• Sibling closed issue closeQuickOpen automation survey — 41 assisted tests can become fully automated #557 / PR [issues/557] Automate 33 assisted integration tests via closeQuickOpen dismissal #561 — closeQuickOpen automation. Did not address silent-pass directly; converted picker-dismissal tests which were never silent-pass candidates.

Suggested ordering when picking this up

1. Read issues/547 regression — paste-to-AI-assistant logs success but webview receives nothing #559 first to understand the concrete regression that motivated this audit.
2. Run the full audit per the Implementation Suggestion phasing in this issue's body.
3. Decide: do conversions land before issues/547 regression — paste-to-AI-assistant logs success but webview receives nothing #559 is fixed (with xit on the visual-paste-delivery tests), or after (with the tests truly green)? Either is defensible; ordering depends on appetite for red CI on unrelated branches.
4. Consider whether to do Lever 1 / Lever 2 above (Dummy AI inspectable paste, or terminal-target [assisted] conversions) as a sibling effort — both address the green-because-blind problem at the infrastructure level, complementary to this issue's waitForHuman→waitForHumanVerdict work.

[couimet]

Follow-up: precision on "115" and where more conversions are realistic

Re-reading the previous comment after a sanity check: 115 is accurate (grep confirms 115 [assisted] tests across the suite, matching QA YAML), but I want to be sharper about what the closeQuickOpen lever has captured vs. what's still left.

The closeQuickOpen lever is exhausted

PR #561 picked up the eligible tests — terminal-picker (12), file-picker (8), status-bar-menu (4), and similar. The remaining 115 are [assisted] for reasons closeQuickOpen can't address:

Category	Approx count	Why structurally [assisted]
Dialog button clicks (dirtyBufferWarning)	18	No API for "click Save & Generate" / "Generate Anyway" buttons
Right-click context menus (4 files)	32	Menu selection IS the test scope; replacing with executeCommand makes the test name a lie
Picker selection drives outcome (bind-to-destination, etc.)	10	Human's pick from the picker is what the test asserts on
Real keybinding chord tests	~7	Test host can't simulate chord dispatch
Input box typed input (goToRangeLink)	7	Human types a link string; that's the test contract
Visual waitForHumanVerdict	~12	Human's eye is the assertion — tooltip presence, menu visibility, etc.
Custom AI / send commands mix	~12	Selection + visual paste verification
Other (file-picker "More files...", etc.)	~17	Various

Where more conversions ARE realistic — and would close the green-because-blind gap

The previous comment mentioned "convert paste-to-terminal [assisted] tests" as a workaround for the inspectable-paste-target problem, but framed it as orthogonal to this audit. It's actually the same lever: tests that currently use waitForHuman for visual verification of paste delivery, where the surface IS programmatically inspectable.

Estimated 5-10 more conversions in this bucket:

• clipboard-preservation-001 / -005 — paste to terminal (capturingPtyHelpers already exists)
• core-send-commands-r-l-002 — R-L to bound text editor (document content readable)
• core-send-commands-r-l-003 — R-L to Dummy AI textarea (inspectable if the fixture is extended with a getter)
• custom-ai-assistant-010 / -011 / -014 / -017 — Tier 1 textarea inserts (same)

These conversions WOULD reduce the green-because-blind gap because they verify actual paste delivery — just to a non-webview surface. Each converted test is one more thing CI catches on every PR. They don't help with #559 directly (that's a webview-specific race), but they raise the floor on what "green CI" actually means.

Realistic floor on [assisted]

If we exploited all named levers:

• 115 today
• −5 to −10 from paste-to-inspectable-surface conversions (the lever above)
• −1 from focusFail Dummy AI fixture extension (clipboard-preservation-010 is a false, not assisted, so actually no change here for the assisted count)
• −1 from bind-to-destination-011 direct-command conversion (per the comment on closeQuickOpen automation survey — 41 assisted tests can become fully automated #557)
• −1 to −3 from bind-007/008/009 partial conversions (mixed effect — picker step automatable, dialog stays manual; test stays [assisted] but with reduced friction)

Realistic floor: ~95-100 [assisted] tests. The remaining ~95-100 is genuinely manual — dialogs, context menus, visual chord verification, typed input, true visual-only verdicts. The CI green-because-blind window narrows from 115 to ~95-100, but doesn't close. Closing it would require either:

• A VS Code test-host API to read webview content (out of our control), or
• Extending test-fixtures/dummy-ai-extension/ to be a fully inspectable AI assistant target, then re-pointing tests through it (significant fixture work).

What this means for the audit scoped in this issue

The waitForHuman → waitForHumanVerdict conversion described in this issue's body is still the right work. The clarification: of the 115 tests, the ones eligible for this audit (those using waitForHuman for visual verification with no machine-verifiable signal) are a subset — probably around 30-50. The rest already use waitForHumanVerdict correctly, or use waitForHuman for non-visual contracts (where the human's role is to dismiss something or perform a setup step, and the assertion is on logs).

A full enumeration is still part of Phase 1 of the original plan. Estimated final post-audit state:

• ~95-100 [assisted] — structural manual
• ~30-50 of those flipped from waitForHuman to waitForHumanVerdict (this issue's scope)
• ~5-10 dropped entirely via paste-to-inspectable-surface conversions (separate lever)

These two levers compose cleanly — they don't conflict and can be done in either order. The paste-to-inspectable-surface lever is probably the highest-ROI single piece of work left for [assisted] reduction.