`closeQuickOpen` automation survey — 41 assisted tests can become fully automated

[couimet]

Spawned from the claude-code-001 implementation: vscode.commands.executeCommand('workbench.action.closeQuickOpen') reliably dismisses an open QuickPick from inside a test, with no human in the loop. The fundamental test-host constraint described in TESTING.md § "QuickPick limitation" — "no API to interact with QuickPick UI" — turns out to be about selecting, not dismissing. Dismissal has always been possible; we just weren't using it.

The proven pattern

From builtInAiAssistants.test.ts claude-code-001 (passed in --with-extensions run on 2026-05-12):

const pickerPromise = vscode.commands.executeCommand(CMD_BIND_TO_DESTINATION);
await settle();
await vscode.commands.executeCommand('workbench.action.closeQuickOpen');
await pickerPromise;
await settle();

const lines = logCapture.getLinesSince('before-cc-001');
const items = extractQuickPickItemsLogged(lines);
// ... assert on items array ...

Fire the command without awaiting → let the showQuickPick log emit → dismiss programmatically → await the original promise (which resolves with outcome: 'cancelled') → inspect the captured items. End-to-end runtime in the live run: 1012ms. No human prompt, no notification, no Cancel button.

Why this is a big deal

Of the 147 currently [assisted] integration tests, 41 only need a human because the existing test asks the human to dismiss the picker after it opens. The instruction text is literally "Press Cmd+R Cmd+D, then Escape" — and the assertions read picker contents from extractQuickPickItemsLogged(lines), not from any selection outcome. The human's role in those tests has been entirely ceremonial. They could have been automated from the day extractQuickPickItemsLogged shipped (#481 / PR #484, 2026-03-21) — we just didn't see it.

Current YAML distribution (qa-test-cases-v1.1.0.yaml):

• automated: true — 86 (35%)
• automated: assisted — 147 (59%)
• automated: false — 16 (6%)

If all 41 Bucket A tests are converted:

• automated: true — 127 (51%)
• automated: assisted — 106 (43%)
• automated: false — 16 (6%)

The "true" line jumps from 35% to 51%, and assisted-mode QA cycle time drops by roughly 40 / 147 ≈ 28% (assuming the median assisted test takes the same human time, which is conservative — picker-dismissal tests are among the fastest, so the actual time saving is larger).

The 41 convertible tests, by file

Highest-density files first. Per-file conversion is mechanical and orthogonal across files.

terminalPicker.test.ts — 12 of 14 convertible

• terminal-picker-001, 002, 003, 004, 005, 006, 007, 008, 011, 012, 013
• bind-to-destination-013 (lives in this file)
• Not convertible: terminal-picker-009, 010 (require "More terminals..." selection)

filePicker.test.ts — 8 of 12 convertible

• file-picker-001, 002, 003, 004, 005, 009, 011, 012
• Not convertible: file-picker-006, 007, 008, 010 (require "More files..." selection)

statusBarMenu.test.ts — 4 of 8 convertible

• status-bar-menu-002, 003, 005, 006
• Not convertible: status-bar-menu-001, 009 (waitForHumanVerdict), 007, 008 (menu selections)

coreSendCommands.test.ts — 4 of 9 convertible

• core-send-commands-r-l-005, core-send-commands-r-p-001, core-send-commands-r-v-001
• send-terminal-selection-004 — convertible IF the test can pre-set the terminal selection programmatically; otherwise it needs the human to select terminal text (verify before converting)
• Not convertible: tests with real Cmd+R Cmd+L / R-V / R-C dispatch

customAiAssistants.test.ts — 2 of 8 convertible

• custom-ai-assistant-003, 007
• Not convertible: the 6 tests that involve selecting a Dummy AI tier and then sending

bindToDestination.test.ts — 2 of 11 convertible

• bind-to-destination-010, bind-to-destination-013 (bind-013 also counted under terminalPicker file above — same TC, just located there)
• The other 9 all require human selection of a destination

editorBindingValidation.test.ts — 1 of 5 convertible

• editor-binding-validation-004 (.png exclusion check via dismissed picker)

clipboardPreservation.test.ts — 1 of 4 convertible

• clipboard-preservation-009 (picker opens because no destination bound; assertion is clipboard unchanged)

goToRangeLink.test.ts — 1 of 8 convertible

• go-to-link-001 (input box dismissed; assertion is on assertInputBoxLogged)

builtInAiAssistants.test.ts — already done

• claude-code-001 (this is what motivated the survey)

Files with 0 convertible tests

• contextMenuTerminal.test.ts (12 tests, all menu selections)
• contextMenuEditorContent.test.ts (11 tests, all menu selections or verdict)
• contextMenuExplorer.test.ts (5 tests, all menu selections or verdict)
• contextMenuEditorTab.test.ts (4 tests, all menu selections)
• dirtyBufferWarning.test.ts (18 tests, all dialog-button interactions or real keybindings)
• releaseNotifier.test.ts (3 tests, all notification button clicks)
• sendFilePath.test.ts (7 tests, all require real keybinding chord dispatch)
• textEditorDestination.test.ts (3 tests, all require Cmd+R Cmd+L / R-V dispatch)
• linkGeneration.test.ts (2 tests, both waitForHumanVerdict)
• platformKeybindings.test.ts (1 test, waitForHumanVerdict)

What disqualifies a test from Bucket A

In rough order of frequency:

1. Human-driven selection from the picker — the test outcome depends on which item the human chooses (bind-to-destination-004 through 012 except 010). Could in principle be replaced by calling the underlying command (e.g., bindToTerminalHere), but that bypasses the picker code path the TC names — different scope of work, not a closeQuickOpen win.
2. Menu / dialog button click — context-menu TCs, dirty-buffer dialog TCs, release-notifier TCs. The VS Code test host has no API to dispatch a context-menu command or click a notification button without the human.
3. Real keybinding chord dispatch — claude-code-002/004/005, send-file-path TCs, sendTerminalSelection TCs. Cmd+R Cmd+L can be partially simulated via executeCommand but the TC scope often requires the chord to actually fire (and visual confirmation that focus + paste worked end-to-end). Out of scope for this lever.
4. waitForHumanVerdict Pass/Fail judgment — 12 tests. The human's eyeball is the assertion. Examples: tooltip presence, menu item visibility, version-info readability.
5. "More terminals..." / "More files..." picker navigation — these are technically picker selections (the human clicks the overflow item) and currently exist to verify the secondary picker contents. Convertible only by extending the closeQuickOpen pattern to handle two pickers in sequence — possible but not the simple case.

Validation status

The pattern is proven on the destination picker (rangelink.bindToDestination → R-D). It is not yet validated on:

• The R-M status bar menu (rangelink.showRangeLinkMenu) — but it uses the same VscodeAdapter.showQuickPick log path so highly likely to work.
• The R-G go-to-link input box (vscode.window.showInputBox) — input boxes are dismissed by workbench.action.closeQuickOpen too in VS Code, but worth confirming on the first conversion.
• The R-V terminal selection picker (when no destination is bound — opens R-D as fallback). Same code path as R-D.
• The R-L / R-P / R-V "no destination bound" fallback pickers — same R-D code path under the hood, same expectation.

Suggested first conversions to validate the pattern across surface area:

• terminal-picker-001 (R-D, terminal-only) — confirms the pattern on a non-AI flavor of R-D
• status-bar-menu-002 (R-M menu) — confirms the pattern on the secondary menu picker code path
• go-to-link-001 (R-G input box) — confirms dismissal works on showInputBox, not just showQuickPick
• core-send-commands-r-l-005 (R-L → fallback R-D) — confirms when picker is opened as side-effect rather than direct command

If those four all pass, the remaining 37 conversions are mechanical and could be batched into one or two PRs.

Caveats worth flagging before mass conversion

1. Timing: await settle() between the command-fire and the closeQuickOpen call is doing real work — it lets the picker render and emit its log. If settle() is too short on a loaded CI machine, the log won't be captured and the test will fail with "Expected showQuickPick log entry". Worth checking SETTLE_MS value and considering an explicit poll for the log entry rather than a fixed delay.
2. Test name should drop [assisted]: once converted, the title must lose the [assisted] prefix so that --no-assisted / --automated runs include it. The --automated mode in test-release-run.sh filters on the [assisted] substring.
3. No banner side effect: printAssistedBanner() in suiteSetup is fine to keep — it doesn't affect non-assisted tests, just prints unused guidance. Not worth churning per-file.
4. QA YAML flips: each conversion needs a corresponding automated: assisted → automated: true flip in qa-test-cases-v1.1.0.yaml. The QA validator (scripts/validate-qa-coverage.sh) checks the test exists; it doesn't check the [assisted] tag matches, so a stale automated: assisted with a non-[assisted] test would still pass validation. Flipping the YAML is purely documentation, not enforcement — but it matters for human-driven QA planning.
5. Per-test verification cost: each conversion needs a pnpm test:release:with-extensions --grep "<id>" run to verify (per release-test-requirement in CLAUDE.md). At ~30s of compile + ~3s of test, that's ~33s per test × 41 tests ≈ 23 min of cumulative runner time if batched.

Recommended path forward

1. Validate the pattern on 4 diverse surfaces (terminal-picker-001, status-bar-menu-002, go-to-link-001, core-send-commands-r-l-005). If any of these has a quirk (e.g., input-box dismissal differs), capture the divergence before fanning out.
2. Batch the remaining 37 conversions by file — one PR per file keeps diffs reviewable. Highest-leverage file first: terminalPicker.test.ts (12 conversions, single suite).
3. Update TESTING.md § "QuickPick limitation" with the new sharper framing: dismissal is fine; only selection needs the human. The cross-reference from CLAUDE.md QA005 already points there, so updating the canonical doc is enough.
4. Optional second pass: revisit Bucket B for tests where the "selection" is structurally a single deterministic item (e.g., bind-to-destination-006 — "select any AI assistant"). If a test only cares that some AI is bound, it could be auto-bound via the appropriate CMD_BIND_TO_* command. Lower-priority — those tests pass the picker route which is part of what they're documenting.

Bottom line

We just discovered a single-line API call (workbench.action.closeQuickOpen) that converts 41 assisted tests to fully automated — about 28% of the assisted test pool. Conversion is mechanical, the pattern is proven, and the projected outcome is automated: true coverage jumping from 35% to 51%. This is the single highest-ROI test-infrastructure lever in sight, and it's strictly more valuable than continuing #483's β-wave-1 because it touches existing passing tests rather than writing new ones against unknown extension behavior.

Pointers

• Proven pattern: packages/rangelink-vscode-extension/src/__integration-tests__/suite/builtInAiAssistants.test.ts — claude-code-001 test
• Log helper: extractQuickPickItemsLogged in helpers/logBasedUiAssertions.ts
• Test runner: pnpm test:release:with-extensions --grep "<test-id>"
• QA YAML: packages/rangelink-vscode-extension/qa/qa-test-cases-v1.1.0.yaml
• TESTING doc to update: packages/rangelink-vscode-extension/TESTING.md § "QuickPick limitation"

[couimet]

Follow-up: complementary lever via direct-command invocation

Surveyed the complementary category to closeQuickOpen: tests where the human currently selects from a QuickPick to trigger a bind/send, but the same outcome could be reached by invoking a more specific command directly (e.g., CMD_BIND_TO_TERMINAL_HERE instead of "Cmd+R Cmd+D, select Terminal X").

Result: much smaller pool than the 41 closeQuickOpen tests. Conservative findings below.

Clean win — 1 test

• bind-to-destination-011 ("Cmd+R Cmd+D → select an AI → Cmd+R Cmd+D → select same AI") — replace both selections with CMD_BIND_TO_CLAUDE_CODE. Asserts on "already bound" info toast, no dialog. Goes from automated: assisted → automated: true.

Split candidates — 2 tests (coverage GAIN, not conversion)

• bind-to-destination-004 and bind-to-destination-005 currently assert on BOTH picker badges/states AND the bind toast that follows selection. They could be split into:
  • Test A (Bucket A — closeQuickOpen): open picker, dismiss, verify item states
  • Test B (new, fully automated): focus the target, CMD_BIND_TO_TERMINAL_HERE / CMD_BIND_TO_TEXT_EDITOR_HERE, assert bind toast
• This adds fully-automated coverage rather than converting an existing assisted test. Original test could stay assisted (with closeQuickOpen automation of the picker-inspection portion). Net effect: more automated coverage, same assisted count.

Partial conversions — 3 tests (friction reduction only)

• bind-to-destination-007/008/009 involve a confirmation dialog AFTER the selection. Picker selection is replaceable with a direct command, but the dialog button click (Yes, replace / No, keep current binding) or Escape requires a human — no VS Code API for dialog button dispatch. Tests stay [assisted]; instruction shortens by ~40%.

Blocked behind missing platform code — 6 tests

All custom-AI binding tests: custom-ai-assistant-010/011/012/013/014 and clipboard-preservation-004. There is no CMD_BIND_TO_CUSTOM_AI_BY_NAME — built-in AI binding has dedicated commands but custom-AI bindings flow through the picker by design.

To unlock these we'd need a new command:

export const CMD_BIND_TO_CUSTOM_AI_BY_NAME = 'rangelink.bindToCustomAiByName';

vscode.commands.registerCommand(CMD_BIND_TO_CUSTOM_AI_BY_NAME, async (displayName: string) => {
  // resolve custom AI definition by displayName, bind via PasteDestinationManager
});

That's a real platform addition with its own design discussion (parameter shape, name resolution, error handling, command visibility in palette). Worth considering as a separate issue if/when the test-automation ROI justifies it.

Tally

Category	Tests	Effect
Clean assisted→automated conversion	1	bind-011
Split into picker-inspection + direct-bind	2	bind-004, bind-005 — coverage gain, not conversion
Partial (picker auto, dialog stays assisted)	3	bind-007, bind-008, bind-009
Blocked on new command	6	all custom AI bind paths

Trade-off worth saying out loud

When we replace picker → selection → bind with a direct command, the test no longer integration-exercises the DestinationPicker.handleQuickPickSelection → PasteDestinationManager.bind* code path. That path becomes covered only by:

• The Bucket A (closeQuickOpen) tests that inspect what the picker SHOWS but never SELECT.
• Unit tests of handleQuickPickSelection.

Probably an acceptable trade-off for the bind path (it's well-unit-tested), but worth flagging per-test when implementing.

Recommended sequencing

Don't pull this in as part of the closeQuickOpen conversion. Land the 4 validation conversions from this issue first, extract helpers, then revisit:

1. bind-to-destination-011 — quick win, lands as a one-off after the closeQuickOpen helpers exist
2. Split decision on bind-004 / bind-005 — discuss whether the coverage gain justifies splitting (or whether they stay as-is once their picker inspection is closeQuickOpen-automated)
3. bind-007/008/009 partial automation — separate small PR, friction reduction only
4. Custom-AI command — separate issue, platform addition decision

Not pursuing custom-AI tests as part of THIS issue's scope.