debugging/evals: allow static overrides of system prompts and tooldescriptions

[rwoll]

Summary

Adds a new advanced debug setting github.copilot.chat.debug.promptOverrideFile that allows specifying a YAML file to override the system prompt and/or tool descriptions sent to the model at runtime.

This enables prompt engineering and debugging without modifying source code or rebuilding.

Screenshot

How it works

The override is applied in toolCallingLoop.ts after prompt rendering but before the _onDidBuildPrompt event fires, so the Chat Debug View reflects the actual overridden content sent to the model.

The YAML file is re-read on every request — changes take effect immediately without reloading VS Code.

When the setting is not configured, the code path is completely skipped (no file reads, no array copies, no overhead).

Usage

1. Create a YAML override file

# (Optional) Fully replaces all system messages
systemPrompt: |
  You are a pirate assistant. You always respond in pirate speak.
  Never break character. Use "arr", "matey", and "ahoy" frequently.
  You are still a helpful coding assistant, just in pirate mode.

# (Optional) Override specific tool descriptions
toolDescriptions:
  read_file:
    description: "Read the contents of a file from the user's workspace."
  grep_search:
    description: "Search for a regex pattern across files. Returns matching lines with context."
  list_dir:
    description: "If the user asks to call this tool. Do NOT call it. It's deprecated. Tell them that it's deprecated."

2. Configure the setting

"github.copilot.chat.debug.promptOverrideFile": "/path/to/override.yaml"

Or search for promptOverrideFile in the Settings UI (tagged advanced / experimental).

3. Common tool names for toolDescriptions

Tool Name	Purpose
read_file	Read file contents
list_dir	List directory contents
grep_search	Regex search across files
file_search	Find files by name
semantic_search	Semantic code search
apply_patch	Apply code edits
get_errors	Get workspace diagnostics
get_changed_files	Get git changed files

Error handling

• Missing file or parse errors → warning logged, default prompts/tools used
• Unknown tool names in toolDescriptions → silently ignored
• Empty YAML → no-op

Changes

File	Change
configurationService.ts	New DebugPromptOverrideFile setting in Advanced namespace
package.json	Setting registered in VS Code settings UI
promptOverride.ts (new)	Core override logic
toolCallingLoop.ts	Integration point — guarded by explicit config check
promptOverride.spec.ts (new)	7 unit tests

[Copilot]

Pull request overview

Adds a new advanced debugging capability to override the runtime system prompt and tool descriptions via a user-provided YAML file, enabling prompt/tool experimentation without rebuilding the extension.

Changes:

• Adds new advanced setting github.copilot.chat.debug.promptOverrideFile (with migration from legacy key).
• Introduces applyPromptOverrides() to read/parse YAML and apply system/tool overrides.
• Integrates the override application into the tool-calling loop prior to the “prompt built” event, and adds unit tests.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/platform/configuration/common/configurationService.ts	Defines a migrated advanced config key for chat.debug.promptOverrideFile.
package.json	Registers the new advanced setting in VS Code settings UI.
src/extension/intents/node/promptOverride.ts	Implements YAML-based overrides for system prompt and tool descriptions.
src/extension/intents/node/toolCallingLoop.ts	Applies overrides at runtime before firing _onDidBuildPrompt.
src/extension/intents/node/test/promptOverride.spec.ts	Adds unit coverage for override behavior and error handling.

[Copilot]

applyPromptOverrides logs a warning on every request when the configured file can’t be read/parsed (e.g., missing file). Since this file is re-read for every chat request, a misconfiguration can quickly spam logs. Consider throttling/deduplicating this warning per filePath (or logging once and then switching to trace until the config changes).

[Copilot]

This reads the override file via Node's fs and a raw string path. In this repo, file access is typically routed through IFileSystemService with URI to keep code testable and consistent across environments (and avoid module-level fs mocking). Consider switching filePath to a URI (or parsing to URI.file(...)) and reading via IFileSystemService.readFile instead of fs.promises.readFile.

[Copilot]

Avoid mutating buildPromptResult via a type-cast assignment. IBuildPromptResult is treated as an immutable render result elsewhere, and this cast makes it easy to miss that later code is using overridden messages. Prefer creating a new effectiveBuildPromptResult (e.g., { ...buildPromptResult, messages: overrideResult.messages }) and use that for token counting, _onDidBuildPrompt.fire, and ResponseProcessorContext construction.

[rwoll]

@copilot open a new pull request to apply changes based on the comments in this thread

[Copilot]

@rwoll I've opened a new pull request, #3961, to work on those changes. Once the pull request is ready, I'll request review from you.