exploration: config-driven model profiles with inheritance

docs/prompts.md

@@ -17,45 +17,83 @@
 
 ## Overview
 
-vscode-copilot-chat uses a **prompt registry system** to map AI models to their optimal prompt structures. Each model provider can have customized prompts that leverage provider-specific strengths.
+vscode-copilot-chat uses two complementary systems to configure model behavior:
 
-### How the Registry Works
+1. **Model Profiles** ([`modelProfiles.ts`](../src/platform/endpoint/common/modelProfiles.ts)) — a config-driven registry that controls model **capabilities**: which edit tools a model uses, where instructions are placed, image support, notebook format, verbosity, etc. Adding a new model that behaves like an existing archetype is a one-line change here.
+
+2. **Prompt Registry** ([`promptRegistry.ts`](../src/extension/prompts/node/agent/promptRegistry.ts)) — maps models to their **TSX system prompt** components. Only needed when a model requires a genuinely different prompt (different wording, structure, or instructions).
+
+### When to use which
+
+| Scenario | What to change |
+|----------|---------------|
+| New model behaves like an existing archetype | Add one entry to `MODEL_PROFILES` in `modelProfiles.ts` |
+| New model needs a capability tweak (e.g. different edit strategy) | Add one entry to `MODEL_PROFILES` with the override |
+| New model needs a custom system prompt | Add a `MODEL_PROFILES` entry **and** create a new TSX prompt + resolver |
+| Quick eval of a new model (zero code changes) | Set `github.copilot.chat.advanced.models.profiles` in `settings.json` |
+
+### How Model Profiles Work
+
+The [`MODEL_PROFILES`](../src/platform/endpoint/common/modelProfiles.ts) table uses **inheritance** (`extends`) so models only override what's different from their base archetype:
+
+```typescript
+// Base archetype — never matched directly
+'_anthropic': {
+    editStrategy: 'multi-replace-string',
+    imageSupport: { urls: true, mcpResults: false },
+    // ...
+},
+
+// Concrete model — inherits everything, overrides instruction placement
+'claude-3.5-sonnet': {
+    extends: '_anthropic',
+    instructionPlacement: 'user-message',
+},
+```
+
+Profile resolution uses **longest-prefix-first** matching against `model.family`, so `gpt-5.1-codex` matches before `gpt-5.1` which matches before `gpt-5` which matches before `gpt`.
+
+The capability functions in [`chatModelCapabilities.ts`](../src/platform/endpoint/common/chatModelCapabilities.ts) (e.g. `modelSupportsApplyPatch()`, `modelNeedsStrongReplaceStringHint()`) are thin wrappers over the profile system. All existing callsites use these functions unchanged.
+
+### BYOK Models
+
+BYOK models get the right profile automatically as long as their model ID starts with a recognized prefix (`claude`, `gpt`, `gemini`, `grok-code`, etc.). Unknown model IDs fall through to the default profile (insert-edit-only, standard settings) — which is the safe behavior. And with the new settings override, users can explicitly map any unknown model to an archetype via `github.copilot.chat.advanced.models.profiles` without rebuilding.
+
+### How the Prompt Registry Works
 
 The [`PromptRegistry`](../src/extension/prompts/node/agent/promptRegistry.ts) matches models to prompts using:
 1. **Custom matchers**: `matchesModel()` functions for complex logic
 2. **Family prefixes**: Simple string matching on model family names
 
-A single prompt resolver can return **different prompts for different models** within the same provider family. For example, you might want to use one prompt for `gpt-5` and a different optimized prompt for `gpt-5-codex`. The resolver's `resolvePrompt()` method receives the endpoint information (including the model name) and can use conditional logic to return the appropriate prompt class:
+A single prompt resolver can return **different prompts for different models** within the same provider family. For example, you might want to use one prompt for `gpt-5` and a different optimized prompt for `gpt-5-codex`. The resolver's `resolveSystemPrompt()` method receives the endpoint information (including the model name) and can use conditional logic to return the appropriate prompt class:
 
 ```typescript
 class MyProviderPromptResolver implements IAgentPrompt {
-	static readonly familyPrefixes = ['my-model'];
-
-	resolvePrompt(endpoint: IChatEndpoint): PromptConstructor | undefined {
-		// Different prompts for different model versions
-		if (endpoint.model?.startsWith('my-model-1')) {
-			return MyModel1Prompt;  // Optimized for 1 variant
-		}
-		if (endpoint.model?.startsWith('my-model-4')) {
-			return MyModel4Prompt;   // Optimized for standard v4
-		}
-		return MyDefaultPrompt;      // Fallback for other models
-	}
+    static readonly familyPrefixes = ['my-model'];
+
+    resolveSystemPrompt(endpoint: IChatEndpoint): SystemPrompt | undefined {
+        if (endpoint.model?.startsWith('my-model-v2')) {
+            return MyModelV2Prompt;
+        }
+        return MyDefaultPrompt;
+    }
 }
 ```
 
-This allows fine-grained control over prompts while keeping all model variants organized in a single provider file.
-
 ### File Locations
 
-Prompts are located in `src/extension/prompts/node/agent/`:
-- **[defaultAgentInstructions.tsx](../src/extension/prompts/node/agent/defaultAgentInstructions.tsx)** - Base prompt and shared components
-- **[promptRegistry.ts](../src/extension/prompts/node/agent/promptRegistry.ts)**
-- **[anthropicPrompts.tsx](../src/extension/prompts/node/agent/anthropicPrompts.tsx)**
-- **[openAIPrompts.tsx](../src/extension/prompts/node/agent/openAIPrompts.tsx)**
-- **[geminiPrompts.tsx](../src/extension/prompts/node/agent/geminiPrompts.tsx)**
-- **[xAIPrompts.tsx](../src/extension/prompts/node/agent/xAIPrompts.tsx)**
-- **[vscModelPrompts.tsx](../src/extension/prompts/node/agent/vscModelPrompts.tsx)**
+**Model capabilities:**
+- **[modelProfiles.ts](../src/platform/endpoint/common/modelProfiles.ts)** — Profile registry, inheritance, and resolution
+- **[chatModelCapabilities.ts](../src/platform/endpoint/common/chatModelCapabilities.ts)** — Public capability functions (thin wrappers over profiles)
+
+**Prompts** (located in `src/extension/prompts/node/agent/`):
+- **[defaultAgentInstructions.tsx](../src/extension/prompts/node/agent/defaultAgentInstructions.tsx)** — Base prompt and shared components
+- **[promptRegistry.ts](../src/extension/prompts/node/agent/promptRegistry.ts)** — Prompt matching and resolution
+- **[anthropicPrompts.tsx](../src/extension/prompts/node/agent/anthropicPrompts.tsx)** — Claude model prompts
+- **[openai/](../src/extension/prompts/node/agent/openai/)** — GPT model prompts (`gpt5Prompt.tsx`, `gpt51Prompt.tsx`, etc.)
+- **[geminiPrompts.tsx](../src/extension/prompts/node/agent/geminiPrompts.tsx)** — Gemini model prompts
+- **[xAIPrompts.tsx](../src/extension/prompts/node/agent/xAIPrompts.tsx)** — Grok model prompts
+- **[vscModelPrompts.tsx](../src/extension/prompts/node/agent/vscModelPrompts.tsx)** — Hidden/unreleased model prompts
 
 ---
 

package.json

@@ -4123,6 +4123,90 @@
 							"experimental"
 						]
 					},
+					"github.copilot.chat.models.profiles": {
+						"type": "object",
+						"default": {},
+						"markdownDescription": "Override model behavior profiles. Keys are model family prefixes, values are partial profiles merged on top of built-in defaults.",
+						"additionalProperties": {
+							"type": "object",
+							"properties": {
+								"extends": {
+									"type": "string",
+									"description": "Base profile to inherit from (e.g. '_anthropic', '_openai-modern')"
+								},
+								"description": {
+									"type": "string",
+									"description": "Human-readable note about this profile"
+								},
+								"editStrategy": {
+									"type": "string",
+									"enum": [
+										"apply-patch",
+										"apply-patch-with-insert-edit",
+										"multi-replace-string",
+										"replace-string",
+										"insert-edit-only"
+									],
+									"description": "How this model edits code"
+								},
+								"instructionPlacement": {
+									"type": "string",
+									"enum": [
+										"system-before-history",
+										"system-after-history",
+										"user-message"
+									],
+									"description": "Where to place system instructions"
+								},
+								"replaceStringHint": {
+									"type": "string",
+									"enum": [
+										"none",
+										"strong"
+									],
+									"description": "Extra prompt verbiage for replace_string"
+								},
+								"replaceStringHealing": {
+									"type": "boolean",
+									"description": "Auto-heal incorrect replace_string edits"
+								},
+								"imageSupport": {
+									"type": "object",
+									"properties": {
+										"urls": {
+											"type": "boolean"
+										},
+										"mcpResults": {
+											"type": "boolean"
+										}
+									},
+									"additionalProperties": false
+								},
+								"notebookFormat": {
+									"type": "string",
+									"enum": [
+										"json",
+										"markdown"
+									],
+									"description": "Notebook representation format"
+								},
+								"verbosity": {
+									"type": "string",
+									"enum": [
+										"low",
+										"medium",
+										"high"
+									],
+									"description": "Response verbosity hint"
+								}
+							},
+							"additionalProperties": false
+						},
+						"tags": [
+							"advanced",
+							"experimental"
+						]
+					},
 					"github.copilot.chat.inlineEdits.diagnosticsContextProvider.enabled": {
 						"type": "boolean",
 						"default": false,

src/extension/prompts/node/agent/test/agentPrompt.spec.tsx

@@ -8,6 +8,7 @@
 import { ChatLocation } from '../../../../../platform/chat/common/commonTypes';
 import { StaticChatMLFetcher } from '../../../../../platform/chat/test/common/staticChatMLFetcher';
 import { CodeGenerationTextInstruction, ConfigKey, IConfigurationService } from '../../../../../platform/configuration/common/configurationService';
+import { getRegisteredProfileKeys } from '../../../../../platform/endpoint/common/modelProfiles';
 import { MockEndpoint } from '../../../../../platform/endpoint/test/node/mockEndpoint';
 import { messageToMarkdown } from '../../../../../platform/log/common/messageStringify';
 import { IResponseDelta } from '../../../../../platform/networking/common/fetch';
@@ -40,15 +41,39 @@
 	'gpt-5.1',
 	'gpt-5.1-codex',
 	'gpt-5.1-codex-mini',
+	'gpt-5.2',
+	'gpt-5.2-codex',
+	'gpt-5.3-codex',
+	'claude-3.5-sonnet',
 	'claude-haiku-4.5',
 	'claude-sonnet-4.5',
 	'claude-opus-4.5',
 	'claude-opus-4.6',
 	'claude-opus-4.6-fast',
+	'Anthropic',
 	'gemini-2.0-flash',
-	'grok-code-fast-1'
+	'gemini-3-pro',
+	'grok-code-fast-1',
+	'o3-mini',
+	'o4-mini',
+	'OpenAI',
 ];
 
+/**
+ * Ensures every model profile key in MODEL_PROFILES has at least one
+ * matching entry in testFamilies. Fails when a new profile is added
+ * without corresponding snapshot coverage.
+ */
+suite('AgentPrompt - profile coverage', () => {
+	test('every model profile key is covered by a snapshot test family', () => {
+		const profileKeys = getRegisteredProfileKeys();
+		const missingKeys = profileKeys.filter(key =>
+			!testFamilies.some(family => family.startsWith(key) || key.startsWith(family))
+		);
+		expect(missingKeys, 'Profile keys missing snapshot coverage — add a matching entry to testFamilies in agentPrompt.spec.tsx').toEqual([]);
+	});
+});
+
 testFamilies.forEach(family => {
 	suite(`AgentPrompt - ${family}`, () => {
 		let accessor: ITestingServicesAccessor;
@@ -139,7 +164,7 @@
 		}
 
 		test('simple case', async () => {
 			await expect(await agentPromptToString(accessor, {
 				chatVariables: new ChatVariablesCollection(),
 				history: [],
 				query: 'hello',
@@ -148,7 +173,7 @@
 
 		test('all tools', async () => {
 			const toolsService = accessor.get(IToolsService);
 			await expect(await agentPromptToString(accessor, {
 				chatVariables: new ChatVariablesCollection(),
 				history: [],
 				query: 'hello',
@@ -163,7 +188,7 @@
 		test('all non-edit tools', async () => {
 			const toolsService = accessor.get(IToolsService);
 			const editTools: Set<string> = new Set([ToolName.ApplyPatch, ToolName.EditFile, ToolName.ReplaceString, ToolName.MultiReplaceString]);
 			await expect(await agentPromptToString(accessor, {
 				chatVariables: new ChatVariablesCollection(),
 				history: [],
 				query: 'hello',
@@ -176,7 +201,7 @@
 		});
 
 		test('one attachment', async () => {
 			await expect(await agentPromptToString(accessor, {
 				chatVariables: new ChatVariablesCollection([{ id: 'vscode.file', name: 'file', value: fileTsUri }]),
 				history: [],
 				query: 'hello',
@@ -190,7 +215,7 @@
 		};
 
 		test('tool use', async () => {
 			await expect(await agentPromptToString(
 				accessor,
 				{
 					chatVariables: new ChatVariablesCollection([{ id: 'vscode.file', name: 'file', value: fileTsUri }]),
@@ -205,7 +230,7 @@
 		});
 
 		test('cache BPs', async () => {
 			await expect(await agentPromptToString(
 				accessor,
 				{
 					chatVariables: new ChatVariablesCollection([{ id: 'vscode.file', name: 'file', value: fileTsUri }]),
@@ -237,7 +262,7 @@
 			};
 			previousTurn.setResponse(TurnStatus.Success, { type: 'user', message: 'response' }, 'responseId', previousTurnResult);
 
 			await expect(await agentPromptToString(
 				accessor,
 				{
 					chatVariables: new ChatVariablesCollection([]),
@@ -263,7 +288,7 @@
 
 		test('custom instructions not in system message', async () => {
 			accessor.get(IConfigurationService).setConfig(ConfigKey.CustomInstructionsInSystemMessage, false);
 			await expect(await agentPromptToString(accessor, {
 				chatVariables: new ChatVariablesCollection(),
 				history: [],
 				query: 'hello',
@@ -273,7 +298,7 @@
 
 		test('omit base agent instructions', async () => {
 			accessor.get(IConfigurationService).setConfig(ConfigKey.Advanced.OmitBaseAgentInstructions, true);
 			await expect(await agentPromptToString(accessor, {
 				chatVariables: new ChatVariablesCollection(),
 				history: [],
 				query: 'hello',
@@ -283,7 +308,7 @@
 		test('edited file events are grouped by kind', async () => {
 			const otherUri = URI.file('/workspace/other.ts');
 
 			await expect((await agentPromptToString(accessor, {
 				chatVariables: new ChatVariablesCollection(),
 				history: [],
 				query: 'hello',

src/platform/configuration/common/configurationService.ts

@@ -665,6 +665,8 @@ export namespace ConfigKey {
 		export const CLIIsolationOption = defineSetting<boolean>('chat.cli.isolationOption.enabled', ConfigType.Simple, false);
 		export const CLISessionController = defineSetting<boolean>('chat.cli.sessionController.enabled', ConfigType.Simple, false);
 		export const RequestLoggerMaxEntries = defineAndMigrateSetting<number>('chat.advanced.debug.requestLogger.maxEntries', 'chat.debug.requestLogger.maxEntries', 100);
+		/** Override model behavior profiles. Keys are model family prefixes, values are partial profiles. */
+		export const ModelProfiles = defineAndMigrateSetting<Record<string, Record<string, unknown>>>('chat.advanced.models.profiles', 'chat.models.profiles', {});
 
 		// Experiment-based settings
 		/** Uses new expanded project labels */