exploration: config-driven model profiles with inheritance#3915
Draft
exploration: config-driven model profiles with inheritance#3915
Conversation
9007853 to
dc5161c
Compare
Replace hardcoded capability functions in chatModelCapabilities.ts with a declarative MODEL_PROFILES registry in modelProfiles.ts. Models now inherit from base archetypes (_anthropic, _openai-modern, _gemini, etc.) and only override what's different. - Add ModelProfile type with EditStrategy, InstructionPlacement, and inheritance via 'extends' - Add MODEL_PROFILES table covering all current model families - Rewrite ~15 capability functions as thin wrappers over getModelProfile() - Add expandEditStrategy() to map strategy names to capability flags - Add github.copilot.chat.advanced.models.profiles setting for overrides - Add 30 unit tests for profile resolution and behavioral equivalence - Update docs/prompts.md with model profiles documentation - All 150 prompt snapshot tests pass with zero diffs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dc5161c to
88c7654
Compare
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When experimenting with how well existing prompts and tools work with new model variants, we currently need custom builds of the extension — even if the new model behaves identically to an existing one. Every new model requires hardcoded changes spliced across multiple boolean functions in
chatModelCapabilities.tsand potentially a new prompt resolver file, all of which require a full PR/build/deploy cycle just to try things out.This makes it harder to iterate quickly during model evaluation and slows down the feedback loop for understanding whether a new variant needs custom behavior or can reuse an existing archetype.
Exploration
This PR explores making model capabilities config-driven so that evaluating a new model variant that behaves like an existing archetype requires zero code changes — just a settings override or a one-line profile entry.
What it does
New:
modelProfiles.ts— a declarative registry where each model inherits from a base archetype and only overrides what's different:Refactored:
chatModelCapabilities.ts— all ~15 capability functions become thin wrappers:All function signatures are unchanged — zero callsite modifications needed.
New setting:
github.copilot.chat.advanced.models.profiles— override profiles fromsettings.jsonfor eval without any code changes:BYOK models
BYOK models get the right profile automatically as long as their model ID starts with a recognized prefix (
claude,gpt,gemini,grok-code, etc.). Unknown model IDs fall through to the default profile (insert-edit-only, standard settings) — which is the safe behavior. And with the new settings override, users can explicitly map any unknown model to an archetype viagithub.copilot.chat.advanced.models.profileswithout rebuilding.Key design decisions
extends— profiles inherit from base archetypes (_anthropic,_openai-modern,_gemini, etc.) and only override what's differentEditStrategyenum — replaces ~6 interrelated booleans with one meaningful concept that expands to the right flag combinationsgpt-5.1-codexmatches beforegpt-5.1beforegpt-5beforegptfamilyPrefixes/matchesModel/registerPrompt) is untouched; profiles handle capabilities, prompt routing stays as-isValidation
.snap) files have been intentionally removed from this PR — snapshot tests that reference them will fail until regeneratedWhat this does NOT change
See
PLAN.mdin the repo root for the full design document.