Skip to content

Comments

exploration: config-driven model profiles with inheritance#3915

Draft
rwoll wants to merge 6 commits intomainfrom
config-driven-model-profiles
Draft

exploration: config-driven model profiles with inheritance#3915
rwoll wants to merge 6 commits intomainfrom
config-driven-model-profiles

Conversation

@rwoll
Copy link
Member

@rwoll rwoll commented Feb 21, 2026

Problem

When experimenting with how well existing prompts and tools work with new model variants, we currently need custom builds of the extension — even if the new model behaves identically to an existing one. Every new model requires hardcoded changes spliced across multiple boolean functions in chatModelCapabilities.ts and potentially a new prompt resolver file, all of which require a full PR/build/deploy cycle just to try things out.

This makes it harder to iterate quickly during model evaluation and slows down the feedback loop for understanding whether a new variant needs custom behavior or can reuse an existing archetype.

Exploration

This PR explores making model capabilities config-driven so that evaluating a new model variant that behaves like an existing archetype requires zero code changes — just a settings override or a one-line profile entry.

What it does

New: modelProfiles.ts — a declarative registry where each model inherits from a base archetype and only overrides what's different:

'_openai-modern': {
    editStrategy: 'apply-patch',
    notebookFormat: 'json',
    // ...inherited defaults
},

'gpt-5.3-codex': {
    extends: '_openai-modern',
    verbosity: 'low',
},

Refactored: chatModelCapabilities.ts — all ~15 capability functions become thin wrappers:

export function modelSupportsApplyPatch(model): boolean {
    return expandEditStrategy(getModelProfile(model).editStrategy).supportsApplyPatch;
}

All function signatures are unchanged — zero callsite modifications needed.

New setting: github.copilot.chat.advanced.models.profiles — override profiles from settings.json for eval without any code changes:

"github.copilot.chat.advanced.models.profiles": {
    "new-model-family": {
        "extends": "_anthropic",
        "editStrategy": "apply-patch"
    }
}

BYOK models

BYOK models get the right profile automatically as long as their model ID starts with a recognized prefix (claude, gpt, gemini, grok-code, etc.). Unknown model IDs fall through to the default profile (insert-edit-only, standard settings) — which is the safe behavior. And with the new settings override, users can explicitly map any unknown model to an archetype via github.copilot.chat.advanced.models.profiles without rebuilding.

Key design decisions

  • Inheritance via extends — profiles inherit from base archetypes (_anthropic, _openai-modern, _gemini, etc.) and only override what's different
  • EditStrategy enum — replaces ~6 interrelated booleans with one meaningful concept that expands to the right flag combinations
  • Longest-prefix-first matchinggpt-5.1-codex matches before gpt-5.1 before gpt-5 before gpt
  • Hash-based models — unreleased models still use SHA-256 hash matching, resolved to profiles internally
  • Backward compatible — prompt registry (familyPrefixes / matchesModel / registerPrompt) is untouched; profiles handle capabilities, prompt routing stays as-is

Validation

  • 27 new unit tests for profile resolution, inheritance, prefix matching, and edit strategy expansion
  • Snapshot (.snap) files have been intentionally removed from this PR — snapshot tests that reference them will fail until regenerated

What this does NOT change

  • TSX prompt components still exist — models that genuinely need different instructions still need custom prompts
  • The prompt registry pattern is untouched
  • All existing callsites are unchanged

See PLAN.md in the repo root for the full design document.

@rwoll rwoll force-pushed the config-driven-model-profiles branch from 9007853 to dc5161c Compare February 21, 2026 04:14
Replace hardcoded capability functions in chatModelCapabilities.ts with
a declarative MODEL_PROFILES registry in modelProfiles.ts. Models now
inherit from base archetypes (_anthropic, _openai-modern, _gemini, etc.)
and only override what's different.

- Add ModelProfile type with EditStrategy, InstructionPlacement, and
  inheritance via 'extends'
- Add MODEL_PROFILES table covering all current model families
- Rewrite ~15 capability functions as thin wrappers over getModelProfile()
- Add expandEditStrategy() to map strategy names to capability flags
- Add github.copilot.chat.advanced.models.profiles setting for overrides
- Add 30 unit tests for profile resolution and behavioral equivalence
- Update docs/prompts.md with model profiles documentation
- All 150 prompt snapshot tests pass with zero diffs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@rwoll rwoll force-pushed the config-driven-model-profiles branch from dc5161c to 88c7654 Compare February 21, 2026 04:17
@rwoll rwoll closed this Feb 21, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@rwoll rwoll reopened this Feb 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant