Skip to content

Make reasoning-channel emission model-capability-aware #254

Description

@jpr5

Problem

aimock synthesizes a reasoning channel — chat-completions delta.reasoning_content / Responses-API response.reasoning_summary_text.delta — from a fixture's abstract reasoning field regardless of the requested model id.

As a result, a backend wired to a non-reasoning model (e.g. gpt-4.1) still appears to emit reasoning under replay, even though the real provider would emit none for that model.

Impact

This masked a CopilotKit showcase llamaindex demo that was wired to gpt-4.1 — under aimock replay it appeared to emit a reasoning channel, hiding the model/capability mismatch that would produce no reasoning against the real provider.

Ask

aimock should make reasoning-channel emission model-capability-aware:

  • do not emit a reasoning channel for known non-reasoning model ids, or
  • at minimum, emit a warning when a reasoning fixture is replayed against a non-reasoning model id

so that model/capability mismatches surface during replay instead of giving false-green reasoning.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions