Problem
aimock synthesizes a reasoning channel — chat-completions delta.reasoning_content / Responses-API response.reasoning_summary_text.delta — from a fixture's abstract reasoning field regardless of the requested model id.
As a result, a backend wired to a non-reasoning model (e.g. gpt-4.1) still appears to emit reasoning under replay, even though the real provider would emit none for that model.
Impact
This masked a CopilotKit showcase llamaindex demo that was wired to gpt-4.1 — under aimock replay it appeared to emit a reasoning channel, hiding the model/capability mismatch that would produce no reasoning against the real provider.
Ask
aimock should make reasoning-channel emission model-capability-aware:
- do not emit a reasoning channel for known non-reasoning model ids, or
- at minimum, emit a warning when a reasoning fixture is replayed against a non-reasoning model id
so that model/capability mismatches surface during replay instead of giving false-green reasoning.
Problem
aimock synthesizes a reasoning channel — chat-completions
delta.reasoning_content/ Responses-APIresponse.reasoning_summary_text.delta— from a fixture's abstractreasoningfield regardless of the requested model id.As a result, a backend wired to a non-reasoning model (e.g.
gpt-4.1) still appears to emit reasoning under replay, even though the real provider would emit none for that model.Impact
This masked a CopilotKit showcase
llamaindexdemo that was wired togpt-4.1— under aimock replay it appeared to emit a reasoning channel, hiding the model/capability mismatch that would produce no reasoning against the real provider.Ask
aimock should make reasoning-channel emission model-capability-aware:
so that model/capability mismatches surface during replay instead of giving false-green reasoning.