fix(chat): run preflight against the resolved LLM, not the default

[Ahmath-Gadji]

Draft — scaffolding only. To be implemented later.

Problem

The chat preflight validates only the default LLM (config.llm), so it is out of step with the LLM that actually answers once partition chat_llm presets are honored:

• Availability — check_llm_model_availability (openrag/api/dependencies/llm.py) probes only config.llm. A partition with a healthy custom chat_llm is rejected (503/404) when the default endpoint is down.
• Token / context sizing — get_max_model_tokens / validate_tokens_limit (openrag/api/routers/user/chat.py) use a single startup-primed global from the default model, so requests are sized against the wrong context window.

Plan

• Resolve the request's effective LLM (partition → chat_llm, mirroring QueryService._resolve_llm) before the availability/token preflight.
• Validate availability against the resolved endpoint — make the check body-aware or move it into the handler (the partition→endpoint mapping isn't known until the body is parsed).
• Size token/context per resolved endpoint; fall back to config.llm_context.
• Tests: default-down + healthy custom chat_llm succeeds; per-endpoint context sizing.

Notes

Pre-existing limitation of the preflight layer, surfaced once answer generation routes through non-default endpoints. Separately, per-endpoint max_llm_context_size / max_output_tokens are being made admin-configurable, which covers the default endpoint's sizing; this PR extends the preflight to the resolved endpoint.

Closes #639.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 34974f03-180e-491d-8c3c-f85529df440f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/639-chat-preflight-resolved-llm

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}