fix(ai-proxy): validate model tool support at Router init (fail fast) by Scra3 · Pull Request #1452 · ForestAdmin/agent-nodejs

Scra3 · 2026-02-05T15:24:12Z

Summary

Move model validation from agent.addAi() to Router constructor (fail fast pattern)
Add AIModelNotSupportedError dedicated error class for better error messages
Make isModelSupportingTools internal (no longer exported from the package)

Breaking Change

isModelSupportingTools is no longer exported from @forestadmin/ai-proxy. This function was never intended to be public API - validation should be handled internally by the Router.

Why this is a bug fix

The validation was incorrectly placed in agent.addAi() instead of Router constructor. This caused:

Inconsistent behavior when Router is used directly (without agent)
Late failure - error should occur at initialization, not when adding AI config

Test plan

Unit tests for AIModelNotSupportedError in router.test.ts
Unit tests for isModelSupportingTools in provider-dispatcher.test.ts
Removed obsolete test from agent.test.ts
All 130 ai-proxy tests pass
All agent tests pass

🤖 Generated with Claude Code

BREAKING CHANGE: isModelSupportingTools is no longer exported from ai-proxy - Add AIModelNotSupportedError for descriptive error messages - Move model validation from agent.addAi() to Router constructor - Make isModelSupportingTools internal (not exported from index) - Error is thrown immediately at Router init if model doesn't support tools This is a bug fix: validation should happen at proxy initialization, not at the agent level. This ensures consistent behavior regardless of how the Router is instantiated. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

The function is only used in Router, so it makes sense to keep it there. This simplifies the provider-dispatcher module and keeps related code together. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Move model support checking logic to supported-models.ts for better organization and separation of concerns. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

… file - Create supported-models.test.ts with direct function tests - Remove duplicate model list tests from router.test.ts - Simplify index.ts exports using export * pattern Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

gpt-3.5-turbo actually supports function calling (tools), verified with real OpenAI API integration tests. - Remove gpt-3.5-turbo and gpt-3.5 from blocklist - Add model-tools-support.integration.test.ts to verify tool support - Update unit tests to reflect correct model support Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Test was only used to investigate which models support tools. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

qltysh · 2026-02-05T20:01:52Z

Coverage Impact

This PR will not change total coverage.

Modified Files with Diff Coverage (3)

Rating	File	% Diff	Uncovered Line #s
	packages/ai-proxy/src/router.ts	100.0%
	packages/ai-proxy/src/errors.ts	100.0%
	packages/ai-proxy/src/supported-models.ts	100.0%
	Total	100.0%

🚦 See full report on Qlty Cloud »

🛟 Help

Diff Coverage: Coverage for added or modified lines of code (excludes deleted files). Learn more.
Total Coverage: Coverage for the whole repository, calculated as the sum of all File Coverage. Learn more.
File Coverage: Covered Lines divided by Covered Lines plus Missed Lines. (Excludes non-executable lines including blank lines and comments.)
- Indirect Changes: Changes to File Coverage for files that were not modified in this PR. Learn more.

Add validation in addAi() to fail fast when an unsupported model is configured, rather than waiting until route creation. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Scra3 · 2026-02-05T20:09:10Z

packages/agent/test/agent.test.ts

        }),
-      ).toThrow(
-        "Model 'gpt-4' does not support function calling (tools). " +
-          'Please use a compatible model like gpt-4o, gpt-4o-mini, or gpt-4-turbo.',


This sentence is removed, not really relevant I guess

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Verify all allowed OpenAI models support tool calls with real API. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Include comprehensive model list from OpenAI types: - GPT-4o family (gpt-4o, gpt-4o-mini, audio/search previews) - GPT-4.1 family (gpt-4.1, mini, nano) - GPT-4 turbo variants - GPT-3.5 family - O-series reasoning models (o1, o3, o4-mini) - Future models (gpt-5.x) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

All models verified with real OpenAI API: - gpt-4o, gpt-4o-mini - gpt-4.1, gpt-4.1-mini, gpt-4.1-nano - gpt-4-turbo - gpt-3.5-turbo - o1, o3, o3-mini, o4-mini Removed o1-mini and o1-preview (no longer available). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Export SUPPORTED_OPENAI_MODELS from supported-models.ts and use it in both unit tests and integration tests to ensure consistency. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Verified with real API - all gpt-5 variants exist and support tools: - gpt-5 - gpt-5-mini - gpt-5-nano - gpt-5.1 - gpt-5.2 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Remove hardcoded SUPPORTED_OPENAI_MODELS list - Fetch models dynamically via openai.models.list() in integration tests - Filter models using isModelSupportingTools() blacklist - Add prefix and pattern-based blacklist for unsupported models - Add explicit error message when tests fail with instructions to update blacklist Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Keep only essential test cases: - Known supported model - Unknown model (allowed by default) - Blacklisted model Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Remove validateModelSupportsTools call from Agent.addAi() - Validation now happens when AiProxyRouter is instantiated during start() - Update test to use real makeRoutes to trigger validation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Scra3 · 2026-02-06T08:08:29Z

packages/ai-proxy/src/supported-models.ts

@@ -0,0 +1,92 @@
+/**


To build the list, I ran integration test with all openai model and if the model returns an error, I added it to the blacklist.

Scra3 · 2026-02-06T08:09:29Z

packages/ai-proxy/test/llm.integration.test.ts

+      if (failures.length > 0) {
+        const failedModelNames = failures.map(f => f.model).join(', ');
+        // eslint-disable-next-line no-console
+        console.error(


This error message is important because I want to tell to the developper to update the blacklist

- Remove -pro pattern that incorrectly blocked o3-pro - Add error handling for OpenAI models.list() API call - Distinguish infrastructure errors from model compatibility errors in tests - Remove unused validateModelSupportsTools export - Add test for multiple AI configurations validation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Scra3 · 2026-02-06T08:25:14Z

packages/ai-proxy/test/llm.integration.test.ts

+          const errorMessage = String(error);
+
+          // Infrastructure errors should fail the test immediately
+          const isInfrastructureError =


avoid false positive to add the model to the blacklist. Not really clean but for a test, i guess it's ok.

…mpletions Models like gpt-5-pro, o1-pro only support v1/responses, not v1/chat/completions. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

nbouliol · 2026-02-06T09:11:55Z

packages/ai-proxy/src/supported-models.ts

+ *
+ * @see https://platform.openai.com/docs/guides/function-calling
+ */
+const OPENAI_MODELS_WITHOUT_TOOLS_SUPPORT_PREFIXES = [


can you rename it to be more explicit ? something like

Suggested change

const OPENAI_MODELS_WITHOUT_TOOLS_SUPPORT_PREFIXES = [

const OPENAI_MODELS_BLACK_LIST = [

- UNSUPPORTED_MODEL_PREFIXES (was OPENAI_MODELS_WITHOUT_TOOLS_SUPPORT_PREFIXES) - UNSUPPORTED_MODEL_PATTERNS (was OPENAI_MODELS_WITHOUT_TOOLS_SUPPORT_PATTERNS) - SUPPORTED_MODEL_OVERRIDES (was OPENAI_MODELS_EXCEPTIONS) Clearer naming reflecting that models must support tool calls. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

This will identify models that don't support parallel_tool_calls: false. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

…_calls O-series reasoning models (o1, o3, o4) don't support parallel_tool_calls parameter which is required for our use case. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

## @forestadmin/ai-proxy [1.4.1](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/ai-proxy@1.4.0...@forestadmin/ai-proxy@1.4.1) (2026-02-06) ### Bug Fixes * **ai-proxy:** validate model tool support at Router init (fail fast) ([#1452](#1452)) ([adf2efd](adf2efd))

## @forestadmin/agent [1.72.4](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/agent@1.72.3...@forestadmin/agent@1.72.4) (2026-02-06) ### Bug Fixes * **ai-proxy:** validate model tool support at Router init (fail fast) ([#1452](#1452)) ([adf2efd](adf2efd)) ### Dependencies * **@forestadmin/ai-proxy:** upgraded to 1.4.1 * **@forestadmin/forestadmin-client:** upgraded to 1.37.10 * **@forestadmin/mcp-server:** upgraded to 1.7.6

alban bertolini and others added 7 commits February 5, 2026 16:23

refactor(ai-proxy): move isModelSupportingTools to router.ts

9f06d04

The function is only used in Router, so it makes sense to keep it there. This simplifies the provider-dispatcher module and keeps related code together. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

refactor(ai-proxy): extract isModelSupportingTools to dedicated file

e4ab3a1

Move model support checking logic to supported-models.ts for better organization and separation of concerns. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

fix: test

ae0354d

chore(ai-proxy): remove model tools support integration test

acbe27b

Test was only used to investigate which models support tools. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

alban bertolini and others added 2 commits February 5, 2026 21:03

fix(agent): validate model tool support in addAi() for early failure

f62081f

Add validation in addAi() to fail fast when an unsupported model is configured, rather than waiting until route creation. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

test(agent): reorder addAi tests

3ff397f

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Scra3 commented Feb 5, 2026

View reviewed changes

alban bertolini and others added 12 commits February 5, 2026 21:39

style(agent): fix prettier formatting in test

394e5c8

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

chore(ai-proxy): remove integration test reference comments

ce09754

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

chore(ai-proxy): remove @internal comment

4d5db25

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

test(ai-proxy): add model tool support verification test

a2f1e0e

Verify all allowed OpenAI models support tool calls with real API. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

refactor(ai-proxy): share SUPPORTED_OPENAI_MODELS between tests

3baa377

Export SUPPORTED_OPENAI_MODELS from supported-models.ts and use it in both unit tests and integration tests to ensure consistency. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

feat(ai-proxy): add gpt-5 family to supported models

7aa150a

Verified with real API - all gpt-5 variants exist and support tools: - gpt-5 - gpt-5-mini - gpt-5-nano - gpt-5.1 - gpt-5.2 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

chore(ai-proxy): upgrade @langchain/openai to 1.2.5 and fix lint

30f653e

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

refactor(ai-proxy): simplify unit tests for isModelSupportingTools

094895e

Keep only essential test cases: - Known supported model - Unknown model (allowed by default) - Blacklisted model Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Scra3 commented Feb 6, 2026

View reviewed changes

fix(ai-proxy): restore -pro pattern for models not supporting chat co…

b9d8314

…mpletions Models like gpt-5-pro, o1-pro only support v1/responses, not v1/chat/completions. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

nbouliol reviewed Feb 6, 2026

View reviewed changes

alban bertolini and others added 3 commits February 6, 2026 10:21

test(ai-proxy): add parallel_tool_calls to model verification test

4847e4e

This will identify models that don't support parallel_tool_calls: false. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

fix(ai-proxy): block o-series models that don't support parallel_tool…

a8df84c

…_calls O-series reasoning models (o1, o3, o4) don't support parallel_tool_calls parameter which is required for our use case. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs(ai-proxy): add JSDoc to isModelSupportingTools

516706d

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Scra3 force-pushed the fix/ai-proxy-model-validation branch from 5957438 to 516706d Compare February 6, 2026 09:36

fix(ai-proxy): update variable names in test error message

f3ceff7

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

nbouliol approved these changes Feb 6, 2026

View reviewed changes

Scra3 merged commit adf2efd into main Feb 6, 2026
27 checks passed

Scra3 deleted the fix/ai-proxy-model-validation branch February 6, 2026 15:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(ai-proxy): validate model tool support at Router init (fail fast)#1452

fix(ai-proxy): validate model tool support at Router init (fail fast)#1452
Scra3 merged 28 commits intomainfrom
fix/ai-proxy-model-validation

Scra3 commented Feb 5, 2026

Uh oh!

qltysh bot commented Feb 5, 2026 •

edited

Loading

Uh oh!

Scra3 Feb 5, 2026

Uh oh!

Scra3 Feb 6, 2026

Uh oh!

Scra3 Feb 6, 2026

Uh oh!

Scra3 Feb 6, 2026

Uh oh!

nbouliol Feb 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	const OPENAI_MODELS_WITHOUT_TOOLS_SUPPORT_PREFIXES = [
	const OPENAI_MODELS_BLACK_LIST = [

Conversation

Scra3 commented Feb 5, 2026

Summary

Breaking Change

Why this is a bug fix

Test plan

Uh oh!

qltysh bot commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Scra3 Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Scra3 Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Scra3 Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Scra3 Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

nbouliol Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

qltysh bot commented Feb 5, 2026 •

edited

Loading