Conversation
|
Greptile OverviewGreptile SummaryThis PR adds Vercel AI Gateway support to enable routing model calls through the gateway, along with model-specific fixes for Kimi and DeepSeek. Key changes:
Concerns:
Confidence Score: 3/5
Important Files Changed
Sequence DiagramsequenceDiagram
participant User
participant LLMProvider
participant AISdkClient
participant AIGateway
participant Model
User->>LLMProvider: getClient(modelName)
alt modelName contains "/"
LLMProvider->>LLMProvider: Parse provider/model (substring)
LLMProvider->>LLMProvider: Check if provider is "gateway"
alt provider is "gateway"
LLMProvider->>AIGateway: createGateway(clientOptions)
AIGateway-->>LLMProvider: gateway provider instance
end
LLMProvider->>AISdkClient: new AISdkClient(model)
AISdkClient-->>LLMProvider: client instance
end
User->>AISdkClient: createChatCompletion(options)
alt model is Kimi
AISdkClient->>AISdkClient: Force temperature=1
end
alt response_model exists AND (DeepSeek OR Kimi)
AISdkClient->>AISdkClient: Append JSON schema instruction
end
AISdkClient->>Model: generateObject/generateText()
Model-->>AISdkClient: response
AISdkClient-->>User: formatted response
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
@Kylejeong2 would users have to define |
Yeah that is the current flow, but I can update it to make more ergonomic |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| import { deepseek, createDeepSeek } from "@ai-sdk/deepseek"; | ||
| import { perplexity, createPerplexity } from "@ai-sdk/perplexity"; | ||
| import { ollama, createOllama } from "ollama-ai-provider-v2"; | ||
| import { gateway, createGateway } from "ai"; |
There was a problem hiding this comment.
The import source may be incorrect. According to the lockfile, the ai package (v5.0.76) includes @ai-sdk/gateway 2.0.0, but gateway and createGateway may not be re-exported from the main ai package. Check if these should be imported from @ai-sdk/gateway instead:
| import { gateway, createGateway } from "ai"; | |
| import { gateway, createGateway } from "@ai-sdk/gateway"; |
Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/core/lib/v3/llm/LLMProvider.ts
Line: 34:34
Comment:
The import source may be incorrect. According to the lockfile, the `ai` package (v5.0.76) includes `@ai-sdk/gateway` 2.0.0, but `gateway` and `createGateway` may not be re-exported from the main `ai` package. Check if these should be imported from `@ai-sdk/gateway` instead:
```suggestion
import { gateway, createGateway } from "@ai-sdk/gateway";
```
How can I resolve this? If you propose a fix, please make it concise.
There was a problem hiding this comment.
incorrect ai is the correct package
There was a problem hiding this comment.
1 issue found across 6 files
Confidence score: 5/5
- Only minor wording clarity issue in
packages/evals/lib/AISdkClientWrapped.ts; it doesn’t affect runtime behavior and is low severity. - Score reflects low risk since the concern is documentation/terminology rather than functional change.
- Pay close attention to
packages/evals/lib/AISdkClientWrapped.ts- update the prompt wording to say JSON schema format for clarity.
Prompt for AI agents (all issues)
Check if these issues are valid — if so, understand the root cause of each and fix them.
<file name="packages/evals/lib/AISdkClientWrapped.ts">
<violation number="1" location="packages/evals/lib/AISdkClientWrapped.ts:153">
P2: Misleading terminology: the prompt says 'zod schema format' but `toJsonSchema()` produces a JSON Schema, not a Zod schema. Consider changing the wording to 'JSON schema format' for clarity.</violation>
</file>
Architecture diagram
sequenceDiagram
participant App as App / Eval Engine
participant LLMP as LLMProvider.ts
participant Client as AISdkClient
participant AISDK as Vercel AI SDK
participant GW as NEW: AI Gateway
participant Provider as Model Provider (Kimi/DeepSeek)
Note over App,Provider: Runtime Request Flow with AI Gateway & Model Optimization
App->>LLMP: getAISDKLanguageModel(provider, model)
alt NEW: Provider is "gateway"
LLMP->>AISDK: createGateway(config)
else Standard Provider
LLMP->>AISDK: createProvider(config)
end
LLMP-->>App: languageModel instance
App->>Client: createChatCompletion(options)
rect rgb(23, 37, 84)
Note right of Client: CHANGED: Model-specific Logic
opt modelId contains "kimi"
Client->>Client: NEW: Force temperature = 1
end
opt modelId contains "deepseek" OR "kimi"
Client->>Client: NEW: Inject Zod Schema & JSON instructions into prompt
end
end
alt generateObject (Structured Data)
Client->>AISDK: generateObject(model, schema, messages)
AISDK->>GW: Request with Gateway Headers
GW->>Provider: Route to Inference API
Provider-->>GW: JSON Response
GW-->>AISDK: Relay Response
AISDK-->>Client: Typed Object
else generateText
Client->>AISDK: generateText(model, messages)
AISDK->>GW: Request with Gateway Headers
GW->>Provider: Route to Inference API
Provider-->>GW: Text Response
GW-->>AISDK: Relay Response
AISDK-->>Client: Text
end
Client-->>App: LLMResponse
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
|
|
||
| formattedMessages.push({ | ||
| role: "user", | ||
| content: `Respond in this zod schema format:\n${parsedSchema}\n |
There was a problem hiding this comment.
P2: Misleading terminology: the prompt says 'zod schema format' but toJsonSchema() produces a JSON Schema, not a Zod schema. Consider changing the wording to 'JSON schema format' for clarity.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/lib/AISdkClientWrapped.ts, line 153:
<comment>Misleading terminology: the prompt says 'zod schema format' but `toJsonSchema()` produces a JSON Schema, not a Zod schema. Consider changing the wording to 'JSON schema format' for clarity.</comment>
<file context>
@@ -136,13 +137,30 @@ export class AISdkClientWrapped extends LLMClient {
+
+ formattedMessages.push({
+ role: "user",
+ content: `Respond in this zod schema format:\n${parsedSchema}\n
+You must respond in JSON format. respond WITH JSON. Do not include any other text, formatting or markdown in your output. Do not include \`\`\` or \`\`\`json in your response. Only the JSON object itself.`,
+ });
</file context>
| content: `Respond in this zod schema format:\n${parsedSchema}\n | |
| content: `Respond in this JSON schema format:\n${parsedSchema}\n |
That's fine let's just cover it in the docs! |
Yeah I can add into docs in separate PR, can you approve? |
packages/core/package.json
Outdated
| "@ai-sdk/gateway": "^3.0.25", | ||
| "@ai-sdk/google": "^2.0.23", |
There was a problem hiding this comment.
is this the right version? usually v3 ai sdk packages = only works with v6
There was a problem hiding this comment.
im thinking about removing, since the actual package that we're pulling from is "ai" not the /gateway i added this per cubic's recommendation
There was a problem hiding this comment.
docs for reference: https://ai-sdk.dev/providers/ai-sdk-providers/ai-gateway
packages/core/package.json
Outdated
| "@ai-sdk/cerebras": "^1.0.25", | ||
| "@ai-sdk/deepseek": "^1.0.23", | ||
| "@ai-sdk/gateway": "^3.0.25", |
There was a problem hiding this comment.
believe we can also just directly import gateway from ai without needing to add a new dependency
3 participants
why
New open source models are released, and don't necessarily have a great inference provider at all times. Seeing as we're already using the AI-SDK, it makes sense to add support for the AI Gateway.
what changed
Added Vercel AI Gateway as a provider to the llm providers list.
test plan
Testing with Evals
Summary by cubic
Adds Vercel AI Gateway as a provider so model calls can route through it, making it easier to support new/open-source models. Enforces JSON structured outputs (and temperature=1) for Kimi and DeepSeek models, and fixes eval model name parsing.
Written for commit 42a1c50. Summary will update on new commits. Review in cubic