Size: M — enabling infra (prerequisite for onboarding image attachments #65).
Problem
callAI(provider, apiKey, systemPrompt, userMessage) (src/lib/ai.ts:278-333) is text-only — it passes a plain string as the sole user message for all four providers. All the models actually configured are vision-capable (Claude claude-sonnet-4, Gemini gemini-2.0-flash, Mistral mistral-large-latest, OpenAI gpt-4o-mini), so nothing but the wrapper prevents multimodal input. Facet extraction (facet-extract.ts:84) and the onboarding reply (profile/chat/route.ts:67) both call this text-only path.
Scope
Add a multimodal variant (e.g. callAIVision(provider, apiKey, system, text, images: {mime, dataOrUrl}[])) that builds provider-specific image content blocks:
- Claude:
content: [{type:"text"}, {type:"image", source:{type:"base64"|"url", ...}}]
- OpenAI:
content: [{type:"text"}, {type:"image_url", image_url:{url}}]
- Gemini:
contents parts with inlineData/fileData
- Mistral:
image_url content parts (pixtral-capable model — may need a model bump)
Keep callAI as the text-only fast path; the vision call is opt-in. Cap image count/size and downscale server-side to control token cost. Handle providers/models that can't do vision by falling back to text + a note.
Blocks: the onboarding attachment feature.
Size: M — enabling infra (prerequisite for onboarding image attachments #65).
Problem
callAI(provider, apiKey, systemPrompt, userMessage)(src/lib/ai.ts:278-333) is text-only — it passes a plain string as the sole user message for all four providers. All the models actually configured are vision-capable (Claudeclaude-sonnet-4, Geminigemini-2.0-flash, Mistralmistral-large-latest, OpenAIgpt-4o-mini), so nothing but the wrapper prevents multimodal input. Facet extraction (facet-extract.ts:84) and the onboarding reply (profile/chat/route.ts:67) both call this text-only path.Scope
Add a multimodal variant (e.g.
callAIVision(provider, apiKey, system, text, images: {mime, dataOrUrl}[])) that builds provider-specific image content blocks:content: [{type:"text"}, {type:"image", source:{type:"base64"|"url", ...}}]content: [{type:"text"}, {type:"image_url", image_url:{url}}]contentsparts withinlineData/fileDataimage_urlcontent parts (pixtral-capable model — may need a model bump)Keep
callAIas the text-only fast path; the vision call is opt-in. Cap image count/size and downscale server-side to control token cost. Handle providers/models that can't do vision by falling back to text + a note.Blocks: the onboarding attachment feature.