Hi, I noticed that the title generation is taking a majority portion of response generation so I make it run in parallel insteal of block the response process. Please see details below:
Problem
The chatbot API endpoint was experiencing severe performance issues with response times exceeding 84 seconds for new chat creation. The root cause was blocking title generation - the system waited for an LLM API call to generate a chat title before returning the stream response.
Solution
Made title generation non-blocking using a two-phase approach:
Generate a temporary title immediately from the first 60 characters of the user's message
Generate the LLM-based title asynchronously in the background and update when ready
This allows the chat stream to start immediately while the polished title is generated asynchronously.
Changes
lib/db/queries.ts
Added updateChatTitleById function to update chat titles after creation
app/(chat)/api/chat/route.ts
Replaced blocking await generateTitleFromUserMessage() with immediate temporary title generation
Added asynchronous real title generation that updates the chat when ready
Added imports for updateChatTitleById and getTextFromMessage
Performance Impact
Before: 19+ seconds (blocking on LLM title generation)
After: <1 second (immediate response with temporary title, real title updates in background)
PR: #1325
Hi, I noticed that the title generation is taking a majority portion of response generation so I make it run in parallel insteal of block the response process. Please see details below:
Problem
The chatbot API endpoint was experiencing severe performance issues with response times exceeding 84 seconds for new chat creation. The root cause was blocking title generation - the system waited for an LLM API call to generate a chat title before returning the stream response.
Solution
Made title generation non-blocking using a two-phase approach:
Generate a temporary title immediately from the first 60 characters of the user's message
Generate the LLM-based title asynchronously in the background and update when ready
This allows the chat stream to start immediately while the polished title is generated asynchronously.
Changes
lib/db/queries.ts
Added updateChatTitleById function to update chat titles after creation
app/(chat)/api/chat/route.ts
Replaced blocking await generateTitleFromUserMessage() with immediate temporary title generation
Added asynchronous real title generation that updates the chat when ready
Added imports for updateChatTitleById and getTextFromMessage
Performance Impact
Before: 19+ seconds (blocking on LLM title generation)
After: <1 second (immediate response with temporary title, real title updates in background)
PR: #1325