Fallback to sync invocation if streaming fails by hubertzub-db · Pull Request #135 · databricks/app-templates

hubertzub-db · 2026-02-25T10:20:27Z

Summary

When the upstream streamText call encounters an error mid-stream, the chat server now transparently falls back to a non-streaming generateText call so the user still gets a response instead of just an error message.
I had to replace the previous writer.merge() approach with a manual drainStreamToWriter helper that reads chunks individually, detects errors, and triggers the fallback path when needed.

Demo/test (simulated error in agent's `stream_handler`)

stream-fallback.mov

Regression test (streaming still works)

stream-regression.mov

Demo of complete failure (both `stream_handler` and `invoke_handler` in agent are failing)

stream-complete-failure.mov

Test plan

Verify normal streaming still works end-to-end (no regression)
Simulate a streaming failure (e.g. upstream timeout or model error) and confirm the fallback fires and the user receives a complete response
Confirm error logging appears in the server console when fallback is triggered
Verify that if both streaming and fallback fail, the client receives a data-error message

hubertzub-db · 2026-02-25T10:54:34Z

+ * Reads all chunks from a UI message stream, forwarding non-error parts to the
+ * writer. Returns whether the stream encountered any errors.
+ */
+async function drainStreamToWriter(


note: unfortunately we have to merge the streams manually, otherwise we won't have enough control to transparently switch over to fallback :/

bbqiu · 2026-02-26T07:23:37Z

+ * Reads all chunks from a UI message stream, forwarding non-error parts to the
+ * writer. Returns whether the stream encountered any errors.
+ */
+async function drainStreamToWriter(


ah sorry i think i wasn't clear, but we only want to try the non-streaming path if the streaming path never starts -- for errors that are within the stream itself, we do not want to falback right? this could result in duplicate content

ex. when there is a network disconnect, we shouldn't be retrying the synchronous path, we should be trying to resume the stream.

pr looking at how the stream resumption works for context: #121

we could maybe? look at whether or not any content was emitted (ex. there was never a SSE w/ content in it?)

ah sorry i think i wasn't clear, but we only want to try the non-streaming path if the streaming path never starts

If the streaming never starts because agent's streaming handler crashes at the beginning, then the chunk.value.type in middleware's drainStreamToWriter loop will have be set to error. That's due to agent framework/aisdk detecting a chunked stream even if the agent's stream function was never registered. In other words, even if agent<->middleware streaming completely fails at the very start, then the stream is still detected and passes initial messages (handshakes etc).

hmm since that's not trivial, maybe let's discuss this offline

bbqiu

thank you for working on this! i wasn't clear in specifying what type of fallback we wanted, but if it's possible to only fallback when the stream doesn't start in the first place, that would be very helpful

also, once we make the change, if we could write a playwright integration test for this, that would be super helpful

hubertzub-db · 2026-02-27T07:42:57Z

@bbqiu I've

updated the logic to wait until first meaningful text chunk
adjusted so it works with transporting traceIds (Add thumbs up/down feedback to chatbot app #94)
added tests

Signed-off-by: Hubert Zub <hubert.zub@databricks.com>

bbqiu · 2026-03-06T03:42:07Z

+
+    const partId = generateUUID();
+    writer.write({ type: 'text-start', id: partId });
+    writer.write({ type: 'text-delta', id: partId, delta: fallback.text });


have we tried the case where the fallback is not just text? would this break for non text only output from the non streaming output?

Ah so initially I didn't consider this scenario since we shouldn't really put other potentially heavy media into a single message (it's just a fallback after all), but now I realized that we should also handle other aisdk message types: reasoning, tool metadata etc.
Unfortunately, there's no way to reuse aisdk's internal code so all the message types are automatically handled 😭, so I handle all of them here:

https://github.com/databricks/app-templates/pull/135/changes#diff-c8036971ac7d2704cc9cd583b0015eb7f324212190f8a58ed4b5276363f3794dR74

Hope this works, at least for a fallback

bbqiu

overall looks great! only one comment abt non-text output when not streaming

bbqiu · 2026-03-06T03:44:02Z

+  uiStream: ReadableStream,
+  writer: UIMessageStreamWriter,
+): Promise<{ failed: boolean; errorText?: string }> {
+  const reader = uiStream.getReader();


nit: should we close the reader

hmm we shouldn't need that - the reader would be either automatically closed in the happy path, or GCed when failed (there's no other readers for the stream). However, let me add an explicit lock release to keep things same as before the change

Signed-off-by: Hubert Zub <hubert.zub@databricks.com>

bbqiu

PR code lgtm -- have we had a chance to test this against an agent that actually calls tools etc? (asking mostly for the fallback generateText logic)

Signed-off-by: Hubert Zub <hubert.zub@databricks.com>

hubertzub-db · 2026-03-12T09:21:22Z

@bbqiu

have we had a chance to test this against an agent that actually calls tools etc? (asking mostly for the fallback generateText logic)

Seems to work fine as well, here's a result for generateText fallback w/ tool use (all the events arrive instant):

hubertzub-db force-pushed the fix-stream-fallback branch from d922582 to cd37de4 Compare February 25, 2026 10:47

hubertzub-db changed the title ~~nice1~~ Fix streaming fallback Feb 25, 2026

hubertzub-db changed the title ~~Fix streaming fallback~~ Fallback to sync invocation if streaming fails Feb 25, 2026

hubertzub-db commented Feb 25, 2026

View reviewed changes

hubertzub-db requested a review from bbqiu February 25, 2026 10:54

hubertzub-db force-pushed the fix-stream-fallback branch from cd37de4 to b4de3b8 Compare February 25, 2026 11:12

bbqiu reviewed Feb 26, 2026

View reviewed changes

hubertzub-db force-pushed the fix-stream-fallback branch from 370f32b to 5b66ad9 Compare February 26, 2026 11:56

bbqiu self-requested a review February 26, 2026 21:09

hubertzub-db force-pushed the fix-stream-fallback branch from 5b66ad9 to f262d66 Compare February 27, 2026 07:15

hubertzub-db added 5 commits March 5, 2026 13:17

stream fallback

801cab8

Signed-off-by: Hubert Zub <hubert.zub@databricks.com>

Fail only at the stream beginning

fe6040d

Signed-off-by: Hubert Zub <hubert.zub@databricks.com>

Tests

c1b3d2d

Signed-off-by: Hubert Zub <hubert.zub@databricks.com>

Comment revert

0d11b53

Signed-off-by: Hubert Zub <hubert.zub@databricks.com>

Adjust traceIds

940cdb3

Signed-off-by: Hubert Zub <hubert.zub@databricks.com>

hubertzub-db force-pushed the fix-stream-fallback branch from 9b9b671 to 940cdb3 Compare March 5, 2026 12:18

hubertzub-db requested review from bbqiu and removed request for bbqiu March 5, 2026 12:18

bbqiu reviewed Mar 6, 2026

View reviewed changes

Comment thread e2e-chatbot-app-next/server/src/routes/chat.ts

bbqiu reviewed Mar 6, 2026

View reviewed changes

Rework

5aa65b1

Signed-off-by: Hubert Zub <hubert.zub@databricks.com>

hubertzub-db requested review from bbqiu and fanzeyi March 6, 2026 14:40

Lock

c7b1b65

Signed-off-by: Hubert Zub <hubert.zub@databricks.com>

bbqiu approved these changes Mar 12, 2026

View reviewed changes

Dynamic

c954777

Signed-off-by: Hubert Zub <hubert.zub@databricks.com>

hubertzub-db merged commit 99b58c6 into databricks:main Mar 12, 2026

Conversation

hubertzub-db commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Demo/test (simulated error in agent's stream_handler)

Regression test (streaming still works)

Demo of complete failure (both stream_handler and invoke_handler in agent are failing)

Test plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hubertzub-db Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bbqiu left a comment

Choose a reason for hiding this comment

Uh oh!

hubertzub-db commented Feb 27, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bbqiu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bbqiu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hubertzub-db commented Mar 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hubertzub-db commented Feb 25, 2026 •

edited

Loading

Demo/test (simulated error in agent's `stream_handler`)

Demo of complete failure (both `stream_handler` and `invoke_handler` in agent are failing)

hubertzub-db Feb 26, 2026 •

edited

Loading

bbqiu left a comment •

edited

Loading