Skip to content

Buffer streamed chunks from stream-result endpoint#132

Merged
jakelazaroff merged 4 commits intomainfrom
jake/buffer-streamed-json
Mar 17, 2025
Merged

Buffer streamed chunks from stream-result endpoint#132
jakelazaroff merged 4 commits intomainfrom
jake/buffer-streamed-json

Conversation

@jakelazaroff
Copy link
Contributor

@jakelazaroff jakelazaroff commented Mar 17, 2025

I ran into an issue with the stream-result endpoint where a single JSON output or result can end up split between multiple "chunks" in the HTTP response stream. One easy way to reproduce this is to stream back a result with an image, which we serialize to base 64; it ends up being split between two chunks, each of which is invalid JSON.

This PR circumvents that by processing each chunk "newline segment by newline segment". There is a value that contains the current chunk, and a buffer that is assumed to contain all unfinished JSON from previous iterations. TL;DR:

  1. The SDK waits to receive a chunk, which is put into value.
  2. The SDK looks in value for the substring until the first newline.
  3. If there is in fact a newline, it appends that substring to the buffer (if any), parses as JSON and yields it. Then it sets the buffer to an empty string and the value to the remaining text in the chunk after the substring. GOTO 2.
  4. If there's no newline, it appends the substring to the buffer. Since the chunk is finished, it waits for the next chunk; GOTO 1.

@jakelazaroff jakelazaroff force-pushed the jake/buffer-streamed-json branch from 4d51dcc to 8eb9ee2 Compare March 17, 2025 22:46
@jakelazaroff jakelazaroff merged commit 2f79539 into main Mar 17, 2025
4 checks passed
@jakelazaroff jakelazaroff deleted the jake/buffer-streamed-json branch March 17, 2025 22:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants