fix(aws): preserve Bedrock image format#1974
Conversation
🦋 Changeset detectedLatest commit: b2a64b0 The changes in this PR will be included in the next version bump. This PR includes changesets to release 36 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| toolResult: { | ||
| toolUseId: msg.callId, | ||
| content: [{ text: msg.output }], | ||
| status: 'success', |
There was a problem hiding this comment.
🔴 Tool error results are always reported as successful to the AI model
Tool call results are always marked as successful (status: 'success' at agents/src/llm/provider_format/aws.ts:84) regardless of whether the tool actually failed, so the AI model cannot distinguish between successful and failed tool executions.
Impact: When a tool call fails, the AI model will treat the error message as a valid result, potentially producing incorrect or misleading responses.
Mechanism: isError field on FunctionCallOutput is ignored
The FunctionCallOutput type has an isError: boolean field (agents/src/llm/chat_context.ts:539). Other provider format adapters properly check this field — for example, the Google adapter at agents/src/llm/provider_format/google.ts:87 uses msg.isError ? { error: msg.output } : { output: msg.output }. The AWS Bedrock Converse API supports status: 'error' for toolResult blocks, but the AWS adapter hardcodes status: 'success' at line 84 without checking msg.isError.
| status: 'success', | |
| status: msg.isError ? 'error' : 'success', |
Was this helpful? React with 👍 or 👎 to provide feedback.
| if (injectDummyUserMessage && (messages.length === 0 || messages[0]?.role !== 'user')) { | ||
| messages.unshift({ role: 'user', content: [{ text: '(empty)' }] }); | ||
| } |
There was a problem hiding this comment.
🔍 Dummy user message injection targets the first message, unlike Google which targets the last
The injectDummyUserMessage parameter is used differently across providers. In agents/src/llm/provider_format/aws.ts:94-96, a dummy user message is prepended (unshift) to ensure the first message is from the user. In contrast, agents/src/llm/provider_format/google.ts:104-106 appends a dummy user message to ensure the last message is from the user. Both behaviors are valid for their respective APIs (Bedrock requires user-first, Gemini requires user-last), but the Bedrock Converse API also requires the last message to be from the user role. If the conversation ends with an assistant message, the current implementation won't add a trailing user message, which could cause API errors. This may be handled by the calling plugin, but it's worth verifying.
Was this helpful? React with 👍 or 👎 to provide feedback.
Summary
Tests
Notes
pnpm --filter @livekit/agents api:checkcurrently fails on existingagents/dist/index.d.tsexport namespace syntax, unrelated to this change.Ported from livekit/agents#6328
Original PR description
Fixes AWS Bedrock chat-context formatting for non-JPEG
ImageContentinputs by deriving the Bedrock image block format from the serialized image MIME type instead of always declaringjpeg.What Problem This Solves
AWS Bedrock image blocks require the declared
formatto match the image bytes being sent.LiveKit already preserves that information when it serializes an
ImageContentdata URL. For example, a PNG data URL is decoded into raw PNG bytes withmime_type="image/png":The bug is in the AWS formatter. It receives the serialized image, but always declares the Bedrock image format as
jpeg:{ "image": { "format": "jpeg", "source": {"bytes": img.data_bytes}, } }That is correct only when the serialized image is actually JPEG. For PNG, GIF, or WebP data URLs, the formatter keeps the original non-JPEG bytes but labels them as JPEG in the Bedrock request.
So the problem is not that LiveKit loses or rewrites the image bytes. The problem is that the AWS request metadata no longer matches those bytes.
AWS Bedrock supports
png,jpeg,gif, andwebpasImageBlock.formatvalues, so the formatter should use the serialized MIME type to choose the matching Bedrock format.Change
Map
SerializedImage.mime_typeto the corresponding Bedrock image format:This keeps the existing JPEG behavior for
VideoFrameinputs, becauseserialize_image()currently encodes frames as JPEG.This also keeps the current Bedrock behavior for external image URLs unchanged: external URLs are still rejected because this formatter path only builds inline Bedrock image blocks.
Evidence
A PNG
ImageContentcan currently be serialized with the correct MIME type but formatted for AWS with the wrong Bedrock format:Before this fix, the generated Bedrock message declares the image as JPEG:
{ "image": { "format": "jpeg", "source": {"bytes": b"<original png bytes>"}, } }After this fix, the declared format matches the serialized image bytes:
{ "image": { "format": "png", "source": {"bytes": b"<original png bytes>"}, } }The same mapping applies to JPEG, GIF, and WebP inputs.
AWS documents
ImageBlock.formatvalid values aspng | jpeg | gif | webp: https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ImageBlock.htmlPossible call chain / impact
The affected path is only AWS Bedrock chat-context conversion for inline image inputs.
This PR does not change:
VideoFrameencoding behaviorSibling provider surfaces already preserve image MIME type differently: OpenAI and Mistral include the MIME type in data URLs, Anthropic passes
media_type, and Google passesmime_typein image parts. AWS was the outlier because it dropped the MIME type and hardcoded the Bedrock format.Validation
uv run --no-sync pytest tests/test_chat_ctx.py -q- passed, 27 passed / 1 skippeduv run --no-sync pytest tests/test_chat_ctx.py -q -k aws_image_content- passed, 5 passeduv run --no-sync ruff check livekit-agents/livekit/agents/llm/_provider_format/aws.py tests/test_chat_ctx.py- passeduv run --no-sync ruff format --check livekit-agents/livekit/agents/llm/_provider_format/aws.py tests/test_chat_ctx.py- passedgit diff --check- passed