feat(aws): add AWS plugin (Bedrock LLM, Transcribe STT, Polly TTS)#1998
feat(aws): add AWS plugin (Bedrock LLM, Transcribe STT, Polly TTS)#1998smorimoto wants to merge 13 commits into
Conversation
Add @livekit/agents-plugin-aws, providing: - LLM via Amazon Bedrock Converse (streaming, tool calling, prompt caching) - STT via Amazon Transcribe streaming (word-level timestamps, channel identification, idle-timeout reconnects) - TTS via Amazon Polly (PCM output, since the framework has no mp3 decoder) Also adds an 'aws' provider format to @livekit/agents for converting ChatContext into Bedrock Converse messages. Nova Sonic realtime is out of scope for this change.
🦋 Changeset detectedLatest commit: d98d3e1 The changes in this PR will be included in the next version bump. This PR includes changesets to release 37 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
flush() ran before write(audioBytes), always draining an empty buffer and dropping the tail of synthesized audio when Polly's byte length isn't an exact 100ms multiple.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Map SDK status errors to APIStatusError so non-retryable 4xx inputs are not retried as connection failures, surface Transcribe event-stream exceptions, and require languageOptions when automatic language identification is enabled.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Map validationException to 400, keep modelStreamError 424 retryable, requeue audio frames across STT reconnects, validate channel options, and surface speaker labels from Transcribe.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Honour request timeouts, preserve Transcribe speaker and language metadata, and close owned AWS clients. Trim redundant tests, exclude test artefacts from the published package, and add API and CI coverage.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Clear the connection timeout once Bedrock has returned a streaming response, while preserving caller-initiated cancellation. Add regression coverage for responses that outlive the connection timeout.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Reuse the shared toAwsApiError classifier in the Bedrock LLM stream instead of duplicating its logic, cache decoded image bytes so vision turns aren't re-decoded on every request, and simplify a couple of pieces of stream-tracking state in the Transcribe and Polly plugins that never actually held more than one item.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d98d3e13c6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const description = tool.description.trim(); | ||
| return { | ||
| toolSpec: { | ||
| name, |
There was a problem hiding this comment.
When a registered tool name contains characters such as ./space or is longer than 64 characters, this serializes it directly as toolSpec.name; the framework only requires tool names to be non-empty, but Bedrock's ToolSpecification requires names to match [a-zA-Z0-9_-]+ and be at most 64 chars. In those cases any chat with that tool is rejected before inference, so validate or map AWS tool names here and apply the same mapping when replaying toolUse history.
Useful? React with 👍 / 👎.
| } from '@livekit/agents'; | ||
| import { type AwsCredentials, createRequestSignal, resolveRegion, toAwsApiError } from './utils.js'; | ||
|
|
||
| const DEFAULT_MODEL = 'amazon.nova-2-lite-v1:0'; |
There was a problem hiding this comment.
When callers rely on the advertised default model, new aws.LLM().chat(...) sends amazon.nova-2-lite-v1:0 directly to ConverseStream. Nova 2 Lite on Bedrock runtime requires an inference-profile ID such as us.amazon.nova-2-lite-v1:0/global.amazon.nova-2-lite-v1:0 for on-demand invocation in the default regions, so the zero-config example fails with a ValidationException unless users discover and override the model; default to a region-appropriate profile or to a model ID that supports on-demand throughput.
Useful? React with 👍 / 👎.
| if (!toolConfig) { | ||
| chatCtx = chatCtx.copy({ excludeFunctionCall: true }); |
There was a problem hiding this comment.
When the framework asks for a follow-up with toolChoice: 'none' after a tool has run (for example after max tool steps are reached), buildToolConfig returns undefined and this drops every function_call/function_call_output from chatCtx. The final Bedrock request therefore no longer contains the tool output it is supposed to answer from, so it can produce a generic answer unrelated to the tool result; only strip tool history when there are no previous tool blocks to replay, or use a separate mechanism to prevent additional tool calls.
Useful? React with 👍 / 👎.
1 participant
Summary
@livekit/agents-plugin-aws, a new provider plugin covering LLM (Amazon Bedrock Converse), STT (Amazon Transcribe streaming) and TTS (Amazon Polly)'aws'ProviderFormatto@livekit/agentsfor convertingChatContextinto Bedrock Converse messages, alongside the existinggoogle/mistralai/openaiformatssampleRateis restricted to those two valuesregion→AWS_REGION→AWS_DEFAULT_REGION→us-east-1examples/src/aws.ts, a plugin README, a root README table entry,turbo.jsonenv vars, and a changesetTest plan
pnpm build(all 38 packages, including examples)pnpm test— unit tests for the provider format and all three components pass without AWS credentials; live tests gate onAWS_ACCESS_KEY_ID/AWS_PROFILEand are skipped otherwisepnpm lint/pnpm format:checknode ./examples/src/aws.ts dev)