feat(tts): Expressiveness Mode#1982
Conversation
Port of livekit/agents#6116 (expressive mode) to agents-js. Co-authored-by: Cursor <cursoragent@cursor.com>
🦋 Changeset detectedLatest commit: cf15251 The changes in this PR will be included in the next version bump. This PR includes changesets to release 36 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
toubatbrian
changed the title
toubatbrian
changed the title
CodeQL flagged js/polynomial-redos on XML_TAG_RE: the lazy [^>]*? and trailing \s* both match whitespace, so unterminated input like '<A' + '\t' * n backtracks polynomially (~18s at n=200k). Replace the ambiguous body with a single [^>]* and detect self-closing tags on the captured body instead. Matching behavior is unchanged. Co-Authored-By: Oz <oz-agent@warp.dev>
toubatbrian
changed the title
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
CodeQL still flagged the previous form: with [^>]* the body can absorb '<', so unterminated input like '<A<A<A…' rescans to end-of-string from every '<' (quadratic). [^<>] bounds each scan at the next '<'. Co-Authored-By: Oz <oz-agent@warp.dev>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
The documented presets.CUSTOMER_SERVICE / presets.CASUAL API was defined but never re-exported, so consumers couldn't reach it (the package only exposes the root entrypoint). Export it as voice.presets and fix the doc example accordingly.
Three fixes from PR review (each also present in the Python source; JS deviates intentionally, verified against livekit/agents main): - instructionsEqual: an Instructions with audio/text additions no longer compares equal to its bare common string. It renders differently, so treating them as equal made realtime session reuse skip instruction updates on handoff when the new agent added modality rules. - Instructions.resolveTemplate: the produced audio/text variants are full template renders, not additions. render(modality) now returns the variant instead of appending it to common, which duplicated the entire template in the prompt. - pipeline reply: base Instructions are re-rendered for the turn modality even when extra per-turn instructions are supplied, so a text turn no longer keeps audio-rendered base rules copied from the session chat context.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Port frontdesk example changes from livekit/agents#6116: expressive CUSTOMER_SERVICE preset with inference gateway STT/LLM/TTS (inworld tts-2), rewritten voice-first instructions, time-of-day onEnter greeting via generateReply, and idle away-state nudge loop.
Relative .js import specifiers fail when running examples directly under node type stripping (e.g. lk agent console). Switch to .ts specifiers with rewriteRelativeImportExtensions so tsc still emits .js in dist output.
| function cleanToOrig(cleanPos: number, tagSpans: [number, number][]): number { | ||
| let orig = cleanPos; | ||
| for (const [tagStart, tagEnd] of tagSpans) { | ||
| if (tagStart < orig) { | ||
| orig += tagEnd - tagStart; | ||
| } else { | ||
| break; | ||
| } | ||
| } | ||
| return orig; |
There was a problem hiding this comment.
🔍 XML-aware tokenizer offset remapping assumes non-overlapping sorted tag spans
The cleanToOrig function at token_stream.ts:75-84 maps clean-text positions back to original positions by iterating through tag spans and accumulating their lengths. It assumes tag spans are sorted by position and non-overlapping (which is guaranteed by matchAll(XML_TAG_RE) on well-formed input). However, the function does a single forward pass: after adjusting orig for a tag, subsequent tags are compared against the adjusted orig. This means if a tag at position 10 (length 20) shifts orig from 15 to 35, a tag at position 25 would be skipped (25 < 35) even though it should contribute. This could cause incorrect offset mapping when multiple tags cluster near a sentence boundary. The tests cover the common cases (tags between sentences, wrapping tags), and the splitSentences off-by-one fix at sentence.ts:85-88 addresses a related boundary issue.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
🔍 concatInstructions removed from public API — breaking change for external callers
The concatInstructions function was removed from the public exports at agents/src/llm/index.ts (previously line 49). This was a public API used to concatenate strings and Instructions while preserving both audio/text variants. The PR replaces it with the new Instructions.render() approach, but any external code calling concatInstructions will break at compile time. The concat method on Instructions and the asModality method were also removed. The changeset marks this as a patch version bump, but these are breaking API changes.
Was this helpful? React with 👍 or 👎 to provide feedback.
Port drive-thru example changes from livekit/agents#6116: expressive CUSTOMER_SERVICE preset with inference gateway STT/LLM/TTS (inworld tts-2), restructured COMMON_INSTRUCTIONS (outcome / voice / how-to-work / hard constraints), and idle away-state nudge loop.
Port survey example changes from livekit/agents#6116: expressive CASUAL preset with inference gateway STT/LLM/TTS (inworld tts-2, multi-language STT), and idle away-state nudge loop.
Port of livekit/agents examples/inference/agent.py (post-#6116): inference gateway STT/LLM/TTS (inworld tts-2 CREATIVE), expressive CASUAL preset, on_enter greeting, idle away-state nudge loop, and the playground RPC controls (set_stt_model / set_llm_model / set_tts_model / set_system_prompt / open_in_builder).
| const pattern = new RegExp(`<(${tags.join('|')})\\b([^>]*[^/])\\s*>`, 'g'); | ||
| return text.replace(pattern, '<$1$2/>'); |
There was a problem hiding this comment.
🟡 Markup normalization silently skips bare self-closing tags without attributes, leaving them unfixed for the TTS
A tag written by the LLM without any attributes (e.g. <break>) is not converted to self-closing form (<break/>) by the normalization regex (normalizeMarkup at agents/src/tts/_provider_format.ts:762), so it passes through to the TTS provider as an opening tag with no closing tag.
Impact: If the LLM emits a bare <break> (or bare <sound>, <expression>, etc.) without attributes, the tag is sent unconverted to the TTS, which may misinterpret or reject it.
Regex requires at least one non-slash character in the tag body
The regex pattern is <(tag)\b([^>]*[^/])\s*>. The [^/] alternation requires at least one character that is not /. For a bare tag like <break>, the body between \b and > is empty — zero characters — so [^>]*[^/] cannot match (the [^/] needs one character). Tags with at least one attribute character (e.g. <break time="1s">) or even a trailing space (<break >) are matched correctly.
The fix is to make the body group optional or use a negative lookahead for the self-closing slash instead, e.g. <(tag)\b([^>]*?)(?<!/)\s*>.
| const pattern = new RegExp(`<(${tags.join('|')})\\b([^>]*[^/])\\s*>`, 'g'); | |
| return text.replace(pattern, '<$1$2/>'); | |
| const pattern = new RegExp(`<(${tags.join('|')})\\b([^>]*?)(?<!/)\\s*>(?!.*<\\/\\1)`, 'g'); | |
| return text.replace(pattern, '<$1$2/>'); |
Was this helpful? React with 👍 or 👎 to provide feedback.
4 participants
Port of livekit/agents#6116 (expressive mode) to agents-js.
Summary
agents/src/tts/markup_utils.ts,_provider_format.ts): provider-keyed dialects for LLM tag instructions, transcript stripping, tag normalization, and conversion to native TTS syntax.TTSexposes amarkupaccessor; plugins override_markupProviderKey()(Cartesia, Inworld; xAI dialect built in).Instructionsredesign (agents/src/llm/chat_context.ts): unifiedcommontext with optionalaudio/textadditions and arender(modality, data)method, replacing modality variants (asModality/concatInstructionsremoved).safeRenderutility fills template placeholders.agents/src/tokenize/): sentence tokenizer treats markup tags as atomic units so tags never split across TTS chunks (xmlAware,maxTokenLength,minTokenLength).agents/src/voice/):ExpressiveOptions+ presets (presets.ts), expressive instruction injection inAgentActivity, markup stripping in room transcription output and the transcript synchronizer,lk.expressionattribute forwarding.SpeakerContext: context models can contribute LLM instructions viasttContextin audio recognition.agents/src/inference/tts.ts): sentence-level normalize/convert of markup before sending to the gateway, expressive chunking.Also fixes an off-by-one in
splitSentences(final sentence end offset) that caused the XML-aware tokenizer to split the last character into a phantom sentence, producing mid-word TTS chunks (regression test added inxml_markup.test.ts).The XML tag regex in
token_stream.tsuses a single[^<>]*body (self-closing detection on the captured text), keeping matching linear-time on untrusted LLM output (CodeQLjs/polynomial-redos). Note: the Python source (livekit/agents#6116) still carries the backtracking-prone form.Review follow-ups
Fixed (JS-only issue):
voice.presetsnamespace is now exported (presets.CUSTOMER_SERVICE/presets.CASUALwere unreachable).Fixed — intentional deviations from Python (bugs verified against
livekit/agentsmain; candidates for upstreaming):instructionsEqual: anInstructionswithaudio/textadditions no longer equals its bare common string (Python__eq__returnsTrue, which lets realtime session reuse skip instruction updates on handoff).Instructions.resolveTemplate+render(modality): variants are full template renders;rendernow returns the variant instead of appending it to common (Python duplicates the whole template:render(audio)→common + "\n\n" + audio).Instructionsare re-rendered for the turn modality even when extra per-turninstructionsare supplied (Python skips the re-render in that branch, leaving audio-rendered base rules in text turns).Examples (ports of the #6116 example updates):
drive-thru,frontdesk,survey_agent.ts,instructions_per_modality.ts; addedinference_agent.ts(playground agent with expressive CASUAL preset and RPC model controls).healthcareexample. It depends on beta workflow tasks that don't exist in JS yet (GetNameTask,GetDOBTask,GetPhoneNumberTask,GetCreditCardTask; JS has onlyTaskGroup,WarmTransferTask,EndCallTool). Should follow once the beta workflows are ported.Kept as-is (matches Python by design):
[...]bracket tags, regardless of expressive mode — documented upstream design decision (tag shapes don't occur in spoken text).expressive: truesilently resolves to disabled for non-inference-gateway TTS — same silent guard as Python.Instructionsremoved fromChatContent— intentional API change matching Python (instructions are resolved to strings before storage).Test plan
xml_markup.test.ts,markup_utils.test.ts,Instructionstests, provider format snapshotspnpm build,pnpm test,pnpm lint,pnpm api:updategreen (agents + touched plugins)getWeathertool call, non-expressive)<emotion value="..."/>tags; speed/volume/break/spell tags verified; no error events (one CLI-side 300s RPC timeout on a single giant run — driver limitation, not pipeline)