feat(tts): unified <expr/> expression marker dialect + transcript pacing fix#1993
feat(tts): unified <expr/> expression marker dialect + transcript pacing fix#1993toubatbrian wants to merge 4 commits into
Conversation
…ing fix Port of livekit/agents#6347 and livekit/agents#6350 on top of the expressive-mode port (#1982). expr marker dialect (#6347): replace the provider-native markup dialects with a single marker tag the LLM is taught everywhere — <expr type="expression|break|sound|prosody|spell" label="..."/>. Shared syntax, per-provider instruction blocks advertising only the types and label vocabularies each provider supports. convertMarkup lowers expr to native syntax before synthesis (dropping unsupported types while keeping the words); normalizeMarkup closes unclosed self-closing markers; the transcript strippers remove expr in a dedicated pre-pass so type/label surfaces as an ExpressiveTag. The six preset bodies are rewritten in expr; native tag tables remain solely as tolerance for hallucinated native tags. pacing fix (#6350): a markup tag with spaces in its attributes is shredded across the synchronizer's word tokens, so the per-token splitAllMarkup never saw a complete tag and paced every fragment as if spoken — delaying the transcript seconds per expressive sentence. Give each segment a stateful TranscriptMarkupStripper (fed the raw pushedText slices so attribute whitespace survives) so tag fragments contribute zero hyphens to pacing while raw tokens are still forwarded unchanged downstream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015im83A991642dZS2Ri9iv1
🦋 Changeset detectedLatest commit: b24acca The changes in this PR will be included in the next version bump. This PR includes changesets to release 36 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| /** @internal Exported for testing purposes. */ | ||
| export class SegmentSynchronizerImpl { |
There was a problem hiding this comment.
🔍 SegmentSynchronizerImpl is now exported — public API surface change
The class SegmentSynchronizerImpl was changed from a file-private class to an exported one (agents/src/voice/transcription/synchronizer.ts:146) with the @internal JSDoc tag. This is done to enable the new test file (synchronizer_markup.test.ts) to directly instantiate it. While marked @internal, if this package uses API Extractor, this export may need to be reflected in the API declarations (pnpm api:update). The defaultTextSyncOptions export at line 577 is also new.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
No API surface change: voice/transcription/index.ts only re-exports _utils and textTransforms, so synchronizer.js module exports (including SegmentSynchronizerImpl) never reach the package entry point — same pattern as the existing SpeakingRateData test export in this file. defaultTextSyncOptions is not new in this PR; it exists on the base branch. api:check output is unchanged.
Generated by Claude Code
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6f1c0de71b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // a wrapping marker (prosody/spell) and its span; non-greedy, instructed not to nest | ||
| const EXPR_WRAP_RE = /<expr\b(?=[^>]*type="(?:prosody|spell)")([^>]*?)>([\s\S]*?)<\/expr\s*>/g; | ||
| // a non-wrapping type the LLM forgot to self-close (normalizeMarkup fixes these) | ||
| const EXPR_UNCLOSED_RE = /(<expr\b(?=[^>]*type="(?:expression|break|sound)")[^>]*[^/>\s])\s*>/g; |
There was a problem hiding this comment.
For Cartesia, prosody is advertised above as a self-closing marker, but this normalization regex only fixes expression, break, and sound. If the model makes the same missing-slash mistake for Cartesia prosody, e.g. <expr type="prosody" label="slow"> One moment, normalizeMarkup('cartesia', ...) leaves it as an unpaired opening tag and convertExpr later removes it in the stray-tag cleanup, so the intended speed/volume control never reaches Cartesia. The self-closing normalization needs to cover Cartesia prosody without changing xAI's wrapping prosody behavior.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 308775a. normalizeMarkup('cartesia', ...) now also closes an unclosed prosody marker (prosody is a self-closing point control there), so <expr type="prosody" label="slow"> One moment lowers to <speed ratio="0.85"/> One moment instead of being dropped by the stray-tag cleanup. xAI keeps the narrower regex since its prosody legitimately wraps. Test added in expr_markup.test.ts.
Note: this behavior matches the merged Python implementation (_EXPR_UNCLOSED_RE covers only expression/break/sound there too), so the same fix is a candidate for upstreaming to livekit/agents.
Generated by Claude Code
| return [clean, rawTags.map(([tag, value]) => ({ type: tag, value }))]; | ||
| const [exprClean, exprTags] = splitExpr(text); | ||
| const [clean, rawTags] = extractAndStrip(exprClean, { xmlTags: ALL_MARKUP_TAGS, brackets: true }); | ||
| return [clean, [...exprTags, ...rawTags.map(([tag, value]) => ({ type: tag, value }))]]; |
There was a problem hiding this comment.
When a transcript contains both expr and tolerated native markup, this appends every expr tag before every native/bracket tag instead of preserving document order. For example, <emotion value="sad"/><expr type="expression" label="happy"/>Hi yields tags in the order happy then sad; RoomIO then calls expressionAttribute(tags) and publishes lk.expression for happy even though the leading tag was sad. This matters in the native-hallucination path this commit keeps supporting, so the two stripping passes need to merge tags by their original offsets.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 308775a. splitExpr now records each marker's offset and removed spans, extractAndStrip optionally reports match offsets, and splitMarkup/splitAllMarkup merge the two passes by position in the original text. Your example now yields [sad, happy] and lk.expression publishes sad. Offsets for tags exposed by unwrapping nested hallucinated markup (fixed-point passes) are approximate but order-monotonic in practice. Tests added in expr_markup.test.ts.
Note: the merged Python implementation has the same ordering behavior (expr_tags + raw_tags), so this fix is a candidate for upstreaming to livekit/agents.
Generated by Claude Code
…ent order Two review findings, both also present in the Python source (candidates for upstreaming to livekit/agents): - normalizeMarkup only fixed unclosed expression/break/sound markers. For Cartesia, prosody is a self-closing point control, so a missing slash left an unpaired opening tag that convertExpr's stray-tag cleanup then dropped — the speed/volume control never reached the TTS. Include prosody in the unclosed fix for Cartesia only; xAI's wrapping prosody stays an opening tag. - splitMarkup/splitAllMarkup appended every expr tag before every native or bracket tag, so a segment opening with a hallucinated native tag published the wrong leading expression via lk.expression. splitExpr now records each marker's offset and removed spans, extractAndStrip optionally reports match offsets, and the two passes merge by position in the original text. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015im83A991642dZS2Ri9iv1
Explicit-dispatch example agent (agentName: expressive-agent-js) wiring inference STT/LLM with inworld TTS and the CASUAL expressive preset, for driving voice-mode e2e assertions against the expr marker dialect with cue-cli. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015im83A991642dZS2Ri9iv1
3 participants
Description
Stacked on #1982 (expressive mode). Ports two follow-up PRs from the Python repo:
<expr/>expression marker dialectexpr marker dialect (#6347): replaces the provider-native markup dialects with a single marker tag — the only dialect the LLM is ever taught, in both
llmInstructions()and the expressive preset bodies:The marker syntax is shared, but each provider gets its own instruction block advertising only the types and label vocabularies it actually supports (Cartesia: discrete emotion vocabulary + point-control prosody + spell, no sounds; Inworld: free-form expression + its sound list + break, no prosody; xAI: its sound cues + break + wrapping prosody, no free-form expression).
convertMarkuplowers expr to each provider's native syntax before synthesis and drops unsupported types — the words always survive, the marker never leaks.normalizeMarkupcloses unclosed self-closing markers, and the transcript strippers remove expr in a dedicated pre-pass so the type/label pair surfaces as anExpressiveTag. The provider-native tag tables remain solely as tolerance for hallucinated native tags.pacing fix (#6350): a markup tag with spaces in its attributes (e.g.
<expr type="expression" label="warm surprise"/>) is shredded across the synchronizer's word tokens, so the per-tokensplitAllMarkupnever saw a complete tag and paced every fragment as if it were spoken — delaying the transcript several seconds per expressive sentence, compounding over the turn. Each segment now gets a statefulTranscriptMarkupStripper(already used by the room output for the same cross-chunk problem) so tag fragments are held until the tag completes and contribute zero hyphens to pacing. Raw tokens are still forwarded unchanged for downstream stripping andlk.expressionextraction.Changes Made
agents/src/tts/_provider_format.ts: per-provider expr instruction blocks replace the native dialects; six preset bodies rewritten in expr;splitExpr/convertExprpre-passes wired intosplitMarkup/splitAllMarkup/normalizeMarkup/convertMarkupagents/src/voice/transcription/synchronizer.ts: statefulTranscriptMarkupStripperfor pacing, fed the rawpushedTextslices so attribute whitespace survives token shredding (JS word tokens don't retain format, unlike Python'sretain_format=True);SegmentSynchronizerImplexported as@internalfor the regression testagents/src/tts/expr_markup.test.ts: port oftests/test_expr_markup.py(33 tests) plus tests for the review fixes belowagents/src/voice/transcription/synchronizer_markup.test.ts: port oftests/test_transcript_sync_markup.py— fails at ~13s / passes at ~2.6s with the fixagents/src/tts/markup_utils.test.ts: xAI assertions updated to the expr dialect (mirrors upstreamtest_tokenizer_xml_markup.pyupdates)examples/src/expressive_agent.ts: explicit-dispatch harness agent (expressive-agent-js) for cue-cli voice-mode e2eAll instruction and preset strings were verified byte-for-byte identical to the merged Python sources.
Review follow-ups
Fixed — intentional deviations from Python (bugs verified against
livekit/agentsmain; candidates for upstreaming). Public function signatures are unchanged across frameworks:normalizeMarkup('cartesia', ...)also closes an unclosed prosody marker (<expr type="prosody" label="slow">→.../>): Cartesia prosody is a self-closing point control, and the missing-slash form was silently dropped by the stray-tag cleanup instead of lowering to<speed>/<volume>. xAI's wrapping prosody is untouched.splitMarkup/splitAllMarkupmerge expr tags and tolerated native/bracket tags by their position in the original text instead of appending all expr tags first, so a segment opening with a hallucinated native tag no longer publishes the wrong leading expression vialk.expression.Pre-Review Checklist
Testing
pnpm build,pnpm test,pnpm lint,pnpm format:check— the only failures are pre-existing, environment-dependent suites that also fail on the base branch: example agents/cerebras needing API keys, silero model download, and the pre-existingapi:checktoolchain issue)expressive-agent-js, TCP transport, inference gatewaydeepgram/nova-3+openai/gpt-4.1-mini+inworld/inworld-tts-2,presets.CASUAL):getWeathertool turn all resolved<expr type="expression" label="speak with bright energy, fast and warm"/> ... <expr type="sound" label="laugh"/>) — labels with spaces, the exact shape the pacing fix targets<expression value=/<emotion value=) in any turninference.LLMAPITimeoutError mid-session; retried and recovered (infra blip, unrelated to this change)restaurant_agent.tsandrealtime_agent.tswork properly (for major changes)Additional Notes
The pacing regression test drives
SegmentSynchronizerImpldirectly (like the Python test drives_SegmentSynchronizerImpl); since JS paces by wall-clock hyphens rather than a speaking-rate detector, the test asserts the drain time against the visible-word budget instead of an audio duration. The room transcription pacing path itself isn't reachable over the cue-cli TCP transport, so #6350's behavior is covered by the deterministic regression test.