feat(plugins): add Gnani plugin#2004
Conversation
🦋 Changeset detectedLatest commit: ee097ad The changes in this PR will be included in the next version bump. This PR includes changesets to release 37 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
|
|
||
| const audioBytes = Buffer.from(await response.arrayBuffer()); | ||
| const requestId = shortuuid(); | ||
| putFrames(this.queue, framesFromAudio(audioBytes, this.opts), requestId, requestId); |
There was a problem hiding this comment.
🔴 REST text-to-speech produces corrupted audio because the file header is not stripped
The audio response is fed directly into the frame decoder (framesFromAudio(audioBytes, this.opts) at plugins/gnani/src/tts.ts:290) without first removing the container header, so the header bytes are treated as audio samples and produce a noise burst at the start of every synthesized utterance.
Impact: Every REST-mode TTS synthesis (the default mode) begins with audible corruption.
WAV header not stripped in REST path but correctly stripped in SSE and WebSocket paths
The decodeAudioChunk() function at plugins/gnani/src/tts.ts:166-168 strips the 44-byte WAV header when container === 'wav' (the default). The SSE path correctly calls decodeAudioChunk() before framesFromAudio() at plugins/gnani/src/tts.ts:342, and the WebSocket path does the same at plugins/gnani/src/tts.ts:481 and plugins/gnani/src/tts.ts:493. However, the REST path at line 290 skips decodeAudioChunk() entirely, passing raw response bytes (including the WAV header) directly to framesFromAudio(). The AudioByteStream inside framesFromAudio() (plugins/gnani/src/tts.ts:170-173) interprets every byte as PCM sample data, so the 44-byte RIFF/WAV header is decoded as audio samples, producing a short burst of noise/distortion at the beginning of the output.
| putFrames(this.queue, framesFromAudio(audioBytes, this.opts), requestId, requestId); | |
| putFrames(this.queue, framesFromAudio(decodeAudioChunk(audioBytes, this.opts), this.opts), requestId, requestId); |
Was this helpful? React with 👍 or 👎 to provide feedback.
| try { | ||
| await Promise.race([this.sendAudio(ws), this.receiveMessages(ws)]); | ||
| } finally { | ||
| ws.close(); | ||
| } |
There was a problem hiding this comment.
🔍 STT streaming race between send and receive may drop final transcripts
In SpeechStream.run() at plugins/gnani/src/stt.ts:299, sendAudio and receiveMessages are raced with Promise.race. When the audio input is exhausted, sendAudio completes and run() enters the finally block which calls ws.close(). This closes the WebSocket before receiveMessages has a chance to process any final transcript messages the server may still be sending. If the Gnani server sends the final transcript after receiving all audio but before the client-side close, those transcripts could be lost. Other STT plugins (e.g. Deepgram) typically send an explicit close/finalize message and wait for the server's response before closing. This depends on the Gnani WebSocket protocol — if the server always sends transcripts before the client finishes sending audio, this is fine.
Was this helpful? React with 👍 or 👎 to provide feedback.
Summary
Ports livekit/agents#6004 for Gnani into agents-js by adding a new
@livekit/agents-plugin-gnanipackage, since the target repo had no existing Gnani counterpart.format,preferredLanguage, anditnNativeNumeralssupport.SynthesizeStream.Source diff coverage
File-by-file classification
livekit-plugins/livekit-plugins-gnani/README.mdplugins/gnani/README.md; content ported and adapted from Python install/import examples and snake_case options to JS package/imports and camelCase options.livekit-plugins/livekit-plugins-gnani/livekit/plugins/gnani/__init__.pyplugins/gnani/src/index.ts; maps Python exports/plugin registration to JS exports andPlugin.registerPlugin(). Source__all__parity kept forSTT,SpeechStream,TTS, andSynthesizeStream.livekit-plugins/livekit-plugins-gnani/livekit/plugins/gnani/log.pylog()helper directly inplugins/gnani/src/stt.tsandplugins/gnani/src/tts.ts.livekit-plugins/livekit-plugins-gnani/livekit/plugins/gnani/stt.pyplugins/gnani/src/stt.ts; ports REST form fields, WebSocket headers, sample-rate validation, API-key env fallback, model/provider properties, and stream language inheritance. Adapted from Pythonaiohttp/websocketsand snake_case args to JSfetch/wsand camelCase options.livekit-plugins/livekit-plugins-gnani/livekit/plugins/gnani/tts.pyplugins/gnani/src/tts.ts; ports voice/sample-rate validation, bitrate payload support, REST/SSE/WebSocket synthesis modes,stream()WebSocket synthesis, andupdateOptions()behavior. Adapted to JStts.ChunkedStream/tts.SynthesizeStreamAPIs.livekit-plugins/livekit-plugins-gnani/pyproject.tomlplugins/gnani/package.json,plugins/gnani/tsconfig.json,plugins/gnani/tsup.config.ts,plugins/gnani/api-extractor.json,pnpm-lock.yaml, and.changeset/green-gnani-speak.md; maps Python package metadata/dependencies/version bump to JS workspace package metadata, build metadata, lockfile, and minor changeset for a new public package.tests/test_plugin_gnani_stt.pyplugins/gnani/src/stt.test.ts; ports the source configuration and stream-construction tests to Vitest with JS option names and hermeticwsstubs.tests/test_plugin_gnani_tts.pyplugins/gnani/src/tts.test.ts; ports the source configuration, routing, update, bitrate, sample-rate, and stream-construction tests to Vitest with JS option names and hermeticfetch/wsstubs.Infrastructure gap
The target repo lacked any Gnani plugin package. This PR ports that missing infrastructure as
plugins/gnanirather than dropping the behavior.Validation
pnpm vitest run plugins/gnanipnpm --filter @livekit/agents-plugin-gnani lintpnpm --filter @livekit/agents-plugin-gnani buildpnpm --filter @livekit/agents-plugin-gnani api:updatepnpm --filter @livekit/agents-plugin-gnani api:checkpnpm buildpnpm lint(passes; existing warnings in unrelated packages remain)Ported from livekit/agents#6004
Original PR description