Skip to content

feat(google): support gemini-3.5-live-translate-preview via translation_config#6358

Open
sahal200 wants to merge 2 commits into
livekit:mainfrom
sahal200:feat/google-realtime-live-translate
Open

feat(google): support gemini-3.5-live-translate-preview via translation_config#6358
sahal200 wants to merge 2 commits into
livekit:mainfrom
sahal200:feat/google-realtime-live-translate

Conversation

@sahal200

@sahal200 sahal200 commented Jul 8, 2026

Copy link
Copy Markdown

Summary

Google launched Gemini 3.5 Live Translate (gemini-3.5-live-translate-preview), a low-latency speech-to-speech translation model on the Live API. Using it requires setting translation_config on LiveConnectConfig, which the Google plugin's RealtimeModel previously had no way to pass.

This PR:

  • adds a translation_config: NotGivenOr[types.TranslationConfig] parameter to RealtimeModel (threaded through _RealtimeOptions into _build_connect_config(), following the same pattern as realtime_input_config / context_window_compression)
  • adds gemini-3.5-live-translate-preview to the LiveAPIModels Literal and to KNOWN_GEMINI_API_MODELS (it is a Gemini API model, so the existing model/API mismatch validation applies)
  • documents in the docstring that live-translate models are audio-to-audio only (AUDIO response modality) and don't support tools/system instructions the way conversational Live models do

The required types.TranslationConfig and LiveConnectConfig.translation_config are available in google-genai 2.8.0 (the version currently locked in uv.lock; the plugin requires google-genai >= 1.67).

Usage

from google.genai import types
from livekit.plugins import google

model = google.realtime.RealtimeModel(
    model="gemini-3.5-live-translate-preview",
    translation_config=types.TranslationConfig(
        target_language_code="es-US",
        echo_target_language=True,
    ),
)

Input language is auto-detected; audio is 16kHz PCM in / 24kHz PCM out, matching what the plugin already sends/expects.

Testing

  • uv run ruff format / uv run ruff check β€” clean
  • uv run python scripts/check_types.py (mypy strict) β€” no issues
  • uv run pytest --unit -k google β€” 34 passed
  • verified by import + construction: RealtimeModel(model="gemini-3.5-live-translate-preview", translation_config=types.TranslationConfig(target_language_code="es-US", echo_target_language=True)) constructs, and _build_connect_config() produces a LiveConnectConfig carrying the translation config

Note: I don't currently have access to the preview model, so this is not verified against the live API end-to-end β€” the change is config plumbing only and existing behavior is unchanged when translation_config is not given.

@sahal200 sahal200 requested a review from a team as a code owner July 8, 2026 23:19
@CLAassistant

CLAassistant commented Jul 8, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

devin-ai-integration[bot]

This comment was marked as resolved.

@sahal200 sahal200 force-pushed the feat/google-realtime-live-translate branch from 83e1239 to 110f7c3 Compare July 8, 2026 23:29

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 new potential issues.

Open in Devin Review

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

πŸ” Initial chat context is still sent to translate models on session connect

When mutable_chat_context is False (as it is for translate models), _build_connect_config sets history_config=HistoryConfig(initial_history_in_client_content=True) (realtime_api.py:1162-1164). The _main_task then unconditionally sends any stored chat context via send_client_content (realtime_api.py:908-912). For translate models that are audio-to-audio only, sending text-based chat history could cause an API error if the chat context is non-empty. In practice this is unlikely since the framework checks mutable_chat_context before calling update_chat_ctx in most paths, and a fresh session starts with an empty context. This follows the same pre-existing pattern as 3.1 models, but it's worth verifying that the translate API gracefully handles an empty initial_history_in_client_content config.

(Refers to lines 1162-1164)

Open in Devin Review

Was this helpful? React with πŸ‘ or πŸ‘Ž to provide feedback.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

πŸ” Translate models still accumulate local chat history from generations

When a generation completes, _mark_current_generation_done (realtime_api.py:1336-1384) unconditionally appends input transcription and output text to self._chat_ctx. For translate models, this means the local chat context grows over time even though it's never sent to the API (since mutable_chat_context=False prevents mid-session updates). On session reconnect, this accumulated context would be sent via send_client_content in _main_task (realtime_api.py:908-912), which could cause issues for a translate-only model. This is the same behavior as 3.1 models and is unlikely to cause problems in practice since translate sessions are typically short-lived, but it's worth being aware of.

(Refers to lines 1336-1384)

Open in Devin Review

Was this helpful? React with πŸ‘ or πŸ‘Ž to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants