Skip to content

feat(google): support gemini-3.5-live-translate-preview via translation_config #6358

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sahal200 wants to merge 2 commits into
base: main
Choose a base branch
from
+34 −7
Open

feat(google): support gemini-3.5-live-translate-preview via translation_config #6358

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
110f7c3
feat(google): support gemini-3.5-live-translate-preview via translati…
sahal200 Jul 8, 2026
1c8e106
fix(google): skip instructions, tools, and voice config for live-tran…
sahal200 Jul 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions livekit-plugins/livekit-plugins-google/livekit/plugins/google/realtime/api_proto.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
# Gemini API models
"gemini-3.1-flash-live-preview",
"gemini-2.5-flash-native-audio-preview-12-2025", # https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-live
"gemini-3.5-live-translate-preview", # https://ai.google.dev/gemini-api/docs/live-api/live-translate
]

Voice = Literal[
Expand Down
40 changes: 33 additions & 7 deletions livekit-plugins/livekit-plugins-google/livekit/plugins/google/realtime/realtime_api.py
View file
Open in desktop
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.

@devin-ai-integration devin-ai-integration Bot Jul 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Initial chat context is still sent to translate models on session connect

When mutable_chat_context is False (as it is for translate models), _build_connect_config sets history_config=HistoryConfig(initial_history_in_client_content=True) (realtime_api.py:1162-1164). The _main_task then unconditionally sends any stored chat context via send_client_content (realtime_api.py:908-912). For translate models that are audio-to-audio only, sending text-based chat history could cause an API error if the chat context is non-empty. In practice this is unlikely since the framework checks mutable_chat_context before calling update_chat_ctx in most paths, and a fresh session starts with an empty context. This follows the same pre-existing pattern as 3.1 models, but it's worth verifying that the translate API gracefully handles an empty initial_history_in_client_content config.

(Refers to lines 1162-1164)

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@devin-ai-integration devin-ai-integration Bot Jul 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Translate models still accumulate local chat history from generations

When a generation completes, _mark_current_generation_done (realtime_api.py:1336-1384) unconditionally appends input transcription and output text to self._chat_ctx. For translate models, this means the local chat context grows over time even though it's never sent to the API (since mutable_chat_context=False prevents mid-session updates). On session reconnect, this accumulated context would be sent via send_client_content in _main_task (realtime_api.py:908-912), which could cause issues for a translate-only model. This is the same behavior as 3.1 models and is unlikely to cause problems in practice since translate sessions are typically short-lived, but it's worth being aware of.

(Refers to lines 1336-1384)

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@
{
"gemini-3.1-flash-live-preview",
"gemini-2.5-flash-native-audio-preview-12-2025",
"gemini-3.5-live-translate-preview",
}
)

Expand Down Expand Up @@ -154,6 +155,7 @@ class _RealtimeOptions:
tool_choice: NotGivenOr[llm.ToolChoice | None] = NOT_GIVEN
thinking_config: NotGivenOr[types.ThinkingConfig] = NOT_GIVEN
session_resumption: NotGivenOr[types.SessionResumptionConfig] = NOT_GIVEN
translation_config: NotGivenOr[types.TranslationConfig] = NOT_GIVEN
credentials: google.auth.credentials.Credentials | None = None


Expand Down Expand Up @@ -223,6 +225,7 @@ def __init__(
http_options: NotGivenOr[types.HttpOptions] = NOT_GIVEN,
media_resolution: NotGivenOr[types.MediaResolution] = NOT_GIVEN,
thinking_config: NotGivenOr[types.ThinkingConfig] = NOT_GIVEN,
translation_config: NotGivenOr[types.TranslationConfig] = NOT_GIVEN,
credentials: google.auth.credentials.Credentials | None = None,
) -> None:
"""
Expand Down Expand Up @@ -263,6 +266,12 @@ def __init__(
tool_response_scheduling (FunctionResponseScheduling, optional): The scheduling for tool response. Default scheduling is WHEN_IDLE.
session_resumption (SessionResumptionConfig, optional): The configuration for session resumption. Defaults to None.
thinking_config (ThinkingConfig, optional): Native audio thinking configuration.
translation_config (TranslationConfig, optional): Speech translation configuration, required by live-translate models
(e.g. "gemini-3.5-live-translate-preview"). Sets the target language (BCP-47 code) via `target_language_code` and
whether to also play back speech in the target language via `echo_target_language`. The input language is detected
automatically. Note that live-translate models are audio-to-audio only (AUDIO response modality) and do not support
tools or system instructions the way conversational Live models do.
See https://ai.google.dev/gemini-api/docs/live-api/live-translate. Defaults to None.
conn_options (APIConnectOptions, optional): The configuration for the API connection. Defaults to DEFAULT_API_CONNECT_OPTIONS.

Raises:
Expand Down Expand Up @@ -293,7 +302,10 @@ def __init__(
else "gemini-2.5-flash-native-audio-preview-12-2025"
)

mutable = "3.1" not in model
# live-translate models are audio-to-audio only and do not accept system
# instructions or tools, so their chat context / instructions are not mutable.
is_translate = "live-translate" in model
mutable = "3.1" not in model and not is_translate
super().__init__(
capabilities=llm.RealtimeCapabilities(
message_truncation=False,
Expand Down Expand Up @@ -381,6 +393,7 @@ def __init__(
media_resolution=media_resolution,
thinking_config=thinking_config,
session_resumption=session_resumption,
translation_config=translation_config,
credentials=credentials,
)

Expand Down Expand Up @@ -1130,10 +1143,19 @@ async def _recv_task(self, session: AsyncSession) -> None:
def _build_connect_config(self) -> types.LiveConnectConfig:
temp = self._opts.temperature if is_given(self._opts.temperature) else None

tools_config = create_tools_config(
self._tools,
tool_behavior=self._opts.tool_behavior,
use_parameters_json_schema=False,
# live-translate models are audio-to-audio only: they translate speech via
# translation_config and do not accept system instructions, tools, or a
# prebuilt voice/speech config. Omit those fields so the API doesn't reject them.
is_translate = "live-translate" in self._opts.model

tools_config = (
None
if is_translate
else create_tools_config(
self._tools,
tool_behavior=self._opts.tool_behavior,
use_parameters_json_schema=False,
)
)
conf = types.LiveConnectConfig(
response_modalities=self._opts.response_modalities,
Expand Down Expand Up @@ -1162,9 +1184,11 @@ def _build_connect_config(self) -> types.LiveConnectConfig:
else None,
),
system_instruction=types.Content(parts=[types.Part(text=self._opts.instructions)])
if is_given(self._opts.instructions)
if is_given(self._opts.instructions) and not is_translate
else None,
speech_config=types.SpeechConfig(
speech_config=None
if is_translate
else types.SpeechConfig(
voice_config=types.VoiceConfig(
prebuilt_voice_config=types.PrebuiltVoiceConfig(voice_name=self._opts.voice)
),
Expand All @@ -1186,6 +1210,8 @@ def _build_connect_config(self) -> types.LiveConnectConfig:
conf.realtime_input_config = self._opts.realtime_input_config
if is_given(self._opts.context_window_compression):
conf.context_window_compression = self._opts.context_window_compression
if is_given(self._opts.translation_config):
conf.translation_config = self._opts.translation_config
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.

return conf

Expand Down
Loading