Inference: strip Gemma 4 <|channel>thought…<channel|> markers (strip_thinking_tokens only knows <think>)

[cursor]

Summary

LiveKit Inference's google/gemma-4-31b-it emits Gemma 4 channel-thought markup that leaks into customer-facing agent output (transcription / TTS / chat history). strip_thinking_tokens only strips <think>/</think>, so Gemma's markers pass through unchanged.

Markers

Per Gemma 4 model card:

• Open: <|channel>thought (often followed by \n)
• Close: <channel|>
• Body: internal reasoning (or empty when thinking is "off" — larger models still emit the empty wrapper)

Example stream content that reaches the agent:

<|channel>thought
Customer wants a $100 refund; check policy.
<channel|>I can help start a return instead.

Or empty:

<|channel>thought
<channel|>Happy to help with that.

Where it leaks

In the Agents SDK Inference LLM path, deltas go through strip_thinking_tokens before becoming ChatChunks:

https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/inference/llm.py

THINK_TAG_START = "<think>"
THINK_TAG_END = "</think>"

def strip_thinking_tokens(content: str | None, thinking: asyncio.Event) -> str | None:
    ...

Those tags don't match Gemma's channel markers, so the markup reaches llm_node → TTS / transcription_node / session history.

Expected

Inference (or strip_thinking_tokens) should treat Gemma channel-thought blocks like <think> blocks: drop the markers and any body between them so only the final answer is customer-visible. Prior-turn thoughts also must not re-enter chat history (Gemma model card: "No Thinking Content in History").

Actual

Raw <|channel>thought / <channel|> (and sometimes the thought body) appear in the LiveKit transcription stream used by text chat UIs, and would be spoken on voice.

Repro (Agents SDK + Inference)

1. AgentSession with inference.LLM(model="google/gemma-4-31b-it")
2. Multimodal/text chat so agent replies ride lk.transcription
3. Ask something that triggers a tool-using support turn (refund / damage claim)
4. Inspect the transcription / chat history for <|channel>thought / <channel|>

Workaround

App-side llm_node filter that strips the Gemma markers before chunks reach TTS/transcription/history (same state-machine pattern as the chain-of-thought recipe, with Gemma's tags).

Related

• Gemini thought-part handling: Properly handle Gemini thought summaries and export them via OpenTelemetry in Google LLM plugin #4190
• strip_thinking_tokens list-delta crash: strip_thinking_tokens crashes on list-shaped streaming delta content (AttributeError: 'list' object has no attribute 'find') #6323

[davidzhao]

Can you share the model configuration please? what is the reasoning effort set to?

[romansanchez]

I had same issue just now with 1.6.5.

llm_adapter = llm.FallbackAdapter(
        [
            inference.LLM(
                model="google/gemma-4-31b-it",
                extra_kwargs={
                    "max_completion_tokens": 1000,
                    "reasoning_effort": "none",
                },
            ),
            inference.LLM(
                model="moonshotai/kimi-k2.5",
                provider="baseten",
                extra_kwargs={"max_completion_tokens": 1000},
            )
        ]
    )

[longcw]

@romansanchez I tried to reproduce the issue but didn't see such markers, could you share an example or the output of the LLM?

[romansanchez]

@longcw sure. session options along with messages

{
   "session.options":{
      "turn_handling":{
         "endpointing":{
            "mode":"dynamic",
            "min_delay":0.3,
            "max_delay":2,
            "alpha":0.9
         },
         "interruption":{
            "enabled":true,
            "discard_audio_if_uninterruptible":true,
            "min_duration":0.5,
            "min_words":2,
            "resume_false_interruption":true,
            "false_interruption_timeout":2,
            "backchannel_boundary":[
               1,
               1
            ],
            "mode":"adaptive"
         },
         "turn_detection":"<livekit.agents.inference.eot.detector.TurnDetector object at 0x2g24bef23981>",
         "preemptive_generation":{
            "enabled":true,
            "preemptive_tts":false,
            "max_speech_duration":10,
            "max_retries":3
         },
         "user_turn_limit":{
            "max_words":100,
            "max_duration":30
         }
      },
      "keyterms_options":{
         "keyterm_detection":{
            "enabled":false,
            "turn_interval":1,
            "timeout":10
         }
      },
      "endpointing_overrides":{
         "mode":"dynamic",
         "min_delay":0.3,
         "max_delay":2
      },
      "max_tool_steps":3,
      "user_away_timeout":12.5,
      "min_consecutive_speech_delay":0,
      "tts_text_transforms":[
         "filter_markdown",
         "filter_emoji"
      ],
      "ivr_detection":false,
      "aec_warmup_duration":3,
      "session_close_transcript_timeout":2
   },
   "session.report_timestamp":1783655738.3529763,
   "agent_name":"receptionist-agent",
   "sdk_version":"1.6.5",
   "usage":[
      {
         "type":"llm_usage",
         "provider":"livekit",
         "model":"google/gemma-4-31b-it",
         "input_tokens":163476,
         "input_cached_tokens":160768,
         "output_tokens":334
      },
      {
         "type":"tts_usage",
         "provider":"livekit",
         "model":"rime/coda",
         "characters_count":1078,
         "audio_duration":68.64000000000001
      },
      {
         "type":"stt_usage",
         "provider":"livekit",
         "model":"deepgram/nova-3-general",
         "audio_duration":114.24999999999983
      },
      {
         "type":"eot_usage",
         "provider":"livekit",
         "model":"turn-detector-v1",
         "total_requests":11
      }
   ],
   "room_id":"xxxxx",
   "job_id":"xxxxx",
   "logger.name":"chat_history",
   "lk.id":"xxxxx"
}

This is the USER message right before:

{
   "chat.item":{
      "message":{
         "id":"item_xxxxx",
         "role":"USER",
         "content":[
            {
               "text":"Yeah. I wonder I wonder what is your earliest availability?"
            }
         ],
         "transcript_confidence":0.87402344,
         "metrics":{
            "started_speaking_at":"2026-07-10T03:54:29.121Z",
            "stopped_speaking_at":"2026-07-10T03:54:33.080Z",
            "transcription_delay":0.19620847702026367,
            "end_of_turn_delay":0.37667179107666016,
            "on_user_turn_completed_delay":0.00007635600002231513
         },
         "created_at":"2026-07-10T03:54:33.277Z"
      }
   },
   "room_id":"xxxxx",
   "job_id":"xxxxx",
   "logger.name":"chat_history",
   "lk.id":"xxxxx"
}

ASSISTANT message

{
   "chat.item":{
      "message":{
         "id":"item_xxxxx",
         "role":"ASSISTANT",
         "content":[
            {
               "text":"Let me check that for you. One moment.\n\n<|channel>thought\n<channel|>"
            }
         ],
         "extra":{
            "":0
         },
         "metrics":{
            "started_speaking_at":"2026-07-10T03:54:33.826Z",
            "stopped_speaking_at":"2026-07-10T03:54:37.909Z",
            "llm_node_ttft":0.3692234920000601,
            "tts_node_ttfb":0.08362462799993864,
            "e2e_latency":0.745974063873291
         },
         "created_at":"2026-07-10T03:54:33.464Z"
      }
   },
   "room_id":"xxxxx",
   "job_id":"xxxxx",
   "logger.name":"chat_history",
   "lk.id":"xxxxx"
}