Feature Request: Utterance timestamps in the ChatContext or Transcript

[zaheerabbas-prodigal]

I want to capture the word level or atleast utterance level timestamps for both the user and agent transcripts and store the timestamp details in the ChatContext. The usecase is to use the ChatContext as the transcript to run some post-processing. Example use cases- displaying the calls in UI, redaction, summarization etc.

Currently the transcription data the SDK exposes for the user transcript is just the text part of the transcript and the utterance or word level timestamps are NOT exposed at all from the SDK.

The agent's transcript however does not even have the timestamps even though elevenlabs and cartesia TTS support timestamps in their API.

Has anyone tried a way to get these timestamp data from the livekit-agents SDK?

I am happy to submit a PR to add this feature. Would the PR be merged if I added this feature?

[hsjun99]

@zaheerabbas-prodigal How have you ended up regarding this issue?

[longcw]

It's added in #2580, it supports the word level timestamp for cartesia TTS or sentence level timestamp for non-streaming TTS using StreamAdapter.

We are not adding the timed transcripts to the chat context, but you can access the it from the transcription_node of the agent when it's supported by the TTS.

[hsjun99]

@longcw Great news thanks. Would it also work for elevenlabs?
Also, what is the best strategy to create a word-level timestamp(relative X, absolute O) for the user transcript?

[longcw]

elevenlabs is not supported right now as we are using the websocket api and they only support timestamps in post api https://elevenlabs.io/docs/api-reference/text-to-speech/stream-with-timestamps

[hsjun99]

@longcw Isn't the normalizedAlignment/Alignment field that provides the timestamps?

[longcw]

oh yeah I'll take a look

[zaheerabbas-prodigal]

Thanks for adding this functionality - @longcw

We are not adding the timed transcripts to the chat context

Any reason for not adding this to ChatContext?

Also - does the above PR handle the word level timestamps for STT (Deepgram plugin) to be accessible in the transcription_node?

[longcw]

@hsjun99 elevenlabs TTS is supported.

[longcw]

@zaheerabbas-prodigal

does the above PR handle the word level timestamps for STT (Deepgram plugin) to be accessible in the transcription_node?

no, the user transcript from STT won't go through the transcription_node, you can obtain them from the user_input_transcribed event. but we may add the word timestamps to that event if STT supports.

[LGR3VE]

Hey @longcw, do you have any updates on this or any idea how to work around it? My application requires at least the sentence level timestamps both for the agent and user transcripts (if supported by STT).

-> Similar to the request #1670

[MINEGHOST007]

I have read through the issue the issue seems resolved can anyone give me a summary of implementation of this

I need start and end time of every conversation item added event