Speech to text?

[wlandau]

Are there plans for speech to text support in the user prompt UI?

[cpsievert]

It's not likely to be a priority in shinychat anytime soon, but here are other efforts in that direction

https://github.com/posit-dev/shinymedia
https://github.com/posit-dev/shinyrealtime

[JamesHWade]

I've been experimenting with adding audio input directly to shinychat's chat UI. The implementation adds an audio_input parameter to chat_ui() with two modes:

• "transcribe": Uses the browser's Web Speech API for free, client-side speech-to-text. The transcribed text is inserted into the chat input field.
• "raw": Captures audio via MediaRecorder and sends it to the server as a base64-encoded blob. This is for multimodal models like accept audio directly.

Would there be interest in this feature? Any feedback on the API design or use cases I should consider?

I expect you'd want to pair it with audio support in ellmer and chatlas.

[cpsievert]

This is definitely something we'd like to get to in 2026. I personally haven't looked much into it, but it also feels like transcription is something we may want to delegate to another tool, like:

https://github.com/jcheng5/sherpa-onnx-r
https://github.com/k2-fsa/sherpa-onnx

And yes, ideally, chatlas and ellmer would have audio support when it lands.

[JamesHWade]

Thanks for sharing those projects. Having a SOTA local model for transcription would be great.

I'm hoping we see some expanded local options from the browsers in the near future. It looks like Chrome has the beginnings of that as of a fall release.

Here's the branch where I played around with audio input a bit: https://github.com/JamesHWade/shinychat/tree/feature/audio-input

Happy to turn this into a PR if there's interest.

[jcheng5]

I'm working hard on sherpa-onnx-r, it works well for what it does but one big limitation is that for Whisper models it truncates the audio stream at 30 seconds. There are solutions to that problem but you need some timestamp data that sherpa-onnx doesn't currently support, so I'm in the hopefully final stages of a PR that adds it (k2-fsa/sherpa-onnx#2945).

Personally I have had great experiences with Whisper and Parakeet models, those are the ones I'm most excited to support.

If the Web Speech APIs are good enough, that would be great for Shiny. I'll still keep pushing ahead with sherpa-onnx-r because I really like the idea of easy access to SOTA transcription models from R regardless of operating system or CPU/GPU/NPU.