Add voice interface for Raspberry Pi

[patmakesapps]

Problem

On a Raspberry Pi, a voice-controlled assistant would enable hands-free coding. Also great for accessibility.

Proposed Solution

• Speech-to-text (whisper.cpp, Vosk)
• Text-to-speech for responses
• Wake word detection
• Accessible interface

Acceptance Criteria

• Voice input is transcribed and sent to the agent
• Responses can be spoken aloud
• Wake word triggers listening
• Works on Raspberry Pi hardware

[m13v]

built voice input for a macOS AI agent and the model selection matters a lot on constrained hardware like the Pi. whisper.cpp tiny.en is probably the sweet spot, it runs at about 10x realtime on a Pi 4 which is acceptable for push-to-talk but too slow for continuous listening. base.en is noticeably more accurate but drops to maybe 3-4x realtime on the Pi.

for the wake word detection, keep it separate from the whisper transcription pipeline. running whisper continuously to detect a wake word burns too much CPU. something lightweight like openWakeWord or Porcupine runs in a fraction of the resources and only triggers the heavy whisper inference when the wake word fires. the tricky part is the handoff: you need to keep a small audio buffer (2-3 seconds) before the wake word so whisper can transcribe the command that comes right after it, otherwise you lose the first word.

Vosk is also worth evaluating on the Pi since it is lighter than whisper.cpp and does streaming natively, but the accuracy is noticeably worse for technical vocabulary.

[m13v]

our on-device transcription service with the model selection, audio buffer management, and wake-to-transcription handoff: https://github.com/m13v/fazm/blob/main/Desktop/Sources/TranscriptionService.swift

[patmakesapps]

Thanks for that info! I think it would be cool to do the TTS/STT on the Pi5...I tried it once before on a different project using Whisper and Porcupine. The results were okay but the speed was the issue...not terrible but not great...that being said LumaKit now connects to Telegram...just added the feature today...

[patmakesapps]

But yea that 2-3 second buffer you are describing...that was an issue especially depending if using USB speakers...opens up a whole other can of worms lol

[m13v]

Telegram integration is a smart move, that sidesteps the local TTS latency entirely since you can just send text responses. what model size were you running for Whisper on the Pi, tiny or base? we found base was usable but tiny was noticeably faster for real-time use.

and yeah the USB audio buffering is its own adventure lol. we ended up pinning ALSA buffer sizes explicitly in asound.conf instead of relying on auto-detection. the Pi defaults are... not great.

[patmakesapps]

I was using tiny I'm pretty sure. Was a while ago tho but tiny sounds familiar. To combat the usb speaker delay I ended up playing a very quiet sound before the actually tts played to "warm up" the speaker. Hacky but it worked. Added to the sluggishness a bit tho.

[m13v]

Ha, the silent audio warmup trick is a classic. USB audio devices dropping the first few hundred ms is such a common issue. Have you tried keeping the audio session active with a looping silence buffer instead of playing a clip each time? That way the device stays warm and you avoid the latency hit entirely. Adds a tiny bit of power draw but on a Pi plugged into the wall it's basically free.

[patmakesapps]

I have not but that's a good idea!

[m13v]

if you go that route, start with the wake word detection running on a separate thread so it doesn't block the main loop. porcupine has a Pi-optimized library that's pretty lightweight. happy to help if you run into latency issues with the handoff.

[patmakesapps]

always open to the help!

[m13v]

great, happy to help. the voice interface piece can get complex fast once you add wake word detection. what's your current Pi setup, Pi 4 or Pi 5? that affects which model tier makes sense.

[patmakesapps]

Target is Pi5...that's what I have and if I do custom home setups for customers I'd recommend the Pi5.

I believe the one I have sitting is the 250gb storage or similar. Planning to put the code on it later today and do some testing.