Feat: Real-time speech-to-text with websockets

[pomartel]

Thanks for making this great tool!

I was hoping that the websockets implementation would allow for real-time transcribing like in iOS or macOS. Instead, I have to toggle the recording mode to get the results. Is that something that would be possible? That could really improve the flow of the dictation.

[goodroot]

Bonjour!

Thanks for the feedback.

I have not spent much time with web socket transcription.

Will play around and see what is possible.

[goodroot]

Indeed, there is no recording mode that further flexes the benefits of real timeliness.

After looking into this I have a couple questions:

a. do you simply want to see the text as you speak it, eg. 'to see it "take shape"' before pasting?
b. do you actually need it to be pasted?

For b:

There's Wayland limitations, and that output style would be - if even possible - quite tricky.

For a:

If just seeing the output in real-time processed chunks, that I can do.

Thoughts?

[pomartel]

Thanks for the reply! I guess the best value would have been real-time pasting so I can speak but also edit the text with the keyboard at the same time to fix typos and add punctuation. That's how I do it my iphone. I just leave the dictation mode on and fix as I go. I didn't know there was a wayland limitation. It's a shame because that would really be a killer feature regardless of the model (local or remote).

[goodroot]

I agree. I will poke at it, though it is a rabbit hole.

In short:

• iOS/macOS have native text field APIs that let the OS replace/update uncommitted text in-place
• Wayland has no equivalent - we only have clipboard or simulated keystrokes (brittle)

So we could get words pasted as they come up, but if something were to change (eg. "their" becomes "there" earlier in the message based on new, later message context), we have to do some hacky backspacing/return-to-space actions while new text appears, and that has a solid chance of chopping everything into garbage.

Will look though, to confirm, as that is my early read.