streaming tts with piper

[bchinnari]

Piper official repo, as of now, splits the input text into sentences using phonemize().
It generates audio chunk for whole sentence at once and then "yields" audio for each sentence one aftyer the other.

So, if the first sentence in the input text is long, it will some time to generate the audio.

Does this repo follow the same if i select piperEngine? or does this repo somehow genrate audio for each word by word instead of sentence by sentence? If it produces word by word , latency will be much lower.

Genral Question: Is there a opern source TTS system which has bith

1. generates speech word by word or like sub-sentence level, so that latency is lower
2. Has script for fine tuning the pre-trained model with our custom data.

[KoljaB]

RealtimeTTS does this in the same way. Not with phonemize but with stream2sentence library. I would not recommend splitting word by word, because TTS systems need context or they lose prosody.

[KoljaB]

There are systems like Coqui XTTS that use real output streaming and can achieve very low latencies way below 200 ms quite independent of input sentence length.

[bchinnari]

Kindly clarify one last thing. This is very important for my work.

1. Is Coqui XTTS-v2 commercially usable?
2. could you suggest a tts system which has voice cloning feature ( we can do apriori) and low-latency in true relatime sense.
3. Piper model is giving me around 800ms latency ( on T4 gpu) if the first sentence is "Hello World!", do you think it is large ? This insight can help me look for optimization further.

[KoljaB]

1. No. It's under coqui public model license which only allows non-commercial use. The company behind coqui does not exist anymore. There were commercial license to buy for a quite cheap price, but since there is no company anymore to pay this to there is currently no way to get a safe license.
2. ZipVoice
3. Yes, that's a lot. I'd aim for 300-500ms max for realtime usage.

[KoljaB]

Zipvoice does not support output streaming though. It's the same procedure like in Piper, split into sentences then synthesize those. It's crazy fast, but I can't say if it's fast enough for realtime usage on a T4. It might have a slightly better output quality than coqui XTTS, but it does only support english and chinese. XTTS supports 17 languages.