1300 offline voices via SherpaOnnx

[willwade]

FYI - Im working on this here willwade#1

I cant say its fully working yet.. and its messy. If anyone wants to help out please be my guest. Will support around 1100 languages - and offline models by sherpaonnx. Just realise absolutley no SSML in this Update I'll push a release to https://github.com/willwade/NaturalVoiceSAPIAdapter/releases/tag/v0.0.1 - NB: Its signed so no virus warnings - let me know if you do get any

[willwade]

Update - I've got this now working. I'm going to tweak things to deal with the different models like piper kokoro and enhance the User experience..

[gexgd0419]

First of all, thank you for your interest in my project!

Actually, I have an ambitious plan for the next version (v0.3 maybe), which would decouple the adapter from Windows/SAPI and make this project cross-platform. However, I haven't invested time or effort on it yet...

Anyway, here's the plan:

• Migrate from registry-based configuration to text configuration files
• Migrate from Win32 GUI APIs to a cross-platform framework, such as wxWidgets or Qt
  • This also simplifies i18n
  • The GUI is only for configuration, so wxWidgets is enough
  • Qt would be better for cross-platform apps and have more features, but may need to consider support for Windows 7 and licensing
• Replace the TTS tester TtsApplication with Qt hello_speak, or another cross-platform TTS program
• Separate the adapter into SAPI5 frontend and NaturalVoice backend
  • Frontend has all SAPI5-related logics, such as SSML conversion
  • Backend runs the Azure Speech SDK, and provides a standard API for the frontend
  • Backend should be run out-of-process
  • Provide a CLI program for using/testing the backend directly without SAPI5
  • Backend should use a cross-platform build system, such as CMake
• Add more frontends for each speech API on each OS, making it cross-platform
  • For macOS, it's AVSpeechSynthesisProviderAudioUnit. Unfortunately, network access isn’t allowed in speech synthesizers.
  • For Linux, it's a speech-dispatcher module.
  • Not needed for Android. MultiTTS is good enough.
  • Not possible for iOS. Technically it's possible, but I don't think it can be made into the App Store.
• Add extensibility to the backend, so that more voices could be added later
  • For cross-platform support, use scripting languages such as Lua or JavaScript (QuickJS) to handle network-based voices, which needs an API specification

It's a little too much for me, and I'm not that experienced in programming either. I still have to decide on some details, such as the format of config files (JSON? XML? YAML? TOML?), the GUI framework to use (wxWidgets or Qt?), and so on.

Feel free if you have something to say about my plan.

[willwade]

Cool work on this. I’ve had a few stabs at this project before—this is actually my third attempt at something like this!

I’m wondering if there is an even simpler, more modular way to handle the cross-platform side. My previous work with OpenVoiceProxy used a REST/WebSocket interface backed by js-tts-wrapper, and it felt like a solid direction (I have done a bit of the QT route and its a bit of a pain to manage and look spot on cross platform..)

Maybe theres a simpler approach?

The real key is js-tts-wrapper. It provides a unified interface for all TTS engines, including standardized SSML and language codes. I did wonder about writing this in something like rust but it's not a lot of gain as so many services already have node/rest interfaces.. and node/bun are pretty fast. So here's my idea - but lets iterate this as somewhere in between your idea and mine might be the solution..

Heres my rough idea:

• The Backend: A Node-based streaming server to do the heavy lifting. We could easily add ElevenLabs or OpenAI-compatible endpoints here (addressing Support OpenAI Compatible Endpoint #68 - nb I already have elevenlabs equivalent endpoint for that.. it needs refactoring).
• The Bridge: Your SAPI adapter (or an AVSynth wrapper) acts as the thin client connecting to that local server - or of course you talk directly to the rest server.
• The UI: A small Electron wrapper or even native .NET/Swift interfaces to talk to the REST endpoints. It keeps things way lighter than Qt or Tkinter (which is... let's face it, pretty hideous). The one major benefit of electron is packaging, signing and updates - it's a very well trodden path.. and it's far easier. I just hate the chrome bundling thing..

I previously built a pipe service system (AACSpeakHelper), but doing it in Python/PyQt was a mistake—it’s fragile, slow, and overly complex.

What I’ve always struggled with in my SAPI experiments is the actual SAPI interface—it is notoriously difficult to debug. You’ve clearly cracked that! I was actually working on this installer, got totally stuck, and then realized I’d starred your project ages ago!

Mobile is the only real headache. iOS is a massive pain because you can't have an AVSynth engine with web access; those platforms would likely need to be handled differently. and yeah - your idea or mine is pretty dead in the water there (you could do something with mono/dotnet but the wins aren't enough)..

Anyway, I think a Node webserver for the streaming + your SAPI wrapper for the Windows integration could work.. but still not entirely sure..

NB: For my use cases I really need a Translation/Transliteration option too - so you type in one language and it can speak in another..

Going to tag in the awesome @synesthesiam who I know has made another stab at TTS server instance like this in the past - or knows of some other repos doing something similar (and just to be aware of this neat project ;) )

[gexgd0419]

My project goal is to build a bridge between the OS's native speech API and Microsoft's natural voices (and possibly more voices). Since the Azure Speech SDK has support for different OSes, and online voices are platform-agnostic by nature, it would be natural to expand this project to other platforms.

I started this project as a proof of concept after I found that the "Narrator-exclusive" natural voices could actually be utilized through Azure Speech SDK. And although there were already tools that use the Edge online voices to synthesize audio, no one tried to adapt them to SAPI5, so I did that as well. As a result, this can work with any app that needs TTS and supports SAPI5 voices, such as screen readers, ebook readers, and chat text-to-speech.

Quirks of SAPI5

SAPI5 is a rather open standard based on COM. Third-party voice vendors can develop their own voices and add them into SAPI5's voice list, so any compatible client can use them. SAPI5 is quite old, and it has been supported since Windows XP, but now we are still stuck on SAPI5, since the "next generation" speech API, OneCore speech, isn't as open and extensible.

And yes, SAPI5 brought me a lot of pain in the process.

SPVTEXTFRAG conversion

SAPI5 uses a linked list of SPVTEXTFRAGs to pass the parsed text to the engine. It's nice that we don't need to parse the XML ourselves, right? No. Azure Speech SDK (and most of the other modern speech systems) requires SSML, so I have to convert and assemble the text frags back to SSML. It took quite a while to properly convert and support as many SSML tags as I could. I also had to add workarounds for Edge voices, since a lot of SSML tags are not supported, and when the Edge voice server finds an unknown tag, it rejects the whole request instead of ignoring it.

In-process requirement

SAPI5 voice engines are essentially DLL files that will be loaded in every TTS client processes. This presents some challenges.

Debugging: As it's a DLL file, debugging the DLL requires loading the DLL into a host process first. My solution is to use TtsApplication as the host process. I changed the debugger settings so that clicking Run runs the TtsApplication program and attaches to it. Now I can choose an adapter voice and let it speak, and debug the code in the adapter DLL.

C runtime (CRT) version collision: When I tried to update the Azure Speech SDK version in the released ZIP, some user reported crashes in some apps. Turned out that newer Azure Speech SDK needs newer CRT DLLs, which is incompatible with older CRT DLLs. If the client process has already loaded its own older CRT DLL before my adapter is loaded, the Azure Speech SDK will use the CRT that was loaded before, and then crash because of incompatibility. Unfortunately, as a DLL, you can't choose which CRT DLL you will use, or replace the loaded, older CRT with your own newer CRT. I have to downgrade to older Azure Speech SDK to prevent crashes.

ARM64 support: When I attempted to build an ARM64 version, I found that since x64 and ARM64 programs share the same registry location, I have to make the same DLL compatible with both x64 and ARM64. You can achieve that by building ARM64X DLLs, but unfortunately, Azure Speech SDK DLLs have no ARM64X version, only separate x64 and ARM64 ones. So I had to build an Arm64XForwarder, whose job is to load the appropriate DLLs for x64 or ARM64. If the Azure Speech SDK was loaded out of process, I wouldn't need a forwarder like that, since I could choose whatever architecture for the host process.

Token enumerator

Online voice lists can change from time to time, so the adapter will attempt to pull the latest list data periodically. But in order to actually add the voices into SAPI5 dynamically, a token enumerator is required.

Each SAPI5 voice is represented by an object token, which contains information needed to identify the voice and the voice engine. Object tokens are usually stored as registry keys under HKLM\Software\Microsoft\Speech\Voice\Tokens. You can add your voice tokens directly as registry keys there, but it's more difficult to add more voices later, since you'll need administrator's privilege to write to that registry key, let alone generating voice list dynamically.

So SAPI5 allows you to add a "token enumerator", which returns a list of your tokens to SAPI5. Your self-defined tokens can be plain-old registry keys, but can also be your own COM objects, as long as they implement ISpDataKey.

Tokens in the registry use their registry paths as their token ID. Tokens from a token enumerator, on the other hand, use the token enumerator's registry path, plus a self-defined name, as if the tokens were actual registry keys under the token enumerator's key, for example HKLM\Software\Microsoft\Speech\Voice\TokenEnums\MyEnum\MyVoice. This registry path does not need to exist; it only means that the token belongs to the enumerator MyEnum, so that SAPI5 can contact this enumerator once this token is selected.

Undocumented features

Remember when I said SAPI5 is "open"? Well, not completely. Later versions of SAPI5 had the ability to parse and preserve most of the SSML tags, by using a new interface ISpTTSEngineSSML and passing SPVSSMLTEXTFRAGs, which is nearly a 1-to-1 mapping to actual SSML tags. Old SPVTEXTFRAG is based on SAPI5's proprietary XML format, which lacks some features of SSML, so those features would be lost in the translation from SSML input to text frags to SSML again. With the new SSML text frags, this is no longer the case.

However, this new SAPI5 feature isn't open to voice vendors and isn’t documented, and there are no declarations in the header files either, so only system built-in voices supports them.

I only knew its existence because of the source code of TtsEngineSSML in .NET framework. My adapter hasn't supported this yet, but might support it in the future.

My plan

As you might notice, I haven't released too many versions in the past two years. The first release was on May 10, 2024, but the first commit was on Dec 14, 2023, when I had my initial plan for this project, and pretty much procrastinated for the next few months. So even though I've listed my plan, I don't have any estimated finish time for this, or it might end up never happen at all, who knows.

I haven't looked at this repo's code for quite a while now, and I think it's quite messy and needs some refactoring. I have no serious programming working experience though, so I might be bad at this. Maybe I can learn something more from your code.

So here's what I'm thinking of.

A basic UI

For the UI part, I only want a basic, lightweight UI, instead of a modern, fancy UI, to allow users to tweak some configurations, just like the Properties dialog on Windows. Users can also modify the config files directly, but having a UI is obviously better. I used Win32 APIs, since MFC was considered outdated, but Win32 APIs might be just as bad. But I think that using a whole Electron to build the UI might be a little overkill. So, wxWidgets maybe?

Compatibility with older Windows

You might notice that this adapter supports Windows XP. That's right, and I've actually tested it in an XP virtual machine. I use YY-Thunks and intentionally use basic UI to keep XP compatibility (although only online voices work on XP). I don't know how many users are still using XP, but it's kind of nostalgic for me. If XP compatibility is not possible, then I hope to have at least Windows 7 compatibility. I wrote a patcher that modifies the Azure Speech SDK DLLs to use shims in YY-Thunks to make it work on Windows 7.

A cross-platform core

This still needs some consideration.

As for now, the core only needs to host the Azure Speech SDK and expose interfaces for the frontends, when it only supports Microsoft natural voices. But if I want it to support more voices, I need to consider its structure.

One of the options to add is to support online voices. They almost all support some kind of web protocol, such as HTTP or WebSocket, but their web APIs are different. So I plan to add a scripting language, such as JavaScript, into the core, so that new voices can be quickly added via scripts, and the community can share their own scripts for new voices. Some other projects, such as MultiTTS and tts-server-android, also use JavaScript for custom online voices.

Your idea is to use Node.js, which already provides a rich JavaScript environment. But Node.js seems a little too heavy, and it's not easy to port to older Windows systems. Also, I'd like to restrict what the script can do, so that a possibly malicious script won't do much harm to user's environment.

My idea is to use QuickJS, plus a series of scripting interfaces provided by the core. This way, the script is under total control of the host, but it also means that I have to implement a whole set of APIs for scripts to allow them to access online voice servers.

What about other local voices? I have no idea yet. Local voices are useful, but it's totally out of control, as it can execute arbitrary code. Also, local voices are platform-dependent.

Yeah ... Maybe I'm overcomplicating things. But I welcome your ideas on this. Maybe you can help me sort this out.