Low quality cloning

[toroidalGames]

Hi I've tried with vatious different clips of a friend and I mean this respectfully it sounded nothing like him, can there at times be something wrong with the installation that causes this?

I've tried clips from 42 seconds and longer

[GitHubGuy89]

I've noticed this same thing. I'm on a Windows installation. Also unsure if it could be something wrong with my installation, or just the nature of how this cloning works.

I also noticed on the demo page there are examples of fine-grained pitch control, but seems that isn't available thru this repo.

https://sparkaudio.github.io/spark-tts/#fine-grained_control

[AcTePuKc]

#10 (comment) - check this - copy the code and place them where I wrote - then just use the cli - the batch file is a test - and you have to replace 2 of the original files as I've fixed few things there and made available the seed option - you can ... just read :) Hope this helps you both

[toroidalGames]

#10 (comment) - check this - copy the code and place them where I wrote - then just use the cli - the batch file is a test - and you have to replace 2 of the original files as I've fixed few things there and made available the seed option - you can ... just read :) Hope this helps you both

thanks but it has barely made a difference I'm not sure if you're the dev I wish you the best of luck but this appears NOTHING like the examples it isn't even remotely close.

[the-xentropy]

Couple of things:

1. voice cloning with audio + audio transcription is currently broken. I created a pull request to fix this ( Fix: voice cloning with 'audio prompt' and associated text. #90 ) but you can make the same trivial changes if you want. You can set the audio prompt text to None if you want in the short term, but in my testing the results were quite unsatisfactory.
2. Be very careful of how you try to perform voice cloning - setting gender, pitch or speed to anything but None (not the string none, but the python value) is going to make it silently not use the audio file you provided, silently perform TTS with a pre-trained model without voice cloning, or fail at some stage. Also, despite the model inference code saying you need a Path object, you have to pass it a string. I suspect that literally any issue with the audio tokenization makes it fail open silently, actually.

In my own testing the model performs extremely well, but you do have to pay attention to the model code a bit at the moment since it has a lot of strange behaviors you can only work around reliably by understanding it.

[GitHubGuy89]

@the-xentropy so with your change of adding:
"<|end_semantic_token|>",

Should this basic example from the README.md work extremely well at cloning a voice? Given that I enter in the --prompt_text and path for --prompt_speech_path?

python -m cli.inference --text "text to synthesis." --device 0 --save_dir "path/to/save/audio" --model_dir pretrained_models/Spark-TTS-0.5B --prompt_text "transcript of the prompt audio" --prompt_speech_path "path/to/prompt_audio"

I tried it and still seems to not even be close to original voice. I'll try to mess with it more and see with other prompt texts and prompt audios.

[toroidalGames]

Couple of things:

1. voice cloning with audio + audio transcription is currently broken. I created a pull request to fix this ( [Fix: voice cloning with 'audio prompt' and associated text. #90](https://github.com/SparkAudio/Spark-TTS/pull/90) ) but you can make the same trivial changes if you want. You can set the audio prompt text to `None` if you want in the short term, but in my testing the results were quite unsatisfactory.

2. Be very careful of how you try to perform voice cloning - setting gender, pitch or speed to anything but `None` (not the string none, but the python value) is going to make it silently not use the audio file you provided, silently perform TTS with a pre-trained model without voice cloning, or fail at some stage. Also, despite the model inference code saying you need a Path object, you have to pass it a string. I suspect that literally any issue with the audio tokenization makes it fail open silently, actually.

In my own testing the model performs extremely well, but you do have to pay attention to the model code a bit at the moment since it has a lot of strange behaviors you can only work around reliably by understanding it.

I don't see an option for changing the tone etc only to put in the audio clip and the text to transcribe from the voice via the audio clip. The cloning only has a slight hint of the original voice when I try it, even adding an American accent when the voice is British.

Does anyone have a video demonstrating how it should look and its effectiveness?