Development plan for VERSA

[ftshijt]

We are now actively developing VERSA for general-purpose speech/audio evaluation.

• Additional Metrics to add

  • torch-squim (metrics done, pending adding to the interface) https://pytorch.org/audio/main/tutorials/squim_tutorial.html
  • torch-log-wmse-audio-quality https://github.com/crlandsc/torch-log-wmse-audio-quality
  • WARP-Q https://github.com/wjassim/WARP-Q
  • WAWEnets https://github.com/NTIA/WEnets
  • Peceptual Audio https://github.com/pranaymanocha/PerceptualAudio
  • VQScore https://github.com/JasonSWFu/VQscore
  • LMScore https://github.com/soumimaiti/speechlmscore_tool
  • Audio metrics in https://github.com/haoheliu/audioldm_eval?tab=readme-ov-file#evaluation-metrics

• Additional integration

  • WER (ASR evaluation from ESPnet)
  • SingMOS https://github.com/South-Twilight/SingMOS

• Software development

  • Webpage for documentation
  • API level documentation
  • CI test
  • User interface for command line + local multiprocessing (currently only with slurm)

Please let us know if you have other metrics to be added to the toolkit as a general evaluation toolkit.

[Emrys365]

DNSMOS Pro (https://github.com/fcumlin/DNSMOSPro) might be a good option.

[Emrys365]

How about adding more TTS objective metrics such as those in https://github.com/AI-Unicamp/TTS-Objective-Metrics?

[Wkstr]

AECMOS might be a good option.