RuntimeError: Inconsistent WER statistics between WER and alignment calculation for 'errors'

[cako2025]

Hi,

While I was trying to find out how the different alignments work, I got this error:

Traceback (most recent call last): File "/home/USER/Python-3.10.14/Lib/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/USER/Python-3.10.14/Lib/runpy.py", line 86, in _run_code exec(code, run_globals) File "/home/USER/Dokumente/meeteval_test/test/meeteval_orc/test_meeteval_orc.py", line 70, in <module> main() File "/home/USER/Dokumente/meeteval_test/test/meeteval_orc/test_meeteval_orc.py", line 36, in main eval_ts_asr.create_viz() File "/home/USER/Dokumente/meeteval_test/src/eval_ts_asr.py", line 56, in create_viz create_viz_folder( File "/home/USER/Dokumente/meeteval_test/.venv/lib/python3.10/site-packages/meeteval/viz/__main__.py", line 108, in create_viz_folder save_name = _get_av_file_path(av.data) File "/home/USER/Python-3.10.14/Lib/functools.py", line 981, in __get__ val = self.func(instance) File "/home/USER/Dokumente/meeteval_test/.venv/lib/python3.10/site-packages/meeteval/viz/visualize.py", line 563, in data d = get_visualization_data( File "/home/USER/Dokumente/meeteval_test/.venv/lib/python3.10/site-packages/meeteval/viz/visualize.py", line 462, in get_visualization_data raise RuntimeError( RuntimeError: Inconsistent WER statistics between WER and alignment calculation for 'errors': errors from WER: 26 errors from alignment calculation: 0 WER details: {'assignment': ('A', 'A'), 'deletions': 13, 'error_rate': 2.0, 'errors': 26, 'hypothesis_self_overlap': {'overlap_rate': Decimal('0'), 'overlap_time': 0, 'total_time': Decimal('6')}, 'insertions': 13, 'length': 13, 'reference_self_overlap': {'overlap_rate': Decimal('0'), 'overlap_time': 0, 'total_time': Decimal('16')}, 'substitutions': 0} Alignment details: {'A': {'deletions': 0, 'error_rate': 0.0, 'errors': 0, 'hypothesis_self_overlap': None, 'insertions': 0, 'length': 13, 'reference_self_overlap': None, 'substitutions': 0}}

It happens on the following three alignments: "greedy_tcorc,greedy_dicp,greedy_ditcp"

It does not happen on the following five alignments: "tcp,cp,tcorc,orc,greedy_orc"

If I comment out the raise, then the create_viz_folder does work till the end and creates the index.html. But I am not sure if the calculated error rate is correct at this point. Do I miss something, or is it maybe the way my ref.stm and hyp.stm are.

My hyp.stm:

recordingA 1 A 1 1 Dies testet die meeteval Funktion.
recordingA 1 C 1 2 Ich teste ebenfalls meeteval.
recordingA 1 A 2 3 
recordingA 1 I 2 3 Ich lasse mich nicht so leicht veräppeln!
recordingB 1 I 3 4 Im Dezember ist es so kalt.
recordingB 1 C 4 5 Ich kann es kaum glauben.
recordingB 1 C 4 5 Wir haben nun hier ein insertion Fehler von genau elf Buchstaben.
recordingA 1 C 5 6 Warten wird belohnt.
recordingC 1 A 1 1 Das testet die meeteval Funktion?
recordingC 1 I 1 2 Ich teste ebenfalls meeteval.
recordingD 1 A 2 3 
recordingD 1 I 2 3 Ist das wirklich so gewollt?
recordingC 1 A 3 4 Kann ich der leider nicht sagen.
recordingD 1 C 4 5 Hauptsache schmeckt!
recordingD 1 C 5 6 Morgen!
recordingE 1 A 3 4 Dies ist ein Test für Zeitversetzung.
recordingE 1 A 5 6 Ebenfalls gibt es hier kein Self-Overlapping mehr!

My ref.stm:

recordingA 1 C 1 1 Dies testet die meeteval Funktion.
recordingA 1 I 1 2 Ich teste ebenfalls meeteval.
recordingA 1 A 2 3 
recordingA 1 C 2 3 Ich lasse mich nicht so leicht veräppeln!
recordingB 1 A 3 4 Im Dezember ist es so kalt.
recordingB 1 I 4 5 Ich kann es kaum glauben.
recordingA 1 I 5 6 Warten wird belohnt.
recordingC 1 C 1 1 Dies testet die meeteval Funktion.
recordingC 1 I 1 2 Ich teste ebenfalls meeteval.
recordingD 1 A 2 3 
recordingD 1 C 2 3 Ist das wirklich so gewollt?
recordingC 1 A 3 4 Kann ich dir leider nicht sagen.
recordingD 1 I 4 5 Hauptsache es schmeckt!
recordingD 1 I 5 6 Guten Morgen!
recordingE 1 A 12 13 Dies ist ein Test für Zeitversetzung.
recordingE 1 A 14 16 Ebenfalls gibt es hier kein Self-Overlapping mehr!

My code.py:

from meeteval.viz.__main__ import create_viz_folder


def main():
    output_folder = "./test/meeteval_orc/output/"
    hyp = "./test/meeteval_orc/hyp.stm"
    ref = "./test/meeteval_orc/ref.stm"

    alignments = 'tcp,cp,tcorc,orc,greedy_orc,greedy_tcorc,greedy_dicp,greedy_ditcp'

    create_viz_folder(
        reference=ref,
        hypothesiss={"test": hyp},
        out=output_folder,
        alignments=alignments,
        normalizer=None
    )


if __name__ == "__main__":
    main()

[thequilo]

Hey @cako2025 !

Does it work if you load the files with meeteval.io.load before passing them to the function?

This hints at an internal error. I'll investigate it, but it'll take a few days. I don't have much time over the holidays

[cako2025]

Hi @thequilo,
no worries and happy holidays btw.

I've just tried it like this:

import meeteval

from meeteval.viz.__main__ import create_viz_folder


def main():
    output_folder = "./test/meeteval_orc/output/"
    hyp = "./test/meeteval_orc/hyp.stm"
    ref = "./test/meeteval_orc/ref.stm"

    alignments = 'tcp,cp,tcorc,orc,greedy_orc,greedy_tcorc,greedy_dicp,greedy_ditcp'

    hyp_loaded = meeteval.io.load(hyp)
    ref_loaded = meeteval.io.load(ref)

    create_viz_folder(
        reference=ref_loaded,
        hypothesiss={"test": hyp_loaded},
        out=output_folder,
        alignments=alignments,
        normalizer=None
    )


if __name__ == "__main__":
    main()

and like this:

import meeteval

from meeteval.viz.__main__ import create_viz_folder


def main():
    output_folder = "./test/meeteval_orc/output/"
    hyp = "./test/meeteval_orc/hyp.stm"
    ref = "./test/meeteval_orc/ref.stm"

    alignments = 'tcp,cp,tcorc,orc,greedy_orc,greedy_tcorc,greedy_dicp,greedy_ditcp'

    hyp_loaded: meeteval.io.SegLST = meeteval.io.asseglst(meeteval.io.load(hyp))
    ref_loaded: meeteval.io.SegLST = meeteval.io.asseglst(meeteval.io.load(ref))

    create_viz_folder(
        reference=ref_loaded,
        hypothesiss={"test": hyp_loaded},
        out=output_folder,
        alignments=alignments,
        normalizer=None
    )


if __name__ == "__main__":
    main()

Both tries still end with the same runtime error at the same point.

[thequilo]

I had the fear that this was an error somewhere in the alignment computaion, but it was a rather dumb mistake of wrongly mapping the config to a non-time-constrained alignment when a time-constrained one should be computed. Fixed in #125.

[cako2025]

Thank you for the quick fix! This solved indeed the problem, works like a charm. Case closed.