Getting ORC-WER breakdown

[desh2608]

Is there a simple way to get the WER break-down into ins/del/sub when computing the ORC-WER?

[boeddeker]

At the moment, it is not implemented. But we planned to integrate it. Since you are interested, I will take care to implement it.

Do you have a suggestion, how it should be obtained? ins/del/sub aren't unique values.
Do people expect to get the same as Kaldi?
I thought about adding my Kaldi wrapper, so that the optimization is done in python/cython, but the final WER is obtained with Kaldi (Editdistance is the same, but Kaldi additionally reports the ins/del/sub).

[desh2608]

One way would be to get the reference assignment and then use the kaldialign package to compute the ins/del/sub. This is the WER package used in icefall, for example, and is kaldi-compatible.

[boeddeker]

Thanks for the pointer, I forgot that kaldialign exists. The code will be much cleaner with kaldialign, then a wrapper around kaldi could be, since adding a pip dependency is simple.

[boeddeker]

Now the code uses kaldialign to calculate insertions, deletions and substitutions. Hence, the generated files have now this information:

.../meeteval/example_files$ python -m meeteval.wer orcwer -h 'hyp*.stm' -r 'ref*.stm'
Wrote: .../hyp_orcwer_per_reco.json
Wrote: .../hyp_orcwer.json

.../meeteval/example_files$ cat hyp_orcwer.json
{
  "errors": 18,
  "length": 184,
  "insertions": 0,
  "deletions": 14,
  "substitutions": 4,
  "error_rate": 0.09782608695652174
}

.../meeteval/example_files$ cat hyp_orcwer_per_reco.json
{
  "recordingA": {
    "errors": 4,
    "length": 124,
    "insertions": 0,
    "deletions": 0,
    "substitutions": 4,
    "error_rate": 0.03225806451612903,
    "assignment": [
      "Alice",
      "Alice",
      "Bob",
      "Bob",
      "Alice",
      "Alice",
      "Alice",
      "Alice"
    ]
  },
  "recordingB": {
    "errors": 14,
    "length": 60,
    "insertions": 0,
    "deletions": 14,
    "substitutions": 0,
    "error_rate": 0.23333333333333334,
    "assignment": [
      "Bob",
      "Bob",
      "Alice",
      "Alice"
    ]
  }

Let me know, when you need more features or if a modification could simplify an integration in another framework.