Safety check on average DER fails

[mdeegen]

When computing the DER for multiple recordings I get this error:

Traceback (most recent call last):
  File "/homes/eva/q/qdeegen/micromamba/envs/diarizen/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/homes/eva/q/qdeegen/micromamba/envs/diarizen/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/mnt/matylda5/qdeegen/deploy/software/meeteval/meeteval/der/__main__.py", line 96, in <module>
    cli()
  File "/mnt/matylda5/qdeegen/deploy/software/meeteval/meeteval/der/__main__.py", line 92, in cli
    cli.run()
  File "/mnt/matylda5/qdeegen/deploy/software/meeteval/meeteval/wer/__main__.py", line 766, in run
    return fn(**kwargs)
  File "/mnt/matylda5/qdeegen/deploy/software/meeteval/meeteval/der/__main__.py", line 47, in dscore
    results = dscore(
  File "/mnt/matylda5/qdeegen/deploy/software/meeteval/meeteval/der/api.py", line 47, in dscore
    results = dscore_multifile(
  File "/mnt/matylda5/qdeegen/deploy/software/meeteval/meeteval/der/nryant_dscore.py", line 190, in dscore_multifile
    result = md_eval_22_multifile(
  File "/mnt/matylda5/qdeegen/deploy/software/meeteval/meeteval/der/md_eval.py", line 301, in md_eval_22_multifile
    raise RuntimeError(
RuntimeError: The error rate of md-eval-22.pl on all recordings (0.15625)
does not match the average error rate of md-eval-22.pl applied to each recording (0.1563).

There seems to be a slight difference in the averaged values. However, the rttm files seem to be correct.

[thequilo]

Hi @mdeegen!

I looked into this and the issue may not be the numerial precision, but the default rounding behavior of Decimal.quantize:

In [1]: from decimal import Decimal, ROUND_HALF_UP

In [1]: Decimal('0.05').quantize(Decimal('0.1'))
Out[1]: Decimal('0.0')

In [2]: Decimal('0.051').quantize(Decimal('0.1'))
Out[2]: Decimal('0.1')

In [2]: Decimal('0.05').quantize(Decimal('0.1'), rounding=ROUND_HALF_UP)
Out[2]: Decimal('0.1')

It rounds down at .5 by default, while md-eval-22.pl rounds up. So the fix may be as simple as adding rounding=ROUND_HALF_UP to the call to quantize. @mdeegen could you check if this resolves the issue?

[boeddeker]

It could also be, that there are special cases, where the numbers of md-eval cannot be perfectly reproduced with example wise calls.
If that is the case, we may want to raise only a warning, when we hit the quantization error.
The error would then only be thrown, when the numbers are more off (ideally, it would never trigger, which was also the assumption of the current error).

[thequilo]

Yes, the rounding option solves this one special case, but we now have another example where it fails even with the different rounding behavior.

I expected that the example-wise calls reproduce it up to two decimal places because the error durations are reported with high precision. But it can of course always hit the boundary and fail the check.

The problem with a warning is that everybody will ignore it. But it may be a better solution in terms of usability than failing the computation.

[thequilo]

In #122, I re-used the check from the DiaErrorRate for a difference below 0.00007, which passes for the examples I have. The error appeared for small inaccuracies around a .00005 mark, so cases where a small error caused a different rounding.

It now prints a warning when the numbers do not match exactly (what was the exception before) and raises an exception when the difference is larger than 0.00007 (more than the last digit differs). The tolerance used for the exception should not be exceeded for reasonable inputs (but I guess that it could happen in extreme cases with thousands of sessions in one file, where larger errors could accumulate).