[Lanre Shittu (Shizoqua)]

Summary

Adds tests/test_docvqa_evaluator.py, direct unit tests for skillopt/envs/docvqa/evaluator.py. The evaluator scores every DocVQA rollout (soft is the ANLS value and hard is anls >= 0.999 in rollout.py) but had no tests of its own, so a regression in the metric would only show up as a silent change in benchmark numbers.

No runtime code changes.

What the tests cover

The tests follow the ANLS definition from Biten et al. (2019): for each gold answer, NL is the Levenshtein distance divided by the length of the longer string; the score is 1 - NL when NL < 0.5 and 0 otherwise; the item score is the maximum over gold answers.

• Levenshtein distance on known pairs (kitten/sitting = 3, flaw/lawn = 2, identical and empty strings), in both argument orders.
• Threshold: the default is 0.5, and the exact NL = 0.5 boundary (abcd vs ab) scores 0, as the definition requires.
• Partial credit: abcd vs abce scores 0.75 and stays below the hard cutoff used by the rollout.
• Maximum over gold answers, with the best match first and last, so an implementation that kept only the first or last gold answer would fail.
• Normalization: case and repeated whitespace are ignored; empty prediction and empty gold match only each other.
• Gold answer shapes the loader accepts: lists, JSON strings, Python-literal strings, dicts with answers or ground_truth, lists of dicts with text, answer or value, plain strings, numbers, None and empty values.
• <answer> extraction: case-insensitive tags, the last tag wins, and the last non-empty line is the fallback.

Testing

• python -m pytest tests/test_docvqa_evaluator.py -q: 32 passed.
• python -m pytest -q: 2101 passed, 12 skipped, 4 xfailed (32 more than main at 6ee29a0).
• python -m ruff check tests/test_docvqa_evaluator.py and python -m ruff format --check tests/test_docvqa_evaluator.py: clean.
• python -m mkdocs build --strict: passed.

To check that the tests catch real regressions, I made each of these changes to the evaluator in turn, ran the suite, and reverted. Every one makes at least one test fail:

Change to the evaluator	Failing tests
Threshold check >= changed to >	1
Max over gold answers changed to min	2
Only the last gold answer counted	2
Only the first gold answer counted	2
NL divided by the shorter length	2
Lowercasing removed	1
Substitution cost doubled	4
First <answer> tag used instead of last	1

Risk and compatibility

Test-only change plus a CHANGELOG line under Unreleased / Tests. No dependencies added; the tests are pure Python and run without network access.