[Lanre Shittu (Shizoqua)]

Summary

Fixes #377.

OfficeQA answer normalization kept a sentence-ending period, so a correct answer such as 42. did not match the gold 42 and scored zero exact match and zero F1. A period is now dropped unless a digit follows it, so decimals are unaffected.

User impact: hard is exact match and soft is F1, so a correct numeric answer that ended with a period got no reward during training and counted as a miss in reported scores.

Proof: the regression tests fail on upstream 6ee29a0 and pass with this change.

Root cause

normalize_answer() in skillopt/envs/officeqa/evaluator.py removes punctuation except characters in _NUMERIC_CHARS (0123456789.-) and %. Keeping . is needed for decimals, but it also kept periods that end a sentence. 42. stayed 42., which differs from 42 for exact match and is a different token for F1.

Changes

• After the punctuation filter, normalize_answer() removes any . that is not followed by a digit (re.sub(r"\.(?!\d)", "", text)).
• Decimal points, signs and percent signs still affect the match: 1.5 vs 15, 0.5 vs 5 and -3 vs 3 still score 0.
• New tests/test_officeqa_evaluator.py, testing through evaluate(), which is what the rollout calls.
• CHANGELOG entry under Unreleased / Fixed.

Reproduction

from skillopt.envs.officeqa.evaluator import evaluate

for prediction, gold in [("42", "42"), ("42.", "42"), ("12.5%.", "12.5%"), ("$1,234.5 million.", "1234.5")]:
    result = evaluate(prediction, gold)
    print(repr(prediction), repr(gold), result["em"], result["f1"])

Before:

'42' '42' 1.0 1.0
'42.' '42' 0.0 0.0
'12.5%.' '12.5%' 0.0 0.0
'$1,234.5 million.' '1234.5' 0.0 0.6666666666666666

After: 1.0 1.0 for all four.

Testing

• python -m pytest tests/test_officeqa_evaluator.py -q: on upstream 6ee29a0, 6 failed (assert 0.0 == 1.0 and similar) and 4 passed; with the fix, 10 passed.
• python -m pytest tests/test_officeqa_evaluator.py tests/test_officeqa_parallel_search.py tests/test_benchmark_task_output_paths.py -q: 58 passed.
• python -m pytest -q: 2079 passed, 12 skipped, 4 xfailed.
• python -m ruff check on the changed files: one I001 import-spacing finding on lines that are unchanged from main.
• python -m mkdocs build --strict: passed.

Risk and compatibility

Normalization is applied to both prediction and gold, so any pair that matched before still matches. Abbreviations such as U.S. now normalize to us, which makes U.S. and US match. This raises OfficeQA scores for runs whose target model ends answers with a period, so results from before and after this fix are not directly comparable. Numeric equivalence such as 3.20 vs 3.2 is unchanged. No backend, config key, file format or CLI change.