[Lanre Shittu (Shizoqua)]

Summary

Fixes #376.

A LiveMathematicianBench answer written as (B), [B] or **B** was scored wrong even when B was the correct choice. These formats are now parsed as the label they name. B, B. and B) were already accepted and are unchanged.

User impact: hard is the exact-match result, so a correct answer in one of these common formats got zero reward during training and counted as a miss in reported accuracy. The optimizer could also be pushed toward learning answer formatting instead of the task.

Proof: the regression tests fail on upstream 6ee29a0 and pass with this change.

Root cause

normalize_label() in skillopt/envs/livemathematicianbench/evaluator.py was:

return str(text).strip().upper().rstrip(".):")

It removed trailing punctuation but nothing at the front, so (B) became (B, which is not a valid label. parse_choice_label() then fell through to returning that invalid label. [B] and **B** were never stripped at all.

Changes

• normalize_label() also strips surrounding Markdown bold, a leading ( or [, and a trailing ], in addition to the existing trailing ., ) and :.
• Wrong labels such as (A) against B, free text such as Option B, and empty answers still score 0.
• The dataloader has its own _normalize_label for gold data, where labels are plain letters. It is unchanged; predictions only pass through the evaluator.
• New tests/test_livemath_evaluator.py, testing through evaluate(), which is what the rollout calls.
• CHANGELOG entry under Unreleased / Fixed.

Reproduction

from skillopt.envs.livemathematicianbench.evaluator import evaluate

choices = [{"label": "A", "text": "2"}, {"label": "B", "text": "3"}, {"label": "C", "text": "5"}]
for response in ["<answer>B)</answer>", "<answer>(B)</answer>", "<answer>[B]</answer>", "<answer>**B**</answer>"]:
    result = evaluate(response, choices[1], choices)
    print(response, result["em"], repr(result["predicted_label"]))

Before: 1.0 'B', 0.0 '(B', 0.0 '[B]', 0.0 '**B**'

After: 1.0 'B' for all four.

Testing

• python -m pytest tests/test_livemath_evaluator.py -q: on upstream 6ee29a0, 4 failed (assert '(B' == 'B' and similar) and 6 passed; with the fix, 10 passed.
• python -m pytest tests/test_livemath_evaluator.py tests/test_benchmark_task_output_paths.py -q: 26 passed.
• python -m pytest -q: 2079 passed, 12 skipped, 4 xfailed.
• python -m ruff check on the changed files: all checks passed.
• python -m mkdocs build --strict: passed.

Risk and compatibility

Only answers that previously normalized to an invalid label change outcome. Any answer that already matched a label is scored the same. This changes LiveMathematicianBench scores upward for runs whose target model wraps its choice, so results from before and after this fix are not directly comparable. No backend, config key, file format or CLI change.