Add a model-agnostic evaluation harness for the QA/MCQ/RRB benchmarks#10
Open
Gazzara-lab wants to merge 2 commits into
Open
Add a model-agnostic evaluation harness for the QA/MCQ/RRB benchmarks#10Gazzara-lab wants to merge 2 commits into
Gazzara-lab wants to merge 2 commits into
Conversation
The repo ships benchmark generation plus the three diagnostic datasets, but no turnkey way to score a model on them. This adds an `evaluate/` package: - `eval-qa` / `eval-mcq` / `eval-rrb` CLIs and metrics (accuracy, Recall@k, hit@k, MRR, nDCG@k) with 95% bootstrap confidence intervals, robust answer parsing, and unparseable-response counting so non-compliant output is visible rather than silently mis-scored. - A `--dry-run` stub path that runs end-to-end with no API calls (CI-friendly, standard-library only) and 27 known-answer unit tests for the metrics/parsers. - A README "Evaluation" section with a no-full-install quickstart (the evaluator needs only the standard library plus a model client), an optional `eval` extra, and REPRODUCTION.md (scope notes, offline dataset-integrity facts, and an example run with google/gemma-3-12b-it). This is an inference-only, in-context evaluation; it is not a reproduction of the paper's trained-ToolGen Rc@50, as documented in REPRODUCTION.md. Closes SAP#9. Signed-off-by: Franco A. Gazzara <287811783+Gazzara-lab@users.noreply.github.com>
|
|
There was a problem hiding this comment.
Pull request overview
Adds a new evaluate/ Python package that provides a model-agnostic scoring harness for the repo’s three shipped diagnostic datasets (QA, MCQ, RRB), including CLI entry points, core metrics (with bootstrap CIs), robust response parsing, and offline unit tests. This fills the current gap where the repo provides dataset generation + data, but not a turnkey scorer for downstream users.
Changes:
- Introduces
eval-qa,eval-mcq, andeval-rrbCLIs with deterministic shuffles, parsing, metrics, and results-card output. - Implements standard classification/ranking metrics plus percentile-bootstrap confidence intervals and paired bootstrap lift CIs.
- Adds documentation (
READMEEvaluation section +REPRODUCTION.md) and unit tests for metrics/parsers.
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| REPRODUCTION.md | Adds detailed evaluation-scope and reproducibility notes plus example results and dataset-integrity facts. |
| README.md | Documents evaluation quickstart, CLI usage, and example outputs. |
| pyproject.toml | Adds eval-* console scripts, an eval optional dependency, and includes evaluate/ in wheels. |
| evaluate/init.py | Package-level documentation and exports for the evaluation harness. |
| evaluate/common.py | Shared JSONL IO, deterministic shuffling, tool referent rendering, and robust response parsers + mocks. |
| evaluate/metrics.py | Implements accuracy/ranking metrics and bootstrap confidence intervals. |
| evaluate/model_client.py | Adds a lightweight model client mirroring the repo’s proxy-vs-direct call pattern. |
| evaluate/eval_qa.py | CLI to score QA (yes/no) with parsing, accuracy, bootstrap CI, and artifact writing. |
| evaluate/eval_mcq.py | CLI to score MCQ with deterministic option shuffle, parsing, accuracy, bootstrap CI, and artifacts. |
| evaluate/eval_rrb.py | CLI to score RRB rankings with Recall@k/hit@k/MRR/nDCG@k, tier breakdowns, and artifacts. |
| evaluate/tests/init.py | Initializes the evaluation test package. |
| evaluate/tests/test_metrics.py | Adds known-answer unit tests for metrics and parsing robustness/determinism. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Resolves the 9 findings from the PR SAP#10 automated review: - parse_choice_letter: normalize a case-insensitively matched decision letter to uppercase (+ regression test) so answers like "Final answer: c" are scored against the uppercase gold labels. - Persist the full raw model response in the QA/MCQ prediction artifacts (was truncated to 200 chars) and add raw + n_parsed to the RRB artifacts, keeping runs auditable and re-parseable as the docs promise. - Guard model_client.complete against max_retries < 1 (previously raised None). - Derive the RRB results-card candidate-pool size from the data instead of a hard-coded literal. - Rename package evaluate -> toolsense_eval to avoid shadowing the HuggingFace evaluate distribution; update imports, console-script targets, wheel include/exclude, and docs. - Fix the module docstring to describe the dual client path (OpenAI-compatible proxy when LITELLM_BASE_URL is set, else litellm directly). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
The repo ships benchmark generation + the three diagnostic datasets, but no turnkey way to score a model on them. This PR adds an
evaluate/package:eval-qa/eval-mcq/eval-rrbCLIs + metrics (accuracy, Recall@k / hit@k / MRR / nDCG@k by tier) with 95% bootstrap confidence intervals, robust answer parsing with unparseable-response counting, a no-API--dry-runpath, 27 unit tests, and docs (README "Evaluation" +REPRODUCTION.md).Scope
Inference-only / in-context; not a reproduction of the paper's trained-ToolGen
Rc@50. Reports--tool-ref name|description|noneso the tool-knowledge lift over a hidden-identity control is measurable (and here, significant).Example run
google/gemma-3-12b-it, full datasets, 0 unparseable; accuracy rises none < name < description (lift CIs exclude 0) and retrieval degrades easy → hard. CIs in the result cards; full table inREPRODUCTION.md. These are illustrativetemperature=0numbers, reproducible up to model/provider nondeterminism.Testing
pytest evaluate/tests(27 pass);eval-* --dry-runruns end-to-end with no API calls.Notes
Apache-2.0; DCO signed-off. Written with AI assistance (Claude) and reviewed/verified by me; original code (standard metrics from scratch, no third-party code copied), consistent with SAP's AI-generated code guidelines.
Closes #9.