Add new HF QA tasks and enhance existing benchmarks

README.md

@@ -2,7 +2,7 @@
 
 A benchmarking framework for evaluating LLM-as-optimizer algorithms, built on [OpenTrace](https://github.com/AgentOpt/Trace).
 
-Trace-Bench provides a **CLI**, **Gradio UI**, and **notebook workflows** to run reproducible experiments across multiple benchmarks (LLM4AD, VeriBench, KernelBench) with fair comparisons between trainers, optimizers, and LLM backends.
+Trace-Bench provides a **CLI**, **Gradio UI**, and **notebook workflows** to run reproducible experiments across LLM4AD, VeriBench, KernelBench, and HuggingFace QA tasks with fair comparisons between trainers, optimizers, and LLM backends.
 
 **Full documentation:** [docs/](docs/README.md)
 
@@ -25,8 +25,8 @@ pip install -e ../OpenTrace
 # Install Trace-Bench (core)
 pip install -e .
 
-# Install with optional extras (can be combined):
-pip install -e ".[hf]"          # HuggingFace QA benchmarks (HotpotQA, BBEH, …)
+# Optional extras can be combined:
+pip install -e ".[hf]"          # HuggingFace QA benchmarks
 pip install -e ".[dspy]"        # DSPy trainer adapter
 pip install -e ".[all-external]"  # everything
 ```
@@ -46,7 +46,17 @@ trace-bench ui --runs-dir runs
 
 ## Real-Mode Setup
 
-To run benchmarks with actual LLM calls, configure an API provider:
+To run benchmarks with actual LLM calls, configure a provider through environment variables or a config `llm` block.
+
+Direct OpenAI:
+
+```bash
+export OPENAI_API_KEY="sk-..."
+export TRACE_DEFAULT_LLM_BACKEND="LiteLLM"
+export TRACE_LITELLM_MODEL="gpt-5.4-nano"
+```
+
+OpenRouter-compatible endpoint:
 
 ```bash
 export OPENROUTER_API_KEY="sk-or-v1-..."
@@ -70,6 +80,7 @@ trace_bench/           # Python package
                        # OpenTrace and are discovered automatically.
 benchmarks/
   LLM4AD/                  # 65 algorithm design tasks
+  hf_qa/                   # HuggingFace QA tasks
   KernelBench/             # CUDA kernel optimization
   Veribench/               # Lean 4 formal verification
 configs/                   # YAML experiment configs
@@ -84,6 +95,9 @@ runs/                      # Output directory (gitignored)
 ### LLM4AD (65 tasks)
 Algorithm design tasks from [LLM4AD](https://github.com/Optima-CityU/LLM4AD): optimization (basic, constructive, CO-Bench), machine learning, and scientific discovery.
 
+### HuggingFace QA
+Config-driven QA tasks under `hf:*`: HotpotQA, BBEH, MuSiQue, GSM8K, ARC-Challenge, QASC, DROP, QuALITY, and Qasper. Install with `pip install -e ".[hf]"`.
+
 ### VeriBench (~140 tasks)
 Formal verification tasks translating Python to Lean 4. Integrated via `trace_bench/veribench_adapter.py` with dual discovery: local entrypoint module or [HuggingFace dataset](https://huggingface.co/datasets/allenanie/veribench_with_prompts) fallback.
 
@@ -115,7 +129,7 @@ See [docs/running-experiments.md](docs/running-experiments.md) for full CLI usag
 
 | Extra | Installs | Enables |
 |-------|----------|---------|
-| `hf` | `datasets>=2.0` | `hf:hotpot_qa`, `hf:bbeh/*` tasks |
+| `hf` | `datasets>=2.0` | `hf:*` QA tasks |
 | `dspy` | `dspy-ai>=2.0` | `DSPyTrainer` in `trace_bench/trainers/` |
 | `all-external` | all of the above | everything |
 

benchmarks/hf_qa/bbeh.py

@@ -5,13 +5,20 @@
 
 Reference: ``Predict`` / ``BigBenchGuide`` classes in ``bbeh_trace.py``.
 """
+
 from __future__ import annotations
 
 import re
-from typing import Any
+from typing import Any, Tuple
+
+try:
+    from .common import response_text
+except ImportError:
+    from common import response_text
 
 try:
     from opto.trainer.guide import Guide
+
     _OPTO_AVAILABLE = True
 except Exception:
     _OPTO_AVAILABLE = False
@@ -21,6 +28,7 @@
 # Data utilities
 # ---------------------------------------------------------------------------
 
+
 def format_context(raw: Any) -> str:
     """No-op — BBEH tasks have no context field."""
     return ""
@@ -56,15 +64,18 @@ def eval_metric(true: str, prediction: str) -> bool:
 # ---------------------------------------------------------------------------
 
 if _OPTO_AVAILABLE:
+
     class BBEHGuide(Guide):
         """Evaluation guide for BigBenchExtraHard tasks.
 
         Framework-agnostic: works with any agent that produces a string answer.
         Uses ``eval_metric`` which handles both multiple-choice and free-form answers.
         """
 
-        def get_feedback(self, task: str, response: Any, info: str, **kwargs: Any):
-            response_str = str(getattr(response, "data", response))
+        def get_feedback(
+            self, task: str, response: Any, info: str, **kwargs: Any
+        ) -> Tuple[float, str]:
+            response_str = response_text(response)
             if eval_metric(info, response_str):
                 return 1.0, "The answer is correct."
             return 0.0, (
@@ -74,6 +85,7 @@ def get_feedback(self, task: str, response: Any, info: str, **kwargs: Any):
             )
 
 else:
+
     class BBEHGuide:  # type: ignore[no-redef]
         pass
 

benchmarks/hf_qa/common.py

@@ -0,0 +1,191 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+import re
+import string
+from typing import Any, Dict, List, Sequence, Tuple
+
+
+@dataclass
+class HFQATask:
+    """A single QA example with formatted context and optional task metadata."""
+
+    question: str
+    context: str
+    answer: str
+    meta: Dict[str, Any] = field(default_factory=dict)
+
+
+def normalize_text(text: str) -> str:
+    """Normalize answer text for exact-match and F1-style comparisons."""
+    normalized = str(text).lower()
+    normalized = re.sub(r"\b(a|an|the)\b", " ", normalized)
+    normalized = normalized.translate(str.maketrans("", "", string.punctuation))
+    return " ".join(normalized.split())
+
+
+def token_f1(prediction: str, gold: str) -> float:
+    """Return token-level F1 between normalized prediction and gold text."""
+    pred_tokens = normalize_text(prediction).split()
+    gold_tokens = normalize_text(gold).split()
+    if not pred_tokens and not gold_tokens:
+        return 1.0
+    if not pred_tokens or not gold_tokens:
+        return 0.0
+
+    counts: Dict[str, int] = {}
+    for token in pred_tokens:
+        counts[token] = counts.get(token, 0) + 1
+
+    overlap = 0
+    for token in gold_tokens:
+        if counts.get(token, 0) > 0:
+            overlap += 1
+            counts[token] -= 1
+    if overlap == 0:
+        return 0.0
+
+    precision = overlap / len(pred_tokens)
+    recall = overlap / len(gold_tokens)
+    return 2 * precision * recall / (precision + recall)
+
+
+def extract_last_number(text: str) -> str:
+    """Extract the final integer or decimal number from a model response."""
+    matches = re.findall(r"[-+]?\d[\d,]*\.?\d*", str(text))
+    return matches[-1].replace(",", "") if matches else ""
+
+
+def extract_last_choice(text: str) -> str:
+    """Extract the final parenthesized or standalone choice label."""
+    parenthesized = re.findall(r"\(([A-Za-z0-9]+)\)", str(text))
+    if parenthesized:
+        return parenthesized[-1]
+    standalone = re.findall(r"\b([A-Z])\b", str(text))
+    return standalone[-1] if standalone else ""
+
+
+def response_text(response: Any) -> str:
+    """Extract assistant text from common raw LLM response containers."""
+    value = getattr(response, "data", response)
+
+    choices = (
+        value.get("choices")
+        if isinstance(value, dict)
+        else getattr(value, "choices", None)
+    )
+    if choices:
+        first_choice = choices[0]
+        message = (
+            first_choice.get("message")
+            if isinstance(first_choice, dict)
+            else getattr(first_choice, "message", None)
+        )
+        if message is not None:
+            content = (
+                message.get("content")
+                if isinstance(message, dict)
+                else getattr(message, "content", None)
+            )
+            if content is not None:
+                return str(content)
+        text = (
+            first_choice.get("text")
+            if isinstance(first_choice, dict)
+            else getattr(first_choice, "text", None)
+        )
+        if text is not None:
+            return str(text)
+
+    content = (
+        value.get("content")
+        if isinstance(value, dict)
+        else getattr(value, "content", None)
+    )
+    if content is not None:
+        return str(content)
+    return str(value)
+
+
+def choice_pairs(raw: Any) -> List[Tuple[str, str]]:
+    """Return normalized (label, text) pairs from common HF MCQ schemas."""
+    if isinstance(raw, dict):
+        raw_texts = raw.get("text") or raw.get("texts") or raw.get("options") or []
+        texts = [str(text) for text in raw_texts]
+        raw_labels = raw.get("label") or raw.get("labels") or []
+        labels = [str(label) for label in raw_labels]
+        if not labels:
+            labels = [chr(65 + index) for index in range(len(texts))]
+        return list(zip(labels, texts))
+
+    if isinstance(raw, list):
+        if raw and isinstance(raw[0], dict):
+            pairs: List[Tuple[str, str]] = []
+            for index, item in enumerate(raw):
+                label = str(item.get("label", chr(65 + index)))
+                text = str(item.get("text", item.get("choice", "")))
+                pairs.append((label, text))
+            return pairs
+        return [(chr(65 + index), str(choice)) for index, choice in enumerate(raw)]
+
+    return []
+
+
+def answer_to_choice_label(
+    raw_answer: Any,
+    pairs: Sequence[Tuple[str, str]],
+    index_base: int = 0,
+) -> str:
+    """Map a raw HF answer value to the option label used by the prompt."""
+    if not isinstance(index_base, int) or isinstance(index_base, bool):
+        raise TypeError(f"index_base must be an integer, got {index_base!r}")
+
+    if isinstance(raw_answer, int) and not isinstance(raw_answer, bool):
+        index = raw_answer - index_base
+        if 0 <= index < len(pairs):
+            return pairs[index][0]
+        return str(raw_answer)
+
+    answer = str(raw_answer).strip()
+    for label, text in pairs:
+        if answer == label or normalize_text(answer) == normalize_text(text):
+            return label
+    return answer
+
+
+def format_mcq_prompt(
+    question: str, pairs: Sequence[Tuple[str, str]], context: str = ""
+) -> str:
+    """Format a multiple-choice prompt with optional context."""
+    prefix = f"Context:\n{context}\n\n" if context else ""
+    options = "\n".join(f"({label}) {text}" for label, text in pairs)
+    return f"{prefix}Question: {question}\nOptions:\n{options}\nAnswer:"
+
+
+def split_aliases(raw: str) -> List[str]:
+    """Split a delimiter-separated alias string into non-empty aliases."""
+    return [item.strip() for item in str(raw).split("||") if item.strip()]
+
+
+def normalize_aliases(aliases: Sequence[str]) -> List[str]:
+    """Deduplicate aliases by normalized text while preserving display text."""
+    output: List[str] = []
+    seen = set()
+    for alias in aliases:
+        key = normalize_text(alias)
+        if key and key not in seen:
+            seen.add(key)
+            output.append(str(alias).strip())
+    return output
+
+
+def best_alias_score(prediction: str, aliases: Sequence[str]) -> float:
+    """Score a prediction against aliases using exact, contains, then token F1."""
+    if not aliases:
+        return 0.0
+    normalized_prediction = normalize_text(prediction)
+    if any(normalized_prediction == normalize_text(alias) for alias in aliases):
+        return 1.0
+    if any(normalize_text(alias) in normalized_prediction for alias in aliases):
+        return 1.0
+    return max(token_f1(prediction, alias) for alias in aliases)