inspect.signature in _call_user_fn_args raises NameError under PEP 649 with TYPE_CHECKING-only annotations

[willfrey]

Summary

_call_user_fn_args in braintrust/framework.py calls inspect.signature(fn) without specifying annotation_format. On Python 3.14 (PEP 649), this defaults to Format.VALUE, which eagerly evaluates annotations. If a callback uses TYPE_CHECKING-only imports in its annotations, the signature call raises NameError.

The bare except on line ~467 catches the error, but the fallback behavior (return [], kwargs) passes all kwargs through unfiltered. For task functions, this means Braintrust can't detect whether the task accepts a hooks parameter (line ~1468), so it silently omits hooks — crashing the task on the missing required argument.

Reproduction

# Python 3.14+
from typing import TYPE_CHECKING, Any
from braintrust import EvalCase, EvalHooks, Eval

if TYPE_CHECKING:
    from some_package import SomeType  # not available at runtime

async def my_task(
    input: str,
    hooks: EvalHooks[frozenset[SomeType]],  # <-- NameError at runtime
) -> dict:
    return {"answer": input}

async def my_scorer(
    input: str,
    output: dict,
    expected: frozenset[Any],
    **_: object,
) -> float:
    return 1.0

Eval(
    "repro",
    data=[EvalCase(input="hello", expected=frozenset())],
    task=my_task,
    scores=[my_scorer],
)

inspect.signature(my_task) raises NameError: name 'SomeType' is not defined because PEP 649's lazy annotation evaluation is triggered by inspect.signature in VALUE format.

Suggested fix

Use annotation_format=annotationlib.Format.FORWARDREF (or inspect.Format.FORWARDREF) in the inspect.signature call. FORWARDREF wraps unresolvable names as ForwardRef objects instead of raising, and parameter name/kind inspection (which is all _call_user_fn_args needs) works identically.

import annotationlib

try:
    signature = inspect.signature(fn, annotation_format=annotationlib.Format.FORWARDREF)
except:
    return [], kwargs

This also applies to the inspect.signature call on line ~1468 that checks whether the task accepts a hooks argument.

Workaround

Annotate the hooks parameter with EvalHooks[Any] instead of a concrete type. Any is always available at runtime.

Environment

• Python 3.14.3
• braintrust 0.14.0

[AbhiPrasad]

We need to just make our typing story more robust here in general. Expand our test suite to hit mypy, ty, and pyright all at once.

[willfrey]

I appreciate you humoring me with the type annotation issues :)