pytest-pyeval

A pytest plugin integrating pydantic-evals

Run evals via pytest with the power of fixtures and using a familiar Arrange, Act, Evaluate pattern.

Example

from pyeval import dataset, execute, Case, EqualsExpected, Contains


def uppercase_text(text: str) -> str:
    return text.upper()


@dataset(
    Case(
        name="uppercase_basic",
        inputs="hello world",
        expected_output="HELLO WORLD",
    ),
    Case(
        name="uppercase_with_numbers",
        inputs="hello 123",
        expected_output="HELLO 123",
    ),
)
def eval_uppercase(case: Case):
    result = execute(uppercase_text, case)

    result.evaluate(EqualsExpected())
    result.evaluate(Contains(value="HELLO", case_sensitive=True))

$ uv run pyeval

============================== test session starts ==============================
platform darwin -- Python 3.13.1, pytest-9.0.2, pluggy-1.6.0
plugins: anyio-4.12.1, pyeval-0.1.0
collected 2 items

tests/evals/eval_example.py ●●                                                                         [100%]

============================= 2 evaluated in 0.02s ==============================

Installation

uv add --dev pytest-pyeval

Running evals

pytest-pyeval keeps evals separate from your regular test suite. Evals are excluded from pytest by default, since they are typically slower, hit live APIs, and run on a different cadence to unit tests.

Command	What runs
pytest	Regular tests only (test_*.py)
pytest --evals	Eval tests only (eval_*.py)
pyeval	Shorthand for pytest --evals

pyeval                     # discover and run all evals in the project
pyeval evals/              # run evals under a specific path
pyeval evals/eval_foo.py   # run a single eval file

Logfire integration

pytest-pyeval automatically sends evaluation results to Logfire as experiment traces when Logfire is configured.

Configure Logfire before your evals run using a session-scoped autouse fixture in your conftest.py:

# tests/evals/conftest.py
import logfire
import pytest


@pytest.fixture(scope="session", autouse=True)
def configure_logfire():
    logfire.configure(
        send_to_logfire="if-token-present",
    )

That's it! With LOGFIRE_TOKEN set in your environment, evaluation traces will appear in the Logfire web UI under the Evals view.

To install Logfire:

uv add --dev logfire