feat: support structured criteria scoring in verifier results

[andrew-stelmach-fleet]

Summary

• Update verifier result processing (SyncVerifierFunction, AsyncVerifierFunction, DecoratorVerifierFunction) to preserve full dict results when they contain structured criteria data (e.g. {"result": 0.84, "criteria": [...]})
• Previously, dict results were always reduced to a float score, which dropped the criteria breakdown data needed by the frontend
• Companion to orchestrator PR: https://github.com/fleet-ai/theseus/pull/1801 and frontend PR: https://github.com/fleet-ai/client/pull/1737

Changes

fleet/verifiers/verifier.py (sync)

• __call__(): Accept dict returns with "result" key (not just "score")
• remote(): Removed -> float return annotation to allow dict returns
• _process_result(): When dict contains "criteria" key, return full dict instead of extracting just the score

fleet/_async/verifiers/verifier.py (async)

• Same changes applied to async versions of __call__, remote, _process_result

fleet/verifiers/decorator.py

• __call__(): Same dict-awareness pattern — returns full dict when result contains "result" key

tests/test_verifier_criteria.py (new)

• Tests for sync verifier: float return, dict-with-criteria return, dict-without-criteria return
• Tests for async verifier: same coverage
• Tests for decorator verifier: float and dict-with-criteria returns

How it works

When a verifier function returns a dict like:

{
    "result": 0.84,
    "criteria": [
        {"criteria": "Accuracy", "score": 0.95, "score_out_of": 1.0},
        {"criteria": "Quality", "score": 0.6, "score_out_of": 1.0}
    ]
}

The SDK now preserves the full dict instead of reducing it to 0.84. This allows the orchestrator (theseus PR #1801) to store the full result in verifier_executions.result jsonb column, and the frontend (client PR #1737) to render the criteria breakdown.

Tests

• tests/test_verifier_criteria.py — 8 test cases covering all verifier types
• Run: pytest tests/test_verifier_criteria.py -v

Test plan

• Verify pytest tests/test_verifier_criteria.py -v passes
• Verify existing tests still pass: pytest tests/ -v
• Integration test: verifier returning {"result": 0.84, "criteria": [...]} flows through to Supabase verifier_executions.result

🤖 Generated with Claude Code

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}

[cursor]

Misleading test masks coverage gap in score-only dict path

Medium Severity

test_call_returns_dict_with_score_key claims to verify that a dict with a "score" key is returned as-is, but the test data also includes a "criteria" key. It's the "criteria" key that triggers the full-dict return path, not the "score" key alone. A dict like {"score": 0.84} (without "criteria") would actually be reduced to the float 0.84 by __call__. The test accidentally passes and provides a false sense of coverage for the score-only dict path through __call__, which is a behavioral change from the previous code (which returned full dicts for any dict with "score").

 

[cursor]

Unused verifier variable in error test

Low Severity

In test_call_error_returns_zero, the variable v is created from a lambda-based verifier on line 86 but is never called or asserted against. The test only exercises v2. This appears to be leftover scaffolding code that adds confusion without contributing to test coverage.