feat: adds ability to use inverted judges

[andrewklatzke]

Requirements

• I have added test coverage for new or changed functionality
• I have followed the repository's pull request submission guidelines
• I have validated my changes against all supported platform versions

Describe the solution you've provided

Implements handling for "inverted" judges.

Describe alternatives you've considered

This gets feature parity with our online evals functionality; no alternatives considered.

Additional context

When a metric has is_inverted set, it's intended that the evaluation of the score flips from >= to <=. This adds a util _judge_passed to handle that logic and implements it throughout. We don't surface the inverted property in the SDK, so we fetch the judge directly to get this information.

---

Note

Medium Risk
Changes core judge pass/fail semantics and adds per-judge REST calls (get_ai_config) during config-driven runs, which could affect optimization outcomes and introduce new failure/performance modes if the API is unavailable or slow.

Overview
Adds first-class support for inverted judges (where lower scores are better) by introducing a shared judge_passed helper and using it for pass/fail decisions in OptimizationClient and in prompt feedback generation.

Extends OptimizationJudge with an is_inverted flag and, for optimize_from_config, fetches each judge’s isInverted value via api_client.get_ai_config when building options. Updates logging to include the inverted status, and adds targeted tests covering the helper, mixed inverted/standard evaluation, config building behavior, and variation_prompt_feedback output.

^{Reviewed by Cursor Bugbot for commit a8f14de. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a8f14de. Configure here.}

[cursor]

Inverted judge logic missed in fallback threshold branch

Medium Severity

In variation_prompt_feedback, the else branch (when optimization_judge.threshold is None) still uses passed = score >= 1.0 instead of calling judge_passed(score, 1.0, optimization_judge.is_inverted). For inverted judges hitting this path, a low score (which should pass) would be marked as FAILED, and only a perfect 1.0 would pass — the exact opposite of the intended behavior. The other two equivalent locations in client.py correctly default to 1.0 and pass it through judge_passed.

 

^{Reviewed by Cursor Bugbot for commit a8f14de. Configure here.}