Skip to content

Revert "Revert "add default try catch to evaluator function""#302

Merged
mayinghan merged 9 commits intomainfrom
revert-301-revert-297-try-catch-on-eval-func
Oct 30, 2025
Merged

Revert "Revert "add default try catch to evaluator function""#302
mayinghan merged 9 commits intomainfrom
revert-301-revert-297-try-catch-on-eval-func

Conversation

@mayinghan
Copy link
Copy Markdown
Collaborator

@mayinghan mayinghan commented Oct 29, 2025

Reverts #301


Note

Catches exceptions in evaluator functions to produce invalid zero-score results, preserves/updates statuses appropriately, deep-copies eval metadata, and adds comprehensive tests.

  • Pytest evaluator:
    • Add try/except around execute_pytest in both pointwise and groupwise modes to set evaluation_result (score=0.0, is_score_valid=False, detailed reason) and mark eval_metadata.status = Status.error(...).
    • Only set eval_finished() if status is unset or still RUNNING; avoid overriding error states.
    • Deep-copy eval_metadata per row via eval_metadata.model_copy(deep=True).
  • Postprocessing:
    • When is_score_valid is False, set Status.score_invalid() only if status is missing or not already an error.
  • Exception config:
    • Minor cleanup of DEFAULT_RETRYABLE_EXCEPTIONS entries/trailing commas.
  • Tests:
    • Add tests/pytest/test_pytest_evaluator_error_handling.py covering pointwise/groupwise exception handling, status codes, edge cases, and input_rows flow.

Written by Cursor Bugbot for commit 138d537. This will update automatically on new commits. Configure here.

@mayinghan mayinghan requested a review from dphuang2 October 29, 2025 22:28
cursor[bot]

This comment was marked as outdated.

Copy link
Copy Markdown
Collaborator

@dphuang2 dphuang2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@mayinghan mayinghan merged commit ad98650 into main Oct 30, 2025
9 checks passed
@mayinghan mayinghan deleted the revert-301-revert-297-try-catch-on-eval-func branch October 30, 2025 07:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants