fix(eval): make _EvalMetricResultWithInvocation.expected_invocation optional#5221
Open
giulio-leone wants to merge 1 commit intogoogle:mainfrom
Open
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Collaborator
|
Response from ADK Triaging Agent Hello @giulio-leone, thank you for your contribution! Before we can merge this pull request, you'll need to sign the Contributor License Agreement (CLA). You can do so by following the instructions in the "cla/google" check at the bottom of the pull request. Thanks! |
…ptional conversation_scenario eval cases intentionally pass expected_invocation=None from local_eval_service (matching the public EvalMetricResultPerInvocation model), but the private _EvalMetricResultWithInvocation required a non-None Invocation, causing a pydantic ValidationError. Changes: - Make expected_invocation Optional[Invocation] with default None - Guard attribute access in _print_details when expected_invocation is None Fixes google#5214
e757949 to
6668a79
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #5214 —
AgentEvaluator.evaluate()crashes with apydantic ValidationErrorwhen evaluatingconversation_scenarioeval cases because_EvalMetricResultWithInvocation.expected_invocationis typed as requiredInvocation, whilelocal_eval_serviceintentionally passesNonefor scenario-based cases.Changes
src/google/adk/evaluation/agent_evaluator.pyMake
expected_invocationoptional (line 93):This aligns with the public
EvalMetricResultPerInvocationmodel ineval_metrics.py:323.Guard downstream attribute access in
_print_details(lines 439–456):Added
if per_invocation_result.expected_invocation else Noneguards foruser_content,final_response, andintermediate_dataaccesses. Both_convert_content_to_textand_convert_tool_calls_to_textalready handleNonegracefully.Tests
Added
tests/unittests/evaluation/test_eval_metric_result_with_invocation.pywith 5 regression tests:None/ omitted / realexpected_invocation_get_eval_metric_results_with_invocationpropagatesNone_print_detailsdoes not crash withNoneexpected invocationAll 5 tests pass. Existing eval tests unaffected (2 pre-existing failures in
test_custom_metric_evaluator.pyconfirmed onmain).