Skip to content

fix: make _EvalMetricResultWithInvocation.expected_invocation Optional for conversation_scenario support#5215

Open
ASRagab wants to merge 1 commit intogoogle:mainfrom
ASRagab:fix/optional-expected-invocation
Open

fix: make _EvalMetricResultWithInvocation.expected_invocation Optional for conversation_scenario support#5215
ASRagab wants to merge 1 commit intogoogle:mainfrom
ASRagab:fix/optional-expected-invocation

Conversation

@ASRagab
Copy link
Copy Markdown

@ASRagab ASRagab commented Apr 8, 2026

Summary

  • _EvalMetricResultWithInvocation.expected_invocation is typed as Invocation (required), but local_eval_service.py:285-287 intentionally sets it to None when eval_case.conversation is None (i.e., conversation_scenario user-simulation cases)
  • The public model EvalMetricResultPerInvocation in eval_metrics.py:323 already types this field as Optional[Invocation] = None
  • This mismatch causes a pydantic ValidationError during post-processing in _get_eval_metric_results_with_invocation, after all metrics have been computed

Changes

  • Make expected_invocation Optional[Invocation] = None in _EvalMetricResultWithInvocation
  • Guard the three attribute accesses in _print_details to handle None (fall back to actual_invocation.user_content for the prompt column, None for expected response/tool calls)
  • Both _convert_content_to_text and _convert_tool_calls_to_text already accept Optional parameters

Testing Plan

Verified with a pytest-based evaluation using AgentEvaluator.evaluate() against an evalset containing conversation_scenario cases (LLM-backed user simulation, no explicit conversation arrays).

Before fix — crashes after ~33 minutes of metric computation during post-processing:

pydantic_core._pydantic_core.ValidationError: 1 validation error for _EvalMetricResultWithInvocation
expected_invocation
  Input should be a valid dictionary or instance of Invocation [type=model_type, input_value=None, input_type=NoneType]

.venv/lib/python3.11/site-packages/google/adk/evaluation/agent_evaluator.py:639: ValidationError

After fix — the ValidationError is eliminated. The None expected_invocation flows through correctly because:

  1. The field now accepts Optional[Invocation], matching the upstream EvalMetricResultPerInvocation model
  2. _print_details gracefully handles None by falling back to actual_invocation.user_content for the prompt column and passing None to _convert_content_to_text/_convert_tool_calls_to_text (both already accept Optional inputs)

Reproduction evalset (any evalset with conversation_scenario triggers this):

{
  "eval_set_id": "test",
  "eval_cases": [{
    "eval_id": "scenario_1",
    "conversation_scenario": {
      "starting_prompt": "Hello",
      "conversation_plan": "Ask the agent a question and accept the answer."
    },
    "session_input": {"app_name": "my_agent", "user_id": "user1", "state": {}}
  }]
}
@pytest.mark.asyncio
async def test_scenario():
    await AgentEvaluator.evaluate("my_agent", "path/to/evalset.json", num_runs=1)

Fixes #5214

When using conversation_scenario for user simulation, expected_invocation
is None because conversations are dynamically generated. The public model
EvalMetricResultPerInvocation already types this as Optional[Invocation],
but the private _EvalMetricResultWithInvocation requires non-None, causing
a pydantic ValidationError during post-processing.

- Make expected_invocation Optional[Invocation] = None
- Guard attribute accesses in _print_details to handle None
- Fall back to actual_invocation.user_content for the prompt column

Fixes google#5214
@adk-bot adk-bot added the eval [Component] This issue is related to evaluation label Apr 8, 2026
@adk-bot
Copy link
Copy Markdown
Collaborator

adk-bot commented Apr 8, 2026

Response from ADK Triaging Agent

Hello @ASRagab, thank you for submitting this pull request!

To help the reviewers, could you please add a testing plan section to your PR description explaining how you verified the fix? For example, did you run the evaluation with a conversation_scenario?

Including the logs or a screenshot showing that the ValidationError is gone after your change would also be very helpful.

You can find more details in our contribution guidelines. Thanks!

@rohityan rohityan self-assigned this Apr 9, 2026
@ASRagab ASRagab changed the title fix: make _EvalMetricResultWithInvocation.expected_invocation Optional for conversation_scenario support fix: make _EvalMetricResultWithInvocation.expected_invocation Optional for conversation_scenario support Apr 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

eval [Component] This issue is related to evaluation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AgentEvaluator crashes with ValidationError when evaluating conversation_scenario eval cases

4 participants