Skip to content

fix: skip eval invocations without user content to prevent ValidationError#6134

Closed
tcconnally wants to merge 3 commits into
google:mainfrom
Perseus-Computing-LLC:fix/eval-user-content-validation-error
Closed

fix: skip eval invocations without user content to prevent ValidationError#6134
tcconnally wants to merge 3 commits into
google:mainfrom
Perseus-Computing-LLC:fix/eval-user-content-validation-error

Conversation

@tcconnally

Copy link
Copy Markdown

Problem

When a session contains invocations without user-authored events (author != "user"), convert_events_to_eval_invocations() creates an Invocation with an empty Content(parts=[]). This causes a Pydantic ValidationError because Invocation.user_content is typed as genai_types.Content and empty Content triggers validation failure.

Error when calling POST /apps/{app_name}/eval-sets/{eval_set_id}/add-session:

ValidationError on Invocation.user_content

Fix

Skip invocations entirely when no user event is found. Evaluations without user input are not meaningful, so filtering them out is both a correctness fix and an efficiency improvement.

Before

for invocation_id, events in events_by_invocation_id.items():
    user_content = Content(parts=[])
    ...
    invocations.append(Invocation(user_content=user_content, ...))

After

for invocation_id, events in events_by_invocation_id.items():
    user_content = Content(parts=[])
    ...
    if not user_content.parts:  # No user event found
        continue
    invocations.append(Invocation(user_content=user_content, ...))

Closes #3760

@rohityan rohityan self-assigned this Jun 15, 2026
@haranrk haranrk assigned haranrk and unassigned rohityan Jun 16, 2026
@rohityan rohityan added eval [Component] This issue is related to evaluation request clarification [Status] The maintainer need clarification or more information from the author labels Jun 17, 2026
@rohityan

Copy link
Copy Markdown
Collaborator

Hi @tcconnally , Thank you for your contribution! We appreciate you taking the time to submit this pull request. Can you please fix the failing mypy-diff tests before we can proceed with the review.

…Error

When a session contains invocations without user-authored events,
convert_events_to_eval_invocations() created an Invocation with an
empty Content(parts=[]). This caused a Pydantic ValidationError because
Invocation.user_content is typed as genai_types.Content (non-optional)
and an empty Content with no parts triggered validation failure.

Fix: skip invocations entirely when no user event is found in the
event list, since evaluations without user input are not meaningful.

Closes google#3760
@tcconnally

Copy link
Copy Markdown
Author

Fixed the mypy issue — added user_content None guard alongside the empty parts check. Rebased on main.

@tcconnally tcconnally force-pushed the fix/eval-user-content-validation-error branch from 41a49b0 to 9352f62 Compare June 17, 2026 18:40
@rohityan rohityan added needs review [Status] The PR/issue is awaiting review from the maintainer and removed request clarification [Status] The maintainer need clarification or more information from the author labels Jun 17, 2026
@haranrk

haranrk commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator

Thanks for your contribution @tcconnally! This is a duplicate of #5287
#5287 (comment)

@haranrk haranrk closed this Jun 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

eval [Component] This issue is related to evaluation needs review [Status] The PR/issue is awaiting review from the maintainer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evaluation Invocation.user_content ValidationError when session has invocations without user events

3 participants