Skip to content

Fix the auto-evaluation triggering in pr review agentic workflows#728

Merged
JanKrivanek merged 2 commits into
mainfrom
dev/jankrivanek/pr-triage-autoeval-fix
Jun 5, 2026
Merged

Fix the auto-evaluation triggering in pr review agentic workflows#728
JanKrivanek merged 2 commits into
mainfrom
dev/jankrivanek/pr-triage-autoeval-fix

Conversation

@JanKrivanek
Copy link
Copy Markdown
Member

Fix: pr-triage worker no longer skips evaluate-now due to placeholder pr-status runs

Symptom. PRs were correctly classified as ready-for-eval and labeled pr-state/ready-for-eval, but the evaluate-now label was never applied, so evaluation.yml's label-driven entry point never fired. Repro on #703 (head a8bb6fc) and #720 (head 638726129): every hourly worker run logged eval-trigger: a workflow run already exists for <sha> — skipping.

Root cause. eval_run_exists_for_head() in .github/scripts/pr-triage-act.sh counted every evaluation.yml run for the head SHA. But evaluation.yml always posts a pr-status run (event=pull_request) on each push just to set the initial commit status — that run is not a real evaluation trigger. Its presence made the gate think evaluation had already run, so the worker skipped applying evaluate-now.

Fix. Filter the runs query to only events that actually trigger evaluation:

  • issue_comment (/evaluate)
  • workflow_dispatch (manual)
  • pull_request_target with conclusion != "skipped" (the label-driven gate job runs; the status-only fork variant ends skipped)

Verification.

  • Stuck SHAs (a8bb6fc, 638726129): filtered count = 0 → worker will now apply evaluate-now.
  • SHA with a real /evaluate run (01050ea83): filtered count = 5 → gate still suppresses re-trigger as intended.

Unsticking existing PRs. This change ships in main, but worker checkouts already use the default branch, so the next hourly batch (cron :17) will fire evaluation on all currently-stuck pr-state/ready-for-eval PRs automatically. No manual label nudge required.

Copilot AI review requested due to automatic review settings June 5, 2026 10:48
@JanKrivanek JanKrivanek enabled auto-merge (squash) June 5, 2026 10:49
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes the PR-triage worker’s “evaluation already ran” gate so it doesn’t get tripped by placeholder evaluation.yml runs that only post an initial evaluation-status commit status, allowing the evaluate-now label to be applied and the label-driven evaluation entry point to fire.

Changes:

  • Update .github/scripts/pr-triage-act.sh to count only evaluation workflow runs that represent real evaluation triggers (vs status-only placeholder runs).
  • Switch the workflow-run query to a paginated scan with a jq filter over event types/conclusions.
Show a summary per file
File Description
.github/scripts/pr-triage-act.sh Refines the “evaluation run exists” detection logic used by the triage worker before applying evaluate-now.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 1/1 changed files
  • Comments generated: 1

Comment thread .github/scripts/pr-triage-act.sh Outdated
@github-actions github-actions Bot added the waiting-on-author PR state label label Jun 5, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 5, 2026

👋 @JanKrivanek — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

…nclusion

Fork PRs' fork-pr-status placeholder run is pull_request_target with conclusion=success, so the previous event/conclusion filter still counted it as a real evaluation and suppressed the evaluate-now label. Inspect each run's jobs and treat a run as a real evaluation only when its gate or discover job is non-skipped. Addresses Copilot review feedback on PR #728.
@JanKrivanek JanKrivanek merged commit 2604067 into main Jun 5, 2026
34 checks passed
@JanKrivanek JanKrivanek deleted the dev/jankrivanek/pr-triage-autoeval-fix branch June 5, 2026 11:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-on-author PR state label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants