opencode-review: fail fast on hung model pool (350->30 min step timeout)#348
Open
seonghobae wants to merge 2 commits into
Open
opencode-review: fail fast on hung model pool (350->30 min step timeout)#348seonghobae wants to merge 2 commits into
seonghobae wants to merge 2 commits into
Conversation
The "Run OpenCode PR Review model pool" step routinely ran to its multi-hour bound and hard-failed with "timed out after 350 minutes", never producing a pass/fail judgment and blocking merges org-wide. Root cause: run_opencode_review_model_pool.sh loops over the model candidates indefinitely (while :;) and relies solely on this step's timeout-minutes as its wall-clock bound, because OPENCODE_TOTAL_RETRY_BUDGET_SECONDS is intentionally 0 (asserted by scripts/ci/test_strix_quick_gate.sh). When candidate models hang or are rate-limited, the pool churns until the 350-minute step timeout kills it. Fix (minimal, in-design): lower the step timeout-minutes from 350 to 30, tightening the bound the pool already relies on. A healthy lead model emits a full review well within 30 minutes, so a real APPROVE / REQUEST_CHANGES decision and the downstream publish/approve security gates are unchanged; only a pathological hang now fails in minutes instead of hours. Adjacent comments updated to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AxU2xaupAjp912oDNFuWyd
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
opencode-reviewjob's Run OpenCode PR Review model pool step routinely runs to its multi-hour wall-clock bound and fails with:It never produces a pass/fail judgment, so the required check neither passes nor cleanly declines — it just burns the job budget and hard-fails, blocking merges org-wide (12 open PRs across 3 repos).
Root cause
scripts/ci/run_opencode_review_model_pool.shmain()is an intentional infinite loop (while :;) over the model candidates — its own comment says it continues "until a model succeeds or the GitHub Actions job timeout is reached." Its only internal wall-clock guard is a deadline built fromOPENCODE_TOTAL_RETRY_BUDGET_SECONDS, and the workflow sets that to"0"(which disables the deadline — and is deliberately asserted byscripts/ci/test_strix_quick_gate.sh: "opencode model pool has no script-level retry budget").So by design the pool relies entirely on the step's
timeout-minutesto bound it. That was set to350. When candidate models hang or are rate-limited, the pool churns through cycle after cycle until the 350-minute timeout kills the step — hours later, with no verdict.Fix (minimal, in-design)
Lower the model-pool step
timeout-minutesfrom 350 → 30, tightening the exact bound the pool already relies on. No restructuring of the pool script, its security gates, or the deliberateOPENCODE_TOTAL_RETRY_BUDGET_SECONDS=0design.The only functional change is one line (
timeout-minutes: 350 → 30); the rest of the diff is comment updates.Does NOT weaken security gating
The fix only shortens how long a hung reviewer is allowed to spin. It does not relax any pass/fail logic: the publish gate (
opencode_review_approve_gate.sh), the normalize step, and the "Approve PR if OpenCode review passed" gate all still run exactly as before. A real "review did not pass" still blocks the merge.Known limitation / deeper change (not done here — deliberately out of scope)
This makes the check fail fast, but on an infra timeout the step still ends as
failure, so the required check still blocks (just quickly) rather than exiting neutral. Making an infra-timeout exit neutral/skip — so a hung reviewer doesn't block merges at all, while a genuineREQUEST_CHANGESstill blocks — requires distinguishing "infra timeout" from "review declined" in the pool script's exit contract and wiring that through the job's conclusion / branch protection. That is a larger, riskier change to shared org CI and is intentionally left for a follow-up.Separately,
scripts/ci/test_strix_quick_gate.shalready asserts stale values that don't match the live workflow (e.g.timeout-minutes: 285,OPENCODE_RUN_TIMEOUT_SECONDS: "600"vs live5400), indicating it validates a trusted pinned ref rather than PR-head content. It was left untouched here to keep the diff scoped and avoid unrelated churn.Validation
python3 -c "import yaml; yaml.safe_load(open('.github/workflows/opencode-review.yml'))"→ parses OK.🤖 Generated with Claude Code