Skip to content

opencode-review: fail fast on hung model pool (350->30 min step timeout)#348

Open
seonghobae wants to merge 2 commits into
mainfrom
fix/opencode-review-timeout
Open

opencode-review: fail fast on hung model pool (350->30 min step timeout)#348
seonghobae wants to merge 2 commits into
mainfrom
fix/opencode-review-timeout

Conversation

@seonghobae

Copy link
Copy Markdown
Contributor

Problem

The opencode-review job's Run OpenCode PR Review model pool step routinely runs to its multi-hour wall-clock bound and fails with:

The action Run OpenCode PR Review model pool has timed out after 350 minutes

It never produces a pass/fail judgment, so the required check neither passes nor cleanly declines — it just burns the job budget and hard-fails, blocking merges org-wide (12 open PRs across 3 repos).

Root cause

scripts/ci/run_opencode_review_model_pool.sh main() is an intentional infinite loop (while :;) over the model candidates — its own comment says it continues "until a model succeeds or the GitHub Actions job timeout is reached." Its only internal wall-clock guard is a deadline built from OPENCODE_TOTAL_RETRY_BUDGET_SECONDS, and the workflow sets that to "0" (which disables the deadline — and is deliberately asserted by scripts/ci/test_strix_quick_gate.sh: "opencode model pool has no script-level retry budget").

So by design the pool relies entirely on the step's timeout-minutes to bound it. That was set to 350. When candidate models hang or are rate-limited, the pool churns through cycle after cycle until the 350-minute timeout kills the step — hours later, with no verdict.

Fix (minimal, in-design)

Lower the model-pool step timeout-minutes from 350 → 30, tightening the exact bound the pool already relies on. No restructuring of the pool script, its security gates, or the deliberate OPENCODE_TOTAL_RETRY_BUDGET_SECONDS=0 design.

  • A healthy lead model (o4-mini / o3-mini, the reliable contract-emitters) produces a full review well within 30 minutes, so a genuine APPROVE or REQUEST_CHANGES decision — and the downstream publish-gate and approve-gate security checks — are unchanged.
  • Only a pathological hang / rate-limit stall now fails in ~30 minutes instead of ~6 hours ("minutes, not hours").
  • Adjacent comments updated so the file stays truthful about the new bound.

The only functional change is one line (timeout-minutes: 350 → 30); the rest of the diff is comment updates.

Does NOT weaken security gating

The fix only shortens how long a hung reviewer is allowed to spin. It does not relax any pass/fail logic: the publish gate (opencode_review_approve_gate.sh), the normalize step, and the "Approve PR if OpenCode review passed" gate all still run exactly as before. A real "review did not pass" still blocks the merge.

Known limitation / deeper change (not done here — deliberately out of scope)

This makes the check fail fast, but on an infra timeout the step still ends as failure, so the required check still blocks (just quickly) rather than exiting neutral. Making an infra-timeout exit neutral/skip — so a hung reviewer doesn't block merges at all, while a genuine REQUEST_CHANGES still blocks — requires distinguishing "infra timeout" from "review declined" in the pool script's exit contract and wiring that through the job's conclusion / branch protection. That is a larger, riskier change to shared org CI and is intentionally left for a follow-up.

Separately, scripts/ci/test_strix_quick_gate.sh already asserts stale values that don't match the live workflow (e.g. timeout-minutes: 285, OPENCODE_RUN_TIMEOUT_SECONDS: "600" vs live 5400), indicating it validates a trusted pinned ref rather than PR-head content. It was left untouched here to keep the diff scoped and avoid unrelated churn.

Validation

python3 -c "import yaml; yaml.safe_load(open('.github/workflows/opencode-review.yml'))" → parses OK.

🤖 Generated with Claude Code

Seongho Bae and others added 2 commits July 7, 2026 22:12
The "Run OpenCode PR Review model pool" step routinely ran to its
multi-hour bound and hard-failed with "timed out after 350 minutes",
never producing a pass/fail judgment and blocking merges org-wide.

Root cause: run_opencode_review_model_pool.sh loops over the model
candidates indefinitely (while :;) and relies solely on this step's
timeout-minutes as its wall-clock bound, because
OPENCODE_TOTAL_RETRY_BUDGET_SECONDS is intentionally 0 (asserted by
scripts/ci/test_strix_quick_gate.sh). When candidate models hang or are
rate-limited, the pool churns until the 350-minute step timeout kills it.

Fix (minimal, in-design): lower the step timeout-minutes from 350 to 30,
tightening the bound the pool already relies on. A healthy lead model
emits a full review well within 30 minutes, so a real APPROVE /
REQUEST_CHANGES decision and the downstream publish/approve security
gates are unchanged; only a pathological hang now fails in minutes
instead of hours. Adjacent comments updated to match.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AxU2xaupAjp912oDNFuWyd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant