fix(strix): treat LLM-backend-unavailable outcomes as neutral, not required-check failures#349
Open
seonghobae wants to merge 1 commit into
Open
fix(strix): treat LLM-backend-unavailable outcomes as neutral, not required-check failures#349seonghobae wants to merge 1 commit into
seonghobae wants to merge 1 commit into
Conversation
…eck failures
The Strix required check exits 1 (blocking merges) when its LLM backend is
unavailable — GitHub Models "Too many requests" rate limits or the
413 tokens_limit_reached token cap — even though no scan actually ran and no
vulnerability was found. That is a CI infrastructure outage masquerading as a
security finding.
The strix_quick_gate.sh gate returns exit 1 for BOTH genuine blocking
vulnerabilities and backend-unavailable fail-closed outcomes (with
STRIX_FAIL_ON_PROVIDER_SIGNAL=1). This wraps the "Run Strix (quick)" step to
capture the gate exit code and console output and disambiguate:
* exit 0 -> pass; exit 2 (config) / any non-1 code -> hard fail (unchanged)
* exit 1 -> neutral skip (exit 0 + ::warning) ONLY when a recognized
backend-unavailability signal is present AND the scan confirmed zero
vulnerabilities AND no vulnerability/severity was reported anywhere.
Otherwise the failure is preserved.
Real security gating is untouched: any reported finding, config error, or
non-backend failure still fails the check. The gate script and its test suite
are unchanged; the smoke-test contract (literal `bash "$TRUSTED_STRIX_GATE"`)
is preserved.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AxU2xaupAjp912oDNFuWyd
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The strix required check fails merges (exit 1) whenever its LLM backend is unavailable rather than when it actually finds a vulnerability:
openai.RateLimitError: Too many requests. For more on scraping GitHub...Error code: 413 ... tokens_limit_reached/Max size: 4000 tokensIn these cases no scan completed and zero vulnerabilities were reported, yet the required check goes red and blocks every dependent repo's PRs. This is a CI infrastructure outage masquerading as a security finding.
Root cause:
scripts/ci/strix_quick_gate.shreturns exit 1 for both genuine blocking vulnerabilities and backend-unavailable fail-closed outcomes (the workflow setsSTRIX_FAIL_ON_PROVIDER_SIGNAL: "1", so provider-signal failures fail closed). TheRun Strix (quick)step could not tell the two apart.Fix (minimal, workflow-only)
Wrap the
Run Strix (quick)step to capture the gate's exit code and console output, then disambiguate:Vulnerabilities 0and noseverity:/Vulnerabilities [1-9]appears anywhereexit 0with a::warningThe neutral path fires only when all three conditions hold, so it can never downgrade a run that reported a finding.
Why this does NOT weaken security gating
Vulnerabilities 0or the gate's own "reported zero vulnerabilities before provider infrastructure failure" message) and the absence of anyseverity:/ non-zeroVulnerabilitiesline. Any real finding keeps the failure.RateLimitError,RESOURCE_EXHAUSTED,tokens_limit_reached,Error code: 413,LLM CONNECTION FAILED, the gate's own "unavailable"/"provider infrastructure" verdicts) — bare numeric429/413were intentionally excluded to avoid spurious matches.strix_quick_gate.shand its test harness are not touched. The required-workflow smoke-test contract (literalbash "$TRUSTED_STRIX_GATE") is preserved and the smoke test still passes.Validation
python3 -c "import yaml; yaml.safe_load(open('.github/workflows/strix.yml'))"→ okbash scripts/ci/strix_required_workflow_smoke.sh→ passedbash -non the extracted step body → ok🤖 Generated with Claude Code