Skip to content

fix(strix): treat LLM-backend-unavailable outcomes as neutral, not required-check failures#349

Open
seonghobae wants to merge 1 commit into
mainfrom
fix/strix-infra-neutral
Open

fix(strix): treat LLM-backend-unavailable outcomes as neutral, not required-check failures#349
seonghobae wants to merge 1 commit into
mainfrom
fix/strix-infra-neutral

Conversation

@seonghobae

Copy link
Copy Markdown
Contributor

Problem

The strix required check fails merges (exit 1) whenever its LLM backend is unavailable rather than when it actually finds a vulnerability:

  • GitHub Models rate limiting — openai.RateLimitError: Too many requests. For more on scraping GitHub...
  • The 413 token-cap — Error code: 413 ... tokens_limit_reached / Max size: 4000 tokens
  • LLM connection / warm-up failures.

In these cases no scan completed and zero vulnerabilities were reported, yet the required check goes red and blocks every dependent repo's PRs. This is a CI infrastructure outage masquerading as a security finding.

Root cause: scripts/ci/strix_quick_gate.sh returns exit 1 for both genuine blocking vulnerabilities and backend-unavailable fail-closed outcomes (the workflow sets STRIX_FAIL_ON_PROVIDER_SIGNAL: "1", so provider-signal failures fail closed). The Run Strix (quick) step could not tell the two apart.

Fix (minimal, workflow-only)

Wrap the Run Strix (quick) step to capture the gate's exit code and console output, then disambiguate:

Gate result Outcome
exit 0 pass (unchanged)
exit 2 (config) or any non-1 code hard fail (unchanged)
exit 1 and a backend-unavailability signal is present and the scan confirmed Vulnerabilities 0 and no severity: / Vulnerabilities [1-9] appears anywhere neutral skipexit 0 with a ::warning
exit 1 otherwise (any reported finding, or non-backend failure) fail (unchanged)

The neutral path fires only when all three conditions hold, so it can never downgrade a run that reported a finding.

Why this does NOT weaken security gating

  • A neutral skip requires positive proof of zero vulnerabilities (Vulnerabilities 0 or the gate's own "reported zero vulnerabilities before provider infrastructure failure" message) and the absence of any severity: / non-zero Vulnerabilities line. Any real finding keeps the failure.
  • Config errors (exit 2) and non-backend scan failures still fail.
  • Backend signals are matched with specific strings (RateLimitError, RESOURCE_EXHAUSTED, tokens_limit_reached, Error code: 413, LLM CONNECTION FAILED, the gate's own "unavailable"/"provider infrastructure" verdicts) — bare numeric 429/413 were intentionally excluded to avoid spurious matches.
  • The shared, heavily-tested strix_quick_gate.sh and its test harness are not touched. The required-workflow smoke-test contract (literal bash "$TRUSTED_STRIX_GATE") is preserved and the smoke test still passes.

Validation

  • python3 -c "import yaml; yaml.safe_load(open('.github/workflows/strix.yml'))" → ok
  • bash scripts/ci/strix_required_workflow_smoke.sh → passed
  • bash -n on the extracted step body → ok
  • 9-case classification table (rate limit / 413 / real vuln + noise / clean vuln / below-threshold+infra / config error / non-backend fail / success / Vertex RESOURCE_EXHAUSTED) → all match intended pass/fail/neutral.

🤖 Generated with Claude Code

…eck failures

The Strix required check exits 1 (blocking merges) when its LLM backend is
unavailable — GitHub Models "Too many requests" rate limits or the
413 tokens_limit_reached token cap — even though no scan actually ran and no
vulnerability was found. That is a CI infrastructure outage masquerading as a
security finding.

The strix_quick_gate.sh gate returns exit 1 for BOTH genuine blocking
vulnerabilities and backend-unavailable fail-closed outcomes (with
STRIX_FAIL_ON_PROVIDER_SIGNAL=1). This wraps the "Run Strix (quick)" step to
capture the gate exit code and console output and disambiguate:

  * exit 0 -> pass; exit 2 (config) / any non-1 code -> hard fail (unchanged)
  * exit 1 -> neutral skip (exit 0 + ::warning) ONLY when a recognized
    backend-unavailability signal is present AND the scan confirmed zero
    vulnerabilities AND no vulnerability/severity was reported anywhere.
    Otherwise the failure is preserved.

Real security gating is untouched: any reported finding, config error, or
non-backend failure still fails the check. The gate script and its test suite
are unchanged; the smoke-test contract (literal `bash "$TRUSTED_STRIX_GATE"`)
is preserved.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AxU2xaupAjp912oDNFuWyd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant