From 0fbcc390809896de8b0bdde5eb58d145447e9a1a Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 7 Jul 2026 22:12:54 +0900 Subject: [PATCH] fix(strix): treat LLM-backend-unavailable outcomes as neutral, not check failures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Strix required check exits 1 (blocking merges) when its LLM backend is unavailable — GitHub Models "Too many requests" rate limits or the 413 tokens_limit_reached token cap — even though no scan actually ran and no vulnerability was found. That is a CI infrastructure outage masquerading as a security finding. The strix_quick_gate.sh gate returns exit 1 for BOTH genuine blocking vulnerabilities and backend-unavailable fail-closed outcomes (with STRIX_FAIL_ON_PROVIDER_SIGNAL=1). This wraps the "Run Strix (quick)" step to capture the gate exit code and console output and disambiguate: * exit 0 -> pass; exit 2 (config) / any non-1 code -> hard fail (unchanged) * exit 1 -> neutral skip (exit 0 + ::warning) ONLY when a recognized backend-unavailability signal is present AND the scan confirmed zero vulnerabilities AND no vulnerability/severity was reported anywhere. Otherwise the failure is preserved. Real security gating is untouched: any reported finding, config error, or non-backend failure still fails the check. The gate script and its test suite are unchanged; the smoke-test contract (literal `bash "$TRUSTED_STRIX_GATE"`) is preserved. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01AxU2xaupAjp912oDNFuWyd --- .github/workflows/strix.yml | 48 ++++++++++++++++++++++++++++++++++++- 1 file changed, 47 insertions(+), 1 deletion(-) diff --git a/.github/workflows/strix.yml b/.github/workflows/strix.yml index 9214c44f..5c584bf6 100644 --- a/.github/workflows/strix.yml +++ b/.github/workflows/strix.yml @@ -574,7 +574,53 @@ jobs: export "STRIX_MEMORY_COMPRESSOR_${budget_suffix}=10" export "STRIX_PROCESS_${budget_suffix}_SECONDS=$process_budget_seconds" export "STRIX_TOTAL_${budget_suffix}_SECONDS=1800" - bash "$TRUSTED_STRIX_GATE" + + # Capture the gate exit code plus its console output. The gate returns + # exit 1 both for genuine blocking vulnerabilities AND for + # LLM-backend-unavailable outcomes (GitHub Models "Too many requests" + # rate limits, 413 tokens_limit_reached token-cap, connection/warm-up + # failures) that could not complete a scan. A backend outage is CI + # infrastructure noise, not a security finding, so it must not fail + # the required check and block merges. + strix_run_log="$RUNNER_TEMP/strix_gate_console.log" + strix_rc=0 + set +e + bash "$TRUSTED_STRIX_GATE" 2>&1 | tee "$strix_run_log" + strix_rc="${PIPESTATUS[0]}" + set -e + + if [ "$strix_rc" -eq 0 ]; then + exit 0 + fi + + # Preserve configuration failures (exit 2) and any unexpected exit + # code as hard failures — only the scan-failure code (1) can be an + # infrastructure/backend-unavailability outcome. + if [ "$strix_rc" -ne 1 ]; then + exit "$strix_rc" + fi + + # Recognized signals that the LLM backend was unavailable / starved. + backend_unavailable_signal='RateLimitError|Too many requests\. For more on scraping GitHub|"status"[[:space:]]*:[[:space:]]*"RESOURCE_EXHAUSTED"|tokens_limit_reached|Request body too large|Max size:[[:space:]]*[0-9]+[[:space:]]+tokens|Error code:[[:space:]]*413|LLM CONNECTION FAILED|Could not establish connection to the language model|LLM warm-up failed|Configured model and fallback models were unavailable|Configured Vertex model and fallback models were unavailable|emitted provider infrastructure or failure-signal output|before provider infrastructure failure' + # Any evidence that a vulnerability was actually reported. Its presence + # forces a hard failure so real findings are NEVER downgraded. + reported_vulnerability_signal='Vulnerabilities[[:space:]]+[1-9]|severity[[:space:]]*:' + # Positive confirmation the scan observed zero vulnerabilities. + zero_vulnerabilities_signal='Vulnerabilities[[:space:]]+0([^0-9]|$)|reported zero vulnerabilities before provider infrastructure failure' + + # Neutral skip only when ALL hold: a backend-unavailability signal is + # present, the scan confirmed zero vulnerabilities, and no vulnerability + # was reported anywhere. This preserves real security gating: any + # reported finding, config error, or non-backend failure still fails. + if grep -Eiq "$backend_unavailable_signal" "$strix_run_log" \ + && grep -Eq "$zero_vulnerabilities_signal" "$strix_run_log" \ + && ! grep -Eiq "$reported_vulnerability_signal" "$strix_run_log"; then + echo "::warning title=Strix backend unavailable::Strix could not complete because its LLM backend was unavailable (rate limit / token cap / connection or warm-up failure) and reported zero vulnerabilities. Treating as a neutral skip so an infrastructure outage does not block merges; genuine findings still fail the check. See the strix-reports artifact and the run log." + exit 0 + fi + + echo "Strix reported security findings or failed for a non-backend reason; failing the required check (gate exit ${strix_rc})." >&2 + exit "$strix_rc" - name: Collect Strix reports for artifact upload if: ${{ always() && steps.gate.outputs.enabled == 'true' }}