Skip to content

fix(security): lower thinking effort for security explanations - #470

Draft
johnmatthewtennant wants to merge 5 commits into
mainfrom
jtennant/gordon/security-explanation-effort
Draft

johnmatthewtennant wants to merge 5 commits into
mainfrom
jtennant/gordon/security-explanation-effort

Conversation

@johnmatthewtennant

@johnmatthewtennant johnmatthewtennant commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This fixes a bug: when a security alert has no detector-written explanation, Berd should generate one, but it can fail with "An explanation could not be generated" and nothing is logged.

inferSecurityExplanation runs the explanation in a new hidden session, and that session inherits the user's default thinking effort. At high or max effort, the model can spend the whole 20 s budget thinking before it writes any text. promptForText then returns null, and the catch discards every other error, so the cause never reaches a log.

This PR:

  • sets thinking_effort to low on the hidden session after setModel (which can rebuild that option) and before the prompt. Not off: always-on thinking models such as Fable 5 treat off as high, so it would raise effort for a user whose default is low. OpenAI reasoning models already send low for off. A model without the option keeps its default, and the explanation still runs.
  • logs the failure reason to the app log through logRendererEvent: security explanation failed: no reply text within 20000 ms for a timeout or empty reply, or the formatted ACP error.

Related issue

None found.

Testing

Red/green through the production path: Berd's inferSecurityExplanation ran against the pinned Goose goose serve over ACP. The model was a local stub that records the effort Goose sends and takes longer than the 20 s budget at high, so model latency was simulated.

Model User default Requested by Berd Effort sent Result
Fable 5 high nothing (main) high no explanation (20 s timeout)
Fable 5 low off high no explanation (20 s timeout)
Fable 5 high or low low (this PR) low explanation returned
Opus 5 high low (this PR) low (adaptive) explanation returned
OpenAI o3 high nothing (main) high no explanation (20 s timeout)
OpenAI o3 high low (this PR) low explanation returned

Live calls with a user default of high, two per model and effort, all returned an explanation inside the budget. At low, first text arrived in 1.8 to 2.8 s on Opus 5 (6.8 to 8.7 s at inherited high) and 1.8 to 1.9 s on Fable 5 (2.0 to 5.9 s).

Unit tests in inferExplanation.test.ts cover the fixed contract:

  • effort low is set between setModel and the prompt (fails on main and with off);
  • both failure reasons are logged (both fail on main);
  • an explanation still returns when setting the option fails (a regression guard).

Not verified: opening an alert without a detector-written explanation in a packaged build. The runs above call inferSecurityExplanation directly rather than going through the alert UI.

🤖 Generated with Claude Code

johnmatthewtennant and others added 4 commits November 5, 2026 10:55
A security alert's explanation runs in a hidden session that inherits the
user's default thinking effort. At high or max effort the model can think
past the 20 s budget before writing text, and the failure is never logged.

These tests expect the session to turn thinking off and the failure reason
to reach the app log.

Co-authored-by: Gordon <6581f171b4f294207c04c83d43b057ea88caa717457980f756160f0212d4f429@buzz.block.builderlab.xyz>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: John Matthew Tennant <jtennant@block.xyz>
Set thinking_effort to off on the hidden explanation session after
setModel, so the model answers within the 20 s budget. Models without the
option keep their default.

Log why an explanation failed (no reply text in time, or the error) to the
app log instead of discarding it.

Co-authored-by: Gordon <6581f171b4f294207c04c83d43b057ea88caa717457980f756160f0212d4f429@buzz.block.builderlab.xyz>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: John Matthew Tennant <jtennant@block.xyz>
Some always-on thinking models map off to high at the Goose pin, so
off can raise the explanation's effort above a user's low default.

Co-authored-by: Gordon <6581f171b4f294207c04c83d43b057ea88caa717457980f756160f0212d4f429@buzz.block.builderlab.xyz>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: John Matthew Tennant <jtennant@block.xyz>
Off maps to high on always-on thinking models at the Goose pin, so it
could raise effort for users whose default is low or medium. Low is the
lowest effort every thinking model honors, and OpenAI already sends low
for off.

Co-authored-by: Gordon <6581f171b4f294207c04c83d43b057ea88caa717457980f756160f0212d4f429@buzz.block.builderlab.xyz>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: John Matthew Tennant <jtennant@block.xyz>
@johnmatthewtennant johnmatthewtennant changed the title fix(security): turn thinking off for security explanations fix(security): lower thinking effort for security explanations Nov 5, 2026
Signed-off-by: John Matthew Tennant <jtennant@block.xyz>

Co-authored-by: Gordon <6581f171b4f294207c04c83d43b057ea88caa717457980f756160f0212d4f429@buzz.block.builderlab.xyz>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* origin/main:
  fix(connections): remove retired Block Academy provider (#475)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant