feat: asymmetric trigger detection — reduce false positives for negative queries#5
Merged
melanie531 merged 1 commit intoaws-samples:mainfrom Mar 19, 2026
Conversation
…gative queries
Refactor _detect_skill_trigger_from_parsed into _classify_trigger_signal
that returns signal strength ('tool', 'text', or 'none') instead of a
boolean.
Key change: for should_trigger=false queries, only 'tool' signals
(Read SKILL.md, Bash script execution, Skill tool invocation) count as
triggers. Text-only mentions of the skill name are ignored, since agents
commonly mention skill names (e.g. 'text-summary', 'compliance') in
their output without intending to activate the skill.
For should_trigger=true queries, both 'tool' and 'text' signals count,
preserving existing behavior.
This fixes false positives that caused trigger eval failures for skills
with common-word names (like text-summary, acme-compliance).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Skills with common-word names (e.g.
text-summary,acme-compliance) suffer high false-positive rates in trigger evaluation. When ashould_trigger=falsequery is evaluated, the agent often casually mentions the skill name in its text output (e.g. "I could use text-summary here") without actually activating the skill. The current code treats any text mention as a trigger, causing negative queries to fail.This was observed in CI pipelines where:
acme-compliancetrigger score dropped to 50/100text-summarytrigger eval failed entirelySolution
Refactor
_detect_skill_trigger_from_parsed()into_classify_trigger_signal()that returns signal strength:"tool""text""Using text-summary to...""none"Asymmetric detection logic:
should_trigger=truequeries: bothtoolandtextsignals count as triggers (unchanged)should_trigger=falsequeries: onlytoolsignals count — text mentions are ignoredThis means a negative query won't fail just because the agent casually mentioned the skill name.
Backward Compatibility
_detect_skill_trigger_from_parsed()still exists and returnsbool(wraps_classify_trigger_signal)_detect_skill_trigger()unchangedTests Added
TestClassifyTriggerSignal(8 tests) — signal classification for tool/text/noneTestAsymmetricTriggerDetection(3 tests) — asymmetric behavior for positive vs negative queries