docs(skill): A/B-tuned ecp skill for agent usability (net -26 lines)#532
Merged
Conversation
…26 lines) Two rounds of empirical A/B (Haiku agents executing real navigation tasks end-to-end, scored on which verb they reached) drove these prompt-only edits to the embedded ecp skill source under docs/skills/ecp/: - description reframed from a capability list into a reflex trigger + a question→verb map, with anchors for the two under-used verbs A/B surfaced: where-a-filename-is-read-vs-written→impact --literal, and graph-question-with-no-verb (orphans/all-impls)→cypher. - Directive 1: name the two weak-model traps — 'who calls X' is impact (list) not find (count); an ambiguous-name error means retry with --file/--kind, not fall back to Read. - Directive 3 + ECP.md: a real found:false miss means report 'doesn't exist', never synthesize a caller list / blast radius for a symbol ecp couldn't find. - Collapsed verbose tables (literal/group/schema/architecture) into tight prose. Net effect: SKILL.md 6235→6106 bytes (smaller), and held-out Haiku validation went from fabricating blast radii / falling back to grep → using impact --literal 3/3, cypher for orphans 2/2, with zero over-triggering on string-literal tasks. Embedded via include_dir!; verified `ecp admin claude install skills` emits the tuned files byte-identical.
Contributor
ecp impact cache (0 symbols) — internal, used by
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tunes the embedded
ecpskill source (docs/skills/ecp/{SKILL.md,ECP.md}) for agent — especially Haiku — usability. Prompt-only; token strictly decreases (SKILL.md 6235→6106 bytes, net −26 lines). Distributed via the existinginclude_dir!+ecp admin claude install skillspath (verified byte-identical after a rebuild).Method
Two rounds of empirical A/B: Haiku agents read the skill and execute real navigation tasks end-to-end on this repo; scored on which verb/tool they reached (vs a no-prompt baseline, per the validate-prompt-rules discipline).
Round 1 — correctness
impact(list), notfind(count) — added to Directive 1.found:false, report 'doesn't exist'; never synthesize a caller list / blast radius for a missing symbol (both arms previously invented one). Directive 3 + ECP.md.--file/--kind, don't fall back to Read.Round 2 — coverage (expand ecp's reach)
cypherfor orphans / all-impls (was loopingimpact) — fixed.impact --literalfor filename read/write (was falling back to grep) — needed hoisting the anchor into thedescriptionverb-map; buried triggers fail regardless of wording, the fix is altitude.Validation (held-out, executed)
impact --literal 3/3, cypher-for-orphans 2/2, zero over-trigger on a string-literal guard task,
fabricated:falseon planted fake symbols.Notes
cargo build -p egent-code-plexus --bin ecpgreen (embeds the new files).