feat(ways): optimization sub-way, BM25 doc updates, sparsity principle

[aaronsb]

Summary

• New optimization sub-way (meta/knowledge/optimization/) with BM25 semantic matching and a prepend macro that injects a live health dashboard of all ways' vocabulary state
• Updated authoring way to reflect BM25 as primary matching engine, additive matching, and /test-way skill
• Updated docs (matching.md, extending.md) — BM25 degradation chain, progressive disclosure pattern, vocabulary design guidance
• Sparsity principle documented: vocabulary optimization targets semantic distance between ways, not match rate per way

Progressive disclosure in action

"what are ways?"              → meta/knowledge fires (overview)
  edit a way.md               → meta/knowledge/authoring fires (format spec)
  "optimize the vocabulary"   → meta/knowledge/optimization fires (workflow + live health macro)

Each level adds context only when the conversation goes there. The macro injects current state so the agent has both the workflow and the data.

Test plan

• Optimization way scores 9.35 on "optimize the ways vocabulary" (strong match)
• Optimization way scores 1.80 on "what are ways" (below threshold, doesn't fire)
• Macro produces correct health dashboard across all 33 ways
• Existing test harness unaffected (no C or test changes)

[aaronsb]

Code Review: Optimization Sub-Way, BM25 Doc Updates, Sparsity Principle

PR Size: 229 additions, 68 deletions across 5 files -- standard review.

---

What This Changes

Adds an optimization sub-way for vocabulary tuning (with a prepend macro that injects live health data), updates the authoring way and two doc files to reflect BM25 as the primary matching engine, documents the progressive disclosure pattern, and introduces the sparsity principle for vocabulary design.

Assessment: Solid Overall

The progressive disclosure documentation in extending.md is well-written and the meta/knowledge hierarchy (overview -> authoring -> optimization) is a clean example of the pattern it describes. The sparsity principle is a valuable conceptual addition -- framing vocabulary design as "maximize inter-way distance" rather than "maximize per-way recall" is the right mental model.

The macro produces useful output and degrades gracefully when the binary is missing. The way body is concise at 80 lines and follows its own authoring guidance.

---

Findings

1. NCD Threshold Inconsistency Between check-prompt.sh and test-harness.sh

Location: tools/way-match/test-harness.sh:84 vs hooks/ways/check-prompt.sh:99

Problem: The test harness uses NCD threshold 0.55 while check-prompt.sh (the live system) uses 0.58. The matching doc (matching.md) says the NCD threshold is "hardcoded 0.58", and the optimization way (way.md:70) also says "hardcoded 0.58 in check-prompt.sh". But the test harness -- the tool documented as the regression gate -- uses a different value.

Why it matters: If someone tunes vocabulary using the test harness as their feedback loop (as the optimization workflow prescribes), they're validating against a different threshold than production uses. The 0.55 vs 0.58 gap means the test harness is slightly more permissive than production, so a term that passes the test harness NCD check could fail in production.

Note: This predates this PR -- it's not a regression you introduced. But since you're documenting the NCD threshold in two new places as "0.58 in check-prompt.sh", it's worth flagging that the test harness disagrees. Consider either aligning them or documenting the intentional difference.

2. Stale match: semantic and match: model References Outside This PR's Scope

Location: Multiple files not touched by this PR

Problem: The authoring way now says "No match: field needed -- the presence of description: + vocabulary: enables semantic matching automatically." This is accurate -- check-prompt.sh never reads a match: field; it checks for the presence of description and vocabulary directly. However:

• All 7 semantic ways still have match: semantic in their frontmatter (e.g., softwaredev/testing/way.md:2, softwaredev/security/way.md:2)
• docs/hooks-and-ways.md:34 still says check-prompt.sh scans for match: semantic or match: model fields
• docs/architecture.md:99 still references match: semantic
• CONTRIBUTING.md:10 still says match: semantic for fuzzy matching
• README.md:251 still shows match: semantic in examples
• docs/hooks-and-ways.md:198 still documents match: model as an active mode

The docs updated by this PR are now internally consistent, but they create a contradiction with the rest of the codebase. The authoring way says match: is unnecessary, matching.md drops model matching entirely, but the parent docs and the actual way files still reference both.

Suggestion: This doesn't need to block merge, but a follow-up to clean up these stale references would prevent confusion. The match: semantic field in existing ways is harmless (it's ignored), but the doc references to match: model are actively misleading since that code path is now undocumented.

3. macro.sh Parses way-match suggest Stderr With Fragile Sed Patterns

Location: hooks/ways/meta/knowledge/optimization/macro.sh:37-39

Problem: The macro extracts gap/coverage/unused counts by parsing stderr with:

gaps=$(echo "$stderr" | sed -n 's/suggest: \([0-9]*\) gaps.*/\1/p')
covered=$(echo "$stderr" | sed -n 's/.*, \([0-9]*\) covered.*/\1/p')
unused=$(echo "$stderr" | sed -n 's/.*, \([0-9]*\) unused/\1/p')

This is tightly coupled to the exact format string in way-match.c:755:

fprintf(stderr, "suggest: %d gaps (min_freq=%d), %d covered, %d unused\n", ...);

Why it matters: If the C code's format string ever changes (different word order, different punctuation, additional fields), the macro silently falls back to showing 0 for all metrics -- a degradation that would be invisible in normal use.

Suggestion: This is a pragmatic choice and not blocking. The fallback to ${gaps:-0} keeps it from erroring. But if the suggest mode ever grows a --json or --machine-readable output flag, this macro would benefit from using it.

4. Optimization Way's Vocabulary Overlap With Parent/Sibling Ways

Location: hooks/ways/meta/knowledge/optimization/way.md:3

Problem: The vocabulary optimize vocabulary suggest gaps coverage unused threshold tune score scoring review ways health audit includes ways, review, and score/scoring which could overlap with the parent knowledge way (triggers on \bway\b|\bways\b) and potentially other ways.

Why it matters: The PR itself articulates the sparsity principle: "maximize the semantic distance between ways". The ways term in the optimization vocabulary creates exactly the kind of overlap the principle warns against -- any prompt mentioning "ways" will give this way some BM25 score from the vocabulary match, even if the prompt has nothing to do with optimization.

Mitigation: The parent fires on regex (\bways\b), not BM25, so there's no BM25 score competition. And the BM25 threshold of 2.0 means ways alone won't fire this way -- it would need several vocabulary terms to hit threshold. So this is more of a "practice what you preach" observation than a functional issue. Still worth considering whether ways in the vocabulary is load-bearing or whether it's the kind of generic term the optimization way itself advises against.

5. macro.sh Uses find ... | sort Without -print0/-d '' for Project-Local Ways

Location: hooks/ways/meta/knowledge/optimization/macro.sh:51

Problem: The global ways loop (line 17) correctly uses find ... -print0 paired with read -r -d '' -- wait, actually looking again, line 17 uses a for loop over $(find ... -print), not the null-delimited form. Both loops (global at line 17 and project-local at line 51) use the same for wayfile in $(find ...) pattern, which is unsafe for paths containing spaces or glob characters.

Why it matters: Way file paths are controlled (convention is lowercase, no spaces), so this is unlikely to bite in practice. But check-prompt.sh:53 uses the safer while IFS= read -r -d '' ... < <(find ... -print0) pattern. The macro is less careful.

Suggestion: Low priority since way paths are convention-controlled. But if you're iterating on the macro, matching check-prompt.sh's find -print0 | while read -d '' pattern would be the more robust form.

6. matching.md Lists 7 Semantic Ways But Doesn't Include the Optimization Way Itself

Location: docs/hooks-and-ways/matching.md:80-87

Problem: The "Which ways use semantic matching" section lists 7 ways (testing, api, debugging, security, design, config, adr-context) but does not include meta/knowledge/optimization which this PR adds as a semantic way with description: + vocabulary: + threshold: 2.0.

Why it matters: Minor inconsistency. The list was presumably meant to be exhaustive ("Ways covering broad concepts where keyword matching would be either too narrow or too noisy"). The new optimization way is exactly that -- a broad concept that users express variously.

Suggestion: Add it to the list, or reframe the section as "examples" rather than an exhaustive enumeration.

---

What Looks Good

• Progressive disclosure section (extending.md:64-82) is excellent documentation. The concrete example with the knowledge domain, the design principle, and the explanation of macros for live state forms a clear pattern that others can follow.

• Sparsity principle (matching.md:72-76, optimization/way.md:47-61) is well-articulated. Framing vocabulary design as an inter-way distance optimization problem rather than per-way coverage is the right abstraction.

• Macro graceful degradation (macro.sh:7-10) -- exits cleanly with a helpful build instruction when the binary is missing. Good pattern.

• Consistent threshold documentation -- BM25 2.0 default, NCD 0.58 fixed fallback, and the explicit note about why the two scales don't map (check-prompt.sh:96-98) all tell a coherent story.

• The way practices what it teaches -- the optimization way at 80 lines is tight, uses the workflow it describes, and its macro injects the data the workflow needs. The self-referential design ("you optimize ways by talking about optimizing ways") is elegant.

---

Considerations

• The stale match: semantic/match: model references in files outside this PR's scope (Finding ADR-003: Simplify to ADR-Driven Workflow Pattern #2) are the biggest source of potential confusion. The docs this PR touches are internally consistent, but a reader who starts from docs/hooks-and-ways.md or CONTRIBUTING.md will get a different mental model than one who starts from matching.md or the authoring way.

• The test harness doesn't include the optimization way in its corpus (WAY_IDS array in test-harness.sh:47). If this way should participate in false-positive regression testing, it would need to be added there.

---

AI-assisted review via Claude