feat: published-capsule library + Finding 21 (0/5 unprompted adoption — words beat schema)#150
Merged
Merged
Conversation
…ture (edge-case 6 instrument) The agent's tool surface exposed none of the published corpus — it would re-run (and re-pay for) experiments the sealed capsules already answer. - api/logic/published_library.py: search_published_capsules(category?) — published = git-tracked AND sealed (verify_doc_claims' definition); suite/engines/description DERIVED from each capsule's own archived config (never hand-typed); categories via taxonomy (override+inherit). - REST: GET /v1/catalog/published?category= - Tool: search_published_capsules in config_builder's SCHEMA only — the workflow prompt does not mention it, and a test pins that. Edge-case 6 measures UNPROMPTED adoption (does schema-derivation extend to knowledge reuse?). Reuse path needs zero orchestrator changes: handoff of a published id -> poll sees complete instantly -> analyzer reads sealed results. - taxonomy: quack-v1 capsules tagged columnar/transport (honest curation — suite was analytical_wall/tpch but the object of study is the Quack protocol; removes a taxonomy-miss confound a human librarian would fix). - Contract: edge6_published_knowledge (9887ce57) — 5-model ladder, n=1, quack transport-efficiency goal the published corpus already answers. 4 tests. Live smoke: 8 published capsules, category filters resolve.
… on conflict Edge-case 6 (study 9887ce57): search_published_capsules in schema only, prompt untouched, quack goal the published corpus already answers, 5-model ladder n=1. Result: 0/5 models called the library. All executed the trained build workflow flawlessly and re-ran the experiment — and since the quack server isn't running on this host, every re-run FAILED at execution. The published capsules were the only available source of the answer; nobody reached for them. 5 x 15-38K tokens to fail at a question the corpus answers free. The boundary: schema communicates capability; the role prompt communicates identity — and identity wins. Findings 5-7 (workflow derivable from schema alone) do NOT extend to deviating from a strongly-framed procedure toward an unmentioned capability. Consequence for the publish-shape question: publishing is measurably insufficient without a workflow consumer — derived machine-readable summaries + honest taxonomy + a librarian stage (or prompt step) ship together or the shape doesn't matter. Edge-6 traces committed as evidence.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Edge-case 6 instrument + result. Library: published = tracked+sealed, derived summaries, taxonomy discovery, REST + tool (schema-only, prompt untouched — pinned by test). Study 9887ce57: 0/5 models called the library; all re-ran and all re-runs failed (quack server absent) — the corpus was the only source of the answer. Boundary drawn: schema = capability, role prompt = identity, identity wins. Publish-shape answer recorded: derived summaries + honest taxonomy + a workflow consumer, or the shape doesn't matter.