Add One-Shot Skills Reliability and Guardrails

.github/skills/document-retrieval/SKILL.md

@@ -0,0 +1,173 @@
+---
+name: document-retrieval
+description: Build and tune retrieval configs that search, rank, and collect ordinance documents in COMPASS. Use whenever a user asks to improve retrieval precision/recall, tune search queries/keywords, or debug acquisition quality before extraction tuning.
+---
+
+# Web Scraper Skill
+
+Use this skill to improve retrieval precision/recall before extraction tuning.
+Applies to both one-shot (schema-driven) and legacy decision-tree extraction
+pipelines.
+
+## When to use
+
+- Download step returns noisy sources (one-shot extraction).
+- Ordinance recall is weak across jurisdictions (one-shot extraction).
+- LLM filtering is compensating for poor search quality.
+
+## Do not use
+
+- Schema feature definition or value extraction logic design.
+- Post-extraction feature/value debugging when retrieval is already correct.
+
+## Expected assistant output
+
+When using this skill, return:
+
+1. The retrieval axis changed (queries, keyword weights, or heuristics).
+2. Evidence from artifacts/logs showing why the change was needed.
+3. The next run command against the same jurisdiction sample.
+
+## Canonical reference
+
+Consult example plugin configurations in `examples/`:
+- `examples/one_shot_schema_extraction/plugin_config.yaml` — standard one-shot config
+- `examples/water_rights_demo/one-shot/plugin_config.yaml` — multi-document edge cases
+
+When creating new tech configs, use `<tech>_plugin_config.yaml` as a recommended
+naming convention (e.g. `geothermal_plugin_config.yaml`).
+
+## Scope
+
+- Query-template strategy.
+- URL ranking and filtering patterns.
+- Heuristic phrase controls before LLM validation.
+
+## Two retrieval phases
+
+COMPASS runs two sequential acquisition passes per jurisdiction:
+
+1. **Search-engine phase** — queries `SerpAPIGoogleSearch` (or configured
+   engine) using `query_templates`. This phase is the primary source of
+   ordinance documents.
+2. **Website crawl phase** — crawls the jurisdiction's official website,
+   ranking pages using `website_keywords`. This phase is a secondary pass
+   and runs only if the search-engine phase did not yield an ordinance
+   context.
+
+Key behaviors:
+- Playwright browser errors during the website crawl phase are **non-fatal**.
+  COMPASS logs the error and continues.
+- `Found 0 potential documents` at the end of the crawl phase is **expected**
+  for jurisdictions without relevant online ordinances.
+- Disable the crawl phase with `perform_website_search: false` in run config
+  when you want faster smoke tests or Playwright is unavailable.
+
+## Key management
+
+For SerpAPI-backed search, keep `api_key` out of committed config and provide
+`SERPAPI_KEY` via environment (for example through `.env` loaded in shell).
+
+Recommended shell setup:
+
+```bash
+set -a
+source .env
+set +a
+```
+
+Avoid spaces around `=` in `.env` assignments.
+
+## Retrieval design pattern
+
+1. Create 3-7 jurisdiction queries with `{jurisdiction}`.
+2. Weight legal document indicators in URL keywords.
+3. Apply exclusions for templates/reports/slides.
+4. Add focused negative tech terms to reduce false positives.
+5. Start with dynamic search, then switch to deterministic known URLs when
+  search infrastructure is unstable.
+
+When using `heuristic_keywords`, use these four lists to guide pre-LLM filtering:
+- `GOOD_TECH_KEYWORDS` — strong indicators of the target technology
+  (e.g., facility types, deployment modes). Documents matching even a
+  few keywords are marked as candidates.
+- `GOOD_TECH_PHRASES` — multi-word phrases that signal relevant
+  ordinance content. Keep specific to avoid false positives.
+- `GOOD_TECH_ACRONYMS` — industry-standard abbreviations for the
+  technology. Narrow list; include only widely recognized acronyms.
+- `NOT_TECH_WORDS` — pre-heuristic filter that rejects documents
+  before keyword matching. Use to exclude adjacent technologies and
+  irrelevant domains (e.g., residential HVAC, unrelated industries).
+  Runs first; prevents wasted keyword evaluation on clearly-wrong
+  documents.
+
+If any required list is missing or empty, COMPASS raises a plugin
+configuration error and extraction quality should be treated as failed.
+
+For first-pass reliability, test retrieval with deterministic known URLs
+before using live web search.
+
+## Technology-specific retrieval controls (template)
+
+- Include target-technology facility/deployment terms.
+- Exclude adjacent and non-target terms (residential/HVAC/PV/etc as needed).
+- Favor jurisdictional legal-code signals like `land use code`,
+  `code of ordinances`, `use table`, and `special use permit`.
+
+## Deterministic smoke-test mode
+For this smoke test, at least one of the following documentation sources must be provided:
+
+- **`known_doc_urls`**: A list of URLs pointing to external documentation that the scraper can access and parse
+- **`known_local_docs`**: A collection of local documentation files available in the repository or system
+
+Use run-config controls to bypass flaky search while tuning:
+
+- supply `known_doc_urls` or `known_local_docs`,
+- set `perform_se_search: false`,
+- set `perform_website_search: false`.
+
+Then validate:
+
+- download artifacts exist,
+- cleaned text exists,
+- ordinance DB rows are non-empty.
+
+## Tuning loop
+
+1. Run SE-search phase on small sample.
+2. Inspect kept vs discarded PDFs (`ordinance_files/`).
+3. Run heuristic filter and review false rejects/accepts (`cleaned_text/`).
+4. Check website crawl phase independently if needed (enable, run, inspect logs).
+5. Update one axis only:
+   - query templates (affects SE phase),
+   - URL weights (affects both phases),
+   - include/exclude heuristic patterns (pre-LLM filter),
+   - `NOT_TECH_WORDS` (upstream document rejection).
+6. Re-run same sample and compare.
+
+## Cross-tech onboarding
+
+When reusing this workflow for any technology:
+
+- keep legal retrieval tokens (`ordinance`, `zoning`, `code`),
+- replace all technology terms in `query_templates`, `website_keywords`,
+  and `heuristic_keywords`,
+- seed `known_doc_urls` with authoritative regulatory documents for smoke
+  testing,
+- avoid copying negatives from previous technologies into the new tech config,
+- verify `NOT_TECH_WORDS` excludes adjacent technologies for your domain.
+
+## Phase gates
+
+- **3 jurisdictions**: ensure major source classes are found.
+- **10 jurisdictions**: verify stability across regions.
+
+
+## Guardrails
+
+- Keep feature extraction logic out of retrieval config.
+- Do not overfit to one county's document style.
+- Preserve auditable rationale for each retrieval change.
+- Keep one canonical retrieval config per active technology.
+- Ensure each run uses a unique `out_dir` to avoid COMPASS aborting early.
+