Phase 30: observation-API unification — dataset(), climate(), observation grain, D-16 covariates, 1.13.0

CHANGELOG.md

@@ -4,6 +4,118 @@ All notable changes to `mostlyright`. The format follows [Keep a Changelog](http
 
 ## [Unreleased]
 
+## [1.13.0] — 2026-07-09 — Observation-API unification: dataset(), climate(), covariates, one mode
+
+Minor release, dual-SDK (PyPI 1.13.0 + npm 1.13.0 in lockstep). Unifies the
+observation API onto a single `dataset()`/`research()` surface, ships a
+standalone `climate()` settlement table, adds an opt-in labels-only covariate
+seam, and starts the deprecation train for the numbered-mode era. Every
+existing entry point is byte-stable at runtime this release; see
+[`docs/migration-1.13.md`](docs/migration-1.13.md) — headline: **migrating a
+pinned feature is a retrain event, not a find-replace.**
+
+### Added
+
+- **`dataset()` — the new canonical name for the training-pairs join (both
+  SDKs).** `research()` is retained as an alias bound to the same callable
+  (`dataset is research`); no deprecation warning on `research()` this
+  release, byte-identical output. Python: `mostlyright.dataset(...)`; TS:
+  `import { dataset } from "mostlyright"`.
+- **`obs(granularity="observation")` — per-report observation grain
+  (Python).** Returns the merged per-report METAR/SPECI rows (window-trimmed,
+  no daily bucketing) with per-row source identity preserved and
+  `observation_type`/`raw_metar` surfaced. Unpinned frames carry the new
+  frame-identity tag `merged.live_v1`, accepted only by the new
+  `schema.observation.merged.v1` — the frozen `schema.observation.v1`
+  rejects it, so fused frames can never masquerade as single-source. The
+  default `granularity="daily"` output is byte-identical to 1.12.1. TS
+  `obs()` gains the same option with a deliberate divergence: its default is
+  `"observation"` (preserving existing TS per-report behavior);
+  `"daily"` throws a documented not-yet-ported error (parity ticket filed).
+  Observation-grain frames stamp `retrieved_at` provenance.
+- **`climate()` — standalone NWS CLI settlement-label table (both SDKs).**
+  The daily `cli_high_f`/`cli_low_f` labels previously visible only inside
+  the `dataset()` join are now a first-class, source-identified domain table
+  in the JOIN column vocabulary, window-trimmed (no whole-year leak), frame
+  tag `cli.archive` with truthful per-row endpoint tag `iem`. Python:
+  `mostlyright.weather.climate(...)`; TS: `import { climate } from
+  "@mostlyrightmd/weather/climate"` (lean subpath — keeps the root barrel
+  inside its size budget). TS `climate()` resolves NWS↔ICAO station codes
+  through the station registry so the `station` column matches Python
+  cross-SDK.
+- **`dataset(include_satellite=, include_cwop=)` — opt-in covariate
+  composition (Python).** LEFT-joins per-LST-settlement-day `sat_*` /
+  `cwop_*` covariate columns onto the composed training frame behind flags
+  defaulting `False`. **Labels-only firewall:** label columns are
+  byte-identical flags-on vs flags-off, row count/order unchanged, missing
+  days are NaN (never dropped), and satellite/CWOP can never become
+  observation sources — the four parity-firewall files are untouched and
+  grep-gated. CWOP covariates are QC-clean-only and read-only (no APRS
+  sockets); satellite covariates count only scans that contribute a usable
+  value (`sat_scan_count`). Requires the `[satellite]` / `[cwop]` extras;
+  missing extras fail loud with a typed error. **Adding a covariate to a
+  trained model is itself a retrain event** — see the migration guide. TS
+  parity for the covariate flags is tracked as a CROSS-SDK-SYNC ticket
+  (browser runtime cannot run local extraction).
+- **`docs/source-identity.md`** — the single cross-vertical kwarg contract
+  page: `source=` is always provenance, `delivery=` is always local|hosted,
+  grain is vertical-defined, frame-vs-row identity, the labels-only
+  firewall, and the train/trade symmetry example.
+
+### Changed
+
+- **Output-knob unification (D-06): `backend=` + `return_type=` is the one
+  output convention.** `obs()` gains `backend`/`return_type`;
+  `return_type="list"` added across the dispatch layer. TS `return_type`
+  accepts `"dataframe"` as the canonical value (the historical `"frame"`
+  remains an alias). Every function's no-argument default return is
+  byte-stable.
+- **Numbered-mode vocabulary retired** across docstrings, runtime strings,
+  and docs (`source must be one of ...`, `source-pinned dispatch requested
+  ...`); forecast surfaces are now described as IEM MOS vs per-NWP-model.
+  TS neutral names `SOURCE_PINNED_SOURCES`/`SourcePinnedSource`/
+  `isSourcePinnedSource` are canonical.
+- **TS weather bundle budget raised 26 → 27 KB** (TS-BUNDLE-01) to admit the
+  station-format guard + pinned-AWC fail-loud hardening; measured 26.23 KB.
+
+### Deprecated
+
+All deprecations warn but behave identically; removal target is ≥2 minor
+releases out. Production pipelines running `-W error` (Python) will surface
+these as errors — see the migration guide before tightening warning filters.
+
+- **`research_by_source()`** → use `obs(source=..., granularity=
+  "observation")`. **Caveat: this is NOT a drop-in swap for pinned
+  features** — the new path returns pre-merge full single-source coverage,
+  a different (typically larger) row composition than `research_by_source`'s
+  post-merge filter. Retrain/re-backtest before switching production
+  strategies (`docs/migration-1.13.md`).
+- **`mostlyright.mode2` import path** → `obs(...)` from
+  `mostlyright.weather` and `assert_source_identity` from
+  `mostlyright.core` (its new canonical home, re-exported).
+- **`as_dataframe=`** on `dataset()`/`research()`, `research_by_source()`,
+  and `obs()` → `return_type=` (`True` → `"dataframe"`, `False` → `"list"`;
+  an explicit `return_type=` wins).
+- **TS `MODE2_SOURCES`/`Mode2Source`/`isMode2Source`** → the
+  `SOURCE_PINNED_*` names (old identifiers remain as aliases binding the
+  same values; removal at 2.0).
+
+### Fixed
+
+- **TS station-format guard on every exported weather network surface**
+  (including the hosted satellite client): malformed station input now fails
+  loud before any URL is built or network dispatched.
+- **TS pinned `source: "awc"` queries fail loud on fetch errors** instead of
+  silently returning an empty array (unpinned merge path unchanged).
+- **Parity case-4 fixture (KMIA rolling year) re-baselined 2026-07-09:**
+  upstream restated 3 of 365 days after the May capture (mean aggregates
+  only; observation counts, settlement labels, and extremes byte-identical),
+  and the v0.14.1 hosted data plane the original fixtures were captured from
+  has since been decommissioned, making a fresh v0.14.1 re-capture
+  permanently impossible. The prior v0.14.1-captured fixture is preserved in
+  git history; provenance documented in `tests/fixtures/parity/README.md`.
+  Cases 1/2/3/5 remain v0.14.1-captured and byte-green.
+
 ## [1.12.1] — 2026-07-08 — #107 follow-ups: cross-SDK consistency + version metadata
 
 Patch release, dual-SDK. Resolves the P3 follow-ups from the 1.12.0 review

CLAUDE.md

@@ -67,12 +67,12 @@ This applies to every PLAN.md generated by `/gsd-plan-phase`, every PROJECT.md "
 
 Standalone live adapter for Citizen Weather Observer Program (PWS) data over the APRS-IS TCP stream. **Deliberately isolated from the parity-critical core** — a hot-rooftop or indoor PWS must never corrupt a Kalshi NHIGH/NLOW settlement target.
 
-- **Never wired into** `research()`, `merge/observations.py` (`SOURCE_PRIORITY`), or `live/_sources.py`. CWOP carries its own `schema.cwop.v1` — `schema.observation.v1`'s `observation_type` enum is untouched. These four files are the parity firewall; do not register "cwop" in any of them. This holds in v0.2: persistence + `history()` add NO wiring into those four files.
+- **Labels-only firewall (D-01 amended, D-16).** CWOP must NEVER become an observation SOURCE — never register "cwop" in the observation merge path of `research()`, `merge/observations.py` (`SOURCE_PRIORITY`), `live/_sources.py`, or `schema.observation.v1`'s `observation_type` enum. CWOP carries its own `schema.cwop.v1`. **These four files are the parity firewall; do not register "cwop" in any of them.** The firewall protects the LABEL computation (obs `max/min` + `cli_*` columns) only — a hot-rooftop/indoor PWS can never enter the settlement aggregation. What the amendment ALLOWS: covariate COLUMNS. `dataset(..., include_cwop=True)` (D-16) LEFT-joins per-settlement-day `cwop_*` columns (QC-clean rows only, ledger left-edge respected) onto the composed frame via the post-aggregation seam in `_covariates.py` (which lives OUTSIDE `_internal/merge/` and runs strictly after the label aggregation — it never touches those four files). Label columns stay byte-identical with the flag on or off. This holds in v0.2: persistence + `history()` add NO wiring into those four files.
 - **Public surface (sync unless noted):** `nearby()` (start-here ICAO→scan), `scan()`, `stream()` (async generator), `snapshot()` (→ `schema.cwop.v1` DataFrame; `persist=True` also writes the backtest cache), `latest()`. v0.2 adds `history(station, from_date, to_date, *, qc_status=None)` (→ `schema.cwop.v1` DataFrame for backtest replay) and `persist_observations(observations)`. Source tag `"cwop.live"` (live) / `"cwop.cache"` (persisted-then-read); `schema.cwop.v1` accepts both via `_registered_sources`. Errors raise `NoCWOPDataError` (subclass of `NoLiveDataError`) — never return `[]`/`None`.
 - **File layout:** `weather/_aprs.py` (in-house MIT parser, no GPLv2 aprslib), `weather/_fetchers/_aprs_stream.py` (`_CWOPReaderThread` daemon owns socket → `threading.Queue` → sync/async consumers; `loop.call_soon_threadsafe` bridges async), `weather/cwop/*` (public API + `_schema.py` + `_registry.py` + v0.2 `_cache.py`/`_history.py`), `weather/qc/_cwop.py` (QC). Pure stdlib (`socket` + `re`) for the live path; persistence pulls `pyarrow`/`filelock`. `[cwop]` extra pulls pandas (for `snapshot()`/`history()` DataFrames).
 - **Persistence (v0.2):** monthly parquet at `$HOME/.mostlyright/cache/cwop/{station}/{year}/{month}.parquet` (honors `MOSTLYRIGHT_CACHE_DIR`). Standalone `weather/cwop/_cache.py` — does NOT import the parity-coupled `weather/cache.py`; `filelock`-guarded read-modify-write merge, dedup by `(station_id, observed_at)` first-seen-wins. **NO current-month skip** (CWOP is ephemeral — unlike re-fetchable AWC/IEM/GHCNh, the current month is the only chance to retain it). CWOP station ids (CW0875, ham callsigns) are NOT ICAO, so the cache uses its own `_CWOP_STATION_RE` path validator, not `STATION_CODE_RE`.
 - **6-layer QC thresholds:** each layer scores 0.0–1.0; combined = weighted geometric mean. `< 0.3` → dropped, `0.3–0.7` → flagged, `> 0.7` → clean. Layers: range (0.20), temporal consistency (0.20), indoor detection (0.15), buddy/ASOS check (0.20), solar-radiation bias (0.10), reliability (0.15). Layers needing history (indoor ≥24h, solar ≥7d) pass (1.0) until enough data accumulates.
-- **Why no `include_cwop` on `research()`:** `research()` aggregates `max/min(temp_f)` source-blind, so a hot-rooftop/indoor PWS would corrupt a Kalshi NHIGH/NLOW target. CWOP reaches models ONLY through `cwop.history()` (a DataFrame the strategy joins itself), never the settlement join. See [`docs/cwop-adapter.md`](docs/cwop-adapter.md).
+- **`include_cwop` covariate columns (D-16), NOT a settlement source:** `research()`/`dataset()` aggregate `max/min(temp_f)` source-blind for the LABEL, so a hot-rooftop/indoor PWS must never enter that aggregation — CWOP stays out of the four firewall files. But `dataset(..., include_cwop=True)` (D-16) attaches per-settlement-day `cwop_*` COVARIATE columns (QC-clean rows only, combined > 0.7; ledger left-edge → NaN before capture start; read-only, no APRS connections) via the labels-only join seam. The label columns are byte-identical with the flag on or off. CWOP also reaches models through `cwop.history()` (a DataFrame the strategy joins itself); either way it never touches the settlement join. See [`docs/cwop-adapter.md`](docs/cwop-adapter.md).
 
 ## Testing
 

README.md

@@ -93,26 +93,36 @@ from mostlyright.markets.catalog import kalshi_nhigh
 
 # 1. What does the contract settle on?
 contract = kalshi_nhigh.resolve("KHIGHNYC", date(2025, 1, 15))
-# 2. Pull the data that decides it.
-df = mostlyright.research(contract.settlement_station, "2025-01-15", "2025-01-15")
+# 2. Pull the data that decides it. dataset() is the primary name;
+#    research() is a fully working alias in this release.
+df = mostlyright.dataset(contract.settlement_station, "2025-01-15", "2025-01-15")
 # 3. Apply the threshold and decide.
 ```
 
 ### Train models with train/infer source parity
 
-`research()` stamps a source identity on every row. The validator catches train/infer mismatches at load time, instead of silently corrupting predictions in production.
+`dataset()` (and its `research()` alias) stamps a source identity on every row.
+Pinning `source=` on `obs()` returns single-source coverage; the validator catches
+train/infer mismatches at load time, instead of silently corrupting predictions in
+production.
 
 ```python
-from mostlyright.mode2 import research_by_source
+from mostlyright.weather import obs
 from mostlyright.core import validate_dataframe
 
-train = research_by_source("KNYC", "iem.archive", "2024-01-01", "2024-12-31")
-# Validator pins the schema's expected source. SourceMismatchError fires if
-# you accidentally route a Mode 1 (fused AWC+IEM+GHCNh) DataFrame through a
-# Mode 2 (iem.archive only) schema.
+train = obs("KNYC", "2024-01-01", "2024-12-31",
+            source="iem.archive", granularity="observation")
+# Validator pins the schema's expected source. SourceMismatchError fires if you
+# accidentally route a source-blind (fused AWC+IEM+GHCNh) DataFrame through an
+# iem.archive-only schema. See the source-identity contract for the full rule.
 validate_dataframe(train, schema_id="schema.observation.v1")
 ```
 
+> Pinning `source=` returns a different row composition than the fused default —
+> migrating a pinned feature is a **retrain event**, not a find-replace. The one
+> contract for `source=` / `delivery=` / grain across every vertical lives in
+> [`docs/source-identity.md`](docs/source-identity.md).
+
 ### Feed AI agents structured public data
 
 Every response carries a stable `schema.*.v1` URI and serializes to JSON-Schema-validated shapes. Drop responses into MCP tool outputs or OpenAI/Anthropic function-call returns without re-shaping.