fix(weather): constellation product parity + probe-measured ingest speed#103
Merged
Conversation
Registers the GOES Downward Shortwave Radiation SECTOR product, absent from PRODUCTS despite living on every GOES bucket. Live-verified 2026-07-05 against real samples on noaa-goes16/17/18: - goes16: CONUS grid (166, 394) — the registry baseline - goes17/goes18: PACUS grid (157, 380) — per-satellite split via _DSRC_SECTOR_SHAPES in the shape gate (same accept-only-your-own-variant discipline as the DSRF split; swapped sectors fail loudly; goes19 publishes NO DSRC and falls back to the registry baseline so an unexpected file still fails loud) - all eras on goes_lat_lon_projection (the existing coarse lat/lon extractor branch — no extractor change needed) - hourly cadence, ~70-150 KB files (vs multi-MB full-disk DSRF decode) - goes16 archive: 2019-12 -> 2024 day-108 — the SAME NOAA product-overhaul cutover as the DSRF coarse->full-disk transition; empty listings outside the window are negligible at hourly cadence Why: for 2019-2024 US-station DSR features this is ~30x cheaper to fetch+decode than full-disk DSRF. As a sector product it correctly gets sector semantics from the coverage filter (cross-side excluded; _is_full_disk_goes_product False — pinned by test). Tests: per-satellite sector acceptance, swapped-sector + garbage rejection, registry presence, sector semantics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex P2 on the DSRC registration: _sources._goes_products() carries a hard-coded fallback product set for base installs (no [satellite] extra), and DSRC was only added to the heavy registry — so a base-install/hosted request for a registered product was rejected as unknown before reaching the hosted path. Adds ABI-L2-DSRC to the fallback AND a drift-guard test that parses the fallback literal out of the function source (ast) and pins it equal to the heavy registry's product set — a future registration that forgets the fallback now fails CI loudly instead of shipping a silent base-install rejection. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Constellation-wide parity audit (8 live-audit agents, every registry cell sampled against real bucket data, 2026-07-05) found 3 P1s that would have silently zeroed data in the full backfill, plus availability-floor waste: P1 — DSRF variants (parity audit): goes17 DSRF is COARSE (326,326) for its ENTIRE 2019-2023 life (retired before the 2024 day-108 overhaul); goes18 DSRF is BIMODAL exactly like goes16 (cutover confirmed at the same day: last coarse 2024/107, first full-disk 2024/108). The gate's bimodal branch was goes16-only — every goes17 DSRF file and all pre-cutover goes18 DSRF raised "rejected before pixel read". New _DSRF_VARIANTS map: goes16/goes18 bimodal (projection-keyed), goes17 coarse-only (a "full-disk goes17" file is mislabeled -> loud), goes19+ full-disk only. P1 — profile size caps: pre-Feb-2024 LVM/LVT files are ~29.4 MiB (live-measured 30,830,368 B; NOAA compressed them to ~7 MiB at 2024 day-038). The 16 MiB default cap zeroed the entire pre-2024 3D-profile era — same failure family as the old ACMC cap. LVMPC/LVTPC join DSRF in the 64 MiB cap set. P2 — per-(satellite, product) availability floors: goes16/17 TPWC/ACHAC/DSRC/LVM/LVT first objects are Dec 2019 (CTPC/DSIC 2019) — the satellite-level 2017 floor alone enumerates years of guaranteed- empty hourly listings. _PRODUCT_AVAILABLE_SINCE + _available_since_for() (max of both floors) wired into the backfill clamp. Floors set conservatively EARLY (too-early only wastes cheap listings). P2 — VIIRS phantom windows: JRR-CloudMask objects begin 2023-06-20 (npp) / 2023-02-19 (n20) / 2023-11-13 (n21), live-listed; the platform-launch floors (2012/2018/2023-01) enumerated up to ~11.5 years of nothing. Speed (probe-measured, in-region Batch job probe-goes-ondemand-121450, n2-standard-8, gcp mirror, sweep 1..64, 20 s wall): knee at 32 workers, derive_rate_cap 118.0 Hz. SOURCE-LIMITS seed records the measurement; shipped constants carry margin: _GOES_S3_RATE_HZ 20->100, _DEFAULT_MAX_WORKERS 8->32 (provenance-lock: shipped <= recorded). Station workflow: n2-standard-16 / 16 workers (compute budget approved). Also: ACMC/AODC era-variable-set comments corrected (2-var ACMC era runs through ~Dec 2021 on goes16, mid-2022 on goes17; AODC gained AE1/AE2 mid-2020). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex P1: the drift-guard test imported the heavy _goes_extract registry unguarded inside test_satellite_sources.py — a module that runs in the BASE no-extra fast suite (it tests the dep-free spine), so the default `uv run pytest -m "not live"` broke without the [satellite] extra. pytest.importorskip: skips on base installs, enforces in the satellite-coverage lane — where a drifted fallback would otherwise ship. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex P2: _GOES_S3_RATE_HZ is the measured PER-TASK AGGREGATE cap (probe-derived SOURCE-LIMITS record), but the limiter is module-local — under executor=process every spawn worker re-imports _goes_s3 and ran the FULL cap (station workflow's 16 workers ~= 1600 Hz nominal, 13x the measured knee and far past the documented limit). _make_executor's process branch now divides the cap across workers via _MOSTLYRIGHT_GOES_RATE_HZ_OVERRIDE (floor 1 Hz), set BEFORE pool creation so spawn children inherit it; _goes_s3 builds its limiter from _effective_rate_hz() at import. Thread mode is unchanged (single process, full cap, shared limiter). Malformed overrides log and fall back to the aggregate cap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
✅ Docs-required check: PASS API-surface change includes docs updates — no reminder needed. API-surface files changed: Docs files changed: |
|
Parity ticket gate: PASSED See |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Directive: full satellite-product parity + fast ingest before the one-shot full backfill. An 8-agent live parity audit (every registry cell sampled against real bucket objects) found 3 P1s that would have silently zeroed data at fleet scale, and the in-region probe finally measured the real throughput knee.
What (5 commits)
ABI-L2-DSRCregistered — cheap sector DSR (hourly ~150 KB vs multi-MB full-disk), live-verified: goes16 CONUS (166,394), goes17/18 PACUS (157,380) via_DSRC_SECTOR_SHAPES; end-to-end extraction validated on real samples (KNYC 400.76 W/m², KSEA 762.48 W/m²)._DSRF_VARIANTS: goes17 DSRF is coarse its whole life; goes18 is bimodal with the same 2024-day-108 cutover as goes16 (both were 100% rejected before)._PRODUCT_AVAILABLE_SINCEper-(satellite, product) floors (TPWC/ACHAC/DSRC/LVM/LVT rolled out Dec-2019 — years of guaranteed-empty listings killed); VIIRS JRR floors corrected (phantom windows up to 11.5 years)._GOES_S3_RATE_HZ20→100,_DEFAULT_MAX_WORKERS8→32 — backed by the measured SOURCE-LIMITS record (in-region probe, knee 32 workers @ 118 Hz); station workflow → n2-standard-16/16 workers.Validation
_GOES_ACTUAL_DATA_END.After merge
Image rebuild → single full-station dispatches (KNYC 2017–2026 ACMC+DSRF+DSRC; RJTT himawari; EGLC viirs). Projection at new constants: ~20–30 min & ~$0.25/station-year.
TS Parity
python_only: true — satellite ingest internals; no public API surface; TS SDK has no satellite ingest.
🤖 Generated with Claude Code