fix(weather): unblock full-disk goes16 DSRF + fast single-station year-shard backfill#99
Merged
Merged
Conversation
goes16 ABI-L2-DSRF switched from the coarse 326x326 lat/lon grid to the full-disk 5424x5424 ABI fixed grid around 2024 day-108 (~Apr 17). The _validate_dataset_shape gate hard-coded goes16 DSRF to the coarse shape by SATELLITE, so every post-cutover full-disk file was rejected "before pixel read" — stalling the entire DSRF ≥ mid-2024 backfill (observed live: a KNYC 2024 pilot uploaded DSRF Jan–Mar then failed every file from day-108 on). The extractor (_extract_from_dataset) already branches per-file on the projection variable (goes_imager_projection vs goes_lat_lon_projection) and is satellite-agnostic — it would decode the full-disk file correctly. Only the shape gate was wrong. Fix: for goes16 DSRF, key the accepted shape on the file's OWN projection variable (coarse lat/lon → 326x326; fixed grid → 5424x5424), mirroring the extractor. goes19 stays strict (full-disk only) so a swapped/mislabeled grid still fails loudly. RED→GREEN: adds a regression that a full-disk goes16 DSRF file passes the gate, plus wrong-shape-on-fixed-grid and full-disk-shape-on-latlon rejection tests. All 46 test_satellite_s3 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A full single-station backfill (one station × all years) had no fast path: `pilot` mode runs it as ONE monolithic Cloud Batch task that blows the 6h maxRunDuration (observed: a KNYC 2017–2026 pilot timed out with only 2 years done), and `full` mode is the 65-station × N-year ~28 TB fleet. Neither gives a bounded, parallel single-station run for a cost/speed measurement. - CLI: extend explicit `--stations` mode to honor the same station×year shard decomposition roster mode already uses. When an array job supplies a shard count > 1 (`--shard-count` or Cloud Batch `BATCH_TASK_COUNT`), each task narrows to exactly one (station, year) via `shard_roster`. Shard count == 1 is a no-op, so the classic single-task pilot / local run stays byte-identical. Reuses the existing per-shard progress key so year-shards never clobber each other's resume markers. - Workflow: add a `station` mode to run-weather-backfill.yml — task_count = stations × years, n2-standard-8 SPOT, parallelism capped at 48, `--executor process` (DSRF full-disk), `--max-workers 8`, R2 + durable progress + CLOUD_LOGGING. One station × 10 years submits as 10 parallel year-shards, each a bounded < 6h unit. Not the 28 TB fleet, so NOT cost-gated. pilot/full modes unchanged (verified: task_count 1 / 650). Tests: explicit-mode station×year decomposition (single + multi station), shard-count-1 no-op, and BATCH_TASK env-driven sharding. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adversarial review caught (and reproduced in bash under set -euo pipefail) that a reversed year window (year_end < year_start), empty, or non-numeric year input in the new `station` backfill mode makes NUM_YEARS <= 0 -> negative TASK_COUNT/PARALLELISM that jq --argjson would bake into an invalid Batch job submitted to the API. Add an explicit numeric + ordering guard that fails loud with an operator-facing ::error:: before the arithmetic. pilot/full modes unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
📝 Docs-required check: REMINDER API-surface change without docs and no opt-out — surfacing reminder. API-surface files changed: Docs files changed: Docs surfaces to consider
How to silence this reminder
This check is advisory — it never blocks the merge. |
|
Parity ticket gate: PASSED See |
Codex closing-pass P2: the previous guard only rejected non-digit input, so a short-year typo like year_start=17 (blank year_end) passed and made NUM_YEARS = current_year - 17 + 1 ≈ 2010 — submitting ~2,010 n2-standard-8 Spot tasks for one station instead of failing early. Match exactly [0-9][0-9][0-9][0-9] before the task_count arithmetic; verified: 17 / 20267 / 99 / abcd / reversed all rejected, 2017..2026 still yields task_count=10. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Two blockers stood between us and a full single-station GOES backfill to measure cost + speed:
ABI-L2-DSRFswitched from the coarse 326×326 lat/lon grid to the full-disk 5424×5424 ABI fixed grid around 2024 day-108 (~Apr 17). The shape gate hard-coded goes16 DSRF to the coarse shape by satellite, so every post-cutover file was rejectedshape (5424, 5424) does not match expected [(326, 326)] — rejected before pixel read. Observed live: a KNYC 2024 pilot uploaded DSRF Jan–Mar then failed every file from day-108 on, stalling the whole DSRF ≥ mid-2024 backfill.pilotmode runs a full station as one monolithic Batch task that blows the 6hmaxRunDuration(a KNYC 2017–2026 pilot timed out with only 2 years done);fullmode is the cost-gated 65-station × N-year ~28 TB fleet. Neither gives a bounded, parallel single-station run.What
fix(weather)— accept full-disk goes16 DSRF._validate_dataset_shapenow keys the accepted DSRF shape for goes16 on the file's own projection variable (goes_lat_lon_projection→ 326×326; else fixed grid → 5424×5424), mirroring_extract_from_dataset's existing per-file branch (which already decodes full-disk files correctly — only the gate was wrong).goes19stays full-disk-only, and a mislabeled/swapped grid still fails loudly.feat(weather)— fast full-single-station backfill. Explicit--stationsmode now honors the same station×yearshard_rosterdecomposition roster mode uses: when an array job supplies a shard count > 1 (--shard-countor BatchBATCH_TASK_COUNT), each task narrows to exactly one(station, year). Shard count == 1 is a no-op (classic pilot / local run stays byte-identical). Newstationworkflow mode:task_count = stations × years,n2-standard-8SPOT, parallelism ≤ 48,--executor process(DSRF full-disk), R2 + durable progress +CLOUD_LOGGING. KNYC 2017–2026 → 10 parallel year-shards, each a bounded < 6h unit.fix(ci)— guard the station-mode year window (reversed/empty/non-numeric → loud::error::before thetask_countmath).How to run the measurement (operator)
Actions → Run weather backfill fleet → mode=station, pilot_station=KNYC, products=ABI-L2-ACMC,ABI-L2-DSRF, year_start=2017. Cost/speed come from the 10 job durations + R2 bytes. Not cost-gated (bounded single station).Tests
test_satellite_s346/46.BATCH_TASKenv-driven.test_cli_roster+test_roster38/38.Review
57dc215. Codexhighindependent pass alongside.research(),merge/*,live/_sources.py, or CWOP.TS Parity
python_only: true — internal satellite backfill infra: a
_fetchersshape-guard fix + the ingest/ops backfill CLI + a GH workflow. Nomostlyright.*public API surface change, and the TS SDK is client-side (query/catalog) with no satellite ingest/backfill, so there is no TS equivalent to keep in parity. (_goes_s3.pymatched the_fetchers/**parity-trigger glob, but the change is an internal grid-shape gate, not public surface.)🤖 Generated with Claude Code