feat(ci): weather-backfill speed knobs — process executor + provisioning#102
Merged
Conversation
Two speed levers measured on the 2026-07-05 KNYC station run: - provisioning (spot|standard, default spot): Spot capacity capped the 10-shard run at 4 concurrent VMs all day and preempted the 2020 shard 104 minutes in (exit 50001) before any month had flushed — the retry restarted from zero. standard = every task starts immediately, no preemption, ~4x VM price (a full 10-shard station run costs ~+$8) — the right trade for measurement runs. Applies to all modes; full mode keeps its spot default unless overridden. - executor (process|thread, default process; mode=station only): the thread executor decodes ONE file at a time regardless of vCPUs — h5netcdf serializes behind HDF5's global mutex (observed live: 2018/ 2019 year-shards past 3h with idle cores). The spawn-context process pool (safe since PR #100) breaks the mutex AND multiplies the per-process rate limiter. thread kept as a dispatch-time fallback; the first station dispatch after a CLI change should smoke process on a cheap 1-year window. Both inputs are validated defensively in the build step; the Batch JSON takes provisioningModel via jq --arg (injection-safe pattern unchanged). Verified: station honors both knobs; pilot/full command lines unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Parity ticket gate: PASSED See |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The KNYC measurement run diagnosed a two-layer speed wall: (1) the thread executor decodes ONE file at a time — h5netcdf serializes behind HDF5's global mutex (2018/2019 year-shards ran 3h+ with idle vCPUs); (2)
CPUS_ALL_REGIONS=32global quota capped the 10-shard run at 4 VMs all day and a Spot preemption erased 104 min of the 2020 shard.What
Two dispatch-time knobs on
run-weather-backfill.yml(no CLI/library change):executor(process|thread, default process; mode=station) — the spawn-context pool (safe since fix(weather): spawn-context process pool; station mode runs thread executor #100) breaks the HDF5 mutex AND multiplies the per-process rate limiter (8 workers ≈ 8× the 20 Hz cap in aggregate). thread stays as a dispatch-time fallback; first post-merge dispatch is a cheap 1-year proving run.provisioning(spot|standard, default spot) — standard = all tasks start immediately, zero preemption; ~+$8 for a full 10-shard station run. For measurement runs.Both inputs validated defensively; Batch JSON via
jq --arg(injection-safe pattern unchanged). Verified: station honors both; pilot/full command lines byte-identical (full keeps its Spot default).Ops context (not in this diff)
_GOES_S3_RATE_HZbump follows separately once the in-region probe (queued Batch job) gets CPU headroom to run — the provenance-lock test requires its SOURCE-LIMITS record.TS Parity
python_only: true — GH workflow ops change only; no public API surface.
🤖 Generated with Claude Code