fix(weather): spawn-context process pool; station mode runs thread executor#100
Merged
Conversation
…ecutor Live outage on the first station-mode fleet run (job weather-backfill-station-28734758679, 2026-07-05): every task died with BrokenProcessPool and exited "0 slices completed, 0 rows written" in 0.2s — 5 tasks reported SUCCEEDED having done nothing. Root cause: --progress-bucket starts the GcsProgressStore gRPC client threads at CLI startup; ProcessPoolExecutor then forks (Linux default start method) and the children inherit a torn copy of those threads' locks -> instant pool breakage. Never reproduced locally (macOS defaults to spawn) or in CI (the process-pool test runs without a progress bucket, so no gRPC threads exist pre-fork). - _backfill._make_executor: pin mp_context=spawn for the process pool. _run_slice + its item tuple are module-level/picklable by design (P1-1), so spawn is safe. Regression test pins the start method and the existing end-to-end process test still passes under spawn. - run-weather-backfill.yml station mode: drop --executor process; run the default THREAD executor — the proven prod path (the 2024 pilot processed + uploaded DSRF with threads; numpy/h5py release the GIL for the heavy decode). Re-enable process only after the spawn path is proven in a container. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
📝 Docs-required check: REMINDER API-surface change without docs and no opt-out — surfacing reminder. API-surface files changed: Docs files changed: Docs surfaces to consider
How to silence this reminder
This check is advisory — it never blocks the merge. |
|
Parity ticket gate: PASSED See |
… boundary Codex P2 on the spawn-context change: with spawn, pool children re-import _backfill fresh, so test_process_executor_for_dsrf's parent-side mock.patch of backfill_goes_satellite never reached them — the "network-free" test silently ran the real 2024 GOES DSRF transport (macOS hit this even before the spawn pin, spawn being its default; CI's fork start method masked it). Rewrite to the same spawn-safe shape as test_process_executor_runs_end_to_end: submit a pre-first-light year (2016 < goes16 first light 2017-05-24) so every slice short-circuits child-side via the available_since clamp with zero I/O, and assert the skip actually held. Runs in ~2s with no network. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
First station-mode fleet run (
weather-backfill-station-28734758679, right after #99 merged) died instantly: every task hitBrokenProcessPooland exited "0 slices completed, 0 rows written" in 0.2s — 5 tasks reported SUCCEEDED having done nothing.Root cause:
--progress-bucketstarts the GcsProgressStore gRPC client threads at CLI startup;ProcessPoolExecutorthen forks (Linux default) and children inherit a torn copy of those threads' locks → instant pool breakage. Why nothing caught it: macOS defaults to spawn (local tests pass), and CI's process-pool test runs without a progress bucket, so no gRPC threads exist pre-fork. The--executor processflag came from #99's station mode; the old pilots used threads, which is why they never hit it.What
_backfill._make_executor: pinmp_context=spawnfor the process pool._run_slice+ item tuple are module-level/picklable by design (P1-1), so spawn is safe. Regression test pins the start method; the existing end-to-end process test passes under spawn.run-weather-backfill.ymlstation mode: drop--executor process→ default thread executor, the proven prod path (2024 pilot processed + uploaded DSRF with threads; numpy/h5py release the GIL for decode). Re-enable process only after spawn is proven in a container.Evidence
Broken run's Cloud Logging shows the gRPC fork warnings + BrokenProcessPool tracebacks + "0 slices" exits. No progress corruption: the only markers written (DSRF 2017 Jan–Apr
local-only) are legitimately empty pre-first-light months; resume is safe.TS Parity
python_only: true — internal satellite backfill executor + ops workflow; no public API surface, no TS equivalent (TS SDK has no satellite ingest).
🤖 Generated with Claude Code