Run the snapshot rebuild before charts prewarm, not after#584
Open
ptrlrd wants to merge 1 commit into
Open
Conversation
The instrumentation (#583) showed the leader logs "refresh cycle: starting" and then nothing — it hangs in the very first cycle step, prewarm_charts(), and never reaches the entity-stats snapshot rebuild. prewarm_charts loads the full ~740k-run frame from Mongo, which under current DB load runs long enough to starve the rebuild indefinitely. That's why prod was frozen serving a stale snapshot (v10): the snapshot rebuild never ran. PR #567 moved prewarm_charts to the FRONT of the cycle so /charts warmed fast after a deploy. That let it block the snapshot rebuild. Move it back to the END: the snapshot rebuild (tier list / Codex Score / metrics — the critical product data) runs first and can't be starved. /charts warms a bit later after a deploy as the trade-off, which is far better than the snapshot never rebuilding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause (finally pinned down)
The instrumentation from #583 gave the definitive answer — the leader logs:
…and then nothing. It hangs in the very first cycle step,
prewarm_charts(), and never reaches the entity-stats snapshot rebuild. That's why prod has been frozen serving a stale snapshot (v10) with tier-list/metrics/composites broken.prewarm_charts()loads the full ~740k-run charts frame from Mongo, and under the current DB load (heavy presence write traffic) that runs long enough to starve the snapshot rebuild indefinitely.This is a regression from PR #567, which moved
prewarm_chartsto the front of the refresh cycle (so /charts warmed fast after a deploy). That reordering let prewarm block the snapshot rebuild that used to run first — which is exactly why the snapshot stopped advancing.Fix
Move
prewarm_chartsback to the end of the cycle. The snapshot rebuild (the critical product data) runs first and can't be starved. /charts warms a little later after a deploy as the trade-off — far better than the snapshot never rebuilding.One-file reorder, ruff + compile clean. After deploy, the logs should show the full sequence through
snapshot rebuilt: N entities across M runs, the snapshot advances to 13, and the composite tier-list/metrics combos populate.