Timeline resource streaming#1929
Open
AaronPlave wants to merge 4 commits into
Open
Conversation
…tch instead to reduce complexity and brittleness.
…riptions Replace batch profile fetching with per-(datasetId, name) subs that pull only segments added since the last seen start_offset. Sim profiles and external dataset profiles share one shape (subscribe → render → unsubscribe) on top of a ref-counted resource-status registry that a new TimelineStatusIndicator reads.
5f1da0c to
24611dc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements streaming of simulation resources and external datasets (profiles) from the database as they are available, closes #1650 . External datasets are now fetched on an as-needed basis instead of fetching all external datasets that could possibly be associated with the plan every time
planDatasetsubscription changed, closes #352.Sim profiles stream segments via a new
createProfileSubscriptionfactory keyed on(datasetId, name). EachsimulationDatasetupdate triggersGET_PROFILE_SINCEfor only the segments past the last seenstart_offset.External dataset profiles stream the same way via a new
createExternalResourceSubscriptionfactory. Metadata source isplanDatasets(live subscription, already in use). When a profile'sdurationadvances, the factory pulls only the new segments viaGET_EXTERNAL_PROFILE_SEGMENTS_SINCE.Removes the batch external profiles fetch.
GET_PROFILES_EXTERNAL,effects.getResourcesExternal, theexternalResources/fetchingResourcesExternalwritable stores, and the +page.svelte reactive statement that re-fetched all external profiles on everyplanDatasetspush are all gone.New
TimelineStatusIndicatornext to the Timeline header. Aggregates loading/error across profile subscriptions, span fetches, constraint runs, external-event raw subscriptions, and active simulation status. Tooltip shows what failed (sim "Profile X" vs external "External profile X").stores/timelineResourceStatus.tsloading/error registry for any resource on the timeline.allResourceTypesnow derives fromplanDatasetsmetadata directly (filtered to plan-level + current-sim rows) instead of the now removed batch fetch.Tests
stores/profile.test.ts,stores/externalResource.test.ts,utilities/profile.test.ts. Cover settling→found, race-defer, settled-missing, sim-tied vs plan-level row preference, duration-advance refetch, profile-row switch reset, refetch error surface, dispose-during-in-flight.utilities/resources.test.tswith sort-invariant regression pins onsampleProfiles.simulation.test.tsStreaming pipeline smoke test — asserts the indicator settles and timeline canvases render non-transparent content across two re-sims. Shared canvas-pixel helper ine2e-tests/utilities/canvas.tsis now used byplan-external-source.test.tstoo.Test plan