Skip to content

Timeline resource streaming#1929

Open
AaronPlave wants to merge 4 commits into
developfrom
feat/resource-streaming
Open

Timeline resource streaming#1929
AaronPlave wants to merge 4 commits into
developfrom
feat/resource-streaming

Conversation

@AaronPlave
Copy link
Copy Markdown
Contributor

@AaronPlave AaronPlave commented May 11, 2026

Summary

This PR implements streaming of simulation resources and external datasets (profiles) from the database as they are available, closes #1650 . External datasets are now fetched on an as-needed basis instead of fetching all external datasets that could possibly be associated with the plan every time planDataset subscription changed, closes #352.

  • Sim profiles stream segments via a new createProfileSubscription factory keyed on (datasetId, name). Each simulationDataset update triggers GET_PROFILE_SINCE for only the segments past the last seen start_offset.

  • External dataset profiles stream the same way via a new createExternalResourceSubscription factory. Metadata source is planDatasets (live subscription, already in use). When a profile's duration advances, the factory pulls only the new segments via GET_EXTERNAL_PROFILE_SEGMENTS_SINCE.

  • Removes the batch external profiles fetch. GET_PROFILES_EXTERNAL, effects.getResourcesExternal, the externalResources/fetchingResourcesExternal writable stores, and the +page.svelte reactive statement that re-fetched all external profiles on every planDatasets push are all gone.

  • New TimelineStatusIndicator next to the Timeline header. Aggregates loading/error across profile subscriptions, span fetches, constraint runs, external-event raw subscriptions, and active simulation status. Tooltip shows what failed (sim "Profile X" vs external "External profile X").

  • stores/timelineResourceStatus.ts loading/error registry for any resource on the timeline.

  • allResourceTypes now derives from planDatasets metadata directly (filtered to plan-level + current-sim rows) instead of the now removed batch fetch.

Tests

  • New: stores/profile.test.ts, stores/externalResource.test.ts, utilities/profile.test.ts. Cover settling→found, race-defer, settled-missing, sim-tied vs plan-level row preference, duration-advance refetch, profile-row switch reset, refetch error surface, dispose-during-in-flight.
  • Expanded: utilities/resources.test.ts with sort-invariant regression pins on sampleProfiles.
  • New e2e: simulation.test.ts Streaming pipeline smoke test — asserts the indicator settles and timeline canvases render non-transparent content across two re-sims. Shared canvas-pixel helper in e2e-tests/utilities/canvas.ts is now used by plan-external-source.test.ts too.

Test plan

  • Load a plan that produces thousands of profile segments such as the demo plan from the Orbiter model. Ensure that resources load in progressively during simulation and reach the end of the simulation once complete. Refresh the page and ensure that the simulation results look identical to the end-of-streaming state.
  • Load a basic model that simulates quickly like Banananation and verify that row data still loads quickly before and during simulation.
  • Upload a multi-thousand-segment external dataset: row plot fills in progressively, timeline indicator stays visible while sim is active.
  • Open a plan whose view references a resource that doesn't exist in the current model: row shows "Failed to load profiles for 1 layer", indicator shows error icon with "Resource not found in attached external datasets" tooltip.
  • Run a failed sim (e.g. invalid activity): indicator settles once status is Failed.

@AaronPlave AaronPlave requested a review from a team as a code owner May 11, 2026 20:43
@AaronPlave AaronPlave self-assigned this May 11, 2026
@AaronPlave AaronPlave requested review from dandelany and removed request for joswig and mattdailis May 11, 2026 20:57
@AaronPlave AaronPlave added the feature New feature or request label May 11, 2026
…tch instead to reduce complexity and brittleness.
…riptions

Replace batch profile fetching with per-(datasetId, name) subs that
pull only segments added since the last seen start_offset. Sim profiles
and external dataset profiles share one shape (subscribe → render →
unsubscribe) on top of a ref-counted resource-status registry that a new
TimelineStatusIndicator reads.
@AaronPlave AaronPlave force-pushed the feat/resource-streaming branch from 5f1da0c to 24611dc Compare May 13, 2026 14:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Streaming to the UI Filter external datasets by layer resource types if possible

1 participant