You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I’m working on the StarRocks engine adapter and have a question about how SQLMesh is expected to handle Materialized Views (MVs) when they depend on incremental models.
Context
Execution engine: StarRocks
I have an incremental model (incremental by time range), depend on a source table.
A downstream model is defined as a Materialized View on top of that incremental model.
All model definitions remain unchanged.
What I do is run the second sqlmesh plan with a new date range, after new data is inserted into the source table.
Observed behavior
During planning, SQLMesh reports that the downstream MV needs to [recreate view].
After execution:
The MV metadata is recreated (e.g. a new create_time is observed, but with the same hash id).
The MV data already contains the newest data.
In other words, the MV has been totaly recreated, and it refreshes the data IMMEDIATEly (a configure parameter.).
Expected / Questioned behavior
From my understanding, in this case:
The upstream incremental model’s logical SQL definition does not change.
Only a new execution time range is processed and new data is appended.
What I would expect (or like to understand if this aligns with SQLMesh design):
Should a downstream Materialized View be recreated when the upstream incremental model only refreshes a new time range?
If not, is the intended behavior that:
The MV remains unchanged and refreshes according to its own configuration, or
A REFRESH MATERIALIZED VIEW (or equivalent) is triggered instead of recreating it?
Is this behavior controlled by:
SQLMesh core (e.g. model change detection / fingerprinting), or
The engine adapter (e.g. StarRocks adapter deciding that incremental upstream refreshes are non-breaking for MVs)?
Are there existing extension points or recommended patterns for this use case?
I’m also open to the possibility that keeping the downstream MV unchanged may not be consistent with SQLMesh’s design, and would appreciate guidance on the intended semantics here.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
Hi SQLMesh team,
I’m working on the StarRocks engine adapter and have a question about how SQLMesh is expected to handle Materialized Views (MVs) when they depend on incremental models.
Context
sqlmesh planwith a new date range, after new data is inserted into the source table.Observed behavior
create_timeis observed, but with the same hash id).Expected / Questioned behavior
From my understanding, in this case:
What I would expect (or like to understand if this aligns with SQLMesh design):
Should a downstream Materialized View be recreated when the upstream incremental model only refreshes a new time range?
If not, is the intended behavior that:
REFRESH MATERIALIZED VIEW(or equivalent) is triggered instead of recreating it?Is this behavior controlled by:
Are there existing extension points or recommended patterns for this use case?
I’m also open to the possibility that keeping the downstream MV unchanged may not be consistent with SQLMesh’s design, and would appreciate guidance on the intended semantics here.
Thanks in advance for any insights.
Beta Was this translation helpful? Give feedback.
All reactions