Problem
For pure irregular TS we should not be doing a gap fill for pseudo irregular. We must actually be checking if it truly local regular and do a check for this before just assuming. Typically 15min is not "local", Mike says it's a 2hour TS interval minimum for that.
The direct SQL timeseries read path appears to classify any interval part beginning with ~ as regular/local-regular and therefore generates expected timestamps plus synthetic missing rows.
This is too broad. Some ~ series, especially short-interval series like:
Baldhill_Dam.Flow-Out.Inst.~15Minutes.0.best / office MVP
appear to be pseudo-irregular/observed series and should not be gap-filled unless CDA can prove the series is truly local-regular.
Current logic:
isRegularSeries(intervalMinutes, intervalPart) returns true when intervalMinutes != 0 || isLocalRegularInterval(intervalPart)
isLocalRegularInterval(intervalPart) returns true when normalized interval starts with ~
For Baldhill_Dam.Flow-Out.Inst.~15Minutes.0.best, catalog metadata currently reports:
- interval:
~15Minutes
- interval-offset:
-2147483648
- time-zone:
US/Central
The direct read path then builds expected timestamps and merges synthetic null rows. In a 2020-present comparison, CDA returned additional null/gap-fill rows that A2W did not return. For all-history reads this also appears to contribute to poor performance/failure.
Expected Behavior
CDA should only perform expected-time generation/gap-fill when the timeseries is truly regular/local-regular under CWMS rules.
Pure irregular or pseudo-irregular ~ series should return observed rows only unless the underlying CWMS/retrieve logic explicitly defines gap-fill behavior for that series.
Acceptance Criteria
- Add tests for a short-interval
~15Minutes pseudo-irregular series proving it does not get synthetic gap rows.
- Keep tests proving true LRTS/local-regular series still get expected-time handling where required.
- Do not rely only on the TSID interval string prefix
~ to decide regularity.
- Use CWMS metadata/package behavior or another authoritative database signal to distinguish:
- true regular
- true local-regular/LRTS
- pseudo-irregular/observed-only
- Preserve data correctness for versioned reads, quality codes, units, trim behavior, and entry dates.
- Add a regression/performance case for a long-window
~15Minutes pseudo-irregular series.
- Must use method to check the timeseries specification to check if it's local regular. Should be a proceedure for this.
Problem
For pure irregular TS we should not be doing a gap fill for pseudo irregular. We must actually be checking if it truly local regular and do a check for this before just assuming. Typically 15min is not "local", Mike says it's a 2hour TS interval minimum for that.
The direct SQL timeseries read path appears to classify any interval part beginning with
~as regular/local-regular and therefore generates expected timestamps plus synthetic missing rows.This is too broad. Some
~series, especially short-interval series like:Baldhill_Dam.Flow-Out.Inst.~15Minutes.0.best/ officeMVPappear to be pseudo-irregular/observed series and should not be gap-filled unless CDA can prove the series is truly local-regular.
Current logic:
isRegularSeries(intervalMinutes, intervalPart)returns true whenintervalMinutes != 0 || isLocalRegularInterval(intervalPart)isLocalRegularInterval(intervalPart)returns true when normalized interval starts with~For
Baldhill_Dam.Flow-Out.Inst.~15Minutes.0.best, catalog metadata currently reports:~15Minutes-2147483648US/CentralThe direct read path then builds expected timestamps and merges synthetic null rows. In a 2020-present comparison, CDA returned additional null/gap-fill rows that A2W did not return. For all-history reads this also appears to contribute to poor performance/failure.
Expected Behavior
CDA should only perform expected-time generation/gap-fill when the timeseries is truly regular/local-regular under CWMS rules.
Pure irregular or pseudo-irregular
~series should return observed rows only unless the underlying CWMS/retrieve logic explicitly defines gap-fill behavior for that series.Acceptance Criteria
~15Minutespseudo-irregular series proving it does not get synthetic gap rows.~to decide regularity.~15Minutespseudo-irregular series.