Skip to content

TimeSeries Endpoint - Pseudo Irregular Fixes #1745

@krowvin

Description

@krowvin

Problem

For pure irregular TS we should not be doing a gap fill for pseudo irregular. We must actually be checking if it truly local regular and do a check for this before just assuming. Typically 15min is not "local", Mike says it's a 2hour TS interval minimum for that.

The direct SQL timeseries read path appears to classify any interval part beginning with ~ as regular/local-regular and therefore generates expected timestamps plus synthetic missing rows.

This is too broad. Some ~ series, especially short-interval series like:

Baldhill_Dam.Flow-Out.Inst.~15Minutes.0.best / office MVP

appear to be pseudo-irregular/observed series and should not be gap-filled unless CDA can prove the series is truly local-regular.

Current logic:

  • isRegularSeries(intervalMinutes, intervalPart) returns true when intervalMinutes != 0 || isLocalRegularInterval(intervalPart)
  • isLocalRegularInterval(intervalPart) returns true when normalized interval starts with ~

For Baldhill_Dam.Flow-Out.Inst.~15Minutes.0.best, catalog metadata currently reports:

  • interval: ~15Minutes
  • interval-offset: -2147483648
  • time-zone: US/Central

The direct read path then builds expected timestamps and merges synthetic null rows. In a 2020-present comparison, CDA returned additional null/gap-fill rows that A2W did not return. For all-history reads this also appears to contribute to poor performance/failure.

Expected Behavior

CDA should only perform expected-time generation/gap-fill when the timeseries is truly regular/local-regular under CWMS rules.

Pure irregular or pseudo-irregular ~ series should return observed rows only unless the underlying CWMS/retrieve logic explicitly defines gap-fill behavior for that series.

Acceptance Criteria

  • Add tests for a short-interval ~15Minutes pseudo-irregular series proving it does not get synthetic gap rows.
  • Keep tests proving true LRTS/local-regular series still get expected-time handling where required.
  • Do not rely only on the TSID interval string prefix ~ to decide regularity.
  • Use CWMS metadata/package behavior or another authoritative database signal to distinguish:
    • true regular
    • true local-regular/LRTS
    • pseudo-irregular/observed-only
  • Preserve data correctness for versioned reads, quality codes, units, trim behavior, and entry dates.
  • Add a regression/performance case for a long-window ~15Minutes pseudo-irregular series.
  • Must use method to check the timeseries specification to check if it's local regular. Should be a proceedure for this.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions