Skip to content

Support hourly granularity end-to-end for cube temporal partitions #2097

@shangyian

Description

@shangyian

Context

DJ currently supports day-level granularity for temporal partition columns. We should add support for hour granularity so cubes and pre-aggregations can refresh on an hourly cadence.

The Granularity enum already includes HOUR and downstream metadata (lookback defaults, cron defaults, pre-aggregation model) supports it. However, cube materialization SQL generation hardcodes day-level format, so setting granularity: hour on a cube's temporal partition does not produce hour-aware filter SQL.

User Outcome

After this lands, a user should be able to:

  1. Define a cube with an hour-grain temporal partition column in YAML:
columns:
  - name: dateint_hour
    type: int
    partition:
      type: temporal
      granularity: hour
      format: yyyyMMddHH
  1. Configure materialization at hourly cadence -- the platform reads granularity: hour and applies the right defaults:
  • lookback_window: 1 HOUR
  • Cron schedule: 0 * * * * (top of every hour)
  • Workflow iterates per-hour over the backfill range
  1. Get hour-aware filter SQL generated automatically. Calling the materialization endpoint produces partition filters like: dateint_hour = CAST(DATE_FORMAT(CAST(DJ_LOGICAL_TIMESTAMP() AS TIMESTAMP), 'yyyyMMddHH') AS INT), or a BETWEEN form with the configured lookback for catch-up runs.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions