Skip to content

Comments

feat: initial pull based ingestion#3125

Draft
Ziinc wants to merge 7 commits intomainfrom
feat/pull-based-ingestion
Draft

feat: initial pull based ingestion#3125
Ziinc wants to merge 7 commits intomainfrom
feat/pull-based-ingestion

Conversation

@Ziinc
Copy link
Contributor

@Ziinc Ziinc commented Feb 2, 2026

This PR implements pull-based jobs, backed by Oban (postgres-based job queue) for consistency and reliability, as well as cross cluster deduplication. This allows unique jobs to be executed across multiple clusters and nodes.

Backed by oban. if this proves successful, we will switch alerts over to using oban as well.

Deferred:

  • BQ pagination
  • verify cross cluster locking
  • specify if cluster has job fetching enabled (enable/disable env var flag like alerts)
  • test BQ max row limit, maybe allow configuration
  • verify no additional jobs are created if uniqueness conflict is present
  • change execution history to be based on system.traces and system.monitoring instead.
  • reduce oban max retention to a few seconds.
  • reduce oban job pre-population
  • docs
  • audit for performance

Part of ANL-1259

@Ziinc Ziinc marked this pull request as ready for review February 13, 2026 16:33
Copy link
Contributor

@chasers chasers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to sync on this one in more detail first

@Ziinc Ziinc marked this pull request as draft February 13, 2026 17:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants