batch asset events#68517
Conversation
68e0576 to
309d811
Compare
c6a5e34 to
4d94b18
Compare
|
Early feedback while CI is running: Can you add some documentation with a very simple example to airflow-core/docs/authoring-and-scheduling/asset-scheduling.rst? Note for reference, this is related to Slack discussion in https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1778258939480149 |
Done |
b11f647 to
12fb085
Compare
12fb085 to
527d8f2
Compare
| Controlling DagRun creation per asset event | ||
| --------------------------------------------- | ||
|
|
||
| By default, when multiple asset events arrive for the same Dag between |
There was a problem hiding this comment.
nit: Can you add a marker that this is added in 3.3.0?
| By default, when multiple asset events arrive for the same Dag between | |
| .. versionadded:: 3.3.0 | |
| By default, when multiple asset events arrive for the same Dag between |
jscheffl
left a comment
There was a problem hiding this comment.
Besides the comment nit I think this is a good solution to fix the problem raised in #53896 and most part of #56750 in my view. (in #56750 part 3 is a bug whereas any background is missing allowing a triage, so maybe this need either more evidence or should be closed).
But besides the positive discussions in Slack would like to have sone Asste Event experts additional opinion, mostly @uranusjr (alternatively @Lee-W / @ashb).
Add
batch_asset_eventsparameter to asset-triggered timetablesIntroduces
batch_asset_events: bool = TrueonAssetTriggeredTimetableandPartitionedAssetTimetable(both SDK and core versions). When set toFalse,asset-triggered Dags create one DagRun per individual asset event instead of
batching all events into a single DagRun.
Behavior
batch_asset_eventsTrue(default)FalseUsage
batch_asset_eventsdefaults toTrue. Set toFalsefor one DagRun per event.Non-partitioned:
Dag(dag_id="per-event", schedule=AssetTriggeredTimetable(assets=asset_1, batch_asset_ events=False))Partitioned:
Dag(dag_id="per-event", schedule=PartitionedAssetTimetable(assets=asset_1, batch_asset _events=False))Related #53896
Was generative AI tooling used to co-author this PR?
Generated-by: codewhale following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.