Search before asking
Motivation
This is the umbrella issue for PIP-30 — moving Paimon's Flink committer from a downstream CommitterOperator to a JobManager-side OperatorCoordinator. The target is Flink jobs writing to unaware-bucket append tables, where this enables region failover and removes the commit-side fan-in shuffle for large-parallelism Flink-to-Paimon write jobs.
Recent design discussion (with updated diagrams that didn't render cleanly on cwiki):
Solution
The work will be split across a series of PRs to keep each one independently reviewable. Tentative breakdown (subject to change as reviews progress):
Links to per-PR issues will be added here as they are opened.
Anything else?
No response
Are you willing to submit a PR?
Search before asking
Motivation
This is the umbrella issue for PIP-30 — moving Paimon's Flink committer from a downstream
CommitterOperatorto a JobManager-sideOperatorCoordinator. The target is Flink jobs writing to unaware-bucket append tables, where this enables region failover and removes the commit-side fan-in shuffle for large-parallelism Flink-to-Paimon write jobs.Recent design discussion (with updated diagrams that didn't render cleanly on cwiki):
Solution
The work will be split across a series of PRs to keep each one independently reviewable. Tentative breakdown (subject to change as reviews progress):
Committer.Contextfrom operator-only assumptions.WriteOperatorCoordinatoras the JobManager-side commit driver.Links to per-PR issues will be added here as they are opened.
Anything else?
No response
Are you willing to submit a PR?