Skip to content

[Feature] PIP-30: Introduce coordinator-based commit for Flink sink #8220

@ifndef-SleePy

Description

@ifndef-SleePy

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

This is the umbrella issue for PIP-30 — moving Paimon's Flink committer from a downstream CommitterOperator to a JobManager-side OperatorCoordinator. The target is Flink jobs writing to unaware-bucket append tables, where this enables region failover and removes the commit-side fan-in shuffle for large-parallelism Flink-to-Paimon write jobs.

Recent design discussion (with updated diagrams that didn't render cleanly on cwiki):

Solution

The work will be split across a series of PRs to keep each one independently reviewable. Tentative breakdown (subject to change as reviews progress):

  • Decouple Committer.Context from operator-only assumptions.
  • Introduce WriteOperatorCoordinator as the JobManager-side commit driver.
  • Wire the unaware-bucket append table to coordinator-based commit.
  • Follow-ups (performance polish, documentation) as needed.

Links to per-PR issues will be added here as they are opened.

Anything else?

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions