Skip to content

[FLUSS] Add Producer Offset Snapshot for Exactly-Once semantics #2434

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
platinumhamburg wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[FLUSS] Add Producer Offset Snapshot for Exactly-Once semantics #2434

platinumhamburg wants to merge 1 commit into from

Conversation

@platinumhamburg
Copy link
Contributor

@platinumhamburg platinumhamburg commented Jan 22, 2026

Core changes:

  • Add ProducerSnapshotManager for lifecycle management with atomic registration
  • Add ProducerSnapshotStore for ZK + remote storage operations
  • Add tryRegisterProducerSnapshot in ZooKeeperClient for atomic check-and-create
  • Add Admin API: registerProducerOffsets, getProducerOffsets, deleteProducerOffsets
  • Add configurable TTL and cleanup interval for producer snapshots

Design highlights:

  • Atomic registration via ZK's NodeExistsException handling
  • Eventually consistent: ZK as commit point, orphan files cleaned periodically
  • UUID-based file naming prevents concurrent upload conflicts

Tests:

  • ProducerSnapshotManagerTest: lifecycle, expiration, concurrent atomicity
  • ProducerSnapshotJsonSerdeTest: JSON format compatibility

Purpose

Linked issue: close #2433

Brief change log

Tests

API and Format

Documentation

@platinumhamburg
[FLUSS] Add producer offset snapshot management for undo recovery
d34c50f 
This commit introduces a producer offset snapshot mechanism to support
undo recovery when Flink jobs fail before completing their first checkpoint.

Key changes:
- Add ProducerSnapshotManager for snapshot lifecycle management including
  registration, retrieval, deletion, and periodic cleanup of expired snapshots
- Add ProducerSnapshotStore for low-level ZooKeeper and remote storage operations
- Add ProducerSnapshot and ProducerSnapshotJsonSerde for snapshot data model
- Extend Admin API with registerProducerOffsets, getProducerOffsets, and
  deleteProducerOffsets operations
- Add RPC protocol definitions for producer offset snapshot management
- Add configuration options for snapshot TTL and cleanup interval
- Add unit tests for ProducerSnapshotManager and JSON serialization
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Server] Add Producer Offset Snapshot Registry as Infrastructure for Exactly-Once Semantics

1 participant

@platinumhamburg