-
Notifications
You must be signed in to change notification settings - Fork 488
[lake] Make FlussLakeTiering pluggable to customize tiering job const… #2364
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
[lake] Make FlussLakeTiering pluggable to customize tiering job const… #2364
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This pull request refactors the FlussLakeTiering entrypoint to improve extensibility by extracting the core job construction logic into a separate pluggable class. This enables developers to customize the tiering job construction (e.g., for security token injection) without modifying the main entrypoint.
Changes:
- Extracted tiering job logic from FlussLakeTieringEntrypoint into a new FlussLakeTiering class with protected fields and methods for extensibility
- Simplified FlussLakeTieringEntrypoint to delegate to FlussLakeTiering
- Updated README documentation to describe the new architecture
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| FlussLakeTieringEntrypoint.java | Refactored to delegate all logic to the new FlussLakeTiering class |
| FlussLakeTiering.java | New class containing the extracted configuration parsing and job execution logic with protected members for extensibility |
| README.md | Updated documentation to describe the two-class architecture |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
...flink/fluss-flink-tiering/src/main/java/org/apache/fluss/flink/tiering/FlussLakeTiering.java
Outdated
Show resolved
Hide resolved
...flink/fluss-flink-tiering/src/main/java/org/apache/fluss/flink/tiering/FlussLakeTiering.java
Outdated
Show resolved
Hide resolved
23c84eb to
efcc2f9
Compare
|
Thanks @luoyuxia! However, the current approach still feels quite hacky—it relies on class overriding and could lead to various issues, such as compilation failures or runtime incompatibilities. If our goal is to make this truly pluggable, we should design a proper plugin mechanism instead—ideally based on a standard pattern like Java SPI. For example, if the only environment-specific customization needed is around Flink execution environment and configuration, we could introduce a clean extension point like this: interface LakeTieringDecoratorPlugin extends Plugin {
String identifier();
LakeTieringDecorator createLakeTieringDecorator();
}
interface LakeTieringDecorator {
// Customize Flink env or configs as needed
void decorate(
StreamExecutionEnvironment env,
Configuration flussConfig,
Configuration dataLakeConfig,
Configuration lakeTieringConfig,
String dataLakeFormat
);
}This way, different environments (e.g., internal vs. public cloud) can provide their own decorator implementations without touching core code, ensuring better maintainability, testability, and forward compatibility. |
a4e7443 to
77f6ae2
Compare
77f6ae2 to
9b34c1d
Compare
…ruct
Purpose
Linked issue: close #2281
Brief change log
Extract the core logic of parse arguments & build tiering job to
FlussLakeTiering, the external implement can overwriteFlussLakeTiering.Some thing like
Tests
API and Format
Documentation