Skip to content

[AMORO-4110] Upgrade Iceberg versions with per-engine compatibility modules#4109

Draft
xxubai wants to merge 7 commits intoapache:masterfrom
xxubai:upgrade-iceberg-version
Draft

[AMORO-4110] Upgrade Iceberg versions with per-engine compatibility modules#4109
xxubai wants to merge 7 commits intoapache:masterfrom
xxubai:upgrade-iceberg-version

Conversation

@xxubai
Copy link
Contributor

@xxubai xxubai commented Mar 5, 2026

Why are the changes needed?

Amoro is upgrading Iceberg, but different engines (Flink/Spark/Hive) have different compatibility windows for Iceberg artifacts and APIs.
To avoid a one-version-for-all conflict, this PR introduces per-module version overrides and engine-specific compatibility modules, so newer engines can use newer Iceberg while older engines keep compatible versions.

Close #4110.

Brief change log

  • Introduce per-module iceberg.version override strategy in mixed-format modules.
  • Upgrade Iceberg for newer engines while keeping compatibility for older engines.
  • Add Flink 1.17 dedicated compatibility modules:
    • amoro-mixed-flink-common-1.17
    • amoro-mixed-flink-common-iceberg-bridge-1.17
  • Adapt Flink-related Iceberg bridge code for new Iceberg APIs:
    • parquet/flink reader path adjustments
    • MiniClusterResource test API compatibility updates
  • Adapt mixed-hive code to new Iceberg parquet API changes.
  • Adapt Spark parquet readers (v3.3/v3.4/v3.5) for Iceberg API updates.
  • Update related tests (including IcebergRewriteExecutorTest) and module wiring in POMs.
  • This change set was generated with Codex assistance.

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Local verification (examples):

  • mvn -pl amoro-format-mixed/amoro-mixed-hive -am -DskipTests compile
  • mvn -pl amoro-format-mixed/amoro-mixed-flink/amoro-mixed-flink-common-iceberg-bridge -am -DskipTests compile
  • mvn -pl amoro-format-mixed/amoro-mixed-flink/v1.17/amoro-mixed-flink-1.17,amoro-format-mixed/amoro-mixed-flink/v1.18/amoro-mixed-flink-1.18 -am -DskipTests compile
  • project package validation (including dist packaging) has been executed in local environment

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not applicable)

@xxubai xxubai marked this pull request as ready for review March 5, 2026 16:51
@xxubai xxubai force-pushed the upgrade-iceberg-version branch from b685b0a to e43126e Compare March 5, 2026 16:54
@xxubai xxubai changed the title Upgrade Iceberg versions with per-engine compatibility modules [AMORO-4110] Upgrade Iceberg versions with per-engine compatibility modules Mar 5, 2026
@xxubai xxubai marked this pull request as draft March 5, 2026 17:04
@github-actions github-actions bot added module:mixed-flink Flink moduel for Mixed Format module:mixed-spark Spark module for Mixed Format module:mixed-hive Hive moduel for Mixed Format type:build type:infra labels Mar 5, 2026
@codecov-commenter
Copy link

Codecov Report

❌ Patch coverage is 0% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 22.42%. Comparing base (e02295b) to head (9aba8b4).
⚠️ Report is 8 commits behind head on master.

Files with missing lines Patch % Lines
.../java/org/apache/amoro/op/MixedOverwriteFiles.java 0.00% 10 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #4109      +/-   ##
============================================
- Coverage     22.44%   22.42%   -0.02%     
  Complexity     2552     2552              
============================================
  Files           458      458              
  Lines         42022    42056      +34     
  Branches       5915     5917       +2     
============================================
  Hits           9433     9433              
- Misses        31777    31811      +34     
  Partials        812      812              
Flag Coverage Δ
trino 22.42% <0.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xxubai xxubai force-pushed the upgrade-iceberg-version branch from f23a886 to 1e4e991 Compare March 13, 2026 08:27
@github-actions github-actions bot removed module:mixed-flink Flink moduel for Mixed Format module:mixed-spark Spark module for Mixed Format type:infra labels Mar 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:mixed-hive Hive moduel for Mixed Format type:build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement]: Upgrade Iceberg with per-engine compatibility modules

2 participants