Refactor fluss-lake-paimon packaging to provide thin and shaded-bundle JARs

[luoyuxia]

Search before asking

• I searched in the issues and found nothing similar.

Description

Background
Currently, the fluss-lake-paimon JAR includes a specific version of Paimon within its packaging. While this makes it easy to use out of the box, it lacks flexibility and leads to dependency conflicts in complex environments.

The Problem
By embedding Paimon classes directly without shading:

• Class Conflicts: If a user’s environment (e.g., Flink/Spark cluster) already has a different version of Paimon, it results in LinkageError or ClassCastException.

• Version Inflexibility: Users cannot easily swap the Paimon version to a bug-fixed or slightly different version without rebuilding the entire Fluss connector.

Proposed Solution
I propose to split the current packaging into two distinct artifacts:

1. fluss-lake-paimon (The "Thin" JAR)

   Behavior: Does not include any Paimon classes. Paimon should be marked as a provided dependency.

   Use Case: Advanced users who want to manage their own Paimon version in their classpath.

   Note: While it is compiled against a specific Paimon version, sticking to Paimon's Public APIs should mitigate most ClassNotFoundException or NoSuchMethodError issues when used with compatible versions.

2. fluss-lake-paimon-bundle (The "Shaded" JAR)

   Behavior: Includes a specific version of Paimon (the one it was compiled against).

   Relocation (Shading): All Paimon classes will be shaded (e.g., relocated to com.fluss.shaded.paimon.*).

   Use Case: Users who want a "plug-and-play" experience without worrying about conflicts with their own application's Paimon version.

Willingness to contribute

• I'm willing to submit a PR!

[luoyuxia]

@wuchong What do you think of it?

[wuchong]

@luoyuxia how to use Paimon filesystem plugins and Paimon catalog plugins with the fluss-lake-paimon-bundle? Will the plugin can't be recognized by the shaded Paimon classes in Fluss connector?

[luoyuxia]

@luoyuxia how to use Paimon filesystem plugins and Paimon catalog plugins with the fluss-lake-paimon-bundle? Will the plugin can't be recognized by the shaded Paimon classes in Fluss connector?

@wuchong
Yes, If use a simple shaded bundle jar, the plugins will not be recognized. To make it recognized, maybe we can consider use an isolated classloader to load the paimon bundle jar.