[Feature] use mergeData for bolt NativeCelebornClient

[zhangxffff]

Feature Category

SQL Function / Operator

Problem / Use Case

In Gluten's Celeborn Java client, pushData is used when the payload exceeds clientPushBufferMaxSize, and mergeData otherwise. However, the current Bolt implementation only use pushData and does not support mergeData.

Additionally, mergeData support depends on the upstream Celeborn PR: apache/celeborn#3611.

Proposed Solution

use mergeData in bolt NativeCelebornClient when payload is large.

References / Prior Art

No response

Importance

Blocker (Cannot use Bolt without this)

[zhangxffff]

@afterincomparableyum This issue has a dependency on your PR: apache/celeborn#3611. We would really appreciate your help here — would you mind taking a look at your convenience?

[frankobe]

@afterincomparableyum Are you interested in contributing this feature to Bolt after apache/celeborn#3611 is merged upstream?

[afterincomparableyum]

@frankobe I can pick up this task after I merge my Celeborn PR.

[frankobe]

@frankobe I can pick up this task after I merge my Celeborn PR.

@afterincomparableyum Awesome! Assigned this issue to you. Looking forward to your contribution!

[afterincomparableyum]

Planning to open a PR for this within the next 2-3 weeks