Support integration with Celeborn C++ client

[SteNicholas]

Celeborn is an intermediate data service for Big Data compute engines (i.e. ETL, OLAP and Streaming engines) to boost performance, stability, and flexibility. Intermediate data typically include shuffle and spilled data. Celeborn is widely used in Spark as a remote shuffle service, which is also used in bytedance. Therefore, Bolt would support integration with Celeborn C++ client to optimize native shuffle.

[zhangxffff]

Currently, Bolt integrates with Celeborn through Gluten. On the shuffle writer side, Gluten provides a JNI-wrapped Celeborn client to the Bolt shuffle writer, which then communicates with the Celeborn service through that client. On the shuffle reader side, Gluten wraps the Celeborn input stream as a Java InputStream and passes it to the Bolt shuffle reader, enabling Bolt to fetch shuffle data from that stream.

We also plan to adopt Celeborn C++ client in Bolt, which may bring several advantages:

1. Bolt could communicate with the Celeborn shuffle service directly, removing the JNI layer.
2. Introduces additional optimization opportunities, such as async shuffle I/O and buffer zero-copy.
3. Makes the parameter-passing interface between Gluten and Bolt more explicit and easier to maintain.
4. Other data compute engines using bolt, in addition to Spark, can also integrate with Celeborn more easily through Bolt.

[SteNicholas]

@zhangxffff, thanks for reply. How could we cooperate integration with Celeborn C++ client？

[zhangxffff]

@zhangxffff, thanks for reply. How could we cooperate integration with Celeborn C++ client？
Thank you for the support from the Celeborn community!

To integrate with the Celeborn C++ client, the following work items may be required from our perspective:

1. Package the Celeborn C++ client as a Conan recipe to manage dependencies properly.
2. Provide deployment scripts or a testing environment for Celeborn within Bolt to support development and verification of the C++ client integration.
3. Implement the partition writer and input reader in Bolt Shuffle using the Celeborn C++ client, and validate them using the setup from step 2.
4. Pass the necessary Celeborn configurations from Gluten to Bolt, and verify end-to-end integration together with Gluten.

Please let us know if you have any concerns or suggestions. We are very happy to collaborate closely with the Celeborn community to bring full C++ client integration into Bolt.

[afterincomparableyum]

Happy to help out in this as well with the community.

[zhangxffff]

@SteNicholas @afterincomparableyum I have filed sub-issue #364 regarding the integration of Bolt's Celeborn C++ client with Gluten. Any feedback or guidance you could offer would be greatly appreciated.

[zhangxffff]

Also, thanks to @SteNicholas for the reminder: Bolt's NativeCelebornClient does not currently support mergeData, which is used in the Celeborn Java client. This is tracked in issue #370 and depends on @afterincomparableyum's upstream PR: apache/celeborn#3611.