Attempt to run HiBench on JDK 17 or higher versions

[Panxuefeng-loongson]

Hi,

I attempted to evaluate the performance of JDK 17 and even more recent versions, such as JDK 21, within a big data context but faced challenges. Upon reviewing the code, I noticed that the highest specified Spark version is 3.1.1[1]. According to the Spark documentation, the highest JDK version supported by Spark 3.1.1 is JDK 11[2]. Does this imply that the maximum compatible JDK version for HiBench is limited to JDK 11? If I aim to conduct tests on JDK 17 or newer versions, what would be the recommended approach? As this is my initial endeavor with HiBench, I may have misunderstood some aspects, so I welcome any corrections or guidance
[1] https://github.com/Intel-bigdata/HiBench/blob/master/sparkbench/pom.xml#L174
[2] https://community.cloudera.com/t5/Community-Articles/Spark-and-Java-versions-Supportability-Matrix/ta-p/383669

[carsonwang]

We haven't tried JDK 17 or newer versions. You can first find out which Spark version supports the JDK version that you want. After that, update HiBench to support that Spark version. You can update the pom file to include a new profile the Spark version.