Dependencies update for S3/HDFS datasources

[ostinru]

Hi, fellas!
I'd like to bump minor versions of dependencies for all S3 and HDFS/Hive datasources.

At this moment I would like to:

• bump minor versions - get low hanging fruits: fix CVEs (if any) and get bugfixes.
• add support for zstd compression support
• test in manually & in CI
• probably fix couple tests from pxf_fdw's pool of broken tests.

[MisterRaindrop]

Regarding CVE, I have been researching for a while.

PXF depends on Spring Boot. Currently, the Spring Boot 3.4.x series has no vulnerabilities. However, their minimum required JDK version is 17.

Hadoop currently likely only supports up to JDK 11.

PXF also depends on the JDK in Cloudberry. Currently, the kernel is JDK 8.

If we need to change the JDK, we must upgrade from Cloudberry to the surrounding Java components in the Cloudberry ecosystem. This change would be very extensive.

[ostinru]

Spring Boot has very short OSS support time (green boxes):

From vmware-pxf changelog it is clear that they are working with spring-boot-2.7.30.
And... from spring-boot changelog: it mostly dependencies updates.
And... almost all dependencies available (except enterprise spring-*).
So, we can live with spring-boot-2.7.x for a while with java 8..21.

However I would like to test it with java 17 (they said that every major JVM version gives several percent boost in performance) and with java 22+ (they said that JEP454 enables java to have faster compression/decompression libraries).

PXF also depends on the JDK in Cloudberry. Currently, the kernel is JDK 8.

f we need to change the JDK, we must upgrade from Cloudberry to the surrounding Java components in the Cloudberry ecosystem. This change would be very extensive.

Could you point on what in Cloudberry requires JVM?
Unless Cloudberry uses cloudberry-pxf java-libraries, it should be possible to run 2 different JVMs on the same machine.

Hadoop currently likely only supports up to JDK 11.

I will check hadoop client-libraries.

PS: java / spring upgrade is out of scope for this issue.

[tuhaihe]

Hi guys, could you see the details on this page: https://github.com/apache/cloudberry-pxf/security/dependabot? Maybe it can be a reference for security updates.

[MisterRaindrop]

Hi guys, could you see the details on this page: https://github.com/apache/cloudberry-pxf/security/dependabot? Maybe it can be a reference for security updates.

404 not found

[ostinru]

Hi guys, could you see the details on this page: https://github.com/apache/cloudberry-pxf/security/dependabot? Maybe it can be a reference for security updates.

We have this page in our fork. There are a lot of notifications for:

• automation tests - there is no urgency to fix
• spring/tomcat - there are vulnerabilities, however it is hard to exploit when PXF is running on the same host as cloudberry (it is the only supported option).
• issues in other components - it can be dangerous (e.g. security issue in avro) - that is what I am want to fix in this ticket.

[MisterRaindrop]

other cloudberry-contrib depends on jdk. see detail https://github.com/cloudberry-contrib/?q=&type=all&language=java&sort=

[MisterRaindrop]

Regarding the Spring issue, I suggest opening a new issue to discuss it ？

[ostinru]

AVRO: #45

[ostinru]

Apache ORC

Changelog 1.6.12 -> 1.7.x:
ORC-757 Add Hashtable implementation for dictionary

Changelog 1.7.x -> 1.8.x:
ORC-1136 Optimize reads by combining multiple reads without significant separation into a single read
ORC-1138 Optimizes the read of streams in ORC by combining multiple nearby reads into a single read, optionally allowing the retention or drop of the extra bytes
ORC-1172 add row count limit config for one stripe

Changelog 1.8.x -> 1.9.x:
no important changes?

Changelog 1.9.x -> 2.0.x:
require Java 17
faster Zstd

Changelog 2.0.x -> 2.1.x:
require Java 17

Changelog 2.1.x -> 2.2.x:
require Java 17
ORC-1903 Support Geometry and Geography types

Version Upgrade

At first glance it is reasonable to upgrade to 1.9.7. However it breaks something in HBASE support. PXF manages dependencies semi-manually (aka transitive = false). It seems that it pulls fresh (unpinned) libraries OR it won't work with protobuf 3.x (it is required to bump). Still investigating.

orc-core	1.6.13	1.7.11	1.8.10	1.9.8
com.google.protobuf:protobuf-java	2.5.0	2.5.0	3.17	X
io.airlift:aircompressor	0.21	0.21	0.21	0.27
org.apache.commons:commons-lang3	3.12.0	3.12.0	3.12	3.12.0
org.apache.hadoop:hadoop-common	2.2.0	2.2.0	2.7.3	2.7.3 (provided)
org.apache.hadoop:hadoop-hdfs	2.2.0	2.2.0	2.7.3	2.7.3 (provided)
org.apache.hive:hive-storage-api	2.7.3	2.8.1	2.8.1	X
org.apache.orc:orc-shims	1.6.14	1.7.11	1.8.10	1.9.8
org.jetbrains:annotations	17	17	17	17
org.slf4j:slf4j-api	1.7.5	1.7.33	1.7.36	2.0.7
org.threeten:threeten-extra	1.5.0	1.5.0	1.7.1	1.7.1

UPD: Surprisingly, pinned aircompressor:0.8 doesn't have ZSTD included, however in documentation it is expected to work well.
UPD: #50

[ostinru]

It seems that I have figured out how to solve HBASE-dependencies issue: #51 . HBASE is not on my priority list, so I moved it to separate issue to keep this knowledge visible.

[ostinru]

Parquet update merged: #56

[ostinru]

Apache Parquet

Changes 1.11.1 -> 1.12.3 ( #56 )

• Bloom filter support

<In previous attempt Hbase blocked next updates. It was addressed in #62 >

Changes 1.12.x -> 1.13.x

• Support LZ4_RAW codec

Changes 1.13.x -> 1.14.x

• Java support for Arrow's float16
• Implement vectored IO in parquet file format
• More optimal memory usage in compression codecs

Changes 1.14.x -> 1.15.x

• nothing interesting, mostly dependencies update

Changes 1.15.x -> 1.16.x

• Remove hadoop-2 support

Changes 1.16.x -> 1.17.x

• Java 11+

Version Upgrade

Dependency tree is small.

+--- org.apache.parquet:parquet-column:1.15.1
|    +--- org.apache.parquet:parquet-common:1.15.1
|    |    +--- org.apache.parquet:parquet-format-structures:1.15.1
|    +--- org.apache.parquet:parquet-encoding:1.15.1
|    |    +--- org.apache.parquet:parquet-common:1.15.1 (*)
+--- org.apache.parquet:parquet-hadoop:1.15.1
|    +--- org.apache.parquet:parquet-column:1.15.1 (*)
|    +--- org.apache.parquet:parquet-format-structures:1.15.1 (*)
|    +--- org.apache.parquet:parquet-common:1.15.1 (*)
|    +--- org.xerial.snappy:snappy-java:1.1.10.7
|    +--- io.airlift:aircompressor:2.0.2
|    +--- commons-pool:commons-pool:1.6
|    +--- com.github.luben:zstd-jni:1.5.6-6
+--- org.apache.parquet:parquet-jackson:1.15.1
+--- org.apache.parquet:parquet-generator:1.15.1
+--- org.apache.parquet:parquet-pig:1.15.1
|    +--- org.apache.parquet:parquet-column:1.15.1 (*)
|    +--- org.apache.parquet:parquet-hadoop:1.15.1 (*)
|    +--- org.apache.parquet:parquet-common:1.15.1 (*)
\--- org.apache.parquet:parquet-format:2.10.0

It has its own thrift library shaded. It doesn't depend on protobuf.
parquet-hadoop in fact expects that there is hadoop-client, hadoop-common, hadoop-annotations and hadoop-mapreduce-client-core provided.

Compression

Overall Parquet compression supports[1]:

• ✅ uncompressed
• ✅ snappy
• ✅ zstd
• ✅ gzip
• 🛑 lz4 -> deprecated[1]
• ✅ lz4_raw - NEW!
• 🛑 lzo - WONTFIX - no updates for this java dependency for 7 years[3][2]
• 🛑 brotli - WONTFIX - not in maven central[2] and no user request

[1] https://github.com/apache/parquet-format/blob/apache-parquet-format-2.12.0/src/main/thrift/parquet.thrift#L633-L651
[2] https://www.jeronimo.dev/compression-algorithms-parquet/
[3] https://mvnrepository.com/artifact/org.anarres.lzo/lzo-hadoop

[ostinru]

Done.