Skip to content

Parquet 1.12.3#56

Merged
tuhaihe merged 5 commits intoapache:mainfrom
open-gpdb:parquet-1.12.3
Feb 6, 2026
Merged

Parquet 1.12.3#56
tuhaihe merged 5 commits intoapache:mainfrom
open-gpdb:parquet-1.12.3

Conversation

@ostinru
Copy link
Collaborator

@ostinru ostinru commented Feb 5, 2026

Update parquet to 1.12.3 (latest version with hadoop-client-2.x)

  • Add support for ZSTD compression
  • Temporary declare LZO as not supported. It causes following error in both current and in pre-upgrade builds:
    ERROR:  PXF server error : Class com.hadoop.compression.lzo.LzoCodec was not found  (seg1 10.11.0.131:6000 pid=2567556)
    
  • Add tests to cover different types of compression

@ostinru
Copy link
Collaborator Author

ostinru commented Feb 5, 2026

Overall compression support[1]:

  • ✅ uncompressed
  • ✅ snappy
  • ✅ gzip
  • 🛑 lzo -> PXF server error : Class com.hadoop.compression.lzo.LzoCodec was not found
  • 🛑 brotli -> PXF server error : Class org.apache.hadoop.io.compress.BrotliCodec was not found
  • 🛑 lz4 -> PXF server error : native lz4 library not available
  • ✅ zstd (New!)
  • 🛑 lz4_raw -> PXF server error : Invalid codec: lz4_raw

[1] https://github.com/apache/parquet-format/blob/apache-parquet-format-2.12.0/src/main/thrift/parquet.thrift#L633-L651

@ostinru ostinru marked this pull request as ready for review February 5, 2026 19:56
@tuhaihe
Copy link
Member

tuhaihe commented Feb 6, 2026

@MisterRaindrop could you please take a look at this PR? Thanks!

Copy link
Collaborator

@MisterRaindrop MisterRaindrop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tuhaihe tuhaihe merged commit ab7f902 into apache:main Feb 6, 2026
25 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants