Skip to content

feat(ipc): Supports compression level configuration#10133

Open
wForget wants to merge 3 commits into
apache:mainfrom
wForget:ARROW-RS-10132
Open

feat(ipc): Supports compression level configuration#10133
wForget wants to merge 3 commits into
apache:mainfrom
wForget:ARROW-RS-10132

Conversation

@wForget

@wForget wForget commented Jun 12, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

Make compression level of ipc writer configurable

What changes are included in this PR?

  • Add a compression level option to IpcWriteOptions
  • Add try_with_compression_level to configure IPC batch compression levels
  • Pass the configured compression level through the IPC write path when constructing the compression codec

Are these changes tested?

I locally modified the test_write_file_with_zstd_compression test case to benchmark write time and output file size across different compression levels.

image

print logs:

ZSTD compression level -999: wrote 4001530 bytes in 1.658125ms
ZSTD compression level 1: wrote 3657530 bytes in 38.953042ms
ZSTD compression level 22: wrote 3638762 bytes in 869.19925ms

Are there any user-facing changes?

Yes. This PR adds a new batch_compression_level option to IpcWriteOptions.

@github-actions github-actions Bot added the arrow Changes to the arrow crate label Jun 12, 2026
@wForget wForget marked this pull request as ready for review June 12, 2026 07:32
@wForget

wForget commented Jun 12, 2026

Copy link
Copy Markdown
Member Author

The failure is unrelated to this PR and may be related to smithy-rs issues smithy-lang/smithy-rs#4695 and smithy-lang/smithy-rs#4696.

   Compiling aws-runtime v1.7.4
error[E0282]: type annotations needed
   --> /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/aws-runtime-1.7.4/src/auth/sigv4.rs:278:22
    |
278 |                     .send(Box::new(SigV4MessageSigner::new(
    |                      ^^^^ cannot infer type of the type parameter `T` declared on the method `send`
...
285 |                     )) as _)
    |                           - type must be known at this point
    |
help: consider specifying the generic argument
    |
278 |                     .send::<T>(Box::new(SigV4MessageSigner::new(
    |                          +++++

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow Changes to the arrow crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

arrow-ipc: Supports compression level configuration for arrow-ipc writer

1 participant