fix: stabilize Vortex runtime for async IO

[JingsongLi]

Summary

Stabilizes Paimon's optional Vortex file format on CI without skipping tests. The hang was tied to Vortex 0.68's async scan/write paths and the dependency set that allowed unicode-segmentation 1.13.3, which had already been noted in this crate as correlating with Vortex test hangs.

Changes

• Pin unicode-segmentation back to =1.13.2, matching the existing comment that 1.13.3 correlates with Vortex test hangs.
• Move Vortex reads and writes onto a dedicated blocking thread with CurrentThreadRuntime so returned streams and writers do not depend on the caller's Tokio runtime lifetime.
• Read Vortex files into memory and open them with open_buffer, avoiding Vortex's async VortexReadAt segment pipeline for Paimon's storage adapters.
• Serialize Vortex writes into an in-memory buffer before writing once to Paimon's OutputFile.
• Avoid Vortex predicate pushdown for the unstable filtered-scan path; include predicate columns in the scan and evaluate filters locally with Arrow kernels.
• Add/keep regression coverage for runtime lifetime, whole-file buffer reads, predicate filtering, unprojected predicate columns, empty projections, and concurrent predicate reads.

Testing

• cargo fmt --all -- --check
• git diff --check
• cargo test -p paimon --features fulltext,vortex arrow::format::vortex::tests:: -- --test-threads=8 --nocapture
• cargo test -p paimon --all-targets --features fulltext,vortex
• cargo test -p paimon-datafusion --features vortex --test vortex_tables
• cargo clippy --all-targets --workspace --features fulltext,vortex -- -D warnings
• GitHub Actions run 27588470873 passed: check, unit/build on Ubuntu/macOS/Windows, and integration.

Notes

This does not skip any Vortex tests. The trade-off is that Vortex reads/writes are buffered in memory while Vortex support remains optional and experimental; this favors deterministic CI behavior over streaming/random IO for now.

[leaves12138]

Thanks for the fix. The runtime isolation approach and the Arrow-side predicate fallback look reasonable, and I verified the focused Vortex tests and clippy locally:

• cargo test -p paimon --features vortex arrow::format::vortex -- --nocapture
• cargo clippy -p paimon --features vortex --all-targets -- -D warnings

I think this needs one fix before merge.

VortexFormatWriter::num_bytes() now returns 0 until close(), because the new writer only updates bytes_written after the whole in-memory Vortex buffer is serialized and written in close(). However DataFileWriter::write and PostponeFileWriter::write rely on current_writer.num_bytes() >= target_file_size after each batch to roll files. With file.format = "vortex", that condition never becomes true before prepare_commit, so target-file-size is ignored and all batches keep accumulating into one open Vortex writer. This is a behavior regression from the previous streaming writer, where CountingPaimonWrite updated bytes_written as Vortex wrote to storage, and it also makes the new in-memory buffering path potentially unbounded.

I reproduced this by adding .option("file.format", "vortex") to test_write_rolling_on_target_file_size: with target-file-size = 1b, the test expects two data files but gets one.

Could you restore observable pre-close progress for Vortex writes, at least enough for target-file-size rolling to work, and add a Vortex-specific rolling test? Tracking queued Arrow/Vortex batch memory as an approximate num_bytes() would be better than always returning zero if exact serialized bytes are unavailable before close().

[JingsongLi]

Thanks for catching this. I updated the Vortex writer so it now tracks the queued RecordBatch memory estimate while arrays are staged before close. num_bytes() now reports that pre-close estimate (or the final written size after close), and in_progress_size() also reflects the staged bytes, so target-file-size rolling can observe progress again.

I also added a Vortex-specific regression test for target-file-size = 1b; it failed with 1 data file before the fix and now produces 2 as expected.

Local verification:

• cargo test -p paimon --features fulltext,vortex table::table_write::tests::test_vortex_write_rolling_on_target_file_size -- --nocapture
• cargo test -p paimon --features fulltext,vortex arrow::format::vortex::tests:: -- --test-threads=8 --nocapture
• cargo test -p paimon --all-targets --features fulltext,vortex
• cargo clippy --all-targets --workspace --features fulltext,vortex -- -D warnings
• cargo fmt --all -- --check
• git diff --check

[QuakeWang]

LGTM overall. I left a couple of minor nits, but nothing blocking from my side.

[QuakeWang]

nit: These Vortex expression conversion helpers are now test-only, while the production read path uses Arrow-side filtering. Keeping this conversion test block may make future readers think Vortex filter pushdown is still active. Could we remove it, or rename/comment it as legacy conversion coverage?

[QuakeWang]

nit: kanal appears to be unused now that the Vortex writer no longer uses the streaming path. Could we remove it from the vortex feature and the dependency list?

[leaves12138]

Thanks for the update. I re-reviewed the latest head and the previous blocker is fixed now.

VortexFormatWriter now tracks the staged RecordBatch memory and reports it through num_bytes() / in_progress_size() before close(), so the table writer can observe progress and roll files instead of accumulating everything into a single Vortex writer. The new test_vortex_write_rolling_on_target_file_size covers the regression case I raised.

I also re-ran the focused validation locally:

• cargo test -p paimon --features vortex arrow::format::vortex -- --nocapture
• cargo test -p paimon --features vortex table::table_write::tests::test_vortex_write_rolling_on_target_file_size -- --nocapture
• cargo clippy -p paimon --features vortex --all-targets -- -D warnings

The full GitHub CI is also green at this head. Looks good to me.

[JingsongLi]

Addressed the two nits from the latest review:

• Removed kanal from paimon direct optional dependencies and the vortex feature. It only remains in Cargo.lock as a transitive Vortex dependency.
• Removed the old Vortex expression conversion helpers and their unit tests, since the production read path now decodes predicate columns and applies Arrow-side filtering instead of Vortex filter pushdown. The existing Arrow-side predicate filtering tests remain in place.

Local verification:

• cargo test -p paimon --features fulltext,vortex arrow::format::vortex::tests:: -- --test-threads=8 --nocapture
• cargo test -p paimon --features fulltext,vortex table::table_write::tests::test_vortex_write_rolling_on_target_file_size -- --nocapture
• cargo test -p paimon --all-targets --features fulltext,vortex
• cargo clippy --all-targets --workspace --features fulltext,vortex -- -D warnings
• cargo fmt --all -- --check
• git diff --check