Skip to content

feat: add Mosaic data file reader#386

Open
QuakeWang wants to merge 1 commit into
apache:mainfrom
QuakeWang:mosaic-read
Open

feat: add Mosaic data file reader#386
QuakeWang wants to merge 1 commit into
apache:mainfrom
QuakeWang:mosaic-read

Conversation

@QuakeWang

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: #378

Paimon Rust currently does not recognize .mosaic data files, so Rust clients cannot read tables written with Mosaic by Java Paimon, PyPaimon, or paimon-mosaic-core.

This PR adds the first read-only Mosaic reader foundation behind the mosaic feature.

Brief change log

  • Add optional paimon-mosaic-core dependency behind the mosaic feature.
  • Add .mosaic reader dispatch when the feature is enabled.
  • Implement a read-only Mosaic format reader using a memory-backed InputFile.
  • Preserve projection order, empty projection, row selection, and unsupported-type errors.
  • Add DataFusion feature passthrough and include mosaic in CI feature checks.

Tests

  • cargo fmt --all -- --check
  • cargo test -p paimon --features mosaic arrow::format::mosaic
  • cargo build --features fulltext,vortex,mosaic
  • cargo clippy --all-targets --workspace --features fulltext,vortex,mosaic -- -D warnings
  • cargo test -p paimon --all-targets --features fulltext,vortex,mosaic

API and Format

Adds an optional mosaic Cargo feature. No writer support or storage format change is introduced.

Documentation

No user-facing documentation is added in this PR. Predicate pushdown and async random-access reading are left as follow-up work.

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant