Skip to content

test(datafusion): cover SQL schema evolution reads#391

Open
liujiwen-up wants to merge 1 commit into
apache:mainfrom
liujiwen-up:test/258-mixed-format-schema-evolution-sql
Open

test(datafusion): cover SQL schema evolution reads#391
liujiwen-up wants to merge 1 commit into
apache:mainfrom
liujiwen-up:test/258-mixed-format-schema-evolution-sql

Conversation

@liujiwen-up

@liujiwen-up liujiwen-up commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: #258

Add DataFusion SQL coverage for schema evolution tables generated from Java/Spark consistency fixtures, focusing on mixed-format reads through SQL. The rename coverage uses schema_evolution_rename_column, whose fixture contains parquet/orc/avro files.

Brief change log

  • Add a small assert_sql helper for table-formatted SQL result assertions.
  • Cover SQL reads for mixed-format add column and type promotion tables.
  • Cover SQL reads for rename, drop, and reorder/move schema evolution cases with SELECT, WHERE, and ORDER BY id.

Tests

  • cargo fmt --check: passed
  • git diff --check: passed
  • cargo test -p paimon-datafusion --test read_tables schema_evolution --no-run: passed
  • cargo test -p paimon-datafusion --test read_tables schema_evolution -- --nocapture: not fully runnable locally because /tmp/paimon-warehouse has not been provisioned and Docker daemon is unavailable

API and Format

No API or storage format changes.

Documentation

No documentation changes required.

ctx.sql(sql).await?.collect().await
}

async fn assert_sql(sql: &str, expected: &[&str]) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this helper compares the full pretty_format_batches output verbatim. It keeps the expected SQL result readable, but also couples the tests to Arrow/DataFusion pretty-printer details such as padding, borders, and NULL rendering. Fine for these small result sets; if we want less brittle tests across dependency upgrades, comparing normalized rows/columns would be more stable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants