Skip to content

fix: scan all-null columns after dropping physical columns#7238

Open
skyshineb wants to merge 1 commit into
lance-format:mainfrom
skyshineb:fix/drop-only-physical-column-scan
Open

fix: scan all-null columns after dropping physical columns#7238
skyshineb wants to merge 1 commit into
lance-format:mainfrom
skyshineb:fix/drop-only-physical-column-scan

Conversation

@skyshineb

Copy link
Copy Markdown

Summary

Closes #7236
Fix scans after dropping the only physical column from a dataset that still has row-bearing all-null schema-evolution columns.

Operation::Project can leave fragments with files = [] while retaining physical_rows. Scans need that row count to synthesize null arrays for the remaining projected columns.

Changes

  • Trust modern Fragment.physical_rows before rejecting file-less fragments in FileFragment::physical_rows().
  • Update fragment validation to allow file-less fragments when physical_rows is present.
  • Use physical_rows for deletion-vector bounds in file-less projected fragments.
  • Add regressions for scanning all-null columns and appending after re-adding a dropped physical column.

Trust modern Fragment.physical_rows before rejecting fragments with no data files, allowing all-null projected columns to synthesize rows after dropping the only physical column.

Update fragment validation to use physical_rows for row-bearing fragments without visible data files and cover scan/append regressions.
@github-actions github-actions Bot added the bug Something isn't working label Jun 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: scan fails after dropping the only original physical column and leaving all-null added columns

1 participant