Skip to content

fix: handle empty batch from deletion in add_columns_from_stream#7233

Open
zhangyang0418 wants to merge 1 commit into
lance-format:mainfrom
zhangyang0418:fix/add-columns-empty-batch-deletion
Open

fix: handle empty batch from deletion in add_columns_from_stream#7233
zhangyang0418 wants to merge 1 commit into
lance-format:mainfrom
zhangyang0418:fix/add-columns-empty-batch-deletion

Conversation

@zhangyang0418

Copy link
Copy Markdown

What

Fixes a panic in add_columns when a fragment's deleted rows fill an entire read batch.

Closes #7232.

Root cause

In add_columns_from_stream, the updater reads each fragment in physical batches and applies the deletion vector via filter_record_batch. When every row in a physical batch is deleted, the filter yields a 0-row batch. The outer loop then sets rows_remaining = 0, the inner while rows_remaining > 0 loop never runs, batches stays empty, and concat_batches(&batches[0].schema(), ..) panics with index out of bounds: the len is 0 but the index is 0.

This was hit in production using a merge-columns workflow over a dataset whose fragment had a deletion file, where the deleted rows happened to align with a read-batch boundary.

Fix

When the updater yields a 0-row batch, feed an empty batch back to the updater (to keep it in sync) and continue:

if rows_remaining == 0 {
    updater.update(RecordBatch::new_empty(stream.schema())).await?;
    continue;
}

Test

Adds test_add_columns_with_fully_deleted_batch: writes a single fragment with 105 rows, deletes the trailing 5 rows so that — read with batch_size=50 — the last batch [100..105) is fully deleted, then verifies add_columns succeeds and the new column has the correct values. The test panics on the unpatched code and passes with the fix.

When an entire read batch has been deleted (e.g., all rows in a batch
fall within the deletion vector), the updater yields a 0-row batch.
The old code would then skip the inner loop, leaving `batches` empty,
causing `concat_batches(&batches[0]..)` to panic with 'index out of
bounds: the len is 0 but the index is 0'.

This fix detects the empty batch case and feeds an empty batch back to
the updater to keep it in sync, then continues to the next batch.

Adds a regression test that creates a dataset with 105 rows, deletes
the trailing 5 rows (forming a fully-deleted trailing batch when read
with batch_size=50), and verifies add_columns succeeds.

Fixes the panic reported in production when using merge_columns on a
dataset with deletion files where deleted rows cluster at batch
boundaries.
@github-actions github-actions Bot added the bug Something isn't working label Jun 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

panic in add_columns_from_stream: index out of bounds when entire read batch is deleted

1 participant