feat(compaction): introduce RowAddrRemap structure to avoid remap OOM caused by HashMap

[zhangyue19921010]

Closes: #7150

Compact Row-Address Remapping for Compaction

Compaction rewrites rows into new fragments, so indices that store physical row addresses need an old-address to new-address mapping without building an O(total rows) HashMap<u64, Option<u64>>.

Layout

Old Rows

old_fragment_id -> (old_offsets, old_rows_before)

• old_offsets: rewritten old row offsets in this old fragment.
• old_rows_before: rewritten row count before this old fragment.

New Rows

Ordered new-fragment ranges:

(fragment_id, new_rows_before, physical_rows)

• new_rows_before: rewritten row count before this new fragment.

Lookup

• An address whose fragment was not rewritten returns None.

• For an address whose fragment was rewritten:

  1. Read (old_offsets, old_rows_before) from the old-row layout.

  2. If offset is not in old_offsets, return Some(None) because the row was deleted.

  3. Otherwise, old_offsets.rank(offset) - 1 is this row's 0-based position among rewritten old rows in this old fragment.

  4. Add old_rows_before to get k, the row's 0-based position among all rewritten old rows.

  5. In the new-row layout, find the range:

     (fragment_id, new_rows_before, physical_rows)
     

     where:

     new_rows_before <= k < new_rows_before + physical_rows
     

  6. The new address is:

     (fragment_id, k - new_rows_before)
     

Ordering

Compact remap does not store each old-to-new row mapping. It computes k from the old-row layout, then maps it to the k-th row written to the new fragments.

This requires the reader-to-writer pipeline to preserve row order.

• old_frag_ids must match the order old fragments are read.

  • Within each old fragment, rewritten rows are interpreted by ascending old row offset.

• new_frags must match the order new rows are written.

• Current compaction satisfies this because it scans selected fragments in order and writes the resulting stream without reordering rows.

[codecov]

Codecov Report

❌ Patch coverage is 94.35666% with 25 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/optimize.rs	65.62%	8 Missing and 3 partials ⚠️
rust/lance-core/src/utils/row_addr_remap.rs	98.48%	3 Missing ⚠️
rust/lance/src/dataset/optimize/remapping.rs	96.92%	1 Missing and 1 partial ⚠️
rust/lance/src/io/exec/knn.rs	0.00%	2 Missing ⚠️
rust/lance-index/src/scalar/fmindex.rs	0.00%	1 Missing ⚠️
rust/lance-index/src/scalar/label_list.rs	0.00%	1 Missing ⚠️
rust/lance-index/src/vector/hnsw/index.rs	0.00%	1 Missing ⚠️
rust/lance/src/index/vector/fixture_test.rs	0.00%	1 Missing ⚠️
rust/lance/src/index/vector/ivf.rs	80.00%	1 Missing ⚠️
rust/lance/src/index/vector/ivf/v2.rs	0.00%	1 Missing ⚠️
... and 1 more

📢 Thoughts on this report? Let us know!