Add RunEndBool encoding for efficient boolean array compression#8467
Add RunEndBool encoding for efficient boolean array compression#8467joseph-isaacs wants to merge 1 commit into
Conversation
Merging this PR will improve performance by 19.09%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | chunked_varbinview_into_canonical[(100, 100)] |
272.2 µs | 307.1 µs | -11.39% |
| ⚡ | Simulation | chunked_bool_canonical_into[(1000, 10)] |
26.8 µs | 16.4 µs | +63.47% |
| ⚡ | Simulation | chunked_varbinview_canonical_into[(1000, 10)] |
191.4 µs | 154.8 µs | +23.66% |
| ⚡ | Simulation | chunked_varbinview_into_canonical[(1000, 10)] |
205.9 µs | 169.8 µs | +21.26% |
| ⚡ | Simulation | eq_i64_constant |
319.8 µs | 290.1 µs | +10.25% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/runend-bool-compression-r148yw (8133d43) with develop (ee2cd67)
Footnotes
-
4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
1fca4d5 to
53d3cdb
Compare
…election Reintroduce the `vortex-runend-bool` encoding on top of the current `develop`. Boolean runs strictly alternate, so a `RunEndBoolArray` stores only the run `ends`, the value of the first run (`start`), and optional validity, rather than a separate values array. The encoding is self-contained against develop's reworked `runend` search API: `find_physical_index` / `find_slice_end_index` are local free functions built on `SearchSortedPrimitiveArray` + `match_each_unsigned_ integer_ptype!`, threading `&mut ExecutionCtx` through every caller (scalar_at, take, filter, slice). The dense run-preserving filter path is preserved. Parent kernels (slice, filter, take, not) are registered via `session.kernels().register_execute_parent_kernel`, and `with_buffers` is implemented per the current `VTable` contract. Wire the encoding into `vortex-file`'s `register_default_encodings` and add a `BoolRunEndScheme` to `vortex-btrblocks` so run-heavy bool arrays are automatically run-end encoded. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
53d3cdb to
8133d43
Compare
Summary
This PR introduces a new
RunEndBoolencoding specialized for compressing boolean arrays with long runs. Since boolean values strictly alternate in run-end encoding, we can store only the run end positions plus a singlestartflag, rather than maintaining a separate values array like the genericRunEndencoding.Changes
New
vortex-runend-boolcrate with:RunEndBoolArraytype and metadata serializationSelf-contained run-end indexing —
RunEndBoolimplements its ownfind_physical_index/find_slice_end_index(ctx-based, mirroringvortex-runend'sSearchSortedPrimitiveArrayapproach).vortex-runenditself is left unchanged.Integration with compression pipeline:
BoolRunEndSchemeinvortex-btrblocksfor automatic bool array compressionendschild for further compressionFile format support:
RunEndBoolregistered in the default session; arrays serialize/deserialize correctly in the Vortex file formattest_run_heavy_bool_uses_runendinvortex-btrblocksasserts the compressor auto-selects the encodingDesign Notes
ihas valuevalue_at_index(i, start)where even indices equalstartand odd indices equal!start