⚡ Bolt: optimize hash join build side for single-batch inputs#285
⚡ Bolt: optimize hash join build side for single-batch inputs#285
Conversation
This optimization speeds up hash joins with a small build side (single batch) by: 1. Bypassing `concat_batches` which is unnecessary for a single batch. 2. Reusing pre-evaluated join key arrays to avoid redundant `evaluate_expressions_to_arrays` calls in both the ArrayMap attempt and standard HashMap fallback. 3. Only performing pre-evaluation for single-column joins to avoid unnecessary work for multi-column joins. The implementation carefully handles the borrow checker by ensuring ownership of the batch vector is only taken when appropriate. Tests passed in datafusion-physical-plan. Co-authored-by: Dandandan <163737+Dandandan@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
1 participant
💡 What: Optimized
collect_left_inputandtry_create_array_mapinHashJoinExecto add a fast path for single-batch build side inputs.🎯 Why: Small joins are very common in analytical queries. Previously, the hash join build phase would always call
concat_batchesand might evaluate join key expressions multiple times (once for ArrayMap check and again for HashMap creation), even when the input was already a single batch.📊 Impact: Reduces CPU usage and memory allocations during the build phase of hash joins with small inputs.
🔬 Measurement: Verified with existing unit tests in
datafusion-physical-plan. All hash join tests passed. Verified withcargo clippy.PR created automatically by Jules for task 18045351061971946219 started by @Dandandan