[arrow-flight][#10125] add dictionary focused benchmarks that hydrate/resend#10126
Open
Rich-T-kid wants to merge 4 commits into
Open
[arrow-flight][#10125] add dictionary focused benchmarks that hydrate/resend#10126Rich-T-kid wants to merge 4 commits into
Rich-T-kid wants to merge 4 commits into
Conversation
Rich-T-kid
changed the title
Contributor
Author
|
This should provide better visibility for any optimizations we to the encoding path in |
Contributor
|
run benchmark flight |
alamb
approved these changes
Jun 11, 2026
alamb
left a comment
alamb
left a comment
Contributor
There was a problem hiding this comment.
looks good to me -- thank you @Rich-T-kid
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/investigate-arrow-flight (ef4ddcd) to 826b808 (merge-base) diff File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Contributor
Author
|
@alamb updating this PR to include the case for 4 columns as well as [1,8]. This is mostly to illustrate a point in #10137. I updated my local benchmarks on main & my branch to use [1,4,8,16] as it shows a more gradual trend line. 16 is overkill I think 8 columns is a good max but 1 -> 8 is too steep of a jump. |
github-actions
Bot
added
arrow
Contributor
Author
|
@Jefffrey do you mind running the arrow-flight benchmarks on this PR? Thx |
Contributor
|
run benchmark flight |
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-t-kid/investigate-arrow-flight (54faeda) to 826b808 (merge-base) diff File an issue against this benchmark runner |
Contributor
Author
|
I have too many PR tabs open, I meant to write that for #10137 |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of
Rationale for this change
Going through the arrow-flight codebase I noticed that by default
DictionaryHandlingis set to Hydrate. This means it expands the arrays out to their logical form. In other words when the variant is set to hydrate,arrow-ipc::IpcDataGenerator::encode_all_dicts()never actually runs.This is important due to the arrow-ipc work that @alamb , @JakeDern & myself have been working on. Efforts are being made to optimize arrow-ipc's use of dictionaries. This PR allows those chanages to be visible through arrow-flight benchmarks
What changes are included in this PR?
This PR adds a benchmark for arrow-flight's
do_putendpoint using dictionary arrays, measuring the latency difference between the two DictionaryHandling variants.Are these changes tested?
changes are benchmarks
Are there any user-facing changes?
no