Skip to content

perf : Optimize count distinct#21456

Open
coderfender wants to merge 11 commits intoapache:mainfrom
coderfender:optimize_count_distinct
Open

perf : Optimize count distinct#21456
coderfender wants to merge 11 commits intoapache:mainfrom
coderfender:optimize_count_distinct

Conversation

@coderfender
Copy link
Copy Markdown
Contributor

@coderfender coderfender commented Apr 8, 2026

Which issue does this PR close?

Remove hashset based accumulators for smaller int data types and use bitmaps. Follow up of : #21453

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions bot added the functions Changes to functions implementation label Apr 8, 2026
@coderfender
Copy link
Copy Markdown
Contributor Author

coderfender commented Apr 8, 2026

benchmark results :

count_distinct i16 bitmap                      1.00      3.3±0.43µs        ? ?/sec    23.87    78.4±0.84µs        ? ?/sec
count_distinct i8 bitmap                       1.00      2.3±0.49µs        ? ?/sec    7.13     16.7±0.55µs        ? ?/sec
count_distinct u16 bitmap                      1.00      3.1±0.18µs        ? ?/sec    25.45    78.8±3.92µs        ? ?/sec
count_distinct u8 bitmap                       1.00      2.3±0.34µs        ? ?/sec    7.37     16.9±0.14µs        ? ?/sec

It seems like we are 25x faster for u16 bitmap based accumulators (or I am sleepy :) )

@Dandandan
Copy link
Copy Markdown
Contributor

I think we can do the same for 16 bit types, it is just 65_536 bytes 8192 if we use a bitmap.

@Dandandan
Copy link
Copy Markdown
Contributor

Oh wait, you're already doing that :)

@coderfender
Copy link
Copy Markdown
Contributor Author

coderfender commented Apr 8, 2026

Query 0 in clickbench_extended dataset (which uses count distinct on u8 is now ~ 11 % faster :

┃ Query     ┃    main_cb ┃ bitmap_cb_2 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │  529.46 ms │   478.99 ms │ +1.11x faster │

(Other queries are faster but I believe that is more around variance )

┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃    main_cb ┃ bitmap_cb_2 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │  529.46 ms │   478.99 ms │ +1.11x faster │
│ QQuery 1  │  107.43 ms │   102.59 ms │     no change │
│ QQuery 2  │  250.89 ms │   240.76 ms │     no change │
│ QQuery 3  │  207.67 ms │   207.49 ms │     no change │
│ QQuery 4  │  391.43 ms │   353.05 ms │ +1.11x faster │
│ QQuery 5  │ 4144.11 ms │  4084.08 ms │     no change │
│ QQuery 6  │  676.03 ms │   622.21 ms │ +1.09x faster │
│ QQuery 7  │  719.78 ms │   599.06 ms │ +1.20x faster │
│ QQuery 8  │  238.30 ms │   207.27 ms │ +1.15x faster │
│ QQuery 9  │ 1531.52 ms │  1406.34 ms │ +1.09x faster │
│ QQuery 10 │  435.27 ms │   403.28 ms │ +1.08x faster │
│ QQuery 11 │ 1043.68 ms │   955.22 ms │ +1.09x faster │
│ QQuery 12 │  113.16 ms │   106.31 ms │ +1.06x faster │
└───────────┴────────────┴─────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary          ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (main_cb)       │ 10388.73ms │
│ Total Time (bitmap_cb_2)   │  9766.66ms │
│ Average Time (main_cb)     │   799.13ms │
│ Average Time (bitmap_cb_2) │   751.28ms │
│ Queries Faster             │          9 │
│ Queries Slower             │          0 │
│ Queries with No Change     │          4 │
│ Queries with Failure       │          0 │
└────────────────────────────┴────────────┘

@coderfender
Copy link
Copy Markdown
Contributor Author

cc : @neilconway , @alamb , @martin-g . Please take a look whenever you get a chance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use bitmap for count_distinct expression for u8/16 and i8/16 [perf]

2 participants