PSplitAgg: split mixed-class aggregations#22
Merged
Conversation
When a global agg or GROUP BY mixes SIMD-friendly aggs (min/max/avg/sum/ count) with aggs that need a different physical operator (median / percentile / approx-quantile), the cascading strategy cond collapses to PScalarAgg — a per-row Clojure reduce loop that ran 6× slower than executing each class separately. The new PSplitAgg physical node partitions aggs by strategy class, plans each subset with the regular strategy chooser, and merges the results. Existing fused filter+agg SIMD kernels are reused unchanged. DuckDB/ClickHouse use a single-scan / per-agg-dispatch design that forgoes filter+agg fusion; PSplitAgg keeps Stratum's fusion advantage while matching their per-class specialization. Performance on 6M rows (NT, vs DuckDB): min+avg+median+max global 457ms → 83ms (1.84× faster than Duck) min+median global 300ms → 77ms (1.88× faster) sum+median global 215ms → 78ms (1.55× faster) GROUP BY min+median+max 1073ms → 79ms (1.62× faster) GROUP BY sum+median 953ms → 76ms (1.67× faster) Single-class queries fall through to the existing path with no plan or execution change. Future fast paths (variance, count-distinct/HLL) plug in by joining the right partition — no executor change needed. All 1012 existing tests pass; 11 new split-agg tests added. Signed-off-by: Christian Weilbach <christian@weilbach.name>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SELECT min(price), avg(price), median(price), max(price) FROM tran ~3× slower than DuckDB on the same data (6M rows: 457ms vs DuckDB 152ms).PScalarAgg(per-row Clojure reduce) whenever the agg list mixes SIMD-friendly aggs withmedian/percentile/approx-quantile. The "fast" siblings get dragged into the slow path.PSplitAggthat partitions aggs by strategy class, plans each subset with the regular chooser, and merges results. Single-class queries are unaffected.Why this design
We studied DuckDB and ClickHouse before settling on the architecture:
UngroupedAggregateExecuteState::Sinkin DuckDB;Aggregator::executeOnIntervalWithoutKeywithAggregateFunctionInstructionarray in ClickHouse). They call each agg'sadd_batchonce per chunk, sharing column data by reference.PMultiAggoperator would require new masked-kernel signatures duplicated across every agg type and would unwind the filter+agg fusion. The theoretical upside (one extra predicate eval saved) is ms-scale.Numbers (6M rows, NT, vs DuckDB JDBC in-process)
Single-class fast-path numbers (min+avg+max only, etc.) are unchanged.
Plan tree shape
Before:
After:
Test plan
stratum.query-test/split-agg-*cover: global / GROUP BY, aliases preserved, predicates respected, single-class does NOT split, percentile-only does NOT split, mixed DOES split, results match individual queries.clojure -M:test→ 1012 tests / 4680 assertions / 0 failures.origin/main(verified via stash-and-test) and is unrelated.Files
src/stratum/query/ir.clj—PSplitAggdefrecord +map-childrenwalk supportsrc/stratum/query/plan.clj—agg-class/partition-aggs-by-class/ cond branch inselect-global-agg-strategyandselect-group-by-strategy+propagate-est-rows+ EXPLAIN printersrc/stratum/query/executor.clj—execute-split-agg(global row-map merge + GROUP BY group-key merge)test/stratum/query_test.clj— 11 new tests